Merge branch 'main' into INFERENG-1867

Author: fynnsu

.github/workflows/linkcheck.yml

@@ -13,10 +13,16 @@ jobs:
   markdown-link-check:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v4
-    - uses: umbrelladocs/action-linkspector@v1
+    - name: Checkout code
+      uses: actions/checkout@v4
+
+    - name: Run linkspector
+      uses: umbrelladocs/action-linkspector@v1
       with:
         github_token: ${{ secrets.github_token }}
         reporter: github-pr-review
         fail_on_error: true
+        filter_mode: nofilter
         config_file: '.github/workflows/linkspector/linkspector.yml'
+        show_stats: true
+        level: info

.github/workflows/quality-check.yaml

@@ -10,7 +10,7 @@ jobs:
     steps:
       - uses: actions/setup-python@v5
         with:
-          python-version: '3.9'
+          python-version: '3.10'
       - uses: actions/checkout@v4
       - name: "⚙️ Install dependencies"
         run: pip3 install .[dev]

README.md

@@ -85,7 +85,7 @@ Applying quantization with `llmcompressor`:
 
 ### User Guides
 Deep dives into advanced usage of `llmcompressor`:
-* [Quantizing with large models with the help of `accelerate`](examples/big_models_with_accelerate/README.md)
+* [Quantizing large models with sequential onloading](examples/big_models_with_sequential_onloading/README.md)
 
 
 ## Quick Tour

docs/developer/contributing.md

@@ -50,7 +50,7 @@ If not, please file a new issue, providing as much relevant information as possi
 
 ### Pull Requests & Code Reviews
 
-Please check the PR checklist in the [PR template](.github/PULL_REQUEST_TEMPLATE.md) for detailed guide for contribution.
+Please check the PR checklist in the [PR template](../../.github/PULL_REQUEST_TEMPLATE.md) for detailed guide for contribution.
 
 ### Thank You
 

docs/getting-started/compress.md

@@ -10,7 +10,7 @@ LLM Compressor provides a straightforward way to compress your models using vari
 
 Before you begin, ensure that your environment meets the following prerequisites:
 - **Operating System:** Linux (recommended for GPU support)
-- **Python Version:** 3.9 or newer
+- **Python Version:** 3.10 or newer
 - **Available GPU:** For optimal performance, it's recommended to use a GPU. LLM Compressor supports the latest PyTorch and CUDA versions for compatibility with NVIDIA GPUs.
 
 ## Select a Model and Dataset

docs/getting-started/deploy.md

@@ -12,7 +12,7 @@ vLLM is a high-performance inference engine designed for large language models,
 
 Before deploying your model, ensure you have the following prerequisites:
 - **Operating System:** Linux (recommended for GPU support)
-- **Python Version:** 3.9 or newer
+- **Python Version:** 3.10 or newer
 - **Available GPU:** For optimal performance, it's recommended to use a GPU. vLLM supports a range of accelerators, including NVIDIA GPUs, AMD GPUs, TPUs, and other accelerators.
 - **vLLM Installed:** Ensure you have vLLM installed. You can install it using pip:
   ```bash

docs/getting-started/install.md

@@ -11,7 +11,7 @@ LLM Compressor can be installed using several methods depending on your requirem
 Before installing LLM Compressor, ensure you have the following prerequisites:
 
 - **Operating System:** Linux (recommended for GPU support)
-- **Python Version:** 3.9 or newer
+- **Python Version:** 3.10 or newer
 - **Pip Version:** Ensure you have the latest version of pip installed. You can upgrade pip using the following command:
 
   ```bash

docs/guides/compression_schemes.md

@@ -6,17 +6,17 @@ Post-training quantization is performed to reduce the precision of quantizable w
 
 | Scheme | Description | Data required? | vLLM Hardware Compatibility |
 |--------|-------------|----------------|-----------------------------|
-| **[W8A8-FP8](../examples/quantization_w8a8_fp8.md)** | 8-bit floating point (FP8) quantization for weights and activations, providing ~2X smaller weights with 8-bit arithmetic operations. Uses channel-wise quantization to compress weights to 8 bits, and uses dynamic per-token or static per-tensor quantization to compress activations to 8 bits. Weights scales that are generated on a per-channel basis or generated on a per-tensor basis are also possible. Channel-wise weights quantization with dynamic per-token activations is the the most performant option. W8A8-FP8 does not require a calibration dataset. Activation quantization is carried out during inference on vLLM. Good for general performance and compression, especially for server and batch inference. | No calibration dataset is required, unless you are doing static per-tensor activation quantization. | Latest NVIDIA GPUs (Hopper and later) and latest AMD GPUs. Recommended for NVIDIA GPUs with compute capability >=9.0 (Hopper and Blackwell) |Recommended for NVIDIA GPUs with compute capability >=9.0 (Hopper and Blackwell) |
-| **[W8A8-INT8](../examples/quantization_w8a8_int8.md)** | 8-bit integer (INT8) quantization for weights and activations, providing ~2X smaller weights with 8-bit arithmetic operations. Uses channel-wise quantization to compress weights to 8 bits using GPTQ, and uses dynamic per-token quantization to compress activations to 8 bits. Weight quantization can be both per-tensor or per-channel for INT8. W8A8-INT8 is good for general performance and compression, especially for server and batch inference. Activation quantization is carried out during inference on vLLM. Activations can be static or dynamic. Additionally, INT8 activations can also be asymmetric. W8A8-INT8 helps improve speed in high QPS scenarios or during offline serving with vLLM. W8A8-INT8 is good for general performance and compression, especially for server and batch inference. | Requires calibration dataset for weight quantization and static per-tensor activation quantization. | Supports all NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators. Recommended for NVIDIA GPUs with compute capability <8.9 (Ampere, Turing, Volta, Pascal, or older). |
-| **[W4A16](../examples/quantization_w4a16/README.md)** | Quantizes only weights to 4-bit integer (INT4) precision, retaining activations in 16-bit floating point (FP16) precision. W4A16 provides ~3.7X smaller weights but requires 16-bit arithmetic operations. W4A16 also supports asymmetric weight quantization. W4A16 provides maximum compression for latency-sensitive applications with limited memory, and useful speed ups in low QPS regimes with more weight compression. The linked example leverages the GPTQ algorithm to decrease quantization loss, but other algorithms like [AWQ](../examples/awq/awq_one_shot.py) can also be leveraged for W4A16 quantization. Recommended for any GPU types. | Requires a calibration dataset. | All NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators |
+| **[W8A8-FP8](../../examples/quantization_w8a8_fp8/README.md)** | 8-bit floating point (FP8) quantization for weights and activations, providing ~2X smaller weights with 8-bit arithmetic operations. Uses channel-wise quantization to compress weights to 8 bits, and uses dynamic per-token or static per-tensor quantization to compress activations to 8 bits. Weights scales that are generated on a per-channel basis or generated on a per-tensor basis are also possible. Channel-wise weights quantization with dynamic per-token activations is the the most performant option. W8A8-FP8 does not require a calibration dataset. Activation quantization is carried out during inference on vLLM. Good for general performance and compression, especially for server and batch inference. | No calibration dataset is required, unless you are doing static per-tensor activation quantization. | Latest NVIDIA GPUs (Hopper and later) and latest AMD GPUs. Recommended for NVIDIA GPUs with compute capability >=9.0 (Hopper and Blackwell) |
+| **[W8A8-INT8](../../examples/quantization_w8a8_int8/README.md)** | 8-bit integer (INT8) quantization for weights and activations, providing ~2X smaller weights with 8-bit arithmetic operations. Uses channel-wise quantization to compress weights to 8 bits using GPTQ, and uses dynamic per-token quantization to compress activations to 8 bits. Weight quantization can be both per-tensor or per-channel for INT8. W8A8-INT8 is good for general performance and compression, especially for server and batch inference. Activation quantization is carried out during inference on vLLM. Activations can be static or dynamic. Additionally, INT8 activations can also be asymmetric. W8A8-INT8 helps improve speed in high QPS scenarios or during offline serving with vLLM. W8A8-INT8 is good for general performance and compression, especially for server and batch inference. | Requires calibration dataset for weight quantization and static per-tensor activation quantization. | Supports all NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators. Recommended for NVIDIA GPUs with compute capability <8.9 (Ampere, Turing, Volta, Pascal, or older). |
+| **[W4A16](../../examples/quantization_w4a16/README.md)** | Quantizes only weights to 4-bit integer (INT4) precision, retaining activations in 16-bit floating point (FP16) precision. W4A16 provides ~3.7X smaller weights but requires 16-bit arithmetic operations. W4A16 also supports asymmetric weight quantization. W4A16 provides maximum compression for latency-sensitive applications with limited memory, and useful speed ups in low QPS regimes with more weight compression. The linked example leverages the GPTQ algorithm to decrease quantization loss, but other algorithms like [AWQ](../../examples/awq/llama_example.py) can also be leveraged for W4A16 quantization. Recommended for any GPU types. | Requires a calibration dataset. | All NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators |
 | **W8A16** | Encodes model weights in 8‑bit integers and activations in 16‑bit integers. W8A16 compression delivers smaller model output size than FP32 and is faster at inferencing on hardware with native 8‑bit integer units. Lower power and memory bandwidth compared to floating‑point.| Requires a calibration dataset. | All NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators |
 | **NVFP4** | 4-bit floating point encoding format introduced with the NVIDIA Blackwell GPU architecture. NVFP4 maintains numerical accuracy across a wide dynamic range of tensor values by using high-precision scale encoding and a two-level micro-block scaling strategy. NVFP4 compression generates a global scale for each tensor, along with local quantization scales for groups of 16 elements. Global scale and local quantization scales are generated for weights and activations. You cannot change the group size. | Requires a calibration dataset. | All NVIDIA Blackwell GPUs or later |
-| [**W8A8-FP8_BLOCK**](../examples/quantization_w8a8_fp8/fp8_block_example.py)| Uses block-wise quantization to compress weights to FP8 in (commonly 128×128 tiles), and dynamic per-token-group (128) quantization for activations. Does not require a calibration dataset. Activation quantization is carried out during inference on vLLM. | Requires a calibration dataset. | All NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators |
+| [**W8A8-FP8_BLOCK**](../../examples/quantization_w8a8_fp8/fp8_block_example.py)| Uses block-wise quantization to compress weights to FP8 in (commonly 128×128 tiles), and dynamic per-token-group (128) quantization for activations. Does not require a calibration dataset. Activation quantization is carried out during inference on vLLM. | Does not require a calibration dataset. | All NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators |
 
 ## Sparsification Compression
 Sparsification reduces model complexity by pruning selected weight values to zero while retaining essential weights in a subset of parameters. Supported formats include:
 
 | Scheme | Description | Data required? | vLLM Hardware Compatibility |
 |--------|-------------|----------------|-----------------------------|
-| **[2:4 Semi-structured Sparsity](../examples/sparse_2of4_quantization_fp8.md)** | Uses semi-structured sparsity (SparseGPT), where, for every four contiguous weights in a tensor, two are set to zero. Uses channel-wise quantization to compress weights to 8 bits and dynamic per-token quantization to compress activations to 8 bits. Useful for better inference than W8A8-FP8, with almost no drop in its [evaluation score](https://neuralmagic.com/blog/24-sparse-llama-fp8-sota-performance-for-nvidia-hopper-gpus/). Small models may experience accuracy drops when the remaining non-zero weights are insufficient to recapitulate the original distribution. | Requires a calibration dataset. | Supported on all NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators. Recommended for compute capability >=9.0 (Hopper and Blackwell) |
+| **[2:4 Semi-structured Sparsity](../../examples/sparse_2of4_quantization_fp8/README.md)** | Uses semi-structured sparsity (SparseGPT), where, for every four contiguous weights in a tensor, two are set to zero. Uses channel-wise quantization to compress weights to 8 bits and dynamic per-token quantization to compress activations to 8 bits. Useful for better inference than W8A8-FP8, with almost no drop in its [evaluation score](https://neuralmagic.com/blog/24-sparse-llama-fp8-sota-performance-for-nvidia-hopper-gpus/). Small models may experience accuracy drops when the remaining non-zero weights are insufficient to recapitulate the original distribution. | Requires a calibration dataset. | Supported on all NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators. Recommended for compute capability >=9.0 (Hopper and Blackwell) |
 | **Unstructured Sparsity** | Unstructured sparsity quantization zeros out individual weights in the model without enforcing a regular pattern. Unlike block or channel pruning, it removes weights wherever they contribute least, yielding a fine‑grained sparse matrix. | Does not require a calibration dataset. | All NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators |

docs/index.md

@@ -39,16 +39,16 @@ Review the [LLM Compressor v0.8.0 release notes](https://github.com/vllm-project
 ## Recent Updates
 
 !!! info "QuIP and SpinQuant-style Transforms" 
-    The newly added [`QuIPModifier`](examples/transform/quip_example.py) and [`SpinQuantModifier`](examples/transform/spinquant_example.py) allow you to quantize models after injecting hadamard weights into the computation graph, reducing quantization error and greatly improving accuracy recovery for low bit-weight and activation quantization.
+    The newly added [`QuIPModifier`](../examples/transform/quip_example.py) and [`SpinQuantModifier`](../examples/transform/spinquant_example.py) allow you to quantize models after injecting hadamard weights into the computation graph, reducing quantization error and greatly improving accuracy recovery for low bit-weight and activation quantization.
 
 !!! info "DeepSeekV3-style Block Quantization Support" 
-    Allows for more efficient compression of large language models without needing a calibration dataset. Quantize a Qwen3 model to [W8A8](examples/quantization_w8a8_fp8.md).
+    Allows for more efficient compression of large language models without needing a calibration dataset. Quantize a Qwen3 model to [W8A8](../examples/quantization_w8a8_fp8/fp8_block_example.py).
 
 !!! info "FP4 Quantization - now with MoE and non-uniform support" 
-    Quantize weights and activations to FP4 and seamlessly run the compressed model in vLLM. Model weights and activations are quantized following the [NVFP4 configuration](https://github.com/neuralmagic/compressed-tensors/blob/f5dbfc336b9c9c361b9fe7ae085d5cb0673e56eb/src/compressed_tensors/quantization/quant_scheme.py#L104). See examples of [FP4 activation support](examples/quantization_w4a4_fp4/llama3_example.py), [MoE support](examples/quantization_w4a4_fp4/qwen_30b_a3b.py), and [Non-uniform quantization support](examples/quantization_non_uniform) where some layers are selectively quantized to FP8 for better recovery. You can also mix other quantization schemes, such as INT8 and INT4.
+    Quantize weights and activations to FP4 and seamlessly run the compressed model in vLLM. Model weights and activations are quantized following the [NVFP4 configuration](https://github.com/neuralmagic/compressed-tensors/blob/f5dbfc336b9c9c361b9fe7ae085d5cb0673e56eb/src/compressed_tensors/quantization/quant_scheme.py#L104). See examples of [FP4 activation support](../examples/quantization_w4a4_fp4/llama3_example.py), [MoE support](../examples/quantization_w4a4_fp4/qwen_30b_a3b.py), and [Non-uniform quantization support](../examples/quantization_non_uniform/README.md) where some layers are selectively quantized to FP8 for better recovery. You can also mix other quantization schemes, such as INT8 and INT4.
 
 !!! info "Llama4 Quantization Support"
-    Quantize a Llama4 model to [W4A16](examples/quantization_w4a16.md) or [NVFP4](examples/quantization_w4a16.md). The checkpoint produced can seamlessly run in vLLM.
+    Quantize a Llama4 model to [W4A16](../examples/quantization_w4a16) or [NVFP4](../examples/quantization_w4a4_fp4/llama4_example.py). The checkpoint produced can seamlessly run in vLLM.
 
 For more information, check out the [latest release on GitHub](https://github.com/vllm-project/llm-compressor/releases/latest).
 

examples/quantization_w4a4_fp4/qwen3_next_example.py

@@ -5,8 +5,7 @@
 from llmcompressor.modifiers.quantization import QuantizationModifier
 from llmcompressor.utils import dispatch_for_generation
 
-# NOTE: Qwen3-Next-80B-A3B-Instruct support is not in transformers<=4.56.2
-# you may need to install transformers from source
+# NOTE: Requires a minimum of transformers 4.57.0
 
 MODEL_ID = "Qwen/Qwen3-Next-80B-A3B-Instruct"