[model_free_ptq] NVFP4A16

[kylesayrs]

Purpose

• Support NVFP4A16 for model_free_ptq

llmcompressor.reindex_fused_weights \
    unsloth/Kimi-K2-Thinking-BF16 \
    Kimi-K2-Thinking-BF16-reindexed \
    --num_workers=10

model_free_ptq(
    model_stub="Kimi-K2-Thinking-BF16-reindexed",
    save_directory="Kimi-K2-Thinking-BF16-NVFP4A16",
    scheme="NVFP4A16",
    ignore=[
        "re:.*gate$",
        "lm_head",
        "re:.*kv_a_proj_with_mqa$",
        "re:.*q_a_proj$",
        "model.embed_tokens",
    ],
    max_workers=15,
    device="cuda:0",
)

Changes

• Restructure files
  • Move validate_scheme to validate.py
  • Move find_safetensors_index_path, find_config_path, find_safetensors_index_file to helpers.py
  • Move process_file to process.py
  • Move validate_scheme to validate.py
  • Break calibrate_weights into calibrate_global_scale and calibrate_scale_zp
• Add extra utility functions
  • match_names_set_eager
  • invert_mapping
• Add microscale/fused module utility functions
  • is_microscale_scheme
  • get_fused_names
• Add process_file_microscale_scheme to separate the fp4 lifecycle from the regular lifecycle (this script should be very trustworthy. By separating the functions, an FP8 user does not have to trust anything about FP4)
• Add llm.compressor.reindex_fused_weights script which reindexes a model's weights so that fused modules are in the same files.
• Fix bug where safetensors index metadata was not being saved correctly

Testing

• Add NVFP4A16 to test_model_free_ptq_matches_oneshot
• Regression tested large mistral model e2e with FP8_BLOCK
• Tested large mistral model e2e with NVFP4A16

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[loqs]

@kylesayrs is it expected that models built using this pull request will fail currently to load in vllm due to an llvm bug vllm-project/vllm#21977?

[kylesayrs]

@loqs Yes, thank you for the clarification. I mentioned in the linked issue that this is relatively trivial to enable on vLLM, I'll make sure to share my branches soon.