Is it possible to use a LoRA with a K-quantized model on GPU

[LostRuins]

I would've sworn it was possible at one point, but perhaps I misremembered it. Currently it doesn't work, as cuda get_rows and vulkan ggml_copy do not support K quants.

[stduhpf]

It works on CPU backend. Maybe it could make sense to add a way to apply LoRA on CPU backend and then move the model back to VRAM for inference?

[leejet]

I'm currently refactoring the LoRA part. Once the refactoring is complete, it will support using LoRA with a K-quantized model.

[LostRuins]

Speaking of which, do you know what's the best way to create all these gguf quants for image models?

I know the recent readmes in wan.md and qwen_image.md have links to GGUFs made by QuantFactory and Phil2Sat but I don't think they are using stable-diffusion.cpp itself to make the quants (especially since they offer k-quants as well).

Is there a pipeline or standard tool for this, or just private conversion scripts?

[stduhpf]

I think they're using an old patched version of llama.cpp (such a patch can be found there https://github.com/city96/ComfyUI-GGUF/tree/main/tools).

Maybe I should update my imatrix branch eventually, results seemed somewhat promising.

[wbruna]

(especially since they offer k-quants as well).

FWIW, mainline sd.cpp can create a few k-quants, even in cases they end up not working (like k-quants with LoRAs). But I believe most available ggufs follow that patched llama.cpp route.

[LostRuins]

It would be good to formalize the conversion process sometime.