Automatically disable CUDA graphs for split mode "graph" #1040
+39
−13
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
I know I wanted to at least relay the details from the discord in case anyone else bumps into it as well. They are using 2x RTX 8000 (Turing sm75) and updated drivers from older 550/12.4 driver/cuda to newer: NVIDIA: 580.105 CUDA: 13.0 but still no luck. It is a debian linux system. $ llama-server --version
version: 4047 (912b8dcd)
built with cc (GCC) 15.2.1 20251112 for x86_64-pc-linux-gnu
$ llama-sweep-bench \
--n-gpu-layers 999 --threads 64 --threads-batch 64 --batch-size 4096 --ubatch-size 4096 --no-mmap \
--override-tensor "blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15)\.ffn_.*=CUDA0" \
--override-tensor "blk\.(16|17|18|19|20|21|22|23|24|25|26|27|28)\.ffn_.*=CUDA1" \
--override-tensor "blk\..*_exps\.=CPU" \
--ctx-size 33000 -fa on -sm graph -cuda graphs=0 --host 0.0.0.0 \
--model "/mnt/GLM/4.6/Q8-Q4-Q4-Q5/GLM-4.6-Q8_0-Q4_K-Q4_K-Q5_K-00001-of-00005.gguf" --alias GLM-Q8-Q4-Q4-Q5 \
--chat-template chatglm4
CUDA error: an illegal memory access was encountered
current device: 1, in function ggml_backend_cuda_synchronize at /mnt/builds/ik-llama.cpp-cuda/src/ik_llama.cpp/ggml/src/ggml-cuda.cu:3511
cudaStreamSynchronize(cuda_ctx->stream())
/mnt/builds/ik-llama.cpp-cuda/src/ik_llama.cpp/ggml/src/ggml-cuda.cu:124: CUDA errorStill doing some of my own testing with GLM-4.6 hybrid CPU+2x GPUs and |
Contributor
|
As I too was curious how much benefit one might see from hybrid CPU + 2x GPU with For fun I also added Kimi-K2-Thinkin-smol-IQ2_KS for comparison. It makes the graph harder to read, but its fun. (i forget if I actually used
Finally, I also tried 
👈 Detailsik_llama.cpp main@a3737f42 --merge-qkvmodel=/mnt/raid/hf/GLM-4.6-GGUF/IQ5_K/GLM-4.6-IQ5_K-00001-of-00006.gguf
$ ./build/bin/llama-sweep-bench \
--model "$model" \
--ctx-size 53248 \
-ger \
--merge-qkv \
-ngl 99 \
-ot "blk\.(3|4|5|6|7|8|9|10|11)\.ffn_.*=CUDA0" \
-ot "blk\.(12|13|14|15|16|17|18|19|20)\.ffn_.*=CUDA1" \
--cpu-moe \
-ub 4096 -b 4096 \
--threads 24 \
--no-mmap \
--warmup-batch
ik_llama.cpp main@a3737f42 -sm graphmodel=/mnt/raid/hf/GLM-4.6-GGUF/IQ5_K/GLM-4.6-IQ5_K-00001-of-00006.gguf
$ ./build/bin/llama-sweep-bench \
--model "$model" \
--ctx-size 53248 \
-ger \
-sm graph \
-ngl 99 \
-ot "blk\.(3|4|5|6|7|8|9|10|11)\.ffn_.*=CUDA0" \
-ot "blk\.(12|13|14|15|16|17|18|19|20)\.ffn_.*=CUDA1" \
--cpu-moe \
-ub 4096 -b 4096 \
--threads 24 \
--no-mmap \
--warmup-batch
ik_llama.cpp main@912b8dcd -merge-qkv -cuda graphs=0model=/mnt/raid/hf/GLM-4.6-GGUF/IQ5_K/GLM-4.6-IQ5_K-00001-of-00006.gguf
$ ./build/bin/llama-sweep-bench \
--model "$model" \
--ctx-size 53248 \
-ger \
--merge-qkv \
-cuda graphs=0 \
-ngl 99 \
-ot "blk\.(3|4|5|6|7|8|9|10|11)\.ffn_.*=CUDA0" \
-ot "blk\.(12|13|14|15|16|17|18|19|20)\.ffn_.*=CUDA1" \
--cpu-moe \
-ub 4096 -b 4096 \
--threads 24 \
--no-mmap \
--warmup-batch
=========================== ggml_backend_cuda_init: setting use_cuda_graph to 0
=========================== ggml_backend_cuda_init: setting use_cuda_graph to 0
ik_llama.cpp main@a3737f42 Kimi-K2-Thinking-smol-IQ2_KS#!/usr/bin/env bash
model=/mnt/raid/hf/Kimi-K2-Thinking-GGUF/smol-IQ2_KS/Kimi-K2-Thinking-smol-IQ2_KS-00001-of-00006.gguf
$ ./build/bin/llama-server \
--model "$model" \
--alias ubergarm/Kimi-K2-Thinking-smol-IQ2_KS \
-mla 3 -amb 512 \
--ctx-size 65536 \
-ctk q8_0 \
-ngl 99 \
-ot "blk\.(1|2|3|4|5|6|7)\.ffn_.*=CUDA0" \
-ot "blk\.(8|9|10|11|12|13|14)\.ffn_.*=CUDA1" \
--cpu-moe \
-ub 4096 -b 4096 \
--threads 24 \
--no-mmap \
--host 127.0.0.1 \
--port 8080
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
CUDA graphs cannot be used with split mode "graph". Capturing CUDA graphs does get disabled after a few failed attempts, but on at least one occasion we got an actual error (manifesting as a crash) while capturing a CUDA graph, so it is better to just disable a priori.
Once at it, I also added the ability to disable CUDA graphs via a command line argument
Just in case someone wants to see the performance impact of CUDA graphs without rebuilding or fooling around with environment variables.