Eval bug: qwen 3 next CUDA still uses a lot of CPU during inference

[fuutott]

Name and Version

PS D:\llama\latest> .\llama-cli.exe --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes
Device 1: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
load_backend: loaded CUDA backend from D:\llama\latest\ggml-cuda.dll
load_backend: loaded RPC backend from D:\llama\latest\ggml-rpc.dll
load_backend: loaded CPU backend from D:\llama\latest\ggml-cpu-sapphirerapids.dll
version: 7300 (2960eb2)
built with Clang 19.1.5 for Windows x86_64

Operating systems

Windows

GGML backends

CUDA

Hardware

RTX PRO 6000 and RTX A6000

Models

qwen next testing with
unsloth\Qwen3-Next-80B-A3B-Instruct-GGUF\Qwen3-Next-80B-A3B-Instruct-UD-Q4_K_XL

Problem description & steps to reproduce

This doesn't seem to affect other models but when inferencing qwen 3 next looks like some things are still run on cpu which results in significant performance loss.
Screenshot was taken during tg stage for qwen next bench

First Bad Commit

No response

Relevant log output

PS D:\llama> .\benchq3nxt.bat

D:\llama>set CUDA_VISIBLE_DEVICES=0

D:\llama>d:/llama/latest/llama-bench.exe   -m d:\models\unsloth\Qwen3-Next-80B-A3B-Instruct-GGUF\Qwen3-Next-80B-A3B-Instruct-UD-Q4_K_XL.gguf   
-p 512   -n 512   -b 1024   -ub 512   -ngl 99  -mmp 0   -fa 1   -o md   -r 3   -d 0
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes
load_backend: loaded CUDA backend from d:\llama\latest\ggml-cuda.dll
load_backend: loaded RPC backend from d:\llama\latest\ggml-rpc.dll
load_backend: loaded CPU backend from d:\llama\latest\ggml-cpu-sapphirerapids.dll
| model                          |       size |     params | backend    | ngl | n_batch | fa | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -: | ---: | --------------: | -------------------: |
| qwen3next ?B Q4_K - Medium     |  42.01 GiB |    79.67 B | CUDA       |  99 |    1024 |  1 |    0 |           pp512 |      1211.32 ± 29.80 |
| qwen3next ?B Q4_K - Medium     |  42.01 GiB |    79.67 B | CUDA       |  99 |    1024 |  1 |    0 |           tg512 |         21.16 ± 0.20 |

build: 2960eb297 (7300)

for comparison gpt-oss on the same build and hardware, cpu remains background level during pp and tg

PS D:\llama> .\bench.bat

D:\llama>set CUDA_VISIBLE_DEVICES=0

D:\llama>d:/llama/latest/llama-bench.exe   -m d:\models\lmstudio-community\gpt-oss-120b-GGUF\gpt-oss-120b-MXFP4-00001-of-00002.gguf   
-p 512   -n 512   -b 1024   -ub 512   -ngl 99  -mmp 0   -fa 1   -o md   -r 3   -d 0
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes
load_backend: loaded CUDA backend from d:\llama\latest\ggml-cuda.dll
load_backend: loaded RPC backend from d:\llama\latest\ggml-rpc.dll
load_backend: loaded CPU backend from d:\llama\latest\ggml-cpu-sapphirerapids.dll
| model                          |       size |     params | backend    | ngl | n_batch | fa | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -: | ---: | --------------: | -------------------: |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | CUDA       |  99 |    1024 |  1 |    0 |           pp512 |     4716.71 ± 175.03 |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | CUDA       |  99 |    1024 |  1 |    0 |           tg512 |        241.51 ± 1.02 |

build: 2960eb297 (7300)
PS D:\llama>

[fuutott]

PS D:\llama> .\run-q3next.bat

D:\llama>set CUDA_VISIBLE_DEVICES=0

D:\llama>d:/llama/latest/llama-server -m d:\models\unsloth\Qwen3-Next-80B-A3B-Instruct-GGUF\Qwen3-Next-80B-A3B-Instruct-UD-Q4_K_XL.gguf -c 131072 --n-gpu-layers 999 --host 0.0.0.0 --flash-attn on --jinja --no-mmap
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes
load_backend: loaded CUDA backend from d:\llama\latest\ggml-cuda.dll
load_backend: loaded RPC backend from d:\llama\latest\ggml-rpc.dll
load_backend: loaded CPU backend from d:\llama\latest\ggml-cpu-sapphirerapids.dll
main: setting n_parallel = 4 and kv_unified = true (add -kvu to disable this)
build: 7300 (2960eb297) with Clang 19.1.5 for Windows x86_64
system info: n_threads = 16, n_threads_batch = 16, total_threads = 32

system_info: n_threads = 16 (n_threads_batch = 16) / 32 | CUDA : ARCHS = 500,610,700,750,800,860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | AMX_INT8 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

init: using 31 threads for HTTP server
start: binding port with default address family
main: loading model
srv    load_model: loading model 'd:\models\unsloth\Qwen3-Next-80B-A3B-Instruct-GGUF\Qwen3-Next-80B-A3B-Instruct-UD-Q4_K_XL.gguf'
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA RTX PRO 6000 Blackwell Workstation Edition) (0000:16:00.0) - 95276 MiB free
llama_model_loader: loaded meta data with 49 key-value pairs and 807 tensors from d:\models\unsloth\Qwen3-Next-80B-A3B-Instruct-GGUF\Qwen3-Next-80B-A3B-Instruct-UD-Q4_K_XL.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3next
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                     general.sampling.top_k i32              = 20
llama_model_loader: - kv   3:                     general.sampling.top_p f32              = 0.800000
llama_model_loader: - kv   4:                      general.sampling.temp f32              = 0.700000
llama_model_loader: - kv   5:                               general.name str              = Qwen3-Next-80B-A3B-Instruct
llama_model_loader: - kv   6:                           general.finetune str              = Instruct
llama_model_loader: - kv   7:                           general.basename str              = Qwen3-Next-80B-A3B-Instruct
llama_model_loader: - kv   8:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   9:                         general.size_label str              = 80B-A3B
llama_model_loader: - kv  10:                            general.license str              = apache-2.0
llama_model_loader: - kv  11:                       general.license.link str              = https://huggingface.co/Qwen/Qwen3-Nex...
llama_model_loader: - kv  12:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv  13:                   general.base_model.count u32              = 1
llama_model_loader: - kv  14:                  general.base_model.0.name str              = Qwen3 Next 80B A3B Instruct
llama_model_loader: - kv  15:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  16:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen3-Nex...
llama_model_loader: - kv  17:                               general.tags arr[str,2]       = ["unsloth", "text-generation"]
llama_model_loader: - kv  18:                      qwen3next.block_count u32              = 48
llama_model_loader: - kv  19:                   qwen3next.context_length u32              = 262144
llama_model_loader: - kv  20:                 qwen3next.embedding_length u32              = 2048
llama_model_loader: - kv  21:              qwen3next.feed_forward_length u32              = 5120
llama_model_loader: - kv  22:             qwen3next.attention.head_count u32              = 16
llama_model_loader: - kv  23:          qwen3next.attention.head_count_kv u32              = 2
llama_model_loader: - kv  24:                   qwen3next.rope.freq_base f32              = 10000000.000000
llama_model_loader: - kv  25: qwen3next.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  26:                qwen3next.expert_used_count u32              = 10
llama_model_loader: - kv  27:             qwen3next.attention.key_length u32              = 256
llama_model_loader: - kv  28:           qwen3next.attention.value_length u32              = 256
llama_model_loader: - kv  29:                     qwen3next.expert_count u32              = 512
llama_model_loader: - kv  30:       qwen3next.expert_feed_forward_length u32              = 512
llama_model_loader: - kv  31: qwen3next.expert_shared_feed_forward_length u32              = 512
llama_model_loader: - kv  32:                  qwen3next.ssm.conv_kernel u32              = 4
llama_model_loader: - kv  33:                   qwen3next.ssm.state_size u32              = 128
llama_model_loader: - kv  34:                  qwen3next.ssm.group_count u32              = 16
llama_model_loader: - kv  35:               qwen3next.ssm.time_step_rank u32              = 32
llama_model_loader: - kv  36:                   qwen3next.ssm.inner_size u32              = 4096
llama_model_loader: - kv  37:             qwen3next.rope.dimension_count u32              = 64
llama_model_loader: - kv  38:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  39:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  40:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  41:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  42:                      tokenizer.ggml.merges arr[str,151387]  = ["─á ─á", "─á─á ─á─á", "i n", "─á t",...
llama_model_loader: - kv  43:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  44:            tokenizer.ggml.padding_token_id u32              = 151654
llama_model_loader: - kv  45:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  46:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  47:               general.quantization_version u32              = 2
llama_model_loader: - kv  48:                          general.file_type u32              = 15
llama_model_loader: - type  f32:  313 tensors
llama_model_loader: - type q8_0:    3 tensors
llama_model_loader: - type q4_K:  390 tensors
llama_model_loader: - type q6_K:   53 tensors
llama_model_loader: - type bf16:   48 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 42.01 GiB (4.53 BPW)
load: printing all EOG tokens:
load:   - 151643 ('<|endoftext|>')
load:   - 151645 ('<|im_end|>')
load:   - 151662 ('<|fim_pad|>')
load:   - 151663 ('<|repo_name|>')
load:   - 151664 ('<|file_sep|>')
load: special tokens cache size = 26
load: token to piece cache size = 0.9311 MB
print_info: arch             = qwen3next
print_info: vocab_only       = 0
print_info: n_ctx_train      = 262144
print_info: n_embd           = 2048
print_info: n_embd_inp       = 2048
print_info: n_layer          = 48
print_info: n_head           = 16
print_info: n_head_kv        = 2
print_info: n_rot            = 64
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 256
print_info: n_embd_head_v    = 256
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 5120
print_info: n_expert         = 512
print_info: n_expert_used    = 10
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 10000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 262144
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 4
print_info: ssm_d_inner      = 4096
print_info: ssm_d_state      = 128
print_info: ssm_dt_rank      = 32
print_info: ssm_n_group      = 16
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = ?B
print_info: model params     = 79.67 B
print_info: general.name     = Qwen3-Next-80B-A3B-Instruct
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 11 ','
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151654 '<|vision_pad|>'
print_info: LF token         = 198 '─è'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 48 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 49/49 layers to GPU
load_tensors:          CPU model buffer size =   166.92 MiB
load_tensors:        CUDA0 model buffer size = 42847.49 MiB
....................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 4
llama_context: n_ctx         = 131072
llama_context: n_ctx_seq     = 131072
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = true
llama_context: freq_base     = 10000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (131072) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
llama_context:  CUDA_Host  output buffer size =     2.32 MiB
llama_kv_cache:      CUDA0 KV buffer size =  3072.00 MiB
llama_kv_cache: size = 3072.00 MiB (131072 cells,  12 layers,  4/1 seqs), K (f16): 1536.00 MiB, V (f16): 1536.00 MiB
llama_memory_recurrent:      CUDA0 RS buffer size =   301.50 MiB
llama_memory_recurrent: size =  301.50 MiB (     4 cells,  48 layers,  4 seqs), R (f32):   13.50 MiB, S (f32):  288.00 MiB
llama_context:      CUDA0 compute buffer size =   484.56 MiB
llama_context:  CUDA_Host compute buffer size =   268.08 MiB
llama_context: graph nodes  = 11401 (with bs=512), 8449 (with bs=1)
llama_context: graph splits = 78 (with bs=512), 6 (with bs=1)
common_init_from_params: added <|endoftext|> logit bias = -inf
common_init_from_params: added <|im_end|> logit bias = -inf
common_init_from_params: added <|fim_pad|> logit bias = -inf
common_init_from_params: added <|repo_name|> logit bias = -inf
common_init_from_params: added <|file_sep|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 131072
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv          init: initializing slots, n_slots = 4
slot         init: id  0 | task -1 | new slot, n_ctx = 131072
slot         init: id  1 | task -1 | new slot, n_ctx = 131072
slot         init: id  2 | task -1 | new slot, n_ctx = 131072
slot         init: id  3 | task -1 | new slot, n_ctx = 131072
srv          init: prompt cache is enabled, size limit: 8192 MiB
srv          init: use `--cache-ram 0` to disable the prompt cache
srv          init: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
srv          init: thinking = 0
init: chat template, chat_template: {%- if tools %}
    {{- '<|im_start|>system\n' }}
    {%- if messages[0].role == 'system' %}
        {{- messages[0].content + '\n\n' }}
    {%- endif %}
    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
    {%- for tool in tools %}
        {{- "\n" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
    {%- if messages[0].role == 'system' %}
        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
    {%- endif %}
{%- endif %}
{%- for message in messages %}
    {%- if message.content is string %}
        {%- set content = message.content %}
    {%- else %}
        {%- set content = '' %}
    {%- endif %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
    {%- elif message.role == "assistant" %}
        {{- '<|im_start|>' + message.role + '\n' + content }}
        {%- if message.tool_calls %}
            {%- for tool_call in message.tool_calls %}
                {%- if (loop.first and content) or (not loop.first) %}
                    {{- '\n' }}
                {%- endif %}
                {%- if tool_call.function %}
                    {%- set tool_call = tool_call.function %}
                {%- endif %}
                {{- '<tool_call>\n{"name": "' }}
                {{- tool_call.name }}
                {{- '", "arguments": ' }}
                {%- if tool_call.arguments is string %}
                    {{- tool_call.arguments }}
                {%- else %}
                    {{- tool_call.arguments | tojson }}
                {%- endif %}
                {{- '}\n</tool_call>' }}
            {%- endfor %}
        {%- endif %}
        {{- '<|im_end|>\n' }}
    {%- elif message.role == "tool" %}
        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
            {{- '<|im_start|>user' }}
        {%- endif %}
        {{- '\n<tool_response>\n' }}
        {{- content }}
        {{- '\n</tool_response>' }}
        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
            {{- '<|im_end|>\n' }}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
{%- endif %}, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: model loaded
main: server is listening on http://0.0.0.0:8080
main: starting the main loop...
srv  update_slots: all slots are idle

[jacekpoplawski]

I confirm high CPU usage on Linux, however I wonder why Q6 on 3x3090 is faster than Q4 on 6000

$ llama-bench -m /mnt/models2/Qwen_Qwen3-Next-80B-A3B-Instruct-Q6_K_L/Qwen_Qwen3-Next-80B-A3B-Instruct-Q6_K_L-00001-of-00002.gguf -p 512 -n 512 -b 1024 -ub 512 -ngl 99 -mmp 0 -fa 1 -o md -r 3 -d 0
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 3 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes

model	size	params	backend	ngl	n_batch	fa	mmap	test	t/s
qwen3next ?B Q6_K	61.20 GiB	79.67 B	CUDA	99	1024	1	0	pp512	565.84 ± 1.13
qwen3next ?B Q6_K	61.20 GiB	79.67 B	CUDA	99	1024	1	0	tg512	42.45 ± 0.02

build: 96fe9ba (7276)

Are there any tools or techniques for profiling a llama model?

[fuutott]

I think the difference is due to cpu performance differences, w5-3435X with 8 channels ddr5 4800.

[fuutott]

shed-debug.txt

[jacekpoplawski]

Your hypothesis sounds reasonable if the CPU/RAM is the bottleneck, but please look at my setup:

Setup

Machine:
Type: Desktop Mobo: ASRock model: X399 Taichi serial:
UEFI-[Legacy]: American Megatrends v: P4.03 date: 01/18/2024
Memory:
System RAM: total: 128 GiB available: 121.43 GiB used: 3.88 GiB (3.2%)
Message: For most reliable report, use superuser + dmidecode.
Array-1: capacity: 512 GiB slots: 8 modules: 4 EC: None
Device-1: Channel-A DIMM 0 type: no module installed
Device-2: Channel-A DIMM 1 type: DDR4 size: 32 GiB speed: 3200 MT/s
Device-3: Channel-B DIMM 0 type: no module installed
Device-4: Channel-B DIMM 1 type: DDR4 size: 32 GiB speed: 3200 MT/s
Device-5: Channel-C DIMM 0 type: no module installed
Device-6: Channel-C DIMM 1 type: DDR4 size: 32 GiB speed: 3200 MT/s
Device-7: Channel-D DIMM 0 type: no module installed
Device-8: Channel-D DIMM 1 type: DDR4 size: 32 GiB speed: 3200 MT/s
CPU:
Info: 12-core model: AMD Ryzen Threadripper 1920X bits: 64 type: MT MCP cache: L2: 6 MiB
Speed (MHz): avg: 1962 min/max: 2200/3500 cores: 1: 1962 2: 1962 3: 1962 4: 1962 5: 1962
6: 1962 7: 1962 8: 1962 9: 1962 10: 1962 11: 1962 12: 1962 13: 1962 14: 1962 15: 1962 16: 1962
17: 1962 18: 1962 19: 1962 20: 1962 21: 1962 22: 1962 23: 1962 24: 1962

I use these build commands:

CUDAHOSTCXX=/usr/bin/g++-12 cmake -DGGML_CUDA=ON -DLLAMA_CURL=OFF ..
cmake --build . --config Release -j 24

nvidia-smi says I am on Driver Version: 580.105.08 and CUDA Version: 13.0

[fuutott]

On paper my machine is nearly twice as fast.
I download releases :) cuda def kicks in on gpt-oss bench from the initial post provided for comparison.

[pwilkin]

n = 64, k = 64 case for SOLVE_TRI needs to be supported on CUDA

[fuutott]

I believe my issue will be fixed by #17793 but anyone on cuda should be also paying attention to #17703