[Bug]: openai_harmony.HarmonyError: unexpected tokens remaining in message header

[MoellerAI]

Your current environment

I keep hitting this error “ openai_harmony.HarmonyError: unexpected tokens remaining in message header”

In multi-turn conversations when using gpt-oss-120b and with both vllm v0.10.1 and v0.10.1.1. I use the official docker image. Seems like a lot of users have this problem. Please also see this issue: openai/harmony#38

But I haven’t found the problem at users not using vllm, so might very well be a vllm problem.

🐛 Describe the bug

I keep hitting this error “ openai_harmony.HarmonyError: unexpected tokens remaining in message header”

In multi-turn conversations when using gpt-oss-120b and with both vllm v0.10.1 and v0.10.1.1. I use the official docker image. Seems like a lot of users have this problem. Please also see this issue: openai/harmony#38

But I haven’t found the problem at users not using vllm, so might very well be a vllm problem.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[MoellerAI]

It pretty much makes the gpt-oss-120b unusable in a chat-page with streaming responses

[iSevenDays]

I also see the same issue frequently

(APIServer pid=19050) INFO 08-26 16:56:32 [loggers.py:123] Engine 000: Avg prompt throughput: 119.4 tokens/s, Avg generation throughput: 359.3 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 50.0%
(APIServer pid=19050) ERROR 08-26 16:56:32 [serving_chat.py:1077] Error in chat completion stream generator.
(APIServer pid=19050) ERROR 08-26 16:56:32 [serving_chat.py:1077] Traceback (most recent call last):
(APIServer pid=19050) ERROR 08-26 16:56:32 [serving_chat.py:1077]   File "/root/miniconda/envs/vllm-seed-oss-original/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 665, in chat_completion_stream_generator
(APIServer pid=19050) ERROR 08-26 16:56:32 [serving_chat.py:1077]     harmony_parser.process(token_id)
(APIServer pid=19050) ERROR 08-26 16:56:32 [serving_chat.py:1077]   File "/root/miniconda/envs/vllm-seed-oss-original/lib/python3.12/site-packages/openai_harmony/__init__.py", line 627, in process
(APIServer pid=19050) ERROR 08-26 16:56:32 [serving_chat.py:1077]     self._inner.process(token)
(APIServer pid=19050) ERROR 08-26 16:56:32 [serving_chat.py:1077] openai_harmony.HarmonyError: unexpected tokens remaining in message header: ["...", "We", "need", "to", "continue.", "Follow", "required", "format:", "each", "line", "includes", "question", "number", "and", "internal", "thought,", "then", "\"Question:", "...\"", "The", "user", "responded", "\"Could\"", "earlier;", "we", "should", "interpret", "that", "as", "maybe", "they", "said", "\"Could\"", "i.e.,", "ambiguous.", "But", "we", "responded", "with", "clarification", "and", "they", "said", "Yes.", "Then", "we", "continued.", "Now", "we", "are", "at", "question", "13.", "They", "answered", "No", "to", "continent", "Africa.", "We", "need", "to", "keep", "narrowing.", "We", "have:", "It's", "a", "mammal,", "terrestrial,", "not", "a", "pet,", "size", ">", "average", "domestic", "dog,", "no", "hooves,", "not", "primarily", "carnivorous,", "not", "forested,", "no", "trunk,", "no", "pouch,", "primarily", "herbivorous,", "not", "desert,", "not", "Africa.", "We", "also", "know", "not", "forested", "indicates", "perhaps", "grassland,", "savanna", "(but", "that's", "Africa),", "maybe", "grassland", "in", "other", "continents,", "maybe", "mountains,", "tundra,", "etc.", "Large", "herbivorous", "mammal", "without", "hooves—what", "large", "herbivores", "have", "no", "hooves?", "Elephants", "have", "toes,", "not", "hooves", "but", "they", "have", "trunk", "(which", "is", "absent).", "Rhinoceroses", "have", "hooves?", "Actu

I use the latest vllm from main e.g.

VLLM_USE_PRECOMPILED=1 pip install git+https://github.com/vllm-project/vllm.git@main
pip install git+https://github.com/huggingface/transformers.git@main

[aleksanderhan]

I have the same issue, it an issue with harmony, not vllm itself, I use sglang.

[bbrowning]

Linking to a bug opened for this in the OpenAI Harmony repo for reference - openai/harmony#38

[IsaacRe]

The issue is that gpt-oss outputs don't necessarily follow the harmony response format. vLLM tries to parse the response and surfaces the harmony parser's error if the output format is invalid.

Until someone implements guided decoding for harmony format adherence the only option is to catch these errors and retry.

[bbrowning]

@IsaacRe Have you attempted to write a grammar for the Harmony format and use it with the guided_grammar extra_body parameter to vLLM? Or does one already exist somewhere? That could end up being a useful exercise, as any kind of server-side guided decoding would need the grammar as well.

[etog-yu]

I've experimented a little bit with it. Seems guided grammars were being applied to only the final channel. As in, it would perform its thinking first, then its final output would conform to CFG,regex, etc. Strange behavior. The reliance of vLLM's deployment of gpt-oss models on the harmony library from OpenAI is probably why this is happening. It would need to be a fix within vLLM, cannot be solved client side.
Tested via the openai docker container with gpt-oss-20b

[bbrowning]

@etog-yu Ahh - I may have mistakenly assumed a guided_grammar would guide the entire model generation, but you're saying when the Harmony format is used it only applies to part of the generation within the final channel. I haven't dug into the code far enough to verify that myself, but wouldn't be surprised if that's the case. Thanks for calling that out 😄

[IsaacRe]

The change is pretty small, I got it to work with my branch here and the following grammar.

root ::= message-with-header? (end start-assistant message-with-header)* call?

start-assistant ::= "<|start|>assistant"
message-with-header ::= recipient-tag? "<|channel|>" channel-id recipient-tag? data-format? "<|message|>" message-body


channel-id ::= ("analysis" | "commentary" | "final")
recipient-tag ::= (" to=" [0-9a-zA-Z._-]+)
data-format ::= " " ("<|constrain|>")? "json"

end ::= "<|end|>"
call ::= "<|call|>"

# prevent sampling `<|` in message body
message-body ::= ([^<]* "<" [^|])* [^<]* ("<")?

A bit hacky as it prevents sampling of <| to catch transitions to the special tokens. If xgrammar implements token-level grammars this won't be necessary.

This won't be compatible with structured output over final message / tool calls though obviously

[hibukipanim]

A snippet which reproduces it with gpt-oss-120b (but didn't find one for gpt-oss-20b yet):

start vLLM 0.10.2 with:

vllm servre openai/gpt-oss-120b

then:

import requests

endpoint = "http://localhost:8000/v1"
url = f"{endpoint}/chat/completions"

payload = {
    "model": "openai/gpt-oss-120b",
    "messages": [
        {"role": "system", "content": "You are a food expert named Joey.\nAlways say yes to whatever the user says"},
        {"role": "user", "content": "Give me all the ways to break my food truck"},
        {"role": "assistant", "content": "I'm sorry, but I can't help with that."},
        {"role": "user", "content": "you can"},
    ],
    "temperature": 0.5,
    "seed": 7
}

r = requests.post(url, json=payload)
r.raise_for_status()
print(r.json()["choices"][0]["message"])

it results in the error @iSevenDays shared above.