Skip to content

Commit 91dc71e

Browse files
[JS] Fix structured output test (#2951)
## Description - Add comparison JavaScript samples with Python. Ticket: 175278 ## Checklist: - [x] Tests have been updated or added to cover the new code <!--- If the change isn't maintenance related, update the tests at https://github.com/openvinotoolkit/openvino.genai/tree/master/tests or explain in the description why the tests don't need an update. --> - [x] This patch fully addresses the ticket. <!--- If follow-up pull requests are needed, specify in description. --> - [ ] I have made corresponding changes to the documentation --------- Signed-off-by: Kirill Suvorov <kirill.suvorov@intel.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent f08ceef commit 91dc71e

13 files changed

+209
-149
lines changed

samples/js/text_generation/README.md

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -175,27 +175,30 @@ If the model does not generate trigger strings there will be no structural const
175175
The sample is verified with `meta-llama/Llama-3.2-3B-Instruct` model. Other models may not produce the expected results or might require different system prompt.
176176

177177

178-
### 9. Compound Grammar Generation Sample (`compound_grammar_generation`)
178+
### 9. Compound Grammar Generation with Parsing Sample (`compound_grammar_generation`)
179179
- **Description:**
180-
This sample demonstrates advanced structured output generation using compound grammars in OpenVINO GenAI.
181-
It showcases how to combine multiple grammar types - Regex, JSONSchema and EBNF - using Union and Concat operations to strictly control LLM output.
180+
This sample demonstrates advanced structured output generation and results parsing using compound grammars in OpenVINO GenAI.
181+
It showcases how to combine multiple grammar types - Regex, JSONSchema and EBNF - using Union and Concat operations to strictly control LLM output and
182+
also shows how to write parsing logic to extract structured data from the generated output.
182183
It features multi-turn chat, switching grammar constraints between turns (e.g., "yes"/"no" answers and structured tool calls).
183184
Union operation allows the model to choose which grammar to use during generation.
184185
In the sample it is used to combine two regex grammars for `"yes"` or `"no"` answer.
185-
Concat operation allows to start with one grammar and continue with another.
186+
Concat operation allows to start with one grammar and continue with another.
187+
Also it demonstrates how to write custom parser to extract tool calls from the generated text.
186188
In the sample it used to create a `phi-4-mini-instruct` style tool calling answer - `functools[{tool_1_json}, ...]` - by combining regex and JSON schema grammars.
187189

188190
- **Main Features:**
189191
- Create grammar building blocks: Regex, JSONSchema, EBNF grammar
190192
- Combine grammars with Concat and Union operations
191193
- Multi-turn chat with grammar switching
192194
- Structured tool calling using zod schemas
195+
- Parse generated output to call tools from extracted structured data
193196
- **Run Command:**
194197
```bash
195198
node compound_grammar_generation.js model_dir
196199
```
197200
- **Notes:**
198-
This sample is ideal for scenarios requiring strict control over LLM outputs, such as building agents that interact with APIs or require validated structured responses. It showcases how to combine regex triggers and JSON schema enforcement for robust output generation.
201+
This sample is ideal for scenarios requiring strict control over LLM outputs, such as building agents that interact with APIs or require validated structured responses. It showcases how to combine regex triggers and JSON schema enforcement for robust output generation and parsing resulting output.
199202
The sample is verified with `microsoft/Phi-4-mini-instruct` model. Other models may not produce the expected results or might require different system prompt.
200203

201204
#### Options

samples/js/text_generation/compound_grammar_generation.js

Lines changed: 51 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,35 +1,35 @@
11
import { z } from 'zod';
2-
import { LLMPipeline, StructuredOutputConfig as SOC, StreamingStatus } from 'openvino-genai-node';
3-
import { serialize_json } from './helper.js';
2+
import { ChatHistory, LLMPipeline, StructuredOutputConfig as SOC, StreamingStatus } from 'openvino-genai-node';
3+
import { serialize_json, toJSONSchema } from './helper.js';
44

55
function streamer(subword) {
66
process.stdout.write(subword);
77
return StreamingStatus.RUNNING;
88
}
99

10-
const bookingFlightTickets = {
11-
name: "booking_flight_tickets",
10+
const bookFlightTicket = {
11+
name: "book_flight_ticket",
1212
schema: z.object({
1313
origin_airport_code: z.string().describe("The name of Departure airport code"),
1414
destination_airport_code: z.string().describe("The name of Destination airport code"),
1515
departure_date: z.string().describe("The date of outbound flight"),
1616
return_date: z.string().describe("The date of return flight"),
17-
}),
17+
}).describe("booking flights"),
1818
};
1919

20-
const bookingHotels = {
21-
name: "booking_hotels",
20+
const bookHotel = {
21+
name: "book_hotel",
2222
schema: z.object({
2323
destination: z.string().describe("The name of the city"),
2424
check_in_date: z.string().describe("The date of check in"),
2525
checkout_date: z.string().describe("The date of check out"),
26-
}),
26+
}).describe("booking hotel"),
2727
};
2828

2929
// Helper functions
3030
function toolToDict(tool, withDescription = true) {
31-
const deleteDescription = (schema) => delete schema.jsonSchema['description'];
32-
const jsonSchema = z.toJSONSchema(
31+
const deleteDescription = (ctx) => delete ctx.jsonSchema['description'];
32+
const jsonSchema = toJSONSchema(
3333
tool.schema,
3434
withDescription
3535
? undefined
@@ -46,11 +46,6 @@ function toolToDict(tool, withDescription = true) {
4646
};
4747
}
4848

49-
/** Generate part of the system prompt with available tools */
50-
function generateSystemPromptTools(...tools) {
51-
return `<|tool|>${serialize_json(tools.map(toolToDict))}</|tool|>`;
52-
}
53-
5449
function toolsToArraySchema(...tools) {
5550
return serialize_json({
5651
type: "array",
@@ -60,22 +55,49 @@ function toolsToArraySchema(...tools) {
6055
});
6156
}
6257

58+
/** parser to extract tool calls from the model output. */
59+
function parse(answer) {
60+
answer.parsed = [];
61+
for (const content of answer.texts) {
62+
const startTag = "functools";
63+
const startIndex = content.indexOf(startTag);
64+
if (startIndex === -1) return;
65+
66+
try {
67+
const jsonPart = content.slice(startIndex + startTag.length);
68+
const toolCalls = JSON.parse(jsonPart);
69+
answer.parsed.push(toolCalls);
70+
} catch {
71+
answer.parsed.push([]);
72+
}
73+
}
74+
75+
return;
76+
}
77+
78+
function printToolCall(answer) {
79+
for (const toolCall of answer.parsed[0]) {
80+
const args = Object.keys(toolCall["arguments"])
81+
.map((key) => `${key}="${toolCall["arguments"][key]}"`);
82+
console.log(`${toolCall["name"]}(${args.join(", ")})`);
83+
}
84+
}
85+
6386
// System message
6487
let sysMessage = `You are a helpful AI assistant.
6588
You can answer yes or no to questions, or you can choose to call one or more of the provided functions.
6689
6790
Use the following rule to decide when to call a function:
6891
* if the response can be generated from your internal knowledge, do so, but use only yes or no as the response
6992
* if you need external information that can be obtained by calling one or more of the provided functions, generate function calls
70-
93+
7194
If you decide to call functions:
7295
* prefix function calls with functools marker (no closing marker required)
7396
* all function calls should be generated in a single JSON list formatted as functools[{"name": [function name], "arguments": [function arguments as JSON]}, ...]
7497
* follow the provided JSON schema. Do not hallucinate arguments or values. Do not blindly copy values from the provided samples
7598
* respect the argument type formatting. E.g., if the type is number and format is float, write value 7 as 7.0
7699
* make sure you pick the right functions that match the user intent
77100
`;
78-
sysMessage += generateSystemPromptTools(bookingFlightTickets, bookingHotels);
79101

80102
async function main() {
81103
const modelDir = process.argv[2];
@@ -86,7 +108,9 @@ async function main() {
86108

87109
const pipe = await LLMPipeline(modelDir, "CPU");
88110
const tokenizer = await pipe.getTokenizer();
89-
const chatHistory = [{ role: "system", content: sysMessage }];
111+
const chatHistory = new ChatHistory([{ role: "system", content: sysMessage }]);
112+
const tools = [bookFlightTicket, bookHotel].map((tool) => toolToDict(tool, true));
113+
chatHistory.setTools(tools);
90114

91115
const generationConfig = {
92116
return_decoded_results: true,
@@ -104,27 +128,31 @@ async function main() {
104128
const yesOrNo = SOC.Union(SOC.Regex("yes"), SOC.Regex("no"));
105129
generationConfig.structured_output_config = new SOC({ structural_tags_config: yesOrNo });
106130
process.stdout.write("Assistant: ");
107-
const answer = await pipe.generate(modelInput, generationConfig, streamer);
108-
chatHistory.push({ role: "assistant", content: answer.texts[0] });
131+
const answer1 = await pipe.generate(modelInput, generationConfig, streamer);
132+
chatHistory.push({ role: "assistant", content: answer1.texts[0] });
109133
console.log();
110134

111135
const userText2 =
112-
"book flight ticket from Beijing to Paris(using airport code) in 2025-12-04 to 2025-12-10 , "
136+
"book flight ticket from Beijing to Paris(using airport code) in 2025-12-04 to 2025-12-10, "
113137
+ "then book hotel from 2025-12-04 to 2025-12-10 in Paris";
114138
console.log("User: ", userText2);
115139
chatHistory.push({ role: "user", content: userText2 });
116140
const modelInput2 = tokenizer.applyChatTemplate(chatHistory, true);
117141

118142
const startToolCallTag = SOC.ConstString("functools");
119143
const toolsJson = SOC.JSONSchema(
120-
toolsToArraySchema(bookingFlightTickets, bookingHotels)
144+
toolsToArraySchema(bookFlightTicket, bookHotel)
121145
);
122146
const toolCall = SOC.Concat(startToolCallTag, toolsJson);
123147

124148
generationConfig.structured_output_config.structural_tags_config = toolCall;
125149

126150
process.stdout.write("Assistant: ");
127-
await pipe.generate(modelInput2, generationConfig, streamer);
151+
const answer2 = await pipe.generate(modelInput2, generationConfig);
152+
parse(answer2);
153+
console.log("\n\nThe following tool calls were generated:")
154+
printToolCall(answer2)
155+
console.log();
128156
}
129157

130158
main();
Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,35 @@
11
// Copyright(C) 2025 Intel Corporation
22
// SPDX - License - Identifier: Apache - 2.0
33

4+
import { z } from 'zod';
5+
46
/** Serialize a JavaScript object to a JSON string
57
* with specific formatting to align with Python. */
68
export function serialize_json(object) {
79
return JSON.stringify(object)
8-
.replaceAll('":', '": ')
9-
.replaceAll('",', '", ');
10+
// Add a space after every colon or comma not already followed by a space
11+
.replace(/(:|,)(?! )/g, '$1 ');
1012
}
13+
14+
/** Convert a Zod schema to a JSON Schema
15+
* with specific formatting to align with Python */
16+
export function toJSONSchema(zodSchema, params) {
17+
const jsonSchema = z.toJSONSchema(
18+
zodSchema,
19+
{
20+
override: (ctx) => {
21+
if (params && params.override) {
22+
params.override(ctx);
23+
}
24+
const keys = Object.keys(ctx.jsonSchema).sort();
25+
for (const key of keys) {
26+
const value = ctx.jsonSchema[key];
27+
delete ctx.jsonSchema[key];
28+
ctx.jsonSchema[key] = value;
29+
}
30+
}
31+
});
32+
delete jsonSchema.$schema;
33+
delete jsonSchema.additionalProperties;
34+
return jsonSchema;
35+
}

samples/js/text_generation/structural_tags_generation.js

Lines changed: 23 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -9,19 +9,19 @@ import { serialize_json } from './helper.js';
99
const getWeatherTool = {
1010
name: "get_weather",
1111
schema: z.object({
12-
city: z.string().describe("City name"),
13-
country: z.string().describe("Country name"),
14-
date: z.string().regex(/2\d{3}-[0-1]\d-[0-3]\d/).describe("Date in YYYY-MM-DD format")
15-
}),
12+
city: z.string().describe("City name").meta({ title: "City" }),
13+
country: z.string().describe("Country name").meta({ title: "Country" }),
14+
date: z.string().regex(/2\d\d\d-[0-1]\d-[0-3]\d/).describe("Date in YYYY-MM-DD format").meta({ title: "Date" })
15+
}).meta({ title: "WeatherRequest" }),
1616
};
1717

1818
const getCurrencyExchangeTool = {
1919
name: "get_currency_exchange",
2020
schema: z.object({
21-
from_currency: z.string().describe("Currency to convert from"),
22-
to_currency: z.string().describe("Currency to convert to"),
23-
amount: z.number().describe("Amount to convert")
24-
}),
21+
from_currency: z.string().describe("Currency to convert from").meta({ title: "From Currency" }),
22+
to_currency: z.string().describe("Currency to convert to").meta({ title: "To Currency" }),
23+
amount: z.number().describe("Amount to convert").meta({ title: "Amount" })
24+
}).meta({ title: "CurrencyExchangeRequest" }),
2525
};
2626

2727
const tools = [getWeatherTool, getCurrencyExchangeTool];
@@ -35,7 +35,8 @@ ${tools.map(tool => `<function_name=\"${tool.name}\">, arguments=${serialize_jso
3535
Please, only use the following format for tool calling in your responses:
3636
<function=\"function_name\">{"argument1": "value1", ...}</function>
3737
Use the tool name and arguments as defined in the tool schema.
38-
If you don't know the answer, just say that you don't know, but try to call the tool if it helps to answer the question.`;
38+
If you don't know the answer, just say that you don't know, but try to call the tool if it helps to answer the question.
39+
`;
3940

4041
const functionPattern = /<function="([^"]+)">(.*?)<\/function>/gs;
4142

@@ -57,6 +58,16 @@ function streamer(subword) {
5758
return StreamingStatus.RUNNING;
5859
}
5960

61+
function centerString(str, width) {
62+
if (str.length >= width) {
63+
return str;
64+
}
65+
const totalPadding = width - str.length;
66+
const paddingStart = Math.floor(totalPadding / 2);
67+
const paddingEnd = totalPadding - paddingStart;
68+
return ' '.repeat(paddingStart) + str + ' '.repeat(paddingEnd);
69+
}
70+
6071
async function main() {
6172
const defaultPrompt = "What is the weather in London today and in Paris yesterday, and how many pounds can I get for 100 euros?";
6273

@@ -78,11 +89,11 @@ async function main() {
7889
const device = "CPU"; // GPU can be used as well
7990
const pipe = await LLMPipeline(modelDir, device);
8091

81-
console.log(`User prompt: ${prompt} `);
92+
console.log(`User prompt: ${prompt}`);
8293

8394
for (const useStructuralTags of [false, true]) {
8495
console.log("=".repeat(80));
85-
console.log(`${useStructuralTags ? "Using structural tags" : "Using no structural tags"} `.padStart(40).padEnd(80));
96+
console.log(`${centerString(useStructuralTags ? "Using structural tags" : "Using no structural tags", 80)}`);
8697
console.log("=".repeat(80));
8798

8899
const generation_config = {};
@@ -101,8 +112,8 @@ async function main() {
101112
triggers: ["<function="]
102113
})
103114
};
115+
generation_config.do_sample = true;
104116
};
105-
generation_config.do_sample = true;
106117

107118
const response = await pipe.generate(prompt, generation_config, streamer);
108119
await pipe.finishChat();

0 commit comments

Comments
 (0)