Skip to content

Conversation

@nicohrubec
Copy link
Member

@nicohrubec nicohrubec commented Nov 11, 2025

This PR adds truncation support for LangChain integration request messages. All messages already get normalized to arrays of messages, so here we need no case distinction for strings.

Adds tests to verify behavior for 1. simple string inputs and 2. conversations in the form of arrays of strings.

Closes #18018

@github-actions
Copy link
Contributor

github-actions bot commented Nov 11, 2025

node-overhead report 🧳

Note: This is a synthetic benchmark with a minimal express app and does not necessarily reflect the real-world performance impact in an application.

Scenario Requests/s % of Baseline Prev. Requests/s Change %
GET Baseline 8,884 - 8,762 +1%
GET With Sentry 1,412 16% 1,394 +1%
GET With Sentry (error only) 6,052 68% 6,136 -1%
POST Baseline 1,181 - 1,200 -2%
POST With Sentry 551 47% 542 +2%
POST With Sentry (error only) 1,067 90% 1,062 +0%
MYSQL Baseline 3,320 - 3,361 -1%
MYSQL With Sentry 513 15% 476 +8%
MYSQL With Sentry (error only) 2,715 82% 2,740 -1%

View base workflow run

@nicohrubec nicohrubec marked this pull request as ready for review November 11, 2025 10:12
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: LLM Prompts: Inconsistent Truncation, Exceeding Limits

Plain LLM string prompts aren't truncated when converted to messages, creating an inconsistency with chat model messages. The extractLLMRequestAttributes function wraps prompts into the same {role, content} message format as chat models but doesn't apply truncateGenAiMessages before stringifying, potentially causing large prompts to exceed byte limits while chat messages get properly truncated.

packages/core/src/utils/langchain/utils.ts#L253-L257

const attrs = baseRequestAttributes(system, modelName, 'pipeline', llm, invocationParams, langSmithMetadata);
if (recordInputs && Array.isArray(prompts) && prompts.length > 0) {
const messages = prompts.map(p => ({ role: 'user', content: p }));
setIfDefined(attrs, GEN_AI_REQUEST_MESSAGES_ATTRIBUTE, asString(messages));

Fix in Cursor Fix in Web


Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Truncation Inconsistency Causes Oversized LLM Data

extractLLMRequestAttributes converts prompts to messages but doesn't apply truncation like extractChatModelRequestAttributes does. This creates inconsistent behavior where chat model messages get truncated to the byte limit but LLM string prompts don't, potentially sending oversized data that exceeds the intended 20KB limit.

packages/core/src/utils/langchain/utils.ts#L254-L259

if (recordInputs && Array.isArray(prompts) && prompts.length > 0) {
const messages = prompts.map(p => ({ role: 'user', content: p }));
setIfDefined(attrs, GEN_AI_REQUEST_MESSAGES_ATTRIBUTE, asString(messages));
}

Fix in Cursor Fix in Web


Copy link
Member

@andreiborza andreiborza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nicohrubec nicohrubec merged commit b6a7773 into develop Nov 11, 2025
195 checks passed
@nicohrubec nicohrubec deleted the nh/langchain-request-truncation branch November 11, 2025 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support truncation for LangChain integration request messages

3 participants