-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
feat(core): Support truncation for LangChain integration request messages #18157
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
node-overhead report 🧳Note: This is a synthetic benchmark with a minimal express app and does not necessarily reflect the real-world performance impact in an application.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bug: LLM Prompts: Inconsistent Truncation, Exceeding Limits
Plain LLM string prompts aren't truncated when converted to messages, creating an inconsistency with chat model messages. The extractLLMRequestAttributes function wraps prompts into the same {role, content} message format as chat models but doesn't apply truncateGenAiMessages before stringifying, potentially causing large prompts to exceed byte limits while chat messages get properly truncated.
packages/core/src/utils/langchain/utils.ts#L253-L257
sentry-javascript/packages/core/src/utils/langchain/utils.ts
Lines 253 to 257 in 02d2b96
| const attrs = baseRequestAttributes(system, modelName, 'pipeline', llm, invocationParams, langSmithMetadata); | |
| if (recordInputs && Array.isArray(prompts) && prompts.length > 0) { | |
| const messages = prompts.map(p => ({ role: 'user', content: p })); | |
| setIfDefined(attrs, GEN_AI_REQUEST_MESSAGES_ATTRIBUTE, asString(messages)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bug: Truncation Inconsistency Causes Oversized LLM Data
extractLLMRequestAttributes converts prompts to messages but doesn't apply truncation like extractChatModelRequestAttributes does. This creates inconsistent behavior where chat model messages get truncated to the byte limit but LLM string prompts don't, potentially sending oversized data that exceeds the intended 20KB limit.
packages/core/src/utils/langchain/utils.ts#L254-L259
sentry-javascript/packages/core/src/utils/langchain/utils.ts
Lines 254 to 259 in a42c548
| if (recordInputs && Array.isArray(prompts) && prompts.length > 0) { | |
| const messages = prompts.map(p => ({ role: 'user', content: p })); | |
| setIfDefined(attrs, GEN_AI_REQUEST_MESSAGES_ATTRIBUTE, asString(messages)); | |
| } | |
andreiborza
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
This PR adds truncation support for LangChain integration request messages. All messages already get normalized to arrays of messages, so here we need no case distinction for strings.
Adds tests to verify behavior for 1. simple string inputs and 2. conversations in the form of arrays of strings.
Closes #18018