-
Notifications
You must be signed in to change notification settings - Fork 750
feat(shared): unify VQA and grounding models into insight model #1432
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Unified MIDSCENE_VQA_MODEL_* and MIDSCENE_GROUNDING_MODEL_* environment variables into a single MIDSCENE_INSIGHT_MODEL_* configuration. Changes: - Updated type definitions to use 'insight' intent instead of 'VQA' and 'grounding' - Unified 12 environment variables into 6 INSIGHT variables - Updated all agent code to use 'insight' intent - Fixed all test cases (140/140 passing) - Added comprehensive documentation for intent-based model configuration - Fixed duplicate case clause warnings in test files Breaking changes: - Replaced TIntent type: 'VQA' | 'grounding' -> 'insight' - Environment variables MIDSCENE_VQA_MODEL_* and MIDSCENE_GROUNDING_MODEL_* are no longer supported Documentation updates: - Added detailed intent-based configuration guide in model-provider.mdx (EN/ZH) - Updated API documentation with modelConfig examples (EN/ZH) - Updated choose-a-model.mdx with task type configuration section (EN/ZH) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
a0ca743 to
803cf00
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR refactors the model configuration system by consolidating the 'VQA' (Visual Question Answering) and 'grounding' (Visual Grounding) intent types into a unified 'insight' intent type, simplifying the model configuration interface and reducing code duplication.
Key Changes:
- Unified two similar intent types ('VQA' and 'grounding') into a single 'insight' intent type
- Updated all environment variables from
MIDSCENE_VQA_MODEL_*andMIDSCENE_GROUNDING_MODEL_*toMIDSCENE_INSIGHT_MODEL_* - Updated type definitions to reflect the new intent types:
TIntent = 'insight' | 'planning' | 'default' - Updated comprehensive documentation (both English and Chinese) to explain the new task type configuration system
Reviewed Changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/web-integration/tests/unit-test/agent.test.ts | Updated test to use 'insight' instead of 'VQA' |
| packages/shared/tests/unit-test/env/modle-config-manager.test.ts | Refactored all tests to use 'insight' instead of 'VQA' and 'grounding', removed grounding-specific test cases |
| packages/shared/tests/unit-test/env/decide-model.test.ts | Updated test cases to use 'insight' instead of 'VQA' |
| packages/shared/src/env/types.ts | Removed VQA and grounding types, added unified INSIGHT types, updated TIntent type definition |
| packages/shared/src/env/model-config-manager.ts | Updated ALL_INTENTS array and model config maps to use 'insight' instead of 'VQA' and 'grounding' |
| packages/shared/src/env/decide-model-config.ts | Updated KEYS_MAP to use INSIGHT_MODEL_CONFIG_KEYS instead of separate VQA and grounding keys |
| packages/shared/src/env/constants.ts | Consolidated VQA_MODEL_CONFIG_KEYS and GROUNDING_MODEL_CONFIG_KEYS into INSIGHT_MODEL_CONFIG_KEYS |
| packages/evaluation/tests/llm-locator.test.ts | Updated to use 'insight' instead of 'grounding' |
| packages/evaluation/src/test-analyzer.ts | Updated to use 'insight' instead of 'grounding' |
| packages/core/tests/unit-test/proxy-integration.test.ts | Updated proxy configuration tests to use 'insight' instead of 'VQA' and 'grounding' |
| packages/core/tests/ai/service/service.test.ts | Updated to use 'insight' instead of 'grounding' |
| packages/core/src/agent/agent.ts | Updated all getModelConfig calls to use 'insight' instead of 'VQA' and 'grounding' |
| apps/site/docs/zh/model-provider.mdx | Added comprehensive documentation for task-type-based model configuration in Chinese |
| apps/site/docs/zh/choose-a-model.mdx | Added section explaining task-type-based model configuration in Chinese |
| apps/site/docs/zh/api.mdx | Updated modelConfig documentation to reflect new intent types and added examples |
| apps/site/docs/en/model-provider.mdx | Added comprehensive documentation for task-type-based model configuration in English |
| apps/site/docs/en/choose-a-model.mdx | Added section explaining task-type-based model configuration in English |
| apps/site/docs/en/automate-with-scripts-in-yaml.mdx | Updated comment to use 'insight' instead of 'VQA' |
| apps/site/docs/en/api.mdx | Updated modelConfig documentation to reflect new intent types and added examples |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
packages/shared/tests/unit-test/env/modle-config-manager.test.ts
Outdated
Show resolved
Hide resolved
packages/shared/tests/unit-test/env/modle-config-manager.test.ts
Outdated
Show resolved
Hide resolved
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Summary
This PR unifies the
MIDSCENE_VQA_MODEL_*andMIDSCENE_GROUNDING_MODEL_*environment variables into a singleMIDSCENE_INSIGHT_MODEL_*configuration. This simplification improves the user experience and aligns with the evolution of vision-language models.Changes
Core Implementation
TIntenttype from'VQA' | 'grounding' | 'planning' | 'default'to'insight' | 'planning' | 'default'MIDSCENE_INSIGHT_MODEL_NAMEMIDSCENE_INSIGHT_MODEL_API_KEYMIDSCENE_INSIGHT_MODEL_BASE_URLMIDSCENE_INSIGHT_MODEL_HTTP_PROXYMIDSCENE_INSIGHT_MODEL_SOCKS_PROXYMIDSCENE_INSIGHT_MODEL_INIT_CONFIG_JSONMIDSCENE_INSIGHT_LOCATOR_MODECode Updates
'insight'intent instead of'VQA'and'grounding'model-config-manager.tsDocumentation
model-provider.mdx(EN/ZH)modelConfigparameter examples (EN/ZH)choose-a-model.mdxwith task type configuration section (EN/ZH)Breaking Changes
TIntenttype has been changed from'VQA' | 'grounding' | 'planning' | 'default'to'insight' | 'planning' | 'default'MIDSCENE_VQA_MODEL_*andMIDSCENE_GROUNDING_MODEL_*are no longer supportedMIDSCENE_INSIGHT_MODEL_*variablesTesting
✅ All tests passing (140/140)
Migration Guide
Before (deprecated):
After (recommended):
Related Issues
Closes #[issue-number] (if applicable)
🤖 Generated with Claude Code