Skip to content

Conversation

@quanru
Copy link
Collaborator

@quanru quanru commented Nov 7, 2025

Summary

This PR unifies the MIDSCENE_VQA_MODEL_* and MIDSCENE_GROUNDING_MODEL_* environment variables into a single MIDSCENE_INSIGHT_MODEL_* configuration. This simplification improves the user experience and aligns with the evolution of vision-language models.

Changes

Core Implementation

  • Type definitions: Updated TIntent type from 'VQA' | 'grounding' | 'planning' | 'default' to 'insight' | 'planning' | 'default'
  • Environment variables: Unified 12 environment variables into 6 INSIGHT variables
    • MIDSCENE_INSIGHT_MODEL_NAME
    • MIDSCENE_INSIGHT_MODEL_API_KEY
    • MIDSCENE_INSIGHT_MODEL_BASE_URL
    • MIDSCENE_INSIGHT_MODEL_HTTP_PROXY
    • MIDSCENE_INSIGHT_MODEL_SOCKS_PROXY
    • MIDSCENE_INSIGHT_MODEL_INIT_CONFIG_JSON
    • MIDSCENE_INSIGHT_LOCATOR_MODE

Code Updates

  • Updated all agent code to use 'insight' intent instead of 'VQA' and 'grounding'
  • Fixed model config resolution logic in model-config-manager.ts
  • Updated 7 test files with new intent names
  • Fixed duplicate case clause warnings in test files

Documentation

  • Added comprehensive intent-based configuration guide in model-provider.mdx (EN/ZH)
  • Updated API documentation with modelConfig parameter examples (EN/ZH)
  • Updated choose-a-model.mdx with task type configuration section (EN/ZH)

Breaking Changes

⚠️ This is a breaking change

  • The TIntent type has been changed from 'VQA' | 'grounding' | 'planning' | 'default' to 'insight' | 'planning' | 'default'
  • Environment variables MIDSCENE_VQA_MODEL_* and MIDSCENE_GROUNDING_MODEL_* are no longer supported
  • Users must migrate to MIDSCENE_INSIGHT_MODEL_* variables

Testing

✅ All tests passing (140/140)

  • Updated unit tests for model config manager
  • Updated integration tests for agents
  • No test failures

Migration Guide

Before (deprecated):

# VQA model configuration
export MIDSCENE_VQA_MODEL_NAME="gpt-4o"
export MIDSCENE_VQA_MODEL_API_KEY="sk-..."
export MIDSCENE_VQA_MODEL_BASE_URL="https://api.openai.com/v1"

# Grounding model configuration
export MIDSCENE_GROUNDING_MODEL_NAME="qwen-vl-plus"
export MIDSCENE_GROUNDING_MODEL_API_KEY="sk-..."
export MIDSCENE_GROUNDING_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"

After (recommended):

# Insight model configuration (unified)
export MIDSCENE_INSIGHT_MODEL_NAME="qwen-vl-plus"
export MIDSCENE_INSIGHT_MODEL_API_KEY="sk-..."
export MIDSCENE_INSIGHT_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
export MIDSCENE_INSIGHT_LOCATOR_MODE="qwen3-vl"

Related Issues

Closes #[issue-number] (if applicable)


🤖 Generated with Claude Code

Unified MIDSCENE_VQA_MODEL_* and MIDSCENE_GROUNDING_MODEL_* environment
variables into a single MIDSCENE_INSIGHT_MODEL_* configuration.

Changes:
- Updated type definitions to use 'insight' intent instead of 'VQA' and 'grounding'
- Unified 12 environment variables into 6 INSIGHT variables
- Updated all agent code to use 'insight' intent
- Fixed all test cases (140/140 passing)
- Added comprehensive documentation for intent-based model configuration
- Fixed duplicate case clause warnings in test files

Breaking changes:
- Replaced TIntent type: 'VQA' | 'grounding' -> 'insight'
- Environment variables MIDSCENE_VQA_MODEL_* and MIDSCENE_GROUNDING_MODEL_*
  are no longer supported

Documentation updates:
- Added detailed intent-based configuration guide in model-provider.mdx (EN/ZH)
- Updated API documentation with modelConfig examples (EN/ZH)
- Updated choose-a-model.mdx with task type configuration section (EN/ZH)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@quanru quanru force-pushed the feat/unify-insight-model-config branch from a0ca743 to 803cf00 Compare November 7, 2025 08:58
@quanru quanru requested a review from Copilot November 7, 2025 09:34
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors the model configuration system by consolidating the 'VQA' (Visual Question Answering) and 'grounding' (Visual Grounding) intent types into a unified 'insight' intent type, simplifying the model configuration interface and reducing code duplication.

Key Changes:

  • Unified two similar intent types ('VQA' and 'grounding') into a single 'insight' intent type
  • Updated all environment variables from MIDSCENE_VQA_MODEL_* and MIDSCENE_GROUNDING_MODEL_* to MIDSCENE_INSIGHT_MODEL_*
  • Updated type definitions to reflect the new intent types: TIntent = 'insight' | 'planning' | 'default'
  • Updated comprehensive documentation (both English and Chinese) to explain the new task type configuration system

Reviewed Changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
packages/web-integration/tests/unit-test/agent.test.ts Updated test to use 'insight' instead of 'VQA'
packages/shared/tests/unit-test/env/modle-config-manager.test.ts Refactored all tests to use 'insight' instead of 'VQA' and 'grounding', removed grounding-specific test cases
packages/shared/tests/unit-test/env/decide-model.test.ts Updated test cases to use 'insight' instead of 'VQA'
packages/shared/src/env/types.ts Removed VQA and grounding types, added unified INSIGHT types, updated TIntent type definition
packages/shared/src/env/model-config-manager.ts Updated ALL_INTENTS array and model config maps to use 'insight' instead of 'VQA' and 'grounding'
packages/shared/src/env/decide-model-config.ts Updated KEYS_MAP to use INSIGHT_MODEL_CONFIG_KEYS instead of separate VQA and grounding keys
packages/shared/src/env/constants.ts Consolidated VQA_MODEL_CONFIG_KEYS and GROUNDING_MODEL_CONFIG_KEYS into INSIGHT_MODEL_CONFIG_KEYS
packages/evaluation/tests/llm-locator.test.ts Updated to use 'insight' instead of 'grounding'
packages/evaluation/src/test-analyzer.ts Updated to use 'insight' instead of 'grounding'
packages/core/tests/unit-test/proxy-integration.test.ts Updated proxy configuration tests to use 'insight' instead of 'VQA' and 'grounding'
packages/core/tests/ai/service/service.test.ts Updated to use 'insight' instead of 'grounding'
packages/core/src/agent/agent.ts Updated all getModelConfig calls to use 'insight' instead of 'VQA' and 'grounding'
apps/site/docs/zh/model-provider.mdx Added comprehensive documentation for task-type-based model configuration in Chinese
apps/site/docs/zh/choose-a-model.mdx Added section explaining task-type-based model configuration in Chinese
apps/site/docs/zh/api.mdx Updated modelConfig documentation to reflect new intent types and added examples
apps/site/docs/en/model-provider.mdx Added comprehensive documentation for task-type-based model configuration in English
apps/site/docs/en/choose-a-model.mdx Added section explaining task-type-based model configuration in English
apps/site/docs/en/automate-with-scripts-in-yaml.mdx Updated comment to use 'insight' instead of 'VQA'
apps/site/docs/en/api.mdx Updated modelConfig documentation to reflect new intent types and added examples

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

quanru and others added 2 commits November 7, 2025 17:47
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@quanru quanru requested review from Copilot and yuyutaotao November 7, 2025 09:48
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@quanru quanru merged commit 6769784 into 1.0 Nov 10, 2025
6 checks passed
@quanru quanru deleted the feat/unify-insight-model-config branch November 10, 2025 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants