Skip to content

Commit f7fdfcd

Browse files
moonbox3moonbox3
authored
Python: Add AzureAIAgent Deep Research Tool support (#13034)
### Motivation and Context The deep research tool was introduced for the AzureAIAgent; however, we didn't have support yet in SK. Adding support now for using the deep research tool for both streaming and non-streaming. Samples added as well. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description - Add support for deep research tool. - Add samples with notes on prerequisites to configure the sample to run properly. - Added support for `AZURE_AI_AGENT_DEEP_RESEARCH_MODEL` env var within `AzureAIAgentSettings` class. - Bumped min allowed `azure-ai-agents` package ver to 1.2.0b3. - Closes #12978 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄
1 parent 86b10b9 commit f7fdfcd

File tree

7 files changed

+4004
-3574
lines changed

7 files changed

+4004
-3574
lines changed

‎python/pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ classifiers = [
2525
dependencies = [
2626
# azure agents
2727
"azure-ai-projects >= 1.0.0b12",
28-
"azure-ai-agents >= 1.1.0b4",
28+
"azure-ai-agents >= 1.2.0b3",
2929
"aiohttp ~= 3.8",
3030
"cloudevents ~=1.0",
3131
"pydantic >=2.0,!=2.10.0,!=2.10.1,!=2.10.2,!=2.10.3,<2.12",

‎python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_deep_research_streaming.py‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
# Copyright (c) Microsoft. All rights reserved.
2+
3+
import asyncio
4+
5+
from azure.ai.agents.models import DeepResearchTool
6+
from azure.identity.aio import AzureCliCredential
7+
8+
from semantic_kernel.agents import AzureAIAgent, AzureAIAgentSettings, AzureAIAgentThread
9+
from semantic_kernel.contents import (
10+
ChatMessageContent,
11+
FunctionCallContent,
12+
FunctionResultContent,
13+
StreamingAnnotationContent,
14+
)
15+
16+
"""
17+
The following sample demonstrates how to create an AzureAIAgent along
18+
with the Deep Research Tool. Please visit the following documentation for more info
19+
on what is required to run the sample: https://aka.ms/agents-deep-research. Please pay
20+
attention to the purple `Note` boxes in the Azure docs.
21+
22+
Note that when you use your Bing Connection ID, it needs to be the connection ID from the project, not the resource.
23+
It has the following format:
24+
25+
'/subscriptions/<sub_id>/resourceGroups/<rg_name>/providers/<provider_name>/accounts/<account_name>/projects/<project_name>/connections/<connection_name>'
26+
"""
27+
28+
TASK = (
29+
"Research the current state of studies on orca intelligence and orca language, "
30+
"including what is currently known about orcas' cognitive capabilities and communication systems."
31+
)
32+
33+
34+
async def handle_intermediate_messages(message: ChatMessageContent) -> None:
35+
for item in message.items or []:
36+
if isinstance(item, FunctionResultContent):
37+
print(f"Function Result:> {item.result} for function: {item.name}")
38+
elif isinstance(item, FunctionCallContent):
39+
print(f"Function Call:> {item.name} with arguments: {item.arguments}")
40+
elif isinstance(item, StreamingAnnotationContent):
41+
label = item.title or item.url or "Annotation"
42+
print(f"Annotation:> {label} ({item.citation_type}) -> {item.url}")
43+
else:
44+
print(f"{item}")
45+
46+
47+
async def main() -> None:
48+
async with (
49+
AzureCliCredential() as creds,
50+
AzureAIAgent.create_client(credential=creds) as client,
51+
):
52+
azure_ai_agent_settings = AzureAIAgentSettings()
53+
# 1. Define the Deep Research tool
54+
deep_research_tool = DeepResearchTool(
55+
bing_grounding_connection_id=azure_ai_agent_settings.bing_connection_id,
56+
deep_research_model=azure_ai_agent_settings.deep_research_model,
57+
)
58+
59+
# 2. Create an agent with the tool on the Azure AI agent service
60+
agent_definition = await client.agents.create_agent(
61+
model="gpt-4o", # Deep Research requires the use of gpt-4o for scope clarification.
62+
tools=deep_research_tool.definitions,
63+
instructions="You are a helpful Agent that assists in researching scientific topics.",
64+
)
65+
66+
# 3. Create a Semantic Kernel agent for the Azure AI agent
67+
agent = AzureAIAgent(client=client, definition=agent_definition, name="DeepResearchAgent")
68+
69+
# 4. Create a thread for the agent
70+
# If no thread is provided, a new thread will be
71+
# created and returned with the initial response
72+
thread: AzureAIAgentThread | None = None
73+
74+
try:
75+
print(f"# User: '{TASK}'")
76+
# 5. Invoke the agent for the specified thread for response
77+
first_chunk = True
78+
async for response in agent.invoke_stream(
79+
messages=TASK,
80+
thread=thread,
81+
on_intermediate_message=handle_intermediate_messages,
82+
):
83+
if first_chunk:
84+
print(f"# {response.name}: ", end="", flush=True)
85+
first_chunk = False
86+
# Print the text chunk
87+
print(f"{response}", end="", flush=True)
88+
# Print any streaming annotations that may arrive in this chunk
89+
for item in response.items or []:
90+
if isinstance(item, StreamingAnnotationContent):
91+
label = item.title or item.url or (item.quote or "Annotation")
92+
print(f"\n[Annotation] {label} -> {item.url}")
93+
thread = response.thread
94+
print()
95+
finally:
96+
# 6. Cleanup: Delete the thread, agent, and file
97+
await thread.delete() if thread else None
98+
await client.agents.delete_agent(agent.id)
99+
100+
"""
101+
Sample Output:
102+
103+
# User: 'Research the current state of studies on orca intelligence and orca language, including what is
104+
currently known about orcas' cognitive capabilities and communication systems.'
105+
Function Call:> deep_research with arguments: {'input': '{"prompt": "Research the current state of studies on
106+
orca intelligence and orca communication, focusing on their cognitive capabilities and language systems.
107+
Provide an overview of key discoveries, critical experiments, and major conclusions about their
108+
intelligence and communication systems. Prioritize primary research papers, reputable academic sources,
109+
and recent updates in the field (from the past 5 years if available). Format as a structured report with
110+
appropriate headings for clarity, and respond in English."}'}
111+
# azure_agent_QhTQHlUs: Title: Current Studies on Orca Intelligence and Communication
112+
113+
Starting deep research...
114+
115+
The user's task is to research orca intelligence, focusing on cognitive capabilities and communication.
116+
【1†Bing Search】
117+
118+
[Annotation] Bing Search: 'orca communication research 2020 killer whale cognitive study' -> https://www.bing.com/search?q=orca%20communication%20research%202020%20killer%20whale%20cognitive%20study
119+
120+
**Weighing options**
121+
122+
I'm examining the research on orca social dynamics, comparing a potential review to a recent journal article
123+
on large-scale unsupervised clustering of orca calls.
124+
125+
**Investigating orca datasets**
126+
127+
OK, let me see. PDF, Interspeech 2020, "ORCA-CLEAN: A Deep Denoising Toolkit for Killer Whale Communication"
128+
seems relevant. They focus on cognitive capabilities, language systems, and communication.
129+
130+
I'm considering if the PDF is relevant and may not need it. ResearchGate's content might need a login
131+
to access. 【1†Bing Search】
132+
133+
[Annotation] Bing Search: '"Social Dynamics and Intelligence of Killer Whales (Orcinus orca)"' -> https://www.bing.com/search?q=%22Social%20Dynamics%20and%20Intelligence%20of%20Killer%20Whales%20%28Orcinus%20orca%29%22
134+
135+
**Evaluating sources**
136+
137+
I'm gathering info on "Social Dynamics and Intelligence of Killer Whales," weighing access to PDFs through
138+
ResearchGate, and considering associated online references for credibility.
139+
140+
**Considering capabilities**
141+
142+
I'm piecing together the intricacies of killer whale creativity under chemical stimuli, as explored
143+
in "Manitzas, Hill, et al 2022." Would love to learn more about their findings.
144+
145+
**Exploring external options**
146+
I'm weighing opening the PDF directly or using an external search. 【1†Bing Search】
147+
148+
[Annotation] Bing Search: 'Manitzas Hill 2022 killer whale creativity cognitive abilities' -> https://www.bing.com/search?q=Manitzas%20Hill%202022%20killer%20whale%20creativity%20cognitive%20abilities
149+
150+
...
151+
"""
152+
153+
154+
if __name__ == "__main__":
155+
asyncio.run(main())

‎python/samples/getting_started_with_agents/azure_ai_agent/step10_azure_ai_agent_deep_research.py‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
# Copyright (c) Microsoft. All rights reserved.
2+
3+
import asyncio
4+
from datetime import timedelta
5+
6+
from azure.ai.agents.models import DeepResearchTool
7+
from azure.identity.aio import AzureCliCredential
8+
9+
from semantic_kernel.agents import AzureAIAgent, AzureAIAgentSettings, AzureAIAgentThread, RunPollingOptions
10+
from semantic_kernel.contents import AnnotationContent, ChatMessageContent, FunctionCallContent, FunctionResultContent
11+
12+
"""
13+
The following sample demonstrates how to create an AzureAIAgent along
14+
with the Deep Research Tool. Please visit the following documentation for more info
15+
on what is required to run the sample: https://aka.ms/agents-deep-research. Please pay
16+
attention to the purple `Note` boxes in the Azure docs.
17+
18+
Note that when you use your Bing Connection ID, it needs to be the connection ID from the project, not the resource.
19+
It has the following format:
20+
21+
'/subscriptions/<sub_id>/resourceGroups/<rg_name>/providers/<provider_name>/accounts/<account_name>/projects/<project_name>/connections/<connection_name>'
22+
"""
23+
24+
TASK = (
25+
"Research the current state of studies on orca intelligence and orca language, "
26+
"including what is currently known about orcas' cognitive capabilities and communication systems."
27+
)
28+
29+
30+
async def handle_intermediate_messages(message: ChatMessageContent) -> None:
31+
for item in message.items or []:
32+
if isinstance(item, FunctionResultContent):
33+
print(f"Function Result:> {item.result} for function: {item.name}")
34+
elif isinstance(item, FunctionCallContent):
35+
print(f"Function Call:> {item.name} with arguments: {item.arguments}")
36+
else:
37+
print(f"{item}")
38+
print()
39+
40+
41+
async def main() -> None:
42+
async with (
43+
AzureCliCredential() as creds,
44+
AzureAIAgent.create_client(credential=creds) as client,
45+
):
46+
azure_ai_agent_settings = AzureAIAgentSettings()
47+
# 1. Define the Deep Research tool
48+
deep_research_tool = DeepResearchTool(
49+
bing_grounding_connection_id=azure_ai_agent_settings.bing_connection_id,
50+
deep_research_model=azure_ai_agent_settings.deep_research_model,
51+
)
52+
53+
# 2. Create an agent with the tool on the Azure AI agent service
54+
agent_definition = await client.agents.create_agent(
55+
model="gpt-4o",
56+
tools=deep_research_tool.definitions,
57+
instructions="You are a helpful Agent that assists in researching scientific topics.",
58+
)
59+
60+
# 3. Create a Semantic Kernel agent for the Azure AI agent
61+
agent = AzureAIAgent(client=client, definition=agent_definition, name="DeepResearchAgent")
62+
63+
# 4. Create a thread for the agent
64+
# If no thread is provided, a new thread will be
65+
# created and returned with the initial response
66+
thread: AzureAIAgentThread | None = None
67+
68+
try:
69+
print(f"# User: '{TASK}'")
70+
# 5. Invoke the agent for the specified thread for response
71+
async for response in agent.invoke(
72+
messages=TASK,
73+
thread=thread,
74+
on_intermediate_message=handle_intermediate_messages,
75+
polling_options=RunPollingOptions(run_polling_timeout=timedelta(minutes=10)),
76+
):
77+
# Note that the underlying Deep Research Tool uses the o3 reasoning model.
78+
# When using the non-streaming invoke, it is normal for there to be
79+
# several minutes of processing before agent response(s) appear.
80+
# For a fast response, consider using the streaming invoke method.
81+
# A sample exists in the `concepts/agents/azure_ai_agent` directory.
82+
print(f"# {response.name}: {response}")
83+
for item in response.items or []:
84+
if isinstance(item, AnnotationContent):
85+
label = item.title or item.url or (item.quote or "Annotation")
86+
print(f"\n[Annotation] {label} -> {item.url}")
87+
thread = response.thread
88+
finally:
89+
# 6. Cleanup: Delete the thread, agent, and file
90+
await thread.delete() if thread else None
91+
await client.agents.delete_agent(agent.id)
92+
93+
"""
94+
Sample Output:
95+
96+
# User: 'Research the current state of studies on orca intelligence and orca language, including what is
97+
currently known about orcas' cognitive capabilities and communication systems.'
98+
99+
Function Call:> deep_research with arguments: {'input': '{"prompt": "Research the current state of studies on
100+
orca intelligence and orca language. Include up-to-date findings on both their cognitive capabilities and
101+
communication systems. Structure the findings as a detailed report with well-formatted headers that break down
102+
topics such as orcas\' cognitive skills (e.g., problem-solving, memory, and social intelligence), their
103+
language or communication methods (e.g., vocalizations, dialects, and symbolic communication), and comparisons
104+
to other highly intelligent species. Include sources, prioritize recent peer-reviewed studies and reputable
105+
marine biology publications, and ensure the report is well-cited and clear."}'}
106+
107+
# DeepResearchAgent: Got it! I'll gather recent studies and findings on orca intelligence and their
108+
# communication systems, focusing on cognitive abilities and the mechanisms of their language.
109+
# I'll update you with a comprehensive overview soon.
110+
111+
Title: Current Research on Orca Intelligence and Language
112+
113+
Starting deep research...
114+
115+
# DeepResearchAgent: cot_summary: **Examining orca intelligence**
116+
The task involves looking into orca cognitive abilities, communication methods, and comparisons with
117+
other intelligent species. It includes recent peer-reviewed studies and credible sources. 【1†Bing Search】
118+
119+
120+
[Annotation] Bing Search: '2024 orca intelligence cognitive study research' -> https://www.bing.com/search?q=2024%20orca%20intelligence%20cognitive%20study%20research
121+
# DeepResearchAgent: cot_summary: **Investigating orca intelligence**
122+
123+
I'm curious about headlines discussing orcas' cognitive abilities and behaviors, especially their tool use,
124+
recent attacks, and comparisons with dolphins. 【1†Bing Search】
125+
126+
127+
[Annotation] Bing Search: 'orca cognition recent peer-reviewed studies orca communication recent study' -> https://www.bing.com/search?q=orca%20cognition%20recent%20peer-reviewed%20studies%20orca%20communication%20recent%20study
128+
# DeepResearchAgent: cot_summary: **Researching social dynamics**
129+
130+
I'm gathering info on killer whale social dynamics from what appears to be a 2024 article on ResearchGate.
131+
132+
# DeepResearchAgent: cot_summary: **Assessing publication type**
133+
I'm exploring if the ResearchGate link points to a preprint, student paper, or journal article.
134+
Progress is steady as I look into its initiation and source.
135+
136+
# DeepResearchAgent: cot_summary: **Investigating PDF issues**
137+
138+
...
139+
"""
140+
141+
142+
if __name__ == "__main__":
143+
asyncio.run(main())

0 commit comments

Comments
 (0)