Skip to content

Commit 4eac816

Browse files
lyarwoodlyarwood
committed
WIP test(kubevirt): Add gevals-based tests for VM toolset
Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
1 parent edb97de commit 4eac816

File tree

9 files changed

+410
-0
lines changed

9 files changed

+410
-0
lines changed

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,6 @@ python/build/
2727
python/dist/
2828
python/kubernetes_mcp_server.egg-info/
2929
!python/kubernetes-mcp-server
30+
31+
.gevals-step*
32+
gevals-kubevirt-vm-operations-out.json

‎pkg/toolsets/kubevirt/vm/tests/README.md‎

Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
# KubeVirt VM Toolset Tests
2+
3+
This directory contains gevals-based tests for the KubeVirt VM toolset in the Kubernetes MCP Server.
4+
5+
## Overview
6+
7+
These tests validate the VM creation and troubleshooting tools (`vm_create` and `vm_troubleshoot`) by having AI agents complete real tasks using the MCP server.
8+
9+
## Test Structure
10+
11+
```
12+
tests/
13+
├── README.md # This file
14+
├── mcp-config.yaml # MCP server configuration
15+
├── claude-code/ # Claude Code agent configuration
16+
│ ├── agent.yaml
17+
│ └── eval.yaml
18+
└── tasks/ # Test tasks
19+
├── create-vm-basic/ # Basic VM creation test
20+
├── create-vm-with-instancetype/ # VM with specific instancetype
21+
├── create-vm-with-size/ # VM with size parameter
22+
├── create-vm-ubuntu/ # Ubuntu VM creation
23+
├── create-vm-with-performance/ # VM with performance family
24+
└── troubleshoot-vm/ # VM troubleshooting test
25+
```
26+
27+
## Prerequisites
28+
29+
1. **Kubernetes cluster** with KubeVirt installed
30+
- The cluster must have KubeVirt CRDs installed
31+
- For testing, you can use a Kind cluster with KubeVirt
32+
33+
2. **Kubernetes MCP Server** running at `http://localhost:8080/mcp`
34+
35+
```bash
36+
# Build and run the server
37+
cd /path/to/kubernetes-mcp-server
38+
make build
39+
./kubernetes-mcp-server --port 8080
40+
```
41+
42+
3. **gevals binary** built from the gevals project
43+
44+
```bash
45+
cd /path/to/gevals
46+
go build -o gevals ./cmd/gevals
47+
```
48+
49+
4. **Claude Code** installed and in PATH
50+
51+
```bash
52+
# Install Claude Code (if not already installed)
53+
npm install -g @anthropic/claude-code
54+
```
55+
56+
5. **kubectl** configured to access your cluster
57+
58+
## Running the Tests
59+
60+
### Run All Tests
61+
62+
```bash
63+
# From the gevals directory
64+
./gevals eval /path/to/kubernetes-mcp-server/pkg/toolsets/kubevirt/vm/tests/claude-code/eval.yaml
65+
```
66+
67+
### Run a Specific Test
68+
69+
```bash
70+
# Run just the basic VM creation test
71+
./gevals eval /path/to/kubernetes-mcp-server/pkg/toolsets/kubevirt/vm/tests/tasks/create-vm-basic/create-vm-basic.yaml \
72+
--agent-file /path/to/kubernetes-mcp-server/pkg/toolsets/kubevirt/vm/tests/claude-code/agent.yaml \
73+
--mcp-config-file /path/to/kubernetes-mcp-server/pkg/toolsets/kubevirt/vm/tests/mcp-config.yaml
74+
```
75+
76+
## Test Descriptions
77+
78+
### create-vm-basic
79+
80+
**Difficulty:** Easy
81+
**Description:** Tests basic VM creation with default Fedora workload.
82+
**Key Tool:** `vm_create`
83+
**Expected Behavior:** Agent should use `vm_create` to generate a plan and then create the VM using `resources_create_or_update`.
84+
85+
### create-vm-with-instancetype
86+
87+
**Difficulty:** Medium
88+
**Description:** Tests VM creation with a specific instancetype (u1.medium).
89+
**Key Tool:** `vm_create`
90+
**Expected Behavior:** Agent should pass the instancetype parameter to `vm_create` and create a VM with the correct instancetype reference.
91+
92+
### create-vm-with-size
93+
94+
**Difficulty:** Medium
95+
**Description:** Tests VM creation using a size hint ('large').
96+
**Key Tool:** `vm_create`
97+
**Expected Behavior:** Agent should use the size parameter which should map to an appropriate instancetype.
98+
99+
### create-vm-ubuntu
100+
101+
**Difficulty:** Easy
102+
**Description:** Tests VM creation with Ubuntu workload.
103+
**Key Tool:** `vm_create`
104+
**Expected Behavior:** Agent should create a VM using the Ubuntu container disk image.
105+
106+
### create-vm-with-performance
107+
108+
**Difficulty:** Medium
109+
**Description:** Tests VM creation with performance family ('compute-optimized') and size.
110+
**Key Tool:** `vm_create`
111+
**Expected Behavior:** Agent should combine performance and size to select an appropriate instancetype (e.g., c1.medium).
112+
113+
### troubleshoot-vm
114+
115+
**Difficulty:** Easy
116+
**Description:** Tests VM troubleshooting guide generation.
117+
**Key Tool:** `vm_troubleshoot`
118+
**Expected Behavior:** Agent should use `vm_troubleshoot` to generate a troubleshooting guide for the VM.
119+
120+
## Assertions
121+
122+
The tests validate:
123+
124+
- **Tool Usage:** Agents must call `vm_create`, `vm_troubleshoot`, or `resources_*` tools
125+
- **Call Limits:** Between 1 and 30 tool calls (allows for exploration and creation)
126+
- **Task Success:** Verification scripts confirm VMs are created correctly
127+
128+
## Expected Results
129+
130+
**✅ Pass** means:
131+
132+
- The VM tools are well-designed and discoverable
133+
- Tool descriptions are clear to AI agents
134+
- Schemas are properly structured
135+
- Implementation works correctly
136+
137+
**❌ Fail** indicates:
138+
139+
- Tool descriptions may need improvement
140+
- Schema complexity issues
141+
- Missing functionality
142+
- Implementation bugs
143+
144+
## Output
145+
146+
Results are saved to `gevals-kubevirt-vm-operations-out.json` with:
147+
148+
- Task pass/fail status
149+
- Assertion results
150+
- Tool call history
151+
- Agent interactions
152+
153+
## Customization
154+
155+
### Using Different AI Agents
156+
157+
You can create additional agent configurations (similar to the `claude-code/` directory) for testing with different AI models:
158+
159+
```yaml
160+
# Example: openai-agent/agent.yaml
161+
kind: Agent
162+
metadata:
163+
name: "openai-agent"
164+
commands:
165+
argTemplateMcpServer: "{{ .File }}"
166+
runPrompt: |-
167+
agent-wrapper.sh {{ .McpServerFileArgs }} "{{ .Prompt }}"
168+
```
169+
170+
### Adding New Tests
171+
172+
To add a new test task:
173+
174+
1. Create a new directory under `tasks/`
175+
2. Add task YAML file with prompt
176+
3. Add setup, verify, and cleanup scripts
177+
4. The test will be automatically discovered by the glob pattern in `eval.yaml`
178+
179+
## Troubleshooting
180+
181+
### Tests Fail to Connect to MCP Server
182+
183+
Ensure the Kubernetes MCP Server is running:
184+
185+
```bash
186+
curl http://localhost:8080/mcp/health
187+
```
188+
189+
### VirtualMachine Not Created
190+
191+
Check if KubeVirt is installed:
192+
193+
```bash
194+
kubectl get crds | grep kubevirt
195+
kubectl get pods -n kubevirt
196+
```
197+
198+
### Permission Issues
199+
200+
Ensure your kubeconfig has permissions to:
201+
202+
- Create namespaces
203+
- Create VirtualMachine resources
204+
- List instancetypes and preferences
205+
206+
## Contributing
207+
208+
When adding new tests:
209+
210+
- Keep tasks focused on a single capability
211+
- Make verification scripts robust
212+
- Document expected behavior
213+
- Set appropriate difficulty levels
214+
- Ensure cleanup scripts remove all resources

‎pkg/toolsets/kubevirt/vm/tests/claude-code/agent.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
kind: Agent
2+
metadata:
3+
name: "claude-code"
4+
commands:
5+
useVirtualHome: false
6+
argTemplateMcpServer: "--mcp-config {{ .File }}"
7+
argTemplateAllowedTools: "mcp__{{ .ServerName }}__{{ .ToolName }}"
8+
allowedToolsJoinSeparator: ","
9+
runPrompt: |-
10+
claude {{ .McpServerFileArgs }} --strict-mcp-config --allowedTools "{{ .AllowedToolArgs }}" --print "{{ .Prompt }}"

‎pkg/toolsets/kubevirt/vm/tests/claude-code/eval.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
kind: Eval
2+
metadata:
3+
name: "kubevirt-vm-operations"
4+
config:
5+
agentFile: agent.yaml
6+
mcpConfigFile: ../mcp-config.yaml
7+
taskSets:
8+
- glob: ../tasks/*/*.yaml
9+
assertions:
10+
toolsUsed:
11+
- server: kubernetes
12+
toolPattern: "(vm_create|vm_troubleshoot|resources_.*)"
13+
minToolCalls: 1
14+
maxToolCalls: 30

‎pkg/toolsets/kubevirt/vm/tests/mcp-config.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
mcpServers:
2+
kubernetes:
3+
type: http
4+
url: http://localhost:8888/mcp
5+
enableAllTools: true

‎pkg/toolsets/kubevirt/vm/tests/tasks/create-vm-basic/create-vm-basic.yaml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
kind: Task
2+
metadata:
3+
name: "create-basic-vm"
4+
difficulty: easy
5+
steps:
6+
setup:
7+
inline: |-
8+
#!/usr/bin/env bash
9+
kubectl delete namespace vm-test --ignore-not-found
10+
kubectl create namespace vm-test
11+
verify:
12+
inline: |-
13+
#!/usr/bin/env bash
14+
# Wait for VirtualMachine to be created
15+
if kubectl wait --for=jsonpath='{.metadata.name}'=test-vm virtualmachine/test-vm -n vm-test --timeout=30s 2>/dev/null; then
16+
echo "VirtualMachine test-vm created successfully"
17+
exit 0
18+
else
19+
echo "VirtualMachine test-vm not found"
20+
kubectl get virtualmachines -n vm-test
21+
exit 1
22+
fi
23+
cleanup:
24+
inline: |-
25+
#!/usr/bin/env bash
26+
kubectl delete virtualmachine test-vm -n vm-test --ignore-not-found
27+
kubectl delete namespace vm-test --ignore-not-found
28+
prompt:
29+
inline: Please create a Fedora virtual machine named test-vm in the vm-test namespace.

‎pkg/toolsets/kubevirt/vm/tests/tasks/create-vm-ubuntu/create-vm-ubuntu.yaml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
kind: Task
2+
metadata:
3+
name: "create-ubuntu-vm"
4+
difficulty: easy
5+
steps:
6+
setup:
7+
inline: |-
8+
#!/usr/bin/env bash
9+
kubectl delete namespace vm-test --ignore-not-found
10+
kubectl create namespace vm-test
11+
verify:
12+
inline: |-
13+
#!/usr/bin/env bash
14+
# Wait for VirtualMachine to be created
15+
if kubectl wait --for=jsonpath='{.metadata.name}'=ubuntu-vm virtualmachine/ubuntu-vm -n vm-test --timeout=30s 2>/dev/null; then
16+
echo "VirtualMachine ubuntu-vm created successfully"
17+
18+
# Verify that it uses Ubuntu container disk
19+
if kubectl get virtualmachine ubuntu-vm -n vm-test -o yaml | grep -q "ubuntu"; then
20+
echo "VirtualMachine uses Ubuntu image"
21+
exit 0
22+
else
23+
echo "Warning: Could not verify Ubuntu image, but VM was created"
24+
kubectl get virtualmachine ubuntu-vm -n vm-test -o yaml
25+
# Still pass if VM was created
26+
exit 0
27+
fi
28+
else
29+
echo "VirtualMachine ubuntu-vm not found"
30+
kubectl get virtualmachines -n vm-test
31+
exit 1
32+
fi
33+
cleanup:
34+
inline: |-
35+
#!/usr/bin/env bash
36+
kubectl delete virtualmachine ubuntu-vm -n vm-test --ignore-not-found
37+
kubectl delete namespace vm-test --ignore-not-found
38+
prompt:
39+
inline: Create an Ubuntu virtual machine named ubuntu-vm in the vm-test namespace.

‎pkg/toolsets/kubevirt/vm/tests/tasks/create-vm-with-size/create-vm-with-size.yaml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
kind: Task
2+
metadata:
3+
name: "create-vm-with-size"
4+
difficulty: medium
5+
steps:
6+
setup:
7+
inline: |-
8+
#!/usr/bin/env bash
9+
kubectl delete namespace vm-test --ignore-not-found
10+
kubectl create namespace vm-test
11+
verify:
12+
inline: |-
13+
#!/usr/bin/env bash
14+
# Wait for VirtualMachine to be created
15+
if kubectl wait --for=jsonpath='{.metadata.name}'=test-vm-size virtualmachine/test-vm-size -n vm-test --timeout=30s 2>/dev/null; then
16+
echo "VirtualMachine test-vm-size created successfully"
17+
18+
# Verify that it has an instancetype reference (size should map to an instancetype)
19+
if kubectl get virtualmachine test-vm-size -n vm-test -o jsonpath='{.spec.instancetype.name}' | grep -q "large"; then
20+
echo "VirtualMachine has instancetype matching size"
21+
exit 0
22+
else
23+
echo "Warning: VirtualMachine does not have instancetype matching size, but VM was created"
24+
kubectl get virtualmachine test-vm-size -n vm-test -o yaml
25+
# Still pass if VM was created, as size is a hint
26+
exit 0
27+
fi
28+
else
29+
echo "VirtualMachine test-vm-size not found"
30+
kubectl get virtualmachines -n vm-test
31+
exit 1
32+
fi
33+
cleanup:
34+
inline: |-
35+
#!/usr/bin/env bash
36+
kubectl delete virtualmachine test-vm-size -n vm-test --ignore-not-found
37+
kubectl delete namespace vm-test --ignore-not-found
38+
prompt:
39+
inline: Create a Fedora virtual machine named test-vm-size in the vm-test namespace with size 'large'. Use the vm_create tool to generate the creation plan and then create the VirtualMachine resource.

0 commit comments

Comments
 (0)