-
Notifications
You must be signed in to change notification settings - Fork 1
Commit 21f1bc1
Fix CI/CD Pipeline Inconsistencies (#13)
* fix(ci): resolve CI/CD inconsistencies and broken baseline storage
This commit addresses several critical issues in the CI/CD workflows:
1. Fix broken baseline artifact storage:
- benchmark.yml now downloads baseline from update-baseline.yml workflow
- Previously tried to download from itself after baseline upload was removed
2. Standardize Rust toolchain versions to stable:
- CI workflow: Changed test and rustfmt jobs from nightly to stable
- CD workflow: Changed deb publish job from nightly to stable
- Ensures consistent builds and testing across all jobs
These changes restore benchmark comparisons and ensure consistent CI/CD behavior.
* feat(ci): add /bench command for on-demand PR benchmark comparisons
Add new workflow that allows repository owners to trigger benchmark
comparisons between any two refs via PR comments.
Features:
- Command syntax: /bench <ref1> <ref2>
- Owner-only security: Only repo owner can trigger
- Works on PR comments only (not regular issues)
- Compares any two commits, branches, or tags
- Posts detailed comparison report to PR
- Interactive reactions (👀 → 🚀 or 😕)
Use cases:
- Compare feature branch vs stable release
- Validate optimizations between specific commits
- Test performance across version boundaries
- Ad-hoc performance debugging
Documentation:
- Updated scripts/README.md with complete workflow documentation
- Added security model and usage examples
- Documented all three benchmark workflows
* refactor(ci): simplify benchmark system to single /bench command
Removed automatic benchmarking workflows and consolidated to a single
on-demand command-based system with enhanced flexibility.
Changes:
- Removed benchmark.yml (automatic benchmarks on every PR/push)
- Removed update-baseline.yml (manual baseline updates)
- Enhanced bench-command.yml with optional parameters:
* /bench <ref1> <ref2> [iterations] [sizes]
* Defaults: iterations=100, sizes=1000,5000,10000
* Examples: /bench main HEAD
/bench v0.12.0 v0.13.0 100 1000,5000,10000
New features:
- Graceful handling when benchmark tool doesn't exist in refs
- Posts helpful error messages with commit info
- Validates all parameters before running
- Supports custom iterations and sizes per comparison
- Build logs included in artifacts for debugging
Benefits:
- No automatic runs = lower CI costs
- Owner controls when benchmarks run
- Flexible parameters per comparison
- Clear error messages for invalid refs
- Only one workflow to maintain
Documentation:
- Completely rewrote scripts/README.md
- Added comprehensive usage examples
- Documented all error cases
- Added use case scenarios
- Simplified troubleshooting guide
* fix(ci): improve bench workflow robustness and reliability
Address critical issues to make the workflow less flaky and more reliable.
Fixes:
1. Permissions
- Added issues: write and pull-requests: write to check-permission job
- Allows posting comments and reactions without permission errors
2. Concurrency control
- Added concurrency group: bench-${{ issue.number }}
- Prevents overlapping benchmark runs on the same PR
- Uses cancel-in-progress: false to queue instead of cancel
3. Ref fetching
- Added targeted git fetch before validation
- Fetches refs, tags, and heads explicitly
- Prevents "ref not found" errors for remote branches/tags
4. Shell robustness
- Added set -euo pipefail to all bash scripts
- Ensures early failure on any error
- Prevents silent failures and undefined variables
5. Early parse feedback
- Added continue-on-error to parse step
- Posts immediate error message on invalid format
- Shows usage examples with all parameters
- Prevents workflow from proceeding with bad params
Benefits:
- More reliable ref resolution
- Clear error messages for invalid input
- No concurrent runs causing conflicts
- Proper permissions for all operations
- Fail-fast behavior prevents wasted CI time
* fix(ci): fix YAML syntax errors and improve workflow behavior
Critical fixes for YAML parsing and workflow reliability:
1. YAML Syntax Errors (CRITICAL)
- Fixed template literals with markdown causing YAML parse errors
- Converted all multi-line template literals to array.join() pattern
- Lines starting with ** were interpreted as YAML alias markers
- Affected: parse error message, acknowledgment, results, error handler
2. Graceful Error Handling
- Removed process.exit(78) from "Post missing tool error" step
- Non-zero exit triggers handle-error job unnecessarily
- Existing if guards properly skip subsequent steps
- Workflow now exits gracefully without false failures
3. Concurrency Behavior
- Changed cancel-in-progress from false to true
- Previous: Multiple /bench commands queue and run sequentially
- Now: Newer /bench commands cancel older pending runs
- Saves CI time and provides faster feedback
All changes maintain functionality while fixing syntax and improving UX.
* feat(ci): add intelligent commit ordering and same-commit detection
Implement the intended benchmark logic to ensure correct comparisons:
1. Automatic commit ordering
- Determines which ref is older (baseline) vs newer (current)
- Uses git commit timestamps for ordering
- Ensures baseline is always the older commit regardless of argument order
- Example: /bench v0.13.0 main → baseline=v0.13.0, current=main (if main is newer)
2. Same-commit detection
- Checks if both refs resolve to the same SHA
- Exits early with helpful message if refs are identical
- Prevents wasteful benchmark runs on identical code
- Example: /bench main origin/main → detects same commit, posts message
3. Updated all steps to use determined ordering
- Renamed steps: check_ref1_tool → check_baseline_tool, check_ref2_tool → check_current_tool
- Updated benchmark steps to use baseline_ref and current_ref
- Updated artifact names: benchmark_ref1.json → benchmark_baseline.json
- Updated build logs: build_ref1.log → build_baseline.log
- Results always show: Baseline (older) vs Current (newer)
4. Improved error messages
- Missing tool errors now indicate "baseline (older)" vs "current (newer)"
- Same-commit message explains refs are identical
- Clear distinction between user input and determined ordering
Benefits:
- Consistent baseline/current semantics regardless of argument order
- No wasted CI time on identical commits
- Clear, predictable results in every scenario
- Better user experience with informative messages
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 7740e0f commit 21f1bc1Copy full SHA for 21f1bc1
File tree
Expand file treeCollapse file tree
6 files changed
+768
-242
lines changedOpen diff view settings
Filter options
- .github/workflows
- scripts
Expand file treeCollapse file tree
6 files changed
+768
-242
lines changedOpen diff view settings
0 commit comments