Skip to content

Commit dc06069

Browse files
lalvareztlalvarezt
committed
feat(bench): add helper scripts
1 parent 3466ca2 commit dc06069

File tree

3 files changed

+649
-0
lines changed

3 files changed

+649
-0
lines changed

‎scripts/README.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,123 @@ python3 scripts/compare_benchmarks.py \
138138
cat comparison.md
139139
```
140140

141+
## Version Comparison Workflow
142+
143+
For comparing performance across multiple commits (e.g., to find when a regression was introduced), use the `compile_benchmark_versions.sh` script.
144+
145+
### `compile_benchmark_versions.sh`
146+
147+
This script compiles the benchmark tool for every commit in a range, making it easy to run performance comparisons across different versions.
148+
149+
**Features:**
150+
151+
- **Idempotent**: Only compiles versions that don't already exist
152+
- **Safe**: Uses git worktrees in temporary directories (doesn't affect your working directory)
153+
- **Convenient**: Stores binaries with commit SHA for easy identification
154+
- **Non-intrusive**: Works even with uncommitted changes in your main working directory
155+
- **Storage**: Uses `$XDG_DATA_HOME/string_pipeline/benchmarks/` (typically `~/.local/share/string_pipeline/benchmarks/`)
156+
157+
**Usage:**
158+
159+
```bash
160+
# Compile all versions from 78594af (stabilized benchmark tool v1.0.0) to HEAD
161+
./scripts/compile_benchmark_versions.sh
162+
163+
# Compile specific range
164+
./scripts/compile_benchmark_versions.sh --start abc1234 --end def5678
165+
166+
# See what would be compiled (dry run)
167+
./scripts/compile_benchmark_versions.sh --dry-run
168+
169+
# List already compiled versions
170+
./scripts/compile_benchmark_versions.sh --list
171+
172+
# Remove all compiled versions
173+
./scripts/compile_benchmark_versions.sh --clean
174+
175+
# Verbose output for debugging
176+
./scripts/compile_benchmark_versions.sh --verbose
177+
```
178+
179+
**Example Workflow - Finding a Performance Regression:**
180+
181+
```bash
182+
# 1. Compile all versions
183+
./scripts/compile_benchmark_versions.sh
184+
185+
# 2. Set up benchmark directory path
186+
BENCH_DIR="${XDG_DATA_HOME:-$HOME/.local/share}/string_pipeline/benchmarks"
187+
188+
# 3. Run benchmarks on two versions
189+
$BENCH_DIR/bench_throughput_abc1234 \
190+
--sizes 10000 \
191+
--iterations 100 \
192+
--output before.json
193+
194+
$BENCH_DIR/bench_throughput_def5678 \
195+
--sizes 10000 \
196+
--iterations 100 \
197+
--output after.json
198+
199+
# 4. Compare results
200+
python3 scripts/compare_benchmarks.py before.json after.json
201+
202+
# 5. If regression found, bisect by testing commits in between
203+
$BENCH_DIR/bench_throughput_xyz9999 --sizes 10000 --iterations 100 --output middle.json
204+
python3 scripts/compare_benchmarks.py before.json middle.json
205+
```
206+
207+
### `compare_benchmark_versions.sh`
208+
209+
After compiling benchmark binaries, use this script to quickly compare performance between two versions using hyperfine.
210+
211+
**Features:**
212+
213+
- **Fast comparison**: Uses hyperfine for accurate benchmark timing
214+
- **Automatic validation**: Checks that both binaries exist before running
215+
- **Flexible parameters**: Customize warmup, runs, and sizes
216+
- **Clear output**: Shows which version is faster with statistical confidence
217+
218+
**Requirements:**
219+
220+
- hyperfine must be installed (`apt install hyperfine` or `brew install hyperfine`)
221+
222+
**Usage:**
223+
224+
```bash
225+
# Basic comparison with defaults
226+
./scripts/compare_benchmark_versions.sh 78594af c5a8a11
227+
228+
# Custom warmup and runs for better accuracy
229+
./scripts/compare_benchmark_versions.sh 78594af c5a8a11 --warmup 5 --runs 20
230+
231+
# Compare with specific benchmark parameters
232+
./scripts/compare_benchmark_versions.sh abc1234 def5678 --sizes 10000
233+
```
234+
235+
**Example Workflow - Performance Comparison:**
236+
237+
```bash
238+
# 1. Compile the versions you want to compare
239+
./scripts/compile_benchmark_versions.sh --start 78594af --end c5a8a11
240+
241+
# 2. Run hyperfine comparison
242+
./scripts/compare_benchmark_versions.sh 78594af c5a8a11
243+
244+
# Output shows:
245+
# - Mean execution time for each version
246+
# - Standard deviation
247+
# - Min/max range
248+
# - Relative speed comparison (e.g., "1.05x faster")
249+
```
250+
251+
**Important Notes:**
252+
253+
- This compares **execution time** of the entire benchmark run, not the benchmark throughput metrics
254+
- Both versions run with identical parameters for fair comparison
255+
- Hyperfine handles warmup runs and statistical analysis automatically
256+
- For more detailed performance analysis, use the benchmark JSON output with `compare_benchmarks.py`
257+
141258
## Configuration
142259

143260
### Benchmark Parameters

‎scripts/compare_benchmark_versions.sh‎

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
#!/usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
# Script to compare two compiled benchmark binaries using hyperfine
6+
# This makes it easy to see performance differences between versions
7+
8+
BENCH_DIR="${XDG_DATA_HOME:-$HOME/.local/share}/string_pipeline/benchmarks"
9+
10+
# Colors for output
11+
RED='\033[0;31m'
12+
GREEN='\033[0;32m'
13+
YELLOW='\033[1;33m'
14+
BLUE='\033[0;34m'
15+
NC='\033[0m' # No Color
16+
17+
# Default values
18+
WARMUP=5
19+
RUNS=50
20+
SIZE="10000"
21+
22+
# Usage information
23+
usage() {
24+
cat <<EOF
25+
Usage: $(basename "$0") <sha1> <sha2> [OPTIONS]
26+
27+
Compare performance of two compiled benchmark binaries using hyperfine.
28+
29+
The script will run both benchmark versions with the same parameters and
30+
use hyperfine to measure and compare their execution time.
31+
32+
ARGUMENTS:
33+
<sha1> Short SHA of first benchmark version (baseline)
34+
<sha2> Short SHA of second benchmark version (current)
35+
36+
OPTIONS:
37+
--warmup N Number of warmup runs (default: $WARMUP)
38+
--runs N Number of benchmark runs (default: $RUNS)
39+
--size SIZE Input size (default: $SIZE)
40+
-h, --help Show this help message
41+
42+
EXAMPLES:
43+
# Basic comparison with defaults
44+
$(basename "$0") 78594af c5a8a11
45+
46+
# Custom warmup and runs
47+
$(basename "$0") 78594af c5a8a11 --warmup 5 --runs 20
48+
49+
# Compare with specific size
50+
$(basename "$0") abc1234 def5678 --size 50000
51+
52+
NOTES:
53+
- Binaries must be compiled first using compile_benchmark_versions.sh
54+
- Both binaries will be run with the same benchmark parameters
55+
- hyperfine must be installed (https://github.com/sharkdp/hyperfine)
56+
- Results show execution time comparison, not benchmark throughput
57+
EOF
58+
}
59+
60+
# Print colored message
61+
log_info() {
62+
echo -e "${BLUE}ℹ${NC} $*"
63+
}
64+
65+
log_success() {
66+
echo -e "${GREEN}✓${NC} $*"
67+
}
68+
69+
log_error() {
70+
echo -e "${RED}✗${NC} $*" >&2
71+
}
72+
73+
# Check if hyperfine is installed
74+
check_hyperfine() {
75+
if ! command -v hyperfine &>/dev/null; then
76+
log_error "hyperfine is not installed"
77+
echo ""
78+
echo "Install hyperfine:"
79+
echo " - Debian/Ubuntu: apt install hyperfine"
80+
echo " - macOS: brew install hyperfine"
81+
echo " - Cargo: cargo install hyperfine"
82+
echo " - GitHub: https://github.com/sharkdp/hyperfine"
83+
echo ""
84+
exit 1
85+
fi
86+
}
87+
88+
# Check if binary exists
89+
check_binary() {
90+
local sha=$1
91+
local binary_path="$BENCH_DIR/bench_throughput_$sha"
92+
93+
if [ ! -f "$binary_path" ]; then
94+
log_error "Benchmark binary not found: bench_throughput_$sha"
95+
echo ""
96+
echo "The binary for commit $sha has not been compiled yet."
97+
echo ""
98+
echo "Compile it first using:"
99+
echo -e " ${YELLOW}./scripts/compile_benchmark_versions.sh --start $sha --end $sha${NC}"
100+
echo ""
101+
echo "Or compile a range of versions:"
102+
echo -e " ${YELLOW}./scripts/compile_benchmark_versions.sh${NC}"
103+
echo ""
104+
exit 1
105+
fi
106+
}
107+
108+
# Parse command line arguments
109+
if [ $# -lt 2 ]; then
110+
usage
111+
exit 1
112+
fi
113+
114+
SHA1=$1
115+
SHA2=$2
116+
shift 2
117+
118+
while [ $# -gt 0 ]; do
119+
case $1 in
120+
--warmup)
121+
WARMUP="$2"
122+
shift 2
123+
;;
124+
--runs)
125+
RUNS="$2"
126+
shift 2
127+
;;
128+
--size)
129+
SIZE="$2"
130+
shift 2
131+
;;
132+
-h | --help)
133+
usage
134+
exit 0
135+
;;
136+
*)
137+
log_error "Unknown option: $1"
138+
echo ""
139+
usage
140+
exit 1
141+
;;
142+
esac
143+
done
144+
145+
# Validate inputs
146+
check_hyperfine
147+
check_binary "$SHA1"
148+
check_binary "$SHA2"
149+
150+
BINARY1="$BENCH_DIR/bench_throughput_$SHA1"
151+
BINARY2="$BENCH_DIR/bench_throughput_$SHA2"
152+
153+
# Prepare hyperfine command
154+
HYPERFINE_ARGS=(
155+
"--warmup" "$WARMUP"
156+
"--runs" "$RUNS"
157+
)
158+
159+
# Print comparison info
160+
echo ""
161+
log_info "Comparing benchmark versions using hyperfine"
162+
echo ""
163+
echo " Baseline: $SHA1"
164+
echo " Current: $SHA2"
165+
echo ""
166+
echo "Benchmark parameters:"
167+
echo " Size: $SIZE"
168+
echo ""
169+
echo "Hyperfine parameters:"
170+
echo " Warmup runs: $WARMUP"
171+
echo " Benchmark runs: $RUNS"
172+
echo ""
173+
174+
# Run hyperfine comparison
175+
hyperfine \
176+
"${HYPERFINE_ARGS[@]}" \
177+
--command-name "$SHA1" \
178+
"$BINARY1 --sizes $SIZE --output /dev/null" \
179+
--command-name "$SHA2" \
180+
"$BINARY2 --sizes $SIZE --output /dev/null"
181+
182+
echo ""
183+
log_success "Comparison complete!"

0 commit comments

Comments
 (0)