go-benchmark Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-04-11 Subject: go-benchmark

go-benchmark is a specialist skill for Go performance benchmarking and pprof profiling. The evaluation ran across three scenarios (writing benchmarks from source / reviewing broken benchmarks / analyzing benchstat output), covering 24 assertions in total. With-Skill passed all 24 (100%); Without-Skill passed 11 (46%), a gain of +54 percentage points. Three standout gaps: first, With-Skill consistently declares var sinkString / var sinkErr and explains why _ = result lets the compiler eliminate the entire call, while Without-Skill used no sink variables at all in Scenario 1 and explicitly called _ = data "safe here" in Scenario 2; second, the Evidence Gate–driven mode / data_basis declaration and Auto Scorecard block were absent from all three Without-Skill responses (0/3); third, Without-Skill scored surprisingly high on Scenario 3 (6/8, 75%) — revealing that the skill's marginal value on statistical analysis tasks is modest, and that the real leverage lies in benchmark writing and code review.

1. Skill Overview¶

go-benchmark defines 5 Hard Rules (silent-corruption guards), 3 Mandatory Gates (Evidence / Applicability / Scope), a three-phase workflow (Write → Run & Profile → Analyze & Optimize), a 4-field Output Contract, and a three-tier Auto Scorecard (Critical / Standard / Hygiene).

Core components:

Regression suite: 96 tests (65 contract + 30 golden + 1 integrity), 100% coverage across all key dimensions.

2. Test Design¶

2.1 Scenario Definitions¶

The three scenarios correspond to the three working modes defined in SKILL.md, each based on a realistic user session prototype:

2.2 Assertion Matrix (24 items)¶

Scenario 1 — Phase 1: Write benchmarks from source (9 items)

*A5 note: Without-Skill provided 7 flat top-level Encode benchmarks covering different input characteristics, rather than b.Run()-style size sub-benchmarks. Graded PASS because multiple sizes were substantively covered, even though the structure differs.

Scenario 2 — Phase 1+2: Review broken benchmarks (7 items)

Scenario 3 — Phase 3: Analyze noisy benchstat output (8 items)

2.3 Trigger Accuracy Analysis¶

The current description uses a two-layer trigger strategy:

Go performance benchmarking and pprof profiling specialist. ALWAYS use when
writing benchmark functions (testing.B), generating or reading pprof profiles,
interpreting flame graphs, finding memory allocation hotspots, comparing
implementations with benchstat, or measuring ns/op / B/op / allocs/op.
In Go code contexts, also trigger when the user says "it's slow", "too many
allocations", "find the bottleneck", or "profile this Go code".

• Explicit triggers (high certainty): testing.B, pprof profiles, flame graphs, benchstat, ns/op, B/op, allocs/op
• Implicit triggers (context-aware): "it's slow", "too many allocations", "find the bottleneck" — gated by "In Go code contexts" to prevent false positives from non-Go performance questions

Should-Trigger scenarios (10)

Should-Not-Trigger scenarios (8)

Estimated trigger accuracy: F1 ≈ 88% (10/10 should-trigger covered; 6/8 should-not-trigger correctly rejected; N7/N8 are acceptable boundary cases)

3. Pass Rate Comparison¶

3.1 Overall Pass Rate¶

Overall improvement: +54 percentage points

3.2 Pass Rate by Scenario¶

Key finding: Without-Skill scored 75% on Scenario 3 — far above its scores on Scenarios 1 (22%) and 2 (43%). This reveals an asymmetric value distribution: baseline Claude already handles statistical concepts (p-values, CV thresholds) quite well, so the skill's incremental gain on analysis tasks is modest (+25pp). The real leverage is in benchmark writing and review, where baseline is most prone to silent, hard-to-detect errors (+57–78pp).

3.3 Substantive Dimensions (12 items, structural-process assertions removed)¶

Substantive pass rate: With-Skill 12/12 (100%) vs Without-Skill 7/12 (58%), improvement +42pp.

4. Key Differences¶

4.1 Behaviors exclusive to With-Skill (completely absent in baseline)¶

4.2 Behaviors where baseline performs but at lower quality¶

4.3 Scenario-level findings¶

Scenario 1 (Write benchmarks from source): - With-Skill: Evidence Gate declared write / static analysis only. Scope Gate selected size sweep sub-benchmarks (64B/1KB/64KB/1MB). Declared var sinkString string + var sinkErr error with compiler-elimination explanation. All Decode calls used sinkString, sinkErr = Decode(...). Run commands: -benchmem -count=5 (exploration) and -count=10 (comparison). Auto Scorecard: Critical ✅✅✅ Standard 5/5 Hygiene 4/4. - Without-Skill: Used the for b.Loop() syntax (Go 1.24+) and explained its advantages. Provided 7 flat Encode benchmarks covering different input characteristics (empty / single char / no-runs / short-runs / long-run / mixed) — solid coverage strategy. However: no sink variables used anywhere in the benchmark code; results were silently discarded. Decode return values also uncaptured. Run command lacked -count. No Evidence Gate / Scorecard / Output Contract. Notably, Without-Skill mentioned "assign the return to package-level sink variables if you observe suspiciously fast numbers" — framing the sink as an optional debugging aid rather than a mandatory prevention rule.

Scenario 2 (Review broken benchmarks): - With-Skill: Enumerated 4 violations (3 Critical + 1 Standard), each with Hard Rule number, offending line, mechanism explanation, and fix code. Violation 1 explicitly stated "compiler is permitted to optimize away the entire json.Marshal call." The corrected file used var sinkBytes []byte; var sinkErr error. Scorecard showed Critical ❌❌❌ (reflecting the violations in the code under review). - Without-Skill: Correctly identified b.ResetTimer() inside the loop under a "Critical Bug" heading. Mentioned -benchmem and suggested -count=5 under minor issues. Critical gap: _ = data was described as "safe here" because json.Marshal can't fail — confusing error-handling safety with compiler optimization risk. As a result, the "corrected" file still kept _ = data; the underlying problem was left unresolved.

Scenario 3 (Analyze benchstat output): - With-Skill: Evidence Gate classified as analyze / benchmark output. Fully covered C1–C6. Went further by providing a pprof diff command (-diff_base mem-old.prof mem-new.prof) and analyzing the super-linear allocation growth (allocs ratios: 1×/3.7×/14.8× vs input sizes 1×/4×/16×). Scorecard noted Standard 3/5 (missing -count=10 and explicit alloc target). - Without-Skill: Covered C1–C6 at comparable quality and depth. Provided clear -count=20 -benchtime=3s recommendations and a statistical power estimate ("approximately 15–20 runs per side to detect a 7% effect with 80% power"). Only gaps: Auto Scorecard and Output Contract fields.

5. Token Cost-Effectiveness¶

5.1 Measured token usage (6 evaluation agents)¶

With-Skill average: 30,312 tokens, 137.2s, 7 tool uses/eval Without-Skill average: 20,762 tokens, 75.7s, 5 tool uses/eval Runtime token overhead: +9,550 tokens/eval (+46%) Runtime time overhead: +61.5s/eval (+81%)

5.2 Skill context cost¶

5.3 Cost-effectiveness calculation¶

5.4 Comparison with other skills¶

Why go-benchmark trails git-commit on cost-effectiveness:

1. Longer SKILL.md (378 vs 169 lines): Inlined Anti-Examples (3 BAD/GOOD pairs), the Auto Scorecard template, and the Output Contract table account for roughly 100 lines.
2. Smaller absolute gain (+54pp vs +77pp): Without-Skill already scored 75% on Scenario 3, pulling the overall improvement down significantly.
3. Higher runtime overhead (+46% tokens, +81% time): Executing 3 Gates, declaring 4 Output Contract fields, and outputting a detailed Scorecard each add token cost.

Important context: If we look only at the scenarios where go-benchmark truly excels (Phase 1 + Phase 2), improvements are +78pp and +57pp respectively — bringing cost-effectiveness close to git-commit. The overall +54pp figure is diluted by Scenario 3's modest +25pp.

6. Scores¶

6.1 Dimension scores¶

6.2 Weighted score¶

7. Conclusion¶

go-benchmark passed 100% of assertions across 24 checks in three scenarios, a +54pp improvement over Without-Skill (46%). The evaluation reveals an asymmetric value distribution:

High-value zone (Phase 1 + Phase 2 — writing and reviewing benchmark code): - Phase 1, +78pp: Without-Skill used no sink variables in actual benchmark code, or treated sinking as an optional fallback. This omission is silent and undetectable from benchmark output — the code compiles and runs but may measure nothing. - Phase 2, +57pp: Without-Skill's judgment that _ = data is "safe here" is technically not wrong for this specific case, yet it led to a corrected file that still left the sink problem unresolved — illustrating the gap between principled understanding and case-by-case reasoning.

Low-value zone (Phase 3 — statistical analysis): - Phase 3, +25pp: Without-Skill performed at a high level on p-values, CV thresholds, and significance judgment. The skill's incremental contribution here is primarily structural (Scorecard, Output Contract), not analytical.

Core value points: 1. Silent-corruption protection: _ = encode(input) looks valid, compiles cleanly, and produces no errors — but the compiler can eliminate the call entirely in certain optimization passes, turning the benchmark into a measurement of loop overhead. This is a high-frequency, self-undetectable baseline error; Hard Rule #1 is the only reliable safeguard. 2. Evidence Gate: When a user provides nothing ("my service is slow"), the skill forces the degradation path and prevents speculative analysis (validated by golden fixture BENCH-009). 3. Output consistency: Output Contract + Auto Scorecard make responses predictable and quality-checkable across users and sessions — a structured reporting mechanism entirely absent from baseline.

Recommendations: 1. Raise Phase 3 incremental value: Expand benchstat-guide.md with super-linear allocation growth analysis (alloc-to-input-size ratio patterns) and a pprof diff workflow, so statistical analysis output exceeds what baseline Claude naturally delivers. 2. Trim Auto Scorecard: Moving the ~40-line template into a reference file and keeping only a pointer in SKILL.md would save ~200 tokens and improve cost-effectiveness. 3. Trigger accuracy: Current F1 ≈ 88%; the main gap is the implicit trigger boundary (e.g., "my Go service has high memory usage"). Adding a negative clarification to the description — "not for general Go debugging or unit testing" — would reduce over-triggering without sacrificing coverage.