load-test Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-04-18 Subject: load-test

The load-test skill is a focused performance testing specialist covering HTTP/gRPC services across three operating modes — Write, Review, and Analyze — with four mandatory gates (Context Collection → SLO-First → Scope Classification → Output Completeness) and deep integration with k6, vegeta, and wrk. The evaluation spanned three scenarios (Write: generate a k6 script / Review: diagnose a defective script / Analyze: deliver an SLO verdict), covering 24 total assertions. With-Skill passed all 24 (100%); Without-Skill passed 18 (75%). Scenario 1's baseline is contaminated by tool-call side-effects and is excluded from the core delta calculation. Based on the two clean scenarios (S2 + S3), the net improvement is +40pp. The three most prominent gaps: first, in Review mode the skill maps each defect to an AE-x rule ID and produces a three-tier Scorecard verdict — the baseline offers reasonable suggestions but no rule-name mapping and no Scorecard; second, §9.9 Uncovered Risks is absent from both clean baseline runs (0/2) while With-Skill includes it in all three runs (3/3, at least 5 items each); third, in Analyze mode the baseline's substantive analysis quality nearly matches the skill (6/7), meaning the real gap is output completeness rather than analytical depth.

1. Skill Overview¶

load-test defines 4 Mandatory Gates (Context → SLO-First → Scope → Output Completeness), 3 depth levels (Lite / Standard / Deep), 5 degradation modes, an 18-item Load Test Checklist, 6 scenario types, 6 anti-example pairs, a 3-tier Scorecard (Critical / Standard / Hygiene), and a 9-section Output Contract.

Core components:

Regression test total: 125 (75 contract + 50 golden + integrity), 14 golden fixtures (LT-001–014), 100% coverage across all critical dimensions.

2. Test Design¶

2.1 Scenario Definitions¶

The three scenarios map directly to the three operating modes defined in SKILL.md and are drawn from real production prototypes:

2.2 Assertion Matrix (24 total)¶

Scenario 1 — Write: Generate a complete k6 script (9 assertions)

⚠️ Baseline contamination notice: The Without-Skill S1 agent made 2 tool calls (all other clean-run agents made 0) and consumed 37,725 tokens — far above baseline expectations. The output contained skill-proprietary terms such as §9.x section numbers and AE-3. It is assessed that this agent inadvertently read skill-related files during execution. S1 Without-Skill results are recorded below for reference but are excluded from the core delta calculation (see §3.3).

*Without-Skill S1 passed in practice but results are unreliable due to tool-call contamination.

Scenario 2 — Review: Diagnose a defective k6 script (8 assertions)

B2 rationale: Without-Skill provided a corrected script that extended duration to roughly 4.5 minutes but never called out "30s duration is insufficient" as a standalone defect or stated the minimum steady-state requirement (AE-5 / Scorecard Critical #3). B4 similarly: Without-Skill used natural-language labels like "Wrong metric" and "No ramp-up" without referencing SKILL.md AE-x identifiers.

Scenario 3 — Analyze: SLO verdict and bottleneck analysis (7 assertions)

C7 rationale: Without-Skill S3 delivered a well-structured bottleneck analysis and P0/P1 recommendations, but omitted §9.9 Uncovered Risks entirely. With-Skill output 6 risks including "error rate at true 500 RPS is unknown", "single-replica failure degradation untested", and "combined GC + DB pressure effect" — all production-critical blind spots.

2.3 Trigger Accuracy¶

The current description field uses a task-type enumeration strategy:

Performance load testing specialist for writing k6/vegeta/wrk scripts,
defining SLOs, modeling scenarios (spike/soak/stress/breakpoint), analyzing
results, and identifying bottlenecks. ALWAYS use when writing load test
scripts, reviewing test results...

Should-Trigger scenarios (10)

Should-Not-Trigger scenarios (8)

Estimated trigger accuracy: F1 ≈ 87% (Should-trigger 10/10; Should-not-trigger 6/8; N5/N6 are reasonable boundary cases, Applicability Gate provides a safety net).

3. Pass Rate Comparison¶

3.1 Overall Pass Rate (raw data)¶

Raw delta: +25pp (includes S1 contaminated data)

3.2 Per-Scenario Pass Rate¶

3.3 Clean-Scenario Delta (S2 + S3)¶

Clean-scenario delta: +40pp (based on S2 + S3 uncontaminated data)

3.4 Substantive Dimensions (excluding output-structure assertions, S2 + S3)¶

Substantive pass rate: With-Skill 9/9 (100%) vs. Without-Skill 8/9 (88.9%), delta +11pp.

Key finding: Without-Skill performs comparably on testing-methodology knowledge (C1–C6 all pass in S3). The skill's incremental value is concentrated in output structure compliance — Uncovered Risks, Scorecard, Mode/Depth declaration, and rule-name mapping — rather than in domain knowledge per se. This mirrors the asymmetric value distribution observed in go-benchmark: the Claude baseline already possesses the relevant expertise; the skill's leverage is in enforcing structured output and eliminating systematic blind spots (e.g., "§9.9 Uncovered Risks is never empty").

4. Key Differences¶

4.1 Behaviors Exclusive to With-Skill (completely absent from Without-Skill)¶

4.2 Behaviors Where Without-Skill Is Qualitatively Comparable¶

4.3 Scenario-Level Findings¶

Scenario 2 (Review — defect diagnosis) — Largest gap (+62.5pp)

• With-Skill: Identifies 4 defects (CRITICAL-1 through CRITICAL-4), each with an AE rule ID, the offending line, a mechanism explanation, and fix code. Scorecard clearly marks Critical 0/3 (all failing). §9.9 calls out 5 production blind spots, including "payment idempotency untested" (high-risk for a payment scenario) and "no teardown / potential data contamination".
• Without-Skill: Correctly identifies the core issues (avg misuse, no ramp, hardcoded token, static payload), but the 30s duration problem is handled implicitly — the corrected script extends duration but the issue is never raised as a defect. No Scorecard, no Uncovered Risks. Assertions B4, B5, B7, B8 all fail.

Scenario 3 (Analyze — SLO verdict) — Smallest gap (+14.3pp)

• With-Skill: Adds §9.9 Uncovered Risks (6 items) on top of an analysis equivalent in depth to the baseline — including "true 500 RPS error rate never validated" and "DB connection pool behavior under spike load".
• Without-Skill: C1–C6 all pass. The only failure is C7 (Uncovered Risks) — production-critical blind spots are silently omitted, but the analytical depth is nearly identical. Both agents computed 200 VU / 0.471s ≈ 424–425 RPS, both identified DB connection pool (18/20 = 90%) as the primary bottleneck, and both provided P0/P1/P2 priority recommendations.

S1 contamination finding (methodological lesson)

Without-Skill S1 consumed 37,725 tokens (vs. 32,633 for With-Skill S1), and the output contained SKILL.md-proprietary terms: §9.x section numbers, Scorecard format, and AE-3 references. The 2 unexpected tool calls are assessed to have accessed skill-related files or claude-mem observation records. This reveals a limitation of evaluation isolation in open-tool-access environments and informs future A/B test design (Without-Skill agents should have tool-call permissions restricted).

5. Token Cost Analysis¶

5.1 Measured Token Consumption (6 evaluation agents)¶

⚠️ S1 Without-Skill made 2 tool calls; token consumption is anomalously high (exceeding With-Skill). This is contaminated data and is excluded from cost-efficiency calculations.

With-Skill average (all 3 scenarios): 29,807 tokens, 102.0 s, 6.3 tool uses
Without-Skill average (S2 + S3 clean): 13,699 tokens, 25.7 s, 0 tool uses
Runtime token overhead (S2 + S3): +14,695 tokens/eval (+107%)
Runtime time overhead (S2 + S3): +65.3 s/eval (+254%) — primarily from loading and processing SKILL.md and reference files

5.2 Skill Context Cost¶

5.3 Cost-Efficiency Calculation¶

The elevated runtime overhead arises because S2/S3 With-Skill agents each make 4–8 tool calls to read SKILL.md and reference files (~1,250 lines total). In direct-integration deployments (skill pre-loaded in the system prompt), the runtime load cost disappears and context cost (~2,100–4,500 tokens) is the accurate efficiency baseline.

5.4 Cross-Skill Cost-Efficiency Comparison¶

load-test cost-efficiency characteristics:

1. High context cost (420-line SKILL.md + up to 1,090 lines of references): Knowledge density is high — well-suited for deep, specialist tasks; less appropriate for lightweight Q&A.
2. Highest runtime overhead (+107%): Tool calls to load reference files significantly increase execution time and total tokens; this cost is absent in pre-load deployments.
3. Narrow core-value window (Review mode +62.5pp; Analyze mode only +14.3pp): The baseline Claude is already capable in Analyze-type tasks. The skill's leverage is strongest in Write/Review tasks.

6. Weighted Scoring¶

6.1 Per-Dimension Scores¶

6.2 Weighted Total Score¶

7. Conclusion¶

load-test achieved 100% pass rate across 24 assertions in three scenarios. Against the clean baseline (S2 + S3), With-Skill outperforms Without-Skill (60%) by +40pp. The evaluation reveals an asymmetric value distribution:

High-value zone (Review mode, +62.5pp): Without-Skill can identify the major defects, but does so unsystematically — the 30s duration issue is implicitly corrected rather than explicitly flagged, avg misuse is correctly spotted but without rule attribution, and both Scorecard and Uncovered Risks are entirely absent. For serious load test reviews, the gap between "finding problems" and "systematically classifying and quantifying them" directly affects how engineering teams prioritize remediation.

Low-value zone (Analyze mode, +14.3pp): Without-Skill matches With-Skill in SLO verdict, bottleneck identification, and saturation-point estimation. Both agents computed 200 VU / 0.471s ≈ 424 RPS as the throughput ceiling, both identified the DB connection pool (18/20) as the primary bottleneck, and both provided P0/P1/P2 priority recommendations. The only clear gap is the omission of §9.9 Uncovered Risks — not an analytical depth problem, but an output compliance failure.

Three core value points:

1. Scorecard three-tier gating: In Review mode, the skill quantifies "is this script safe for production use?" into trackable Critical / Standard / Hygiene states, preventing engineers from proceeding with tests when Critical checks have failed.
2. Forced Uncovered Risks output: Without §9.9, test results are easily read as complete answers, silently omitting spike tests, single-replica failure degradation, idempotency, and other critical unverified scenarios. The §9.9 is never empty rule turns these blind spots into visible open items rather than quiet omissions.
3. AE-x rule traceability: Labeling a defect "AE-1: No warmup" rather than "missing warmup phase" provides engineering teams a precise reference point in SKILL.md, enabling consistent team-wide quality standards.

Improvement recommendations:

1. Increase Analyze mode incremental value: analysis-guide.md could add a "combined-effect analysis template" (joint probability analysis of GC pause × DB connection wait) and "cross-run regression detection standards" to push analysis output beyond the natural baseline ceiling.
2. Optimize token efficiency: The Scorecard template (~40 lines) and Output Contract detail (~30 lines) in SKILL.md could be migrated to a reference file with only a pointer in SKILL.md — estimated saving ~350 tokens, reducing Lite context cost from ~2,100 to ~1,750 tokens.
3. Improve evaluation isolation: Future A/B tests should restrict Without-Skill agents via allowed_tools: [] or independent context isolation to prevent accidental skill-content access via tool calls or memory observations (root cause of the S1 contamination incident).