unit-test Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-03-11 Evaluation subject: unit-test

unit-test is a Go-focused skill for generating and improving unit tests. It is suited for adding or strengthening logic tests, fixing low-signal tests, and designing more targeted test cases for concurrency, boundary, and mapping defects. Its three standout strengths are: high trigger accuracy, reliably distinguishing unit tests from benchmarks, fuzz tests, integration tests, and similar adjacent tasks; emphasis on failure hypotheses, Killer Cases, and boundary checklists, shifting test goals from "coverage chasing" to "catching real bugs"; and consistent use of table-driven tests, t.Run, race detection, and project assertion style adaptation so tests are both standard and aligned with existing codebase practices.

1. Evaluation Overview¶

This evaluation reviews the unit-test skill along two axes: trigger accuracy and actual task performance. Task performance covers 3 different types of Go concurrency/time-sensitive target code. Each target was run with both with-skill and without-skill configurations, for 3 scenarios × 2 configs = 6 independent subagent runs, scored against 34 assertions.

2. Trigger Accuracy¶

2.1 Test Method¶

20 test queries were designed (10 should trigger / 10 should not trigger), covering Chinese and English, various unit-test scenarios, and easily confused adjacent tasks (benchmark, fuzz test, integration test, E2E, CI config, mock generation, documentation, translation, pprof analysis). Independent subagents simulated Cursor’s <agent_skills> trigger path. Each query was judged 3 times independently, for 60 judgments total.

Note on run_eval.py failure: The skill-creator run_eval.py script does not work inside Cursor IDE — the claude -p subprocess fails silently due to lost auth context (error: "Your organization does not have access to Claude"), causing all 60 queries to return triggered=false and meaningless 0% Recall / 50% Accuracy. This report’s trigger evaluation uses a Task subagent simulation instead: each round is evaluated by an independent agent with fresh context.

2.2 Results¶

Overall accuracy:  20/20 (100%)
Recall:            10/10 (100%) — all positive queries correctly triggered (3 rounds consistent)
Precision:         10/10 (100%) — all negative queries correctly excluded (3 rounds consistent)
F1:                100%
Total judgments:   60/60 (TP=30, FN=0, FP=0, TN=30)

2.3 Positive Queries (All Correctly Triggered)¶

2.4 Negative Queries (All Correctly Excluded)¶

2.5 Conclusion¶

The improved Description uses a four-layer strategy for trigger accuracy:

1. Irreplaceability signals — "references/ with killer-case pattern templates", "cannot be reproduced from memory", "mandatory 13-check tiered scorecard" make the model judge that its own knowledge cannot replace the skill
2. Strong imperative tone — "ALWAYS read this skill before writing, reviewing, or fixing ANY Go test file (_test.go)"
3. Broad trigger coverage — 12 keywords in Chinese and English + 4 indirect trigger modes (verify, check for race conditions, improve test quality, coverage is too low)
4. Explicit exclusion scope — "Do NOT use for benchmarks, fuzz tests, integration tests, E2E tests, load tests, or mock generation" effectively isolates 6 adjacent task types

3. Actual Task Performance¶

3.1 Test Method¶

Three Go code files in the repo with no existing tests were selected, covering different testing challenges:

Each scenario ran 1 with-skill + 1 without-skill subagent, 6 runs total.

3.2 Assertion Pass Rate Overview¶

3.3 Per-Item Comparison: Which Assertions Drove the Gap?¶

The 13 assertions that Without-skill failed across all 3 scenarios fall into 4 methodology dimensions:

Key observation: All 13 Without-skill failures are methodology-level, not functional coverage.

3.4 Functional Coverage Comparison¶

On "which code paths are tested", the two sides are similar:

Conclusion: Functional coverage is the same. Without-skill did not test fewer code paths; it lacked the methodology framework around those paths.

4. Skill Differentiator Value Deep Dive¶

4.1 Failure Hypothesis List¶

With Skill: Each scenario produces 7–9 numbered hypotheses (H1–H9), organized by category (Branching, Concurrency, Loop/index, Context/time), each mapped to specific test cases.

Without Skill: No such output. Tests are organized by functional area (Rate Limiting, Success Paths, Retry Exhaustion) but with no formal defect analysis.

Practical value: The Failure Hypothesis List matters not because "one more table" exists, but because it drives test design — first think "what bugs might this code have", then design tests accordingly, rather than "spread coverage by function signature".

4.2 Killer Cases¶

With Skill: Each scenario has 3–4 killer cases, each with: - Linked defect hypothesis (e.g. KC1→H3) - Fault injection description - Critical assertion (with concrete field and value) - Removal Risk Statement ("if this assertion is removed, what bug escapes")

Example (Eval 1 KC1):

Linked hypothesis: H3 — ErrBreakerOpen is retried instead of returned immediately Critical assertion: backoffCalls == 0 — no retry backoff was triggered Removal risk: If removed, the known bug (ErrBreakerOpen not short-circuiting retries) can escape detection — 4 unnecessary backoff+retry cycles would occur.

Without Skill: Tests cover the same paths (e.g. TestDo_BreakerOpenStopsRetry) but with no removal risk analysis. Developers cannot tell which assertion is "critical for regression" vs "nice to have".

4.3 Boundary Checklist¶

With Skill: Standard 12-item checklist, each labeled Covered / N/A + notes:

Without Skill: No such output. Boundary cases are scattered across test functions with no systematic audit.

4.4 Test Organization: Table-Driven vs Separate Functions¶

4.5 Auto Scorecard (13-Check Tiered Scorecard)¶

With-skill output includes a structured 13-item scorecard (3 Critical + 5 Standard + 5 Hygiene) with pass/fail evidence and tiered summary. Without-skill has no such output.

4.6 Additional Findings¶

With-skill runs surfaced real insights in the code that Without-skill did not mention:

5. Comprehensive Analysis¶

5.1 Skill Differentiator Value Map¶

5.2 Skill’s True Value Proposition¶

The skill is not for "testing more paths" — it is for "thinking systematically about why to test a path".

Core value by importance:

1. Defect-hypothesis-driven test design — List "possible bugs" (H1–H9) first, then design tests. Without-skill instead "traverses parameter combinations by API signature". The former finds bugs; the latter spreads coverage.
2. Killer Case + Removal Risk — Each killer case answers "what bug does this assertion prevent from escaping". Without this, maintainers cannot distinguish critical from redundant assertions and may delete them during refactors.
3. Structured quality audit — 13-check Scorecard provides quantifiable quality judgment (Critical tier all pass = mergeable), not subjective "looks well tested".
4. Systematic boundary checklist — 12-item standard checklist ensures nil, empty, boundary, concurrency, context cancellation, etc. are not missed; each item’s Covered/N/A provides audit trail.
5. Consistent test organization — table-driven + t.Run is not just style; it affects maintainability and the cost of adding cases.

5.3 Skill Weaknesses¶

1. No functional coverage difference: In all 3 scenarios, Without-skill covered the same core paths as With-skill (rate limit, circuit breaker, context cancel, concurrency). The skill’s differentiation is entirely at the methodology level, not "what to test".
2. Without-skill sometimes has more test cases: Eval 1 Without-skill produced 17 separate test functions vs With-skill’s 12 subtests. More cases ≠ higher quality, but shows Without-skill is not "testing less".
3. Methodology output value depends on team: Failure Hypothesis List and Killer Cases may be "helpful but not essential" for senior developers; more valuable for test newcomers or code review.
4. Limited evaluation scenarios: Only 3 concurrency/design-pattern scenarios; no database ops, HTTP handlers, pure logic functions, etc.

6. Score Summary¶

6.1 Dimension Scores¶

6.2 Weighted Total Score¶

7. Evaluation Methodology¶

Trigger Evaluation¶

• Method: Subagent simulation of trigger judgment (3 independent rounds × 20 queries = 60 judgments)
• Query design: 10 positive (Chinese/English, direct/indirect trigger modes) + 10 negative (6 adjacent task types: benchmark/fuzz/integration/E2E/load/mock + CI/docs/translation/profiling)
• Environment: Cursor IDE Task subagent (generalPurpose, fast model), fresh context each round
• Limitation: Proxy test, not end-to-end real trigger; does not account for 50+ competing skills

Task Evaluation¶

• Method: 3 scenarios × 2 configs = 6 independent subagent runs
• Target code: All real Go code in repo with no existing tests (not artificially constructed)
• Assertions: 34, covering file creation, methodology output, functional paths, test organization, race safety, quality audit (6 dimensions)
• Scoring: Manual per-assertion comparison with subagent output; record pass/fail + evidence
• Baseline: Same prompt, SKILL.md not read

Evaluation Materials¶

• Trigger evaluation queries: unit-test-workspace/trigger-eval-set.json
• Trigger evaluation results: unit-test-workspace/trigger-eval-results.json
• Eval definitions: unit-test-workspace/evals/evals.json
• Scoring results: unit-test-workspace/iteration-1/{resilience-do,worker-pool,rate-limiter}/{with_skill,without_skill}/grading.json
• Benchmark summary: unit-test-workspace/iteration-1/benchmark.json
• Description improvement report: unit-test-workspace/description-improvement-report.md
• Eval viewer: unit-test-workspace/iteration-1/eval-review.html
• Generated test code: unit-test-workspace/iteration-1/*/outputs/*_test.go
• Generated reports: unit-test-workspace/iteration-1/*/outputs/report.md

8. Round-4 Review Remediation (2026-09-10)¶

An external review raised four findings against the skill and its regression harness. All four were reproduced before any change was made, and three reproduce as executable facts rather than opinions.

8.1 Reproduction, before any fix¶

8.2 What changed¶

Suite: 103 → 138 tests, green. Every round-4 rule was mutation-checked — each guard broken in turn, the matching test required to fail: 17/17 killed against a green baseline, with a skipped test counted as SURVIVED rather than as a kill.

Two of the new tests were themselves defective and were caught by that run: one let SkipTest propagate (unittest reports a skip as green, so the guard could not detect the regression it existed to catch), and one asserted through a build-error branch because its fixture file declared a mismatched package, leaving the branch it meant to test unreachable.

8.3 On finding 4: what the evidence does and does not support¶

The review's substantive point stands and is worth stating plainly in this report: the A/B in §3 shows all 13 of Without-skill's failed assertions were methodology-level, with identical core-path coverage (§3.4). That supports "more auditable, more consistently organised output". It does not establish a higher defect-detection rate — no experiment here measured defects found. §5.1's "value map" and §6's weighted score should be read with that boundary in mind.

The remediation narrows the gap in the one place it can be narrowed without a new experiment: the killer case's central claim is now either executed and quoted, or explicitly labelled unverified. Whether the discipline raises real defect-detection remains an open question that needs a defect-seeded A/B — not re-derivable from §3.

8.4 Not re-measured¶

Sections 2–7 were not re-run for round 4. Trigger accuracy (§2), the A/B assertion pass rates (§3), the dimension scores (§6.1) and the weighted total (§6.2) all still describe the 2026-03-11 evaluation. Round 4 changed the skill document and the harness; those numbers would need a fresh A/B — including a Verification:-aware rubric — to be claimed again.

9. Round-5 Review Remediation (2026-09-10)¶

A follow-up review found three further defects, all reproduced before any change.

9.1 Reproduction, before any fix¶

9.2 What changed¶

Finding 1 — the JSON validator now validates in ordered stages. Types first (from meta.json's json_field_types, which doubles as the required-field list so there is one list instead of two), then ranges, then consistency. Consistency is unconditional, so a wrong type can no longer disable a rule; int rejects true because bool is an int subclass in Python. New rules: tier totals must match the scorecard's tier sizes, 0 <= *_pass <= *_total, percentages in 0..100, cases >= 1, and the missing cross-pillar link — the coverage gate is Critical item 13, so a measured miss (the restricted N/A covers only unmeasured coverage) must appear as a failed Critical item and an overall FAIL. Nine new mutation tests in JsonSummaryContractTests.

Finding 2 — the two claims are now separate, and the difference is executed. The mandatory report item is Kill: Verified / Kill: Unverified (defect injected, failure observed). An assertion-necessity claim requires its own experiment — delete the named assertion with the defect still injected and see the test PASS — and is optional; the blanket "if this assertion is removed, the known bug can escape detection" mandate is retired from all 10 places it appeared. references/killer-case-patterns.md § Verifying the Kill now ships both procedures with different conclusions, states that a still-failing test refutes necessity, and explains that necessity is a property of an (assertion, defect) pair: the identity assertion is indispensable against a reordering defect, the length assertion against a duplication defect, and neither against a dropped tail — which is exactly the reviewer's case. test_behavioral_killer.test_removing_an_assertion_can_still_leave_the_mutation_caught pins that fact by execution.

Finding 3 — the exemplar was rewritten to be true of its own code, and two checks now compare the two. good.md ships a 4-case table-driven test (nil, empty, single, three elements), reports 4 cases, maps H1 and H2 to the cases that actually exercise them, includes the full boundary checklist that justifies its scorecard, records Kill: Verified with the observed failure, and states honestly that no assertion-necessity claim is supported for H1 — with the reason. The grader now runs go test -v, so _grade_report_matches_code can compare sum(targets[].cases) against the cases the toolchain executed, and require an executed case per fixture-declared hypothesis (required_case_patterns).

Suite: 140 → 153 tests, green. SKILL.md's line budget was raised 500 → 520; the reason is recorded in the test.

9.3 The general lesson¶

Findings 1 and 3 are the same defect at two levels: a check that inspects form while the claim it certifies is about content. A JSON fence is not a JSON document; a parsed document is not a coherent report; a coherent report is not a report about the code that ships with it. Each round of this skill's hardening has moved one boundary further from form toward content, and each move was found by an experiment rather than by re-reading the rules.

Finding 2 is different in kind: the rule itself was unsound, not merely unenforced. It mandated a claim that its own recommended experiment could not establish, and the exemplar demonstrated the claim being false. A mandate that cannot be verified by the procedure it ships with will be satisfied by assertion — which is what happened.

10. Round-6 Review Remediation (2026-09-10)¶

Two further gaps, both reproduced before any change, and both variants of the round-5 lesson: a check that inspects form while the claim is about content.

10.1 Reproduction, before any fix¶

10.2 What changed¶

Discovered is not verified. executed_case_names became test_case_results, a name → PASS/FAIL/SKIP map, with verified_cases dropping the skips. A hypothesis is covered only by a matching case that reached a verdict, and this fixture — a pure function with no environment dependency — reports any skip at all (allow_skipped_cases: false).

Every hypothesis owns a mutation. One mutation per fixture was the structural cause of finding 2: H2 could be stated, given an input scenario, matched by name, and never verified, because nothing tested the behaviour it promised. The fixture's three parallel lists (hypothesis_keywords / required_case_patterns / mutation) collapsed into one hypotheses array where each entry carries its wording, its required case and its mutation. The emitted test must now kill each hypothesis's mutation.

The exemplar was corrected — by reading the contract, not the implementation. [] vs null is a real observable difference for any serialized result, so the non-nil promise was made explicit in the system under test's doc comment and then asserted (got == nil is a separate check, because len cannot see it). H2 became a second killer case, K2, with its own executed kill. Had the contract not promised non-nil, the correct fix would have been to narrow H2 — an exemplar must not invent a requirement for the code it tests.

Two exemplar details worth recording: the reported failure lines point at the assertIDs(...) call site rather than the assertion, because t.Helper() reattributes them; and the exemplar's killer-case count went from 1 to 2, since each hypothesis now has a defect-bound case with a verified kill.

The mutation run then found the opposite direction unguarded. Deleting the non-nil sentence from sut.go's doc comment left the exemplar passing — nothing bound H2's assertion to a promise the code actually makes, so an exemplar could invent a requirement as easily as it could omit a verification. Each hypothesis now declares contract_evidence, a regex that must match the source, and validate_fixture rejects an ungrounded hypothesis (or one declaring no evidence) before any response is graded. The two checks point in opposite directions: the mutation proves the hypothesis is verified, the evidence proves it is legitimate.

The same two rules were missing from the skill itself, not just the grader: a skipped case would equally be marked Covered on a boundary checklist. Reporting Integrity now states that --- SKIP is discovered, not verified, and blocks Kill: Verified on it; the Defect-First Workflow requires each hypothesis to name the implementation change that would violate it ("if you cannot name one, the hypothesis is not checkable as written") and states that supplying the input is not verification; references/bug-finding-techniques.md documents the len(nil) == 0 trap for slices, maps and []byte, and routes the decision through the contract rather than the current implementation.

Suite: 154 → 166 tests, green.

10.3 The pattern across rounds 4–6¶

Every finding so far has had the same shape: the checked artifact was one step away from the claimed one.

Each round closed one step and the next review found the following one. The rule that generalises: for every claim, name the observation that would be different if the claim were false, and check that — which is the same discipline the skill demands of a killer case, applied to the skill's own harness.