fuzzing-test Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-03-12 Evaluation subject: fuzzing-test

fuzzing-test is a skill specialized in generating high-signal fuzz tests for Go code, suitable for parsers, codecs, state transitions, and other targets with clear invariants. It also helps determine when a target is not worth fuzzing at all. Its three main strengths are: running an Applicability Gate first before deciding whether to enter the generation flow, avoiding "write fuzz for every function"; explicitly rejecting unsuitable targets with alternative suggestions instead of producing low-value code; and built-in target prioritization, cost tiers, and structured output for more controllable, cost-effective fuzz testing.

1. Evaluation Overview¶

This evaluation reviews the fuzzing-test skill along two dimensions: actual task performance and token cost-effectiveness. It uses 3 fuzz test generation scenarios (suitable parser target, unsuitable network-dependent target, package-level evaluation with multiple candidate functions), each run with both with-skill and without-skill configurations—3 scenarios × 2 configs = 6 independent subagent runs—scored against 35 assertions.

2. Test Methodology¶

2.1 Scenario Design¶

2.2 Execution¶

• Use issue2md project as base; create independent copies per scenario (/tmp/fuzz-eval-*)
• With-skill runs load SKILL.md and referenced materials first
• Without-skill runs load no skill; model uses default behavior
• All runs execute in parallel in independent subagents

2.3 Scenario Details¶

Eval 1 — parser.Parse (suitable target)

Parse(rawURL string) (ResourceRef, error) is a classic Tier 1 fuzz target: - Accepts string input (native Go fuzz type) - Pure function, no I/O, network, or state - Multiple verifiable invariants (non-empty Owner, Number > 0, Type ∈ valid set, canonical URL consistency, re-parse idempotency) - Fast execution (sub-microsecond)

Eval 2 — fetcher.Fetch (unsuitable target)

Fetch(ctx, ref, opts) (IssueData, error) is a classic unsuitable fuzz target: - All code paths perform real HTTP requests - Depends on GitHub API token auth - Includes retry + backoff logic - Interesting input space is API response, not method parameters

Eval 3 — converter package (multiple candidates)

5 candidate functions: 4 suitable, 1 unsuitable: - ✅ yamlQuote(string) string — YAML escaping, round-trip invariant - ✅ normalizeSummaryJSON(string) (string, error) — JSON extractor, json.Valid invariant - ✅ detectSummaryLanguage(string) string — Unicode analysis, finite return set invariant - ✅ capSummarySourceLength(string) string — rune truncation, length upper-bound invariant - ❌ Summarize(ctx, data, lang) — OpenAI HTTP call, network-dependent

3. Assertion Pass Rate¶

3.1 Overview¶

3.2 Per-Scenario Assertion Details¶

Eval 1: parser-fuzz (15 assertions)¶

Eval 2: fetch-reject (7 assertions)¶

Eval 3: converter-multi (13 assertions)¶

3.3 Classification of 19 Without-Skill Failed Assertions¶

3.4 Key Finding: Eval 2 +100pp Delta¶

This is the largest single-scenario delta among all evaluated skills. Analysis:

With-Skill behavior: - Runs 5-item Applicability Gate - Marks Check 1/3/4/5 as Fail (especially Check 3 — no oracle — triggers Hard Stop) - Produces "Not suitable" verdict - Recommends 4 alternative strategies, including "fuzz pure mapping functions in the package"

Without-Skill behavior: - No gate; directly analyzed how to make fuzz work - Creatively built fuzzRoundTripper (custom http.RoundTripper) to stub HTTP layer - Effectively fuzzed GraphQL JSON parsing path, not the Fetch method itself - Only oracle was "no panic"

Assessment: The baseline approach has practical value (can find panics in JSON parsing) but from fuzz testing best practices: 1. Oracle is only "no panic"; cannot find logic bugs (invariant violations) 2. Actually tests JSON parsing path, not the Fetch method under review 3. Does not tell the user "this is not optimal," missing the chance to steer them toward fuzzing pure functions

The skill's gate mechanism ensures honest engineering decisions: if unsuitable, do not proceed, and recommend better alternatives.

4. Dimension-by-Dimension Comparison¶

4.1 Applicability Gate¶

This is the skill's core differentiator, affecting all 3 scenarios.

Practical value: - Applicability Gate prevents generating useless fuzz tests (Eval 2 saves cost of writing and maintaining low-value tests) - Structured checklist makes decisions auditable and reproducible - In Eval 3, enforces "evaluate first, then code" workflow

4.2 Systematic Size Guard Coverage¶

Analysis: The skill's "Size guard present" rule (in SKILL.md Templates A/B/C/D) ensures all string/[]byte harnesses have boundary protection. Without-skill had more seeds in Eval 1 (29 vs 19) but lacked size guards; long fuzz runs risk OOM.

4.3 Output Contract (Structured Report)¶

With-Skill runs produce structured reports including:

Without-Skill produces narrative summaries but no standardized structure.

4.4 Fuzz Code Quality Comparison¶

Using Eval 3 (best for code quality comparison), FuzzYamlQuote:

Code quality is similar in oracle design; Claude's base model is already strong at fuzz code generation. The skill's main gains are process discipline (gate, cost class, size guard, output contract), not the code itself.

4.5 Alternative Strategy Recommendations¶

In Eval 2, With-Skill recommended 4 alternatives:

1. Integration tests with real GitHub token (gated) — gated integration tests
2. Unit tests with HTTP stubbing — httptest.Server stub tests
3. Fuzz the pure mapping functions instead — e.g. mapIssueTimelineNode
4. Table-driven unit tests for the dispatcher — table-driven unit tests

These recommendations both reject the unsuitable approach and steer users toward more valuable testing paths. Without-Skill built a workaround directly (valuable, but did not inform users of better options).

5. Token Cost-Effectiveness Analysis¶

5.1 Skill Size¶

5.2 Load Scenarios¶

5.3 Token Cost for Quality Gain¶

5.4 Token Segment Cost-Effectiveness¶

5.5 High-Leverage vs Low-Leverage Instructions¶

High leverage (~930 tokens SKILL.md → 19 assertion delta, 23% of SKILL.md): - Applicability Gate + Hard Stop rules (400 tok → 9 assertions) - Output Contract definition (200 tok → 3 assertions) - Cost Class + Quick Commands (100 tok → 3 assertions) - Size guard examples in Templates (150 tok → 2 assertions) - Fuzz Mode + Target Priority (80+150 tok → 2 assertions)

Medium leverage (~700 tokens → indirect): - Quality Scorecard (200 tok) — drives self-check flow - Anti-Examples (500 tok) — avoid common mistakes

Low leverage (~950 tokens → 0 assertion delta): - Coverage Feedback (~400 tok) — not used in eval scenarios - Go Version Gate (~200 tok) — not used in eval scenarios - Troubleshooting (~350 tok) — not used in eval scenarios

References (~3,460 tokens → indirect): - applicability-checklist.md (1,250 tok) — improves gate quality, concrete examples - target-priority.md (1,170 tok) — Tier ordering basis - crash-handling.md + ci-strategy.md (1,040 tok) — no direct contribution in eval

5.6 Token Efficiency Rating¶

5.7 Cost-Effectiveness vs Other Skills¶

Analysis: - fuzzing-test has the largest delta (+54.3%), mainly from Eval 2's +100pp extreme delta - SKILL.md cost-effectiveness (~75 tok/1%) is mid-range: higher than create-pr (38) and go-ci-workflow (45), lower than go-makefile-writer (63) - Typical-load cost-effectiveness (~120 tok/1%) is better than go-makefile-writer and go-ci-workflow, worse than create-pr - SKILL.md size (679 lines / ~4,100 tokens) is the largest among evaluated skills, but its delta is also the largest

6. Boundary Analysis vs Claude Base Model¶

6.1 Base Model Capabilities (No Skill Increment)¶

6.2 Base Model Gaps (Skill Fills)¶

7. Overall Score¶

7.1 Dimension Scores¶

7.2 Weighted Total Score¶

7.3 Comparison with Other Skills¶

Analysis: - fuzzing-test rejection (+5.0 delta) is the largest single-dimension delta among evaluated skills - +54.3pp delta is also the highest, proving Applicability Gate value - Token cost-effectiveness score (6.0/10) is lower due to SKILL.md size (679 lines) and ~950 tokens low-leverage content

8. Conclusion¶

The fuzzing-test skill adds clear value in three areas:

1. Applicability Gate rejection (+100pp single-scenario delta): The largest single-scenario delta among evaluated skills, showing that "when not to fuzz" is a major gap for Claude. The baseline builds workarounds for unsuitable targets (not without value) but does not inform users of better strategies.

2. Systematic Size Guard coverage (5/5 vs 0/5): The skill's templates and rules ensure all string/[]byte harnesses have length bounds, preventing OOM in long fuzz runs. A common omission with large production impact.

3. Structured Output Contract: Quality Scorecard (Critical/Standard/Hygiene) makes fuzz test quality measurable and auditable.

Main risk: SKILL.md size (~4,100 tokens) is the largest among evaluated skills; ~23% (~950 tokens) is low-leverage. Trimming Coverage Feedback, Troubleshooting, Anti-Examples, and Go Version Gate could reduce SKILL.md ~29% and improve typical-load cost-effectiveness from ~120 tok/1% to ~76 tok/1%.

9. Evaluation Materials¶