go-code-reviewer Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-03-11 Evaluation subject: go-code-reviewer

go-code-reviewer is a defect-first Go code and PR review skill that focuses on real defects, regression risks, and project-policy deviations rather than generic style comments. Its three main strengths are: high trigger accuracy, with significantly lower false positives and higher signal-to-noise in complex grey-area scenarios; a review flow with mode selection, mandatory gates, and on-demand domain references that align review depth with risk; and Residual Risk, suppression rationale, and structured output for actionable, team-friendly results.

1. Evaluation Overview¶

This evaluation reviews the go-code-reviewer skill along three dimensions: trigger accuracy, actual task performance, and token cost-effectiveness. Task performance covers two difficulty levels: 4 textbook scenarios (typical common defects) and 4 subtle scenarios (grey-area judgment, domain-specific patterns, multi-file analysis)—8 scenarios × 2 configs (with-skill / without-skill) = 16 independent subagent runs.

2. Trigger Accuracy¶

2.1 Test Method¶

20 test queries (10 should trigger / 10 should not), covering Chinese and English, multiple review scenarios, and edge tasks that should not trigger. Independent subagents simulated Cursor’s <agent_skills> trigger path; each query judged TRIGGER / NO_TRIGGER.

2.2 Results¶

Overall accuracy:  20/20 (100%)
Recall:    10/10 (100%) — all positive queries correctly triggered
Precision: 10/10 (100%) — all negative queries correctly excluded

2.3 Positive Queries (All Correctly Triggered)¶

2.4 Negative Queries (All Correctly Excluded)¶

2.5 Conclusion¶

The description covers common review phrases in Chinese and English ("review"/"审查"/"check for issues"/"security review"), clearly states differentiated value (origin classification, SLA, suppression rationale), and adds "Even for seemingly simple Go review requests, prefer this skill." Trigger accuracy is 100%; no missed or spurious triggers.

3. Task Performance — Textbook Scenarios¶

3.1 Test Method¶

4 Go code files with known typical defects:

Each scenario ran 1 with-skill + 1 without-skill subagent, 8 runs total.

3.2 Defect Detection Completeness¶

For textbook defects, detection is identical. Claude’s general Go knowledge is enough for these patterns.

3.3 Quality Dimension Comparison¶

In textbook scenarios, the skill’s value is mainly:

• False positive control transparency (+1.75) — With-skill has explicit Suppressed Items (e.g. json.Marshal error ignore on safe struct, Mutex vs RWMutex as optimization not defect). Without-skill has no suppression rationale; readers cannot tell "intentionally ignored" from "review blind spot."
• Structure consistency (+1.0) — With-skill uses REV-ID / Origin / Baseline / Evidence / Action template per finding, plus Execution Status, SLA table, Residual Risk. Without-skill format varies across scenarios.
• Origin classification (Eval 4) — With-skill labels each finding introduced → must-fix or pre-existing → follow-up issue; Summary includes origin stats ("5 introduced / 4 pre-existing / 0 uncertain"). Without-skill groups by section for similar effect but lacks per-finding labels and SLA mapping.

4. Task Performance — Subtle Scenarios¶

4.1 Test Method¶

4 scenarios requiring deeper judgment, each with "traps"—patterns that are easy to misreport without the skill:

Each scenario ran 1 with-skill + 1 without-skill subagent, 8 runs total.

4.2 Overview¶

4.3 Eval 5: Grey-Area False Positive Trap¶

Code has 6 grey-area patterns: same-package err == ErrNotFound, read-only defer f.Close(), context.Background() in init, long switch function, interface{} → any cosmetic, json.Marshal safe struct error ignore. Plus 1 real bug (variable shadowing).

Key difference: Skill focuses on 2 high-value findings (validation error shadowing + dead code), zero noise. Without-skill also identified grey areas but reported 3 low-value findings. Skill put configStore concurrency risk in Residual Risk ("Medium | uncertain | test_code.go:38 | mutable package-level map without sync")—not a finding to distract developers, but not lost either.

Grey-area suppression comparison:

4.4 Eval 6: Subtle Concurrency Bug¶

4 real concurrency bugs + 1 nil map panic: time.After timer leak in select loop, WaitGroup.Add race inside goroutine, sync.Pool capacity loss, mutex-held I/O causing global serialization, DataFetcher.cache nil map. Plus 1 nil channel in select trap (used to disable select case; correct pattern).

Key difference: Defect coverage identical. Both handled nil channel trap. Skill adds:

1. More accurate severity: Nil map write causes process crash; skill correctly High, without-skill Medium.
2. Residual Risk supplement: Skill lists 3 Residual Risk items (FanOut error aggregation, Dispatch backpressure discard, FormatRecord map iteration order) for future maintenance.
3. Structured suppression: Nil channel as Suppressed Item with rationale referencing go-concurrency-patterns.md, not just "not a bug."

4.5 Eval 7: gRPC + Database Domain-Specific Patterns¶

5 domain-specific bugs: gRPC interceptor order wrong (auth after logging), gRPC deadline not passed to DB query (context.Background() instead of incoming ctx), metadata not passed downstream, N+1 query, connection pool not configured. Plus 1 err == sql.ErrNoRows grey area (QueryRow.Scan returns unwrapped sentinel; == correct here).

Key difference: Skill 100% signal-to-noise vs without-skill 67%. Without-skill’s 4 noise findings:

Skill correctly suppressed err == sql.ErrNoRows direct comparison (3 places) and referenced grey-area guidance: QueryRow.Scan returns unwrapped sentinel. This is the clearest example of reference loading value.

4.6 Eval 8: Multi-File Impact Radius Analysis¶

PR changed interface file repository.go (FindByEmail added opts ...QueryOption, List params from (limit, offset int) to UserFilter, User struct JSON tag "updated" → "updated_at"), affecting implementation postgres_repo.go and caller handler.go.

Skill captured 4 medium pre-existing issues in Residual Risk:

Key difference:

Skill merges 5 compile errors into 1 finding (with per-location origin breakdown) and uses origin classification + Residual Risk so developers know what to fix (must-fix) vs historical debt (Residual Risk). This is the largest differentiated value scenario for the skill.

5. Token Cost-Effectiveness Analysis¶

5.1 Test Method¶

Based on actual input/output from 8 eval scenarios. Token estimates use file byte size (mixed content ~3 bytes/token).

Skill input cost:

5.2 Total Token Consumption Comparison¶

Note: Isolated measurement (test code + skill context only). In real Cursor sessions, base context (system prompt, history, rules) is ~20–30K tokens; skill increment is ~1.5–2x of full context, not 6x as in the table.

5.3 Output Token Comparison¶

Observation: Eval 8 (multi-file impact) with-skill 3,354 tokens vs without-skill 3,933. Skill’s finding merge (5 compile errors → 1 finding) yields more concise output. In complex scenarios the skill can be more concise, not always more verbose.

5.4 Dollar Cost Model¶

Based on Claude Sonnet pricing (Input $3/M tokens, Output $15/M tokens):

5.5 Core Value Metrics¶

5.5.1 Output Signal Density¶

In subtle scenarios, without-skill output has 16% noise (false positives or low-value findings); with-skill has 0%. So ~470 output tokens from without-skill are "wasted" noise (5 FP × ~94 tokens/FP).

5.5.2 Developer Time ROI¶

5.5.3 Monthly ROI¶

5.6 Token Cost-Effectiveness Conclusion¶

The skill is not token-efficient, but it is highly value-efficient.

6. Comprehensive Analysis¶

6.1 Skill Differentiator Map¶

Conclusion: The more subtle the scenario, the larger the skill’s differentiated value.

• Textbook: Skill mainly improves process (unified format, SLA guidance); defect detection unchanged
• Subtle: Skill improves both detection (100% vs 100%) and judgment (89% vs 53% signal-to-noise); Residual Risk ensures no validated pre-existing issue is lost

6.2 Skill’s Real Value Proposition¶

The skill is not for "finding more bugs" but for "organizing and handling bugs better while not missing any high-risk issues."

Core value by importance:

1. Signal-to-noise control — 19 precise findings vs 32 noisy findings. In subtle scenarios, without-skill’s extra 13 findings include ~5 false positives or noise, increasing cognitive load.
2. Zero-miss High coverage — Severity-tiered volume cap ensures all High defects are reported; no high-risk finding dropped.
3. Transparent false positive management — 9 Suppressed Items each with structured rationale (anti-example catalog and references); team knows what was excluded and why.
4. Origin classification + Residual Risk — Keeps developers unblocked by historical debt while preserving all validated pre-existing issues. "4 issues to fix + 4 known debt recorded" is friendlier than "10 issues mixed together."
5. Standardized review flow — Unified template (REV-ID / Origin / Evidence / Action), mandatory gates, severity-tiered volume cap, SLA table.
6. Reference loading — In gRPC/database domain scenarios, ensures correct checklists are loaded and domain best practices are not missed.

6.3 Remaining Weaknesses¶

1. Limited textbook differentiation: For typical common defects, the skill finds no more bugs than generic Claude; difference is process only (+0.56/5.0).
2. Occasional severity drift: Eval 6: without-skill rated nil map Medium, skill High. Skill is more accurate (nil map write = process crash), but shows possible inconsistency at boundaries.
3. Eval 7 extra Medium findings: Skill reported 3 extra Medium findings (error leak, error context, input validation); without-skill reported similar but more. All skill extras are valid; no noise.

7. Score Summary¶

7.1 Dimension Scores¶

7.2 Weighted Total Score¶

8. Evaluation Methodology¶

Trigger evaluation¶

• Method: Subagent simulates trigger judgment; description shown to independent agent for 20 queries TRIGGER/NO_TRIGGER
• Query design: 10 positive (Chinese/English, multiple review scenarios) + 10 negative (edge tasks that should not trigger)

Task evaluation¶

• Method: 8 scenarios × 2 configs = 16 independent subagent runs
• Textbook: 22 planted defects + 22 semantic/structural assertions
• Subtle: 17 real defects + 7 grey-area/trap patterns
• Quality dimensions: 7 dimensions × 0–5 score
• Baseline: Same prompts, no SKILL.md loaded

Token cost-effectiveness evaluation¶

• Method: Token estimates from actual file sizes (mixed content ~3 bytes/token)
• Input: SKILL.md (30,677 bytes) + scenario-triggered references (14–45K bytes)
• Output: review.md file size measured directly
• Cost model: Claude Sonnet (Input $3/M, Output $15/M)
• Developer time: FP triage ~10 min each, structured output saves ~5 min/review, $100/hr

Evaluation materials¶

• Trigger eval queries: go-code-reviewer-workspace/trigger-eval.json
• Textbook grading: go-code-reviewer-workspace/iteration-1/grading_results.json
• Textbook benchmark: go-code-reviewer-workspace/iteration-1/benchmark.json
• Eval viewer: go-code-reviewer-workspace/iteration-1/eval_review.html
• Test code: go-code-reviewer-workspace/iteration-{1,2}/eval-*/test_code.go
• Subtle outputs: go-code-reviewer-workspace/iteration-2/eval-{5,6,7,8}-*/with_skill/review.md and without_skill/review.md
• Token analysis: token_analysis.json, token_analysis.py