security-review Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-03-12 Subject: security-review

security-review is an exploitability-first security review skill for assessing authentication, input, secrets, API, data flow, dependencies, and resource lifecycle risks in code changes, with emphasis on reproducible, actionable security findings. Its three main strengths are: choosing review depth and multi-domain gate coverage first so changes of different risk levels get matching check intensity; every finding emphasizes evidence, confidence, and CWE/OWASP mapping for audit and governance; and it has systematic false-positive suppression and uncovered-risk recording so "real vulnerabilities" are separated from "suspicious points not yet findings".

1. Evaluation Overview¶

This evaluation assesses the security-review skill along two dimensions: actual task performance and Token cost-effectiveness. It uses 3 security review scenarios of increasing complexity (Web Handler review, OpenAI API client review, benign pure-function review with no security risk). Each scenario runs with both with-skill and without-skill configurations, for 3 scenarios × 2 configurations = 6 independent subagent runs, scored against 40 assertions.

2. Test Methodology¶

2.1 Scenario Design¶

2.2 Execution¶

• With-skill runs first read SKILL.md and its referenced Go secure-coding and scenario checklists
• Without-skill runs read no skill; review follows model default security review behavior
• All runs execute in independent subagents

3. Assertion Pass Rate¶

3.1 Summary¶

3.2 Classification of 20 Without-Skill Failed Assertions¶

3.3 Pass Rate by Assertion Category¶

Key finding: The skill’s core value is structural compliance and standards mapping—without-skill pass rate for these categories is 0%. Security analysis quality (finding real vulnerabilities) differs by only 7.7%, so the base model already has strong security review ability; the skill’s incremental value is process discipline, not discovery capability.

4. Dimension-by-Dimension Comparison¶

4.1 Review Depth Selection (Skill-Only Capability)¶

This is the skill’s most distinctive output.

Practical value: Review Depth controls cost-benefit: - Lite mode skips Gates B/C/E, saving ~40% review time - Standard/Deep distinction ensures security-sensitive code gets adequate review - Without-skill applies the same depth to all scenarios, over-reviewing simple code and possibly under-reviewing complex code

4.2 False-Positive Suppression Quality¶

This is the skill’s largest quality delta.

Analysis: Without-skill’s CSRF finding (Eval 1 Finding #1) conflated cost exhaustion (rate limit exhaustion) with CSRF. With-skill correctly attributed the root cause to missing rate limiting (SEC-001 P2), not CSRF—because /convert is stateless with no session/cookie/state mutation. This demonstrates the skill’s suppression discipline: it prevents inflated severity by separating root cause from delivery mechanism.

4.3 Output Structure Comparison¶

4.4 Security Finding Quality Comparison¶

Despite large structural differences, both configurations overlap significantly on core security findings:

Without-skill-only findings: - CSRF on /convert (HIGH) — root-cause misattribution - URL scheme enforcement in parser (MEDIUM) — valid defense-in-depth - Unbounded pagination (MEDIUM) — valid; with-skill mentioned in Gate F - Token via CLI flag (LOW) — valid but out of changed scope - appendThreadText recursion (LOW) — with-skill correctly suppresses

With-skill-only findings: - No core finding is with-skill-only (base model discovery is strong) - With-skill severity calibration is more precise (e.g., SSRF DNS rebinding correctly P3 suspected, not HIGH)

4.5 Eval 3 (Benign Code) — Lite Review Quality¶

This is the skill’s clearest efficiency advantage.

Analysis: Without-skill’s 46-line output correctly concluded "no security vulnerabilities" but lacked audit traceability. With-skill’s 227-line output provides full audit record: why Lite was chosen, why each domain is N/A, which checks were skipped and why. For compliance (e.g., SOC 2 audit), this traceability is required.

5. Token Cost-Effectiveness Analysis¶

5.1 Skill Size¶

Typical load scenarios:

5.2 Token Cost for Quality Gain¶

5.3 Token Segment Cost-Effectiveness¶

SKILL.md content split by functional module:

5.4 High-Leverage vs Low-Leverage Instructions¶

High leverage (~1,710 tokens SKILL.md → 18 assertion delta): - Review Depth Selection (250 tok → 3) - Evidence Confidence (100 tok → 3) - Suppression Rules (180 tok → 2) - Output Contract (500 tok → 3) - Gate D 10-Domain (400 tok → 3) - Gate F Uncovered Risk (80 tok → 3) - Standards Mapping (50 tok → 3) - Gate A pairing (150 tok → 1)

Medium leverage (~1,100 tokens → indirect quality gain): - Anti-Examples (350 tok) — prevents false positives - Scenario Checklists pointer (200 tok) — systematic - Severity Model + SLA (200 tok) — severity calibration - Focused Automation Gate (350 tok) — tool execution discipline

Low leverage (~250 tokens → no contribution in this evaluation): - Baseline Diff Mode (100 tok) — not tested - Language Extension Hooks (150 tok) — Go only tested

References (~5,800 tokens → indirect review depth): - go-secure-coding.md (4,600 tok) — Gate B/D depth support - scenario-checklists.md (1,200 tok) — scenario systematic coverage

5.5 Token Efficiency Rating¶

5.6 Comparison with Other Skills’ Cost-Effectiveness¶

Analysis: security-review’s SKILL.md cost-effectiveness (76 tok/1%) is mid-to-low among evaluated skills, but its absolute pass-rate gain (+50.0%) is highest, meaning the skill addresses a more fundamental gap—the base model has a large gap in security review structural compliance (without-skill structural compliance pass rate 0%), and the skill fully fills it.

References account for ~60% of tokens, but the Go secure-coding reference is required for Gate B/D and cannot be simplified. If selective loading is introduced (Lite skips go-secure-coding.md), Lite scenario token cost could drop from ~9,600 to ~5,000.

6. Boundary Analysis vs Claude Base Model Capabilities¶

6.1 Base Model Capabilities (No Skill Increment)¶

6.2 Base Model Gaps (Skill Fills)¶

7. Overall Score¶

7.1 Dimension Scores¶

7.2 Weighted Total¶

8. Evaluation Artifacts¶