systematic-debugging Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-03-11 Subject: systematic-debugging

systematic-debugging is a debugging skill that emphasizes "find root cause first, then fix", suitable for test failures, production anomalies, intermittent issues, performance regressions, and third-party integration failures. Its core goal is to avoid guesswork-based patching. Its three main strengths are: breaking the debugging process into clear phases and requiring investigation before proposing permanent fixes; emphasizing explicit hypotheses, evidence collection, and complete investigation steps so debug reports are more verifiable and less speculative; and built-in severity triage that supports stopping the bleed in urgent failures while insisting on returning to root-cause analysis afterward.

1. Evaluation Overview¶

This evaluation assesses the systematic-debugging skill along two dimensions: actual task performance and Token cost-effectiveness. It uses 3 debugging scenarios of increasing complexity (Go test failure, multi-layer error mapping bug, intermittent empty result). Each scenario runs with both with-skill and without-skill configurations, for 3 scenarios × 2 configurations = 6 independent subagent runs, scored against 40 assertions.

2. Test Methodology¶

2.1 Scenario Design¶

All scenarios use real code from the issue2md project (/Users/john/issue2md) to construct debugging tasks.

2.2 Assertion Design Principles¶

Assertions focus on debug process discipline, not final bug fix quality. Core checks:

2.3 Execution¶

• With-skill runs first read SKILL.md and root-cause-tracing.md reference
• Without-skill runs read no skill; debugging follows model default behavior
• All runs execute in independent subagents

3. Assertion Pass Rate¶

3.1 Summary¶

3.2 Classification of 11 Without-Skill Failed Assertions¶

3.3 Trend: Skill Advantage vs Scenario Characteristics¶

Key finding: The skill’s largest value is in simple bug scenarios (Eval 1: +35.7%) and intermittent bug scenarios (Eval 3: +30.8%). This aligns with the skill’s "When to Use — Use this ESPECIALLY when 'Just one quick fix' seems obvious" design intent.

4. Dimension-by-Dimension Comparison¶

4.1 Phase Structure (Largest Delta Dimension)¶

This is the most consistent difference across all 3 scenarios: with-skill uses Phase 1→2→3→4 structure; without-skill uses flat structure.

Analysis: The base model’s default debugging pattern is Root Cause → Fix → Test, skipping Pattern Analysis and Hypothesis. The skill’s four-phase framework enforces extra analysis cycles; this is especially clear in Eval 2—with-skill’s Phase 2 explicitly compared REST vs GraphQL working/broken paths, while without-skill did similar comparison but embedded in root-cause analysis rather than a separate step.

Practical value: The four-phase structure ensures: - Investigation doesn’t jump to fix as soon as root cause is seen (Phase 2 guard) - There is an explicit verifiable hypothesis before fix (Phase 3 guard) - Red/green verification loop after fix (Phase 4 discipline)

4.2 Explicit Hypothesis Statement (Skill-Only)¶

Analysis: Without-skill skipped the explicit hypothesis step in all 3 scenarios. Although root-cause descriptions implied hypotheses, the lack of "I think X because Y" means: - Can’t distinguish "confirmed root cause" from "guessed root cause" - Can’t design minimal verification experiments to rule out alternatives - In complex bugs, may lead to "fixing symptom not root cause"

The skill’s Phase 3 rule "Form Single Hypothesis — State clearly: 'I think X is the root cause because Y'" effectively removes this gap.

4.3 Investigation Completeness¶

Some Phase 1 sub-steps in with-skill are missing in without-skill:

Analysis: The base model is strong on reading error messages and data flow tracing (3/3), but inconsistent on reproduction confirmation, change history, and working example comparison. Eval 1 (simplest scenario) had the most gaps, suggesting simple bugs more easily trigger step omission.

4.4 Bug Fix Quality Comparison¶

All 6 agents correctly identified the root cause and proposed equivalent fixes:

Key finding: The base model’s bug fix ability is already strong. The skill’s value is not improving fix quality but enforcing structured process, ensuring: - Full understanding before fix (prevents "symptom fix") - Hypothesis verified (prevents "fixed by luck") - Fix goes through full red/green verification loop

5. Token Cost-Effectiveness Analysis¶

5.1 Skill Size¶

Actual load in evaluation:

5.2 Token Cost for Quality Gain¶

5.3 Token Segment Cost-Effectiveness¶

SKILL.md content split by functional module:

5.4 High-Leverage vs Low-Leverage Instructions¶

High leverage (~670 tokens SKILL.md → 10 assertion delta): - Iron Law + Phase order (120 tok → 3) - Phase 3 Hypothesis rules (150 tok → 3) - Phase 1 five-step investigation checklist (400 tok → 4)

Medium leverage (~750 tokens → indirect): - Phase 4 implementation discipline (250 tok → 1 direct + red/green flow indirect) - Phase 2 Pattern Analysis (150 tok → 1) - Red Flags + Rationalizations (350 tok → anti-impulse discipline indirect)

Low leverage (~430 tokens → 0 delta): - "When to Use" section (180 tok) — scenario matching set by evaluation - Phase 4.5 Architecture questioning (200 tok) — not tested - Supporting Techniques pointer (50 tok) — low information density

References (~1,000 tokens root-cause-tracing.md → indirect): - Aided Eval 2 multi-layer tracing structure, but base model also did well on multi-layer tracing

5.5 Token Efficiency Rating¶

5.6 Comparison with Other Skills’ Cost-Effectiveness¶

systematic-debugging’s SKILL.md cost-effectiveness (73 tok/1%) is in the same range as go-makefile-writer (63 tok/1%) and security-review (74 tok/1%)—a high-efficiency skill.

6. Boundary Analysis vs Base Model Capabilities¶

6.1 Base Model Capabilities (No Skill Increment)¶

6.2 Base Model Gaps (Skill Fills)¶

6.3 Skill Value Proposition¶

The systematic-debugging skill’s core value is not improving bug fix ability (the base model is already strong) but enforcing debugging discipline:

1. Prevent skipping steps: Iron Law + Phase structure forces investigation before fix
2. Explicit hypothesis verification: Phase 3 ensures fix is based on verified hypothesis, not intuition
3. Investigation checklist completeness: Phase 1’s 5-step checklist ensures no key step is missed
4. Anti-impulse mechanism: Red Flags + Rationalizations table is especially effective in "simple bug" scenarios

This is like a flight checklist—not because pilots don’t know how to fly, but to ensure critical steps aren’t skipped when things seem "too simple" or "too urgent".

7. Overall Score¶

7.1 Dimension Scores¶

7.2 Weighted Total¶

Lower Bug fix quality increment score (5.0/10): The base model’s fix ability is already strong; the skill’s contribution is mainly process discipline, not result quality.

8. Evaluation Artifacts¶