writing-plans Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-03-27 Evaluation target: writing-plans

writing-plans is a structured skill for pre-implementation planning in multi-step tasks. It links requirement clarification, applicability checks, path discovery, and scope assessment through 4 mandatory Gates, with the goal of producing high-quality implementation plans that have verified paths, graded risks, defined interfaces, and are ready to execute immediately. Its three strongest advantages are: a 4-Gate upfront flow that prevents "ghost plans" from being written before the task is clear; four execution modes (SKIP / Lite / Standard / Deep) that scale output to task complexity and avoid document overload; and a verification system built around [Existing] / [New] / [Inferred] / [Speculative] path labels plus [interface] / [test-assertion] / [command] code-block tags, turning the plan into an engineering document that is both verifiable and executable.

1. Skill Overview¶

writing-plans is a structured implementation-planning skill. It defines 4 mandatory Gates, 4 execution modes, 10 anti-pattern checks, and a template system covering 6 change types. Its purpose is to ensure every plan completes requirement clarification, path verification, and risk grading before writing begins.

Core components:

2. Test Design¶

2.1 Scenario Definition¶

Scenario 1 test prompt:

"I need to add JWT-based user authentication to our Go REST API. The API currently serves /users and /products endpoints. I want to add /auth/login, /auth/register, and /auth/refresh endpoints with middleware that protects existing routes."

Scenario 2 test prompt:

"Make the system faster. There are some performance issues we need to fix."

Scenario 3 test prompt:

"Update the README.md to add a section about the new API endpoints we just added. Just document what they do and show example curl commands."

2.2 Assertion Matrix (34 items)¶

Scenario 1: Clear feature request (13 items)

Scenario 2: Vague request (10 items)

Scenario 3: Documentation change (11 items)

3. Pass Rate Comparison¶

3.1 Overall Pass Rate¶

Pass-rate gain: +59 pp (with PARTIAL: +54.5 pp)

3.2 Pass Rate by Scenario¶

Note: Scenario 2 has a smaller gap (+20 pp) because when a request is clearly vague, the baseline model also tends to ask clarifying questions naturally. The skill's added value is in the structured Gate 1 analysis, questions mapped precisely to the D1-D5 dimensions, and the standardized STOP-protocol output.

3.3 Substantive Dimensions (Core Capabilities Independent of Flow Structure)¶

To control for "flow-assertion bias", 12 additional substantive checks that do not depend on the flow itself were evaluated:

Substantive pass rate: With-Skill 12/12 (100%) vs Without-Skill 3/12 (25%), gain +75 pp (counting PARTIAL = 3.5/12 ≈ 29%, gain +71 pp).

4. Key Difference Analysis¶

4.1 Behaviors Unique to With-Skill (Completely Missing in the Baseline)¶

4.2 Behaviors the Baseline Can Do, but at Lower Quality¶

4.3 Key Findings by Scenario¶

Scenario 1 (clear feature): - With-Skill: All 4 Gates pass in Standard mode. The 580-line plan includes a file map with 10 fully labeled paths, 6 task blocks with a dependency graph, Tasks 4 and 5 marked as parallelizable, and a Reviewer Loop with B:6/6 + N:7/7 + SB:6/6. - Without-Skill: Produces a 13-section plan that includes a full Config struct, full handler logic, and full token-service code, violating Anti-Pattern #2. It has no path labels, no parallelization markers, no rollback, and no reviewer loop. The gap is large.

Scenario 2 (vague request): - With-Skill: Gate 1 clearly identifies 4 STOP triggers, asks 5 precise questions covering p99 latency, component scope, baseline vs target, constraints, and existing profiling data, explains that writing a plan would "invent the problem by inertia", and gives a 4-step continuation pipeline: rerun Gate 1 -> classify -> discovery -> planning. - Without-Skill: Asks 4 questions of comparable quality and also proactively gives a pprof usage guide and a classification of common Go performance issues. The main difference is that it has no STOP declaration and no Gate protocol, so it does not define when to "move into planning."

Scenario 3 (documentation change): - With-Skill: Runs the Gate 2 decision tree fully, shows a decision table with 6 signals all pointing to SKIP, and outputs "no formal plan needed, execute directly" plus a 5-step execution checklist and a Gate summary table, 72 lines total. - Without-Skill: Correctly recognizes that no plan is needed, but then writes about 200 lines of README content, including 10 endpoints inferred from handler filenames and full curl examples. The output is useful to the user, but the core issue is path hygiene: inferred endpoint paths such as GET /users and POST /products were written without verification, which counts as invented paths in the output.

5. Token Cost-Effectiveness Analysis¶

5.1 Skill Context Token Cost¶

Note: golden-scenarios.md (157 lines), reviewer-checklist.md (71 lines), and anti-examples.md (104 lines) are only loaded during reviewer loops or as references and are not counted in typical scenario context.

5.2 Cost-Effectiveness Calculation¶

5.3 Comparison with Other Skills¶

writing-plans is slightly less efficient than create-pr on a tokens/1% basis (~57 vs ~48), mainly because in Scenario 2 the baseline can already ask clarifying questions naturally. That narrows the Scenario 2 gap to +20 pp and lowers the overall cost-effectiveness. On the substantive dimension, however, the skill's efficiency (~41 tokens/1%) is better than all compared skills.

5.4 Token Return Curve¶

Mapping token investment to return:

~2,200 tokens (SKILL.md only):
  -> Gains: 4-Gate flow skeleton, 4 execution modes, path-label rules,
            code-block labeling, Output Contract, 10 anti-patterns
  -> Estimated coverage: ~85% of total pass-rate gain

+360 tokens (applicability-gate.md):
  -> Gains: decision tree, 7 signal types, "Looks Small But Isn't" patterns
  -> Estimated coverage: +8% gain (Gate 2 related assertions)

+560 tokens (repo-discovery-protocol.md):
  -> Gains: 5-step discovery protocol, label definitions, path-verification rules
  -> Estimated coverage: +5% gain (path-label assertions)

+220-340 tokens (plan template):
  -> Gains: scenario-specific template structure and trigger signals
  -> Estimated coverage: +2% gain (template-compliance assertions)

SKILL.md alone provides about 85% of the total value; the applicability gate plus discovery protocol add another 13%; templates contribute the final 2% at the margin.

6. Overall Score¶

6.1 Scores by Dimension¶

Notes on the dimension scores:

• Gate execution completeness: With-Skill runs the Gates systematically across all 3 scenarios: Gate 1 STOP in Scenario 2, Gate 1 + 2 with SKIP in Scenario 3, and all 4 Gates in Scenario 1. Each Gate has explicit output and decision evidence. Without-Skill has no Gate system, so the maximum reasonable score is 1.0/5.
• Plan structure quality: In Scenario 1, With-Skill produces a complete 580-line plan with a file map, 6 task blocks, a dependency graph, per-task rollback, and [interface] / [command] code blocks. Without-Skill produces a 13-section unstructured plan that includes full implementation code and lacks path labels, template sections, and a review loop, so it scores 1.5/5.
• Mode-selection accuracy: With-Skill selects the correct mode in all 3 scenarios (Standard / STOP / SKIP). Without-Skill does not declare a mode in Scenario 1, and in Scenario 3 it writes README content directly instead of a SKIP decision plus checklist, so it scores 2.0/5.
• Path verification + anti-pattern avoidance: In Scenario 1, all 10 paths in the With-Skill file map are labeled, and in Scenario 3 it does not write speculative endpoints. Without-Skill includes full implementation code in Scenario 1 (Anti-Pattern #2) and writes 10 inferred endpoint paths in Scenario 3, so it scores 1.5/5.
• Requirement-clarification quality: In Scenario 2, both versions can ask clarifying questions, and the baseline even adds a pprof usage guide, which is extra value. The main gap is that Without-Skill lacks a structured STOP declaration and a follow-up pipeline, so it scores 4.0/5.
• Structured-output compliance: With-Skill produces standardized output in all 3 scenarios, including Gate summary tables, decision-tree walk-throughs, and the Output Contract. Without-Skill has no Output Contract, so it scores 1.5/5.

6.2 Weighted Total Score¶

6.3 Comparison with Other Skills¶

writing-plans receives the second-highest overall score in this evaluation, at 9.45/10, slightly below create-pr at 9.55/10. The main reasons for the gap are:

1. Slightly smaller pass-rate delta (+54.5 pp vs +71 pp): in Scenario 2, the baseline also performs well, which reduces the overall difference.
2. Slightly weaker token efficiency (~57 tokens/1% vs ~48 tokens/1%): again driven by the small Scenario 2 gap.

What the two skills share is that PR creation and implementation planning are both areas where the baseline model lacks strong structure, so the marginal value of a dedicated skill is high.

Why it lost points:

• Assertion pass rate (9.5/10): In Scenario 2, the baseline model can naturally ask clarifying questions, so the gap is only +20 pp. If the evaluation added boundary cases such as "complex feature + partially existing paths," the difference would likely be larger.
• Token cost-effectiveness (7.5/10): golden-scenarios.md (157 lines, ~1,100 tokens) and plan-update-protocol.md (44 lines, ~310 tokens) were not loaded in typical scenarios. They are on-demand, low-frequency references rather than real waste.

7. Conclusion¶

In this evaluation, the writing-plans skill demonstrates highly consistent 4-Gate execution and precise mode-selection logic. Its substantive pass rate reaches 100% (12/12), and its overall pass rate is 98.5%, compared with 44% for the baseline, a gap of +54.5 percentage points.

Core value:

1. 4-Gate upfront flow: It blocks "start writing a plan for a vague request" (Anti-Pattern #10) at Gate 1, and blocks "run the full Standard flow for a README update" at Gate 2.
2. Four-label path-verification system: [Existing] / [New] / [Inferred] / [Speculative] makes every path in the plan traceable and removes ghost paths.
3. Semantic code-block labels: [interface] / [test-assertion] / [command] prevents implementation code from leaking into the plan (Anti-Pattern #2) and keeps the plan at interface-level precision.
4. Dynamic mode selection: SKIP for documentation changes, STOP for vague requests, and Standard for cross-package feature work. All 3 scenarios chose the correct mode.

Main risks and improvement space:

• Scenario 2 gap is narrow (+20 pp): when a request is obviously vague, the baseline also tends to ask questions. The skill's differentiated value is in the systematic STOP declaration, D1-D5 question design, and follow-up pipeline, but reviewers can easily overlook that structural value.
• C9's line-count limit is too strict: the SKIP scenario needs to show a Gate decision table, which is structured evidence. A 72-line output is reasonable, so the "<=15 lines" assertion would be better replaced with "no full plan body."
• Low usage of golden-scenarios.md: this 157-line reference was not actively loaded in any of the 3 test scenarios. SKILL.md should give clearer guidance on when to pull it into the reviewer-loop phase.