update-doc Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-03-19 Evaluation subject: update-doc

Scope note (added 2026-07-31). The scores below were measured against the version of update-doc current on 2026-03-19. That version resolved diff scope from git diff --name-only alone, which reports only unstaged edits to tracked files; staged changes and newly created files were outside the gate entirely. The "Project-type routing and diff scope 10/10" figure in §7.2 therefore reflects routing behavior, not a verified diff-scope gate. The skill has since replaced that gate with scripts/discover_doc_scope.sh, which reconciles four change sources. These numbers have not been re-measured against the current version.

A gated harness now exists (scripts/run_live_eval.sh) and its grader is proven to discriminate a grounded update from a fabricated one, but no model has been run against the current version. The harness installs the skill under .claude/skills/ and invokes /update-doc rather than pasting SKILL.md into a prompt, because the pasted form measures a different artifact.

update-doc is a documentation-synchronization skill for patching or rebuilding repository docs after code changes, including README files, codemaps, and related engineering documentation. Its three main strengths are: project-type routing plus lightweight/full output modes, which keep updates scoped to the actual repository shape and the size of the change; evidence-backed diffs, scorecards, and codemap contracts, which make documentation updates traceable instead of ad hoc; and CI drift guardrails plus maintenance guidance, which reduce the chance that documentation falls behind the code again after future changes.

1. Evaluation Overview¶

This evaluation reviews the update-doc skill along two dimensions: actual task performance and token cost-effectiveness. It uses 3 progressively more complex documentation-update scenarios: a lightweight README patch for a CLI tool, a full README update for a backend service, and codemap generation plus README refactoring for a monorepo. Each scenario was run with both with-skill and without-skill configurations, for 3 scenarios x 2 configs = 6 independent subagent runs, scored against 42 assertions.

2. Test Method¶

2.1 Scenario Design¶

2.2 Test Repository Details¶

Eval 1: go-file-converter - cmd/convert/main.go: flag parsing (--format default "json", --output-dir default ".", --verbose) - Existing README.md: missing --output-dir, outdated default for --format (documented as csv) - Focus: patch only the 2 mismatches without rewriting the whole file

Eval 2: go-notification-service - cmd/api/main.go (API server) + cmd/worker/main.go (new Worker mode) - New environment variables: WORKER_CONCURRENCY (default 5), QUEUE_URL (required) - Makefile with 9 targets, docker-compose.yml with 4 services - Existing README.md covers only API mode

Eval 3: platform-monorepo - services/auth/ (Go, port changed from 8080 to 8443 TLS), services/billing/ (Go, new Stripe integration) - packages/ui-kit/ (TS), packages/api-client/ (TS, new AuthClient + BillingClient) - .github/workflows/ci.yml includes markdownlint - Existing README.md is missing billing and api-client, and the auth port is outdated

2.3 Execution Method¶

• Each scenario used an independent Git repository with code and go.mod preloaded.
• With-skill runs first read SKILL.md and its referenced materials (update-doc.md, project-routing.md, ci-drift.md).
• Without-skill runs did not read any skill and updated the docs using the model's default behavior.
• All runs were executed in parallel in independent subagents.

3. Assertion Pass Rate¶

3.1 Overview¶

3.2 Breakdown of the 24 Failed Assertions Without the Skill¶

3.3 Layered Failure Analysis¶

The 24 failures can be grouped into two layers based on whether the base model could reasonably do them on its own:

Interpretation: the core value of the skill is that it injects 17 pieces of methodology discipline, while anti-patterns and the diff-scope gate provide 7 quality guardrails.

3.4 Trend: the Skill Has the Largest Advantage in the Most Complex Scenario¶

Unlike go-makefile-writer, where the largest advantage appeared in the simplest scenario, update-doc shows its biggest advantage in the most complex monorepo scenario. The reason is that Eval 3 requires skill-specific capabilities such as the Codemap Output Contract, multi-module routing, and CI drift detection, which the baseline model is very unlikely to produce on its own.

4. Dimension-by-Dimension Comparison¶

4.1 Project-Type Routing¶

This is the skill's foundational capability because it directly determines whether all later decisions are correct.

Analysis: in Eval 2, without-skill happened to produce a reasonable service README structure, but without explicit routing the behavior is unpredictable. In Eval 3, the same model chose to dump a directory tree instead of creating a module index table. The skill's routing mechanism ensures consistent behavior across scenarios.

4.2 Output Mode Selection¶

Analysis: the over-rewrite in Eval 1 without the skill, which added sections like "How It Works" and "Error Handling", is exactly the behavior the skill's Lightweight mode and Diff Scope Gate are designed to prevent. In Eval 2 and Eval 3, the concise without-skill responses were not verbose, but they missed structured outputs such as the Evidence Map, Scorecard, and Open Gaps.

4.3 Evidence-Backed Accuracy¶

Both configurations performed well on factual accuracy:

Key difference: the skill does not win on accuracy; it wins on auditability. The Evidence Map makes every documentation claim traceable to specific code lines, which supports PR review and later maintenance.

4.4 Anti-Pattern Avoidance¶

4.5 Codemap Quality (Eval 3 Focus)¶

4.6 CI Drift Guardrails (Eval 2/3 Focus)¶

5. Token Cost-Effectiveness Analysis¶

5.1 Skill Size¶

5.2 Breakdown of SKILL.md by Functional Module¶

5.3 High-Leverage vs Low-Leverage Instructions¶

High leverage (~850 tokens -> 17 delta assertions, ~50 tok/delta):

Medium leverage (~750 tokens -> 7 delta assertions, ~107 tok/delta):

Low leverage (~1,000 tokens -> 0 delta assertions):

5.4 Token Efficiency Rating¶

5.5 Cross-Skill Cost-Effectiveness Comparison¶

update-doc has the highest token cost-effectiveness of the three skills, for three reasons:

1. A very large pass-rate delta (+57.1%): the skill injects 17 methodological capabilities that the baseline model does not have at all.
2. Compact reference materials (~540 tokens): much smaller than the ~2,600 tokens of reference material in go-makefile-writer.
3. A reasonable share of high-leverage modules: 40% of the SKILL.md directly drives 71% of the assertion delta.

6. Boundary Analysis Against Claude's Base Model¶

6.1 Capabilities the Base Model Already Has (No Skill Gain)¶

6.2 Capability Gaps in the Base Model (Filled by the Skill)¶

6.3 Boundary Summary¶

The base model is already strong at fact extraction and basic documentation writing. It got environment variables, ports, and routes correct in all cases. But it completely lacks methodological discipline. The skill's value is not that it "makes the model smarter"; it gives the model a repeatable workflow: project classification -> diff scope -> output mode -> structured report -> CI maintenance guidance.

7. Overall Score¶

7.1 Dimension Scores¶

7.2 Weighted Total Score¶

8. Improvement Suggestions¶

8.1 [P1] Trim Low-Leverage Modules¶

Roughly 1,000 tokens (~48% of SKILL.md) produced no incremental gain in this evaluation:

This would likely remove ~400-500 tokens without affecting the high-leverage assertion gains, improving SKILL.md efficiency from 37 tok/1% to roughly 28 tok/1%.

8.2 [P1] Strengthen Monorepo Codemap Guidance¶

The Codemap Output Contract in Eval 3 was one of the largest gaps between with-skill and without-skill. Suggested changes:

• Add a short codemap INDEX.md template to references/codemap-template.md
• State explicitly that INDEX.md must include: overview, child-codemap link table, and cross-module concerns
• For each project type (Service / Monorepo), specify which codemap files are required

8.3 [P2] Clearer Conditional Loading for Reference Files¶

The current 3 reference files (~540 tokens total) are still less than one quarter of SKILL.md when all are read together. Their loading rules can still be stated more clearly:

• Simple patch (Eval 1-like): SKILL.md only, no references needed (~2,100 tokens)
• Full service update (Eval 2-like): SKILL.md + ci-drift.md (~2,250 tokens)
• Monorepo codemap (Eval 3-like): SKILL.md + all references (~2,640 tokens)

8.4 [P2] Add More Evaluation Scenarios¶

Current skill features that were not covered:

8.5 [P3] Consider Moving the Quality Scorecard to a Reference File¶

The 12-item Scorecard (~250 tokens) is always loaded, but it is only used in Full Output Mode. It could be moved to references/scorecard.md, with SKILL.md keeping a short pointer such as "use the 12-item checklist in references/scorecard.md."

9. Evaluation Materials¶