tech-doc-writer Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-03-17 Evaluation subject: tech-doc-writer

tech-doc-writer is a technical-writing skill for drafting, reviewing, and improving structured engineering documents such as runbooks, troubleshooting guides, API docs, and RFC/ADR-style design docs. Its three main strengths are: document-type classification and audience analysis up front, so structure and depth match the reader’s goal; quality gates for metadata, conclusion-first writing, rollback paths, and SPA titles, which make the output more maintainable and easier to use; and review/improve workflows with scorecards, anti-examples, and structured output, so documentation feedback is concrete rather than vague.

1. Evaluation Overview¶

This evaluation reviews the tech-doc-writer skill along two dimensions: actual task performance and token cost-effectiveness. It uses 3 scenarios covering different document types and execution modes (task-document writing, troubleshooting-document writing, and document review/improvement). Each scenario was run with both with-skill and without-skill configurations, for 3 scenarios x 2 configs = 6 independent subagent runs, scored against 38 assertions.

Note: In Eval 3, with-skill was blocked by file-write permissions and only produced review-findings, with no improved-runbook. As a result, 5 assertions could not be scored. Pass rate is calculated only from scorable assertions (with-skill 31/33, without-skill 21/38).

2. Test Method¶

2.1 Scenario Design¶

2.2 Test Repository¶

/tmp/tech-doc-eval/repos/go-order-service (Go 1.24, Gin, GORM, MySQL 8.0, Redis 7, docker-compose) was used as the target repo for Eval 1 and Eval 2. Eval 3 used a manually written flawed MySQL upgrade runbook (45 lines, passing 0 scorecard items).

2.3 Execution Method¶

• With-skill runs first read SKILL.md and its referenced materials (templates.md, writing-quality-guide.md).
• Without-skill runs explored the repository and then produced documents using the model's default behavior.
• All runs were executed in parallel in independent subagents.
• Note: subagents were restricted by file-write permissions, so the actual document content was extracted from the agent transcripts.

2.4 Timing Data¶

Note: with-skill tokens and runtime were inflated in part because subagents repeatedly retried after being blocked by file-write permissions (Eval 1 with-skill used tools 29 times). In a normal production environment with working write access, the main extra overhead from with-skill would be reading SKILL.md and references (~4,000-6,000 tokens). Estimated total with-skill usage would then be about 36,000-42,000 tokens, roughly 20-30% above without-skill.

3. Assertion Pass Rate¶

3.1 Overview¶

3.2 Eval 1, Assertion-by-Assertion Comparison¶

3.3 Eval 2, Assertion-by-Assertion Comparison¶

3.4 Eval 3, Assertion-by-Assertion Comparison¶

3.5 Breakdown of 17 Failed Assertions in Without-Skill¶

4. Dimension-by-Dimension Analysis¶

4.1 Structured Metadata (YAML Frontmatter + applicable_versions)¶

This is the most stable differentiator. With-skill passed it in every eval; without-skill failed it in every eval.

With Skill (Eval 2 example):

---
title: "MySQL: Deadlocks on the orders Table Under High Concurrency"
owner: order-service-team
status: active
last_updated: 2026-03-17
applicable_versions: Go 1.24+, MySQL 8.0, GORM 1.25+
---

Without Skill (Eval 2): No metadata at all.

Practical value: Metadata allows documents to be indexed by automation, checked for staleness, and traced to ownership. applicable_versions prevents readers from applying instructions to the wrong version.

4.2 Conclusion First¶

The gap is most visible in Eval 2 (the troubleshooting doc).

With Skill opening paragraph:

Root cause conclusion: multiple transactions lock the same row or adjacent index ranges in the orders table in different orders, creating a circular wait deadlock. A typical case is concurrent execution of CreateOrder (INSERT) and CancelOrder (UPDATE), where InnoDB gap locks and record locks conflict.

Without Skill opening paragraph:

Under high concurrency, the service frequently prints the following errors... What is a deadlock... A deadlock happens when two or more transactions each hold locks the others need...

The without-skill version explains background knowledge first and only later analyzes the cause. Readers need to get through about 40% of the document before they reach the root cause. Gate 4 in the skill scorecard explicitly requires "Conclusion/core message appears in the first paragraph."

4.3 Document Type Classification and Template Alignment¶

Gate 2 in the skill (Document Type Classification) drives the with-skill runs to choose the right document template:

Analysis: The without-skill structure is not bad, but it is inconsistent. Different runs may produce different structures. The skill uses templates to make structure predictable.

4.4 Differences in Review Mode¶

In Eval 3, the review quality comparison looks like this:

Analysis: Their issue-finding ability is similar. Both covered the key defects thoroughly. The with-skill version is better in presentation structure (quantified scorecard, before/after fixes). The without-skill review reads more like a code review, with problem descriptions and impact notes, but fewer directly actionable fixes.

4.5 Preventive Measures and Monitoring Alerts¶

Eval 2 shows a clear difference here:

With Skill: | Metric | Collection Method | Alert Threshold | |------|---------|---------| | Innodb_deadlocks | Prometheus mysqld_exporter | Increase > 3 within 5 minutes | | Application-layer retry count | Code instrumentation | > 10 within 1 minute | | Slow query | slow_query_log | Single query > 1s |

Without Skill: Recommends enabling deadlock logs and running load tests, but gives no concrete alert thresholds.

The troubleshooting template in the skill requires "Prevention must include at least one monitoring item", so with-skill directly provides deployable monitoring configuration.

4.6 Code Example Quality¶

There is little difference in the quality of the Go code examples. Both produced a self-contained RunInTxWithRetry implementation with imports, error handling, and exponential backoff.

Analysis: Code quality is something the base model already does well. The skill's incremental value is the <!-- UNVERIFIED: ... --> marker (from Gate 0: Execution Integrity). It is a small but useful improvement because it prevents readers from over-trusting unverified code.

5. Token Cost-Effectiveness Analysis¶

5.1 Skill Size¶

tech-doc-writer is a multi-file skill consisting of SKILL.md, 3 reference files, and regression-test scripts.

5.2 Typical Loading Scenarios¶

The skill uses progressive loading (Load References Selectively), so actual token use depends on document type:

5.3 Quality Gain per Token¶

5.4 Cost-Effectiveness by Module¶

5.5 High-Leverage vs Low-Leverage Instructions¶

High leverage (~940 tokens in SKILL.md -> 14 assertions of delta): - Gate 2 document type classification (150 tok -> 2 assertions) - Gate 4 Quality Scorecard (250 tok -> 3 assertions) - Output Contract (200 tok -> 3 assertions) - Phase 5 Metadata (80 tok -> 3 assertions) - Conclusion First (60 tok -> 1 assertion) - SPA title rules (100 tok -> 2 assertions) - Gate 0 UNVERIFIED marker (100 tok -> indirect contribution)

Medium leverage (~550 tokens -> indirect contribution): - Anti-Examples (350 tok) -> drove the before/after repair pattern in Eval 3 - Document Maintenance (200 tok) -> produced maintenance-trigger conditions

Low leverage (~280 tokens -> 0 assertions of delta): - Degradation Strategy (200 tok) -> not tested - Language rules (80 tok) -> not tested

Reference files (~3,630 tokens -> indirect contribution): - templates.md drove structural consistency - writing-quality-guide.md provided BAD/GOOD examples for review mode - docs-as-code.md was not used in this evaluation

5.6 Token Efficiency Rating¶

5.7 Cost-Effectiveness Compared with go-makefile-writer¶

Analysis: The two skills have almost identical SKILL.md cost-effectiveness (~62-63 tok/1%), but tech-doc-writer loads a wider range of references because it covers more document types and modes. Its progressive-loading design makes the total cost for simple scenarios (task docs, ~2,900 tokens) lower than go-makefile-writer, while complex scenarios (review mode + fuller references) are higher (~5,250 tokens).

6. Boundary Analysis vs Claude Base Model¶

6.1 Capabilities the Base Model Already Has (No Skill Gain)¶

6.2 Gaps in the Base Model (Filled by the Skill)¶

6.3 Precision of the Skill Design¶

The 4 mandatory gates in the skill map cleanly to the 6 main gaps in the base model:

Key finding: Gate 1 and Gate 3 add no measurable gain in this evaluation. The base model already performs well at repo scanning and audience analysis. The largest value comes from Gate 4 (Quality Scorecard), which encodes quality checks the model does not apply on its own.

7. Overall Score¶

7.1 Scores by Dimension¶

7.2 Weighted Total Score¶

8. Strengths of the Skill Design¶

8.1 Progressive Loading¶

The Load References Selectively section clearly defines when each reference file should be loaded, avoiding unnecessary token cost. In task-doc scenarios, total usage is only ~2,900 tokens (SKILL.md + the relevant templates.md section), which is in the same range as the minimal load for go-makefile-writer (~2,490 tokens).

8.2 Serial Gate Design¶

The 4 gates run in sequence, and each has a clear STOP condition (ask the user when uncertain). This prevents work from accumulating on top of bad assumptions.

8.3 Degradation Strategy¶

The Level 1/2/3 degradation mechanism handles incomplete-information scenarios elegantly, even though those paths were not triggered in this evaluation.

8.4 Teaching Value of Anti-Examples¶

The 12 Anti-Examples cover common technical-writing mistakes and complement the Quality Scorecard. The scorecard tells the model what to check; the Anti-Examples tell it what to avoid.

8.5 Output Contract¶

The structured output report makes the writing process auditable. Readers can quickly see document type, audience, quality score, and assumptions.

9. Evaluation Materials¶