readme-generator Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-03-19 Evaluation subject: readme-generator

readme-generator is a repository-aware README generation and refactoring skill for producing maintainable, evidence-backed project homepages across services, libraries, CLI tools, and monorepos. Its three main strengths are: project-type routing and template selection, so structure matches the real repository shape instead of generic documentation patterns; evidence mapping, badge detection, and no-fabrication rules, which keep each section grounded in actual files, commands, and configs; and structured output contracts plus maintenance guidance, which make the resulting README easier to review, sustain, and keep aligned with future code changes.

1. Evaluation Overview¶

This evaluation reviews the readme-generator skill along two dimensions: actual task performance and token cost-effectiveness. It uses 3 progressively more complex README generation / refactoring scenarios: creating a README from scratch for a Go service, creating one for a Go CLI tool, and refactoring a flawed README. Each scenario was run with both with-skill and without-skill configurations, for 3 scenarios x 2 configs = 6 independent subagent runs, scored against 42 assertions.

2. Test Method¶

2.1 Scenario Design¶

2.2 Test Repository Structure¶

Eval 1 repository (/tmp/readme-eval/eval-repos/go-service): - cmd/api/main.go - entrypoint (handler -> service -> repository layers) - internal/handler/user.go - 3 HTTP endpoints (GET/POST /users, GET /users/:id) - .env.example - 5 environment variables (DATABASE_URL, REDIS_URL, JWT_SECRET, LOG_LEVEL, PORT) - .github/workflows/ci.yml - GitHub Actions (runs make ci, Go 1.23) - Makefile - 9 targets, COVER_MIN=80, golangci-lint@v1.62.2 - LICENSE - MIT; Go 1.23; module github.com/acme/user-service

Eval 2 repository (/tmp/readme-eval/eval-repos/go-cli): - cmd/root/root.go - Cobra root + 2 global flags (--output/-o, --format/-f) - cmd/generate/generate.go, cmd/validate/validate.go - 2 subcommands - Makefile - 4 targets (build-schema-gen, test, lint, install) - .github/workflows/ci.yml, LICENSE (Apache 2.0), CONTRIBUTING.md - Go 1.22, no .env.example, no sample output files

Eval 3 repository (/tmp/readme-eval/eval-repos/refactor-stale) - preloaded with a flawed README: - Fake badges: Travis CI, Codecov, npm Downloads (the repo actually uses GitHub Actions) - Wrong config section: DB_HOST, DB_PORT, etc. (.env.example actually uses 7 variables such as POSTGRES_DSN, REDIS_ADDR) - Outdated command: go run main.go (the Makefile has make run-server) - Internal labels: the Testing table contains ✅ Verified / ⚠️ Not verified - Actual repo content: .env.example (7 variables), Makefile (9 targets), CONTRIBUTING.md, SECURITY.md, Go 1.24

2.3 Execution Method¶

• Each scenario used an independent Git repository preloaded with code, go.mod, Makefile, and related files.
• With-skill runs first read SKILL.md and followed the skill workflow to generate or refactor the README.
• Without-skill runs did not read any skill and completed the same task using the model's default behavior.
• All 6 runs were executed in parallel.

3. Assertion Pass Rate¶

3.1 Overview¶

3.2 Breakdown of the 16 Failed Assertions Without the Skill¶

3.3 Trend: the Skill Advantage Grows with Scenario Complexity¶

Eval 3 shows the largest advantage because refactoring requires not only fixing known problems, but also proactively discovering missing sections such as community files and maintenance notes. This kind of "scan and fill the gaps" behavior is built into the skill workflow, while without-skill runs tend to stop after fixing the obvious problems.

4. Dimension-by-Dimension Comparison¶

4.1 Output Contract and Structured Reporting¶

This is a skill-only differentiator: 3/3 scenarios produced it with the skill, compared with 0/3 without it.

Practical value: - Reviewers can verify which file supports each section during PR review. - sections_omitted explains why a section was skipped, instead of leaving "why is section X missing?" unanswered. - The layered scorecard (Critical / Standard / Hygiene) helps reviewers quickly locate quality issues.

4.2 Documentation Maintenance Notes¶

This comes from Hygiene Tier H1 in the skill. It passed in 3/3 scenarios with the skill and 0/3 without it.

Example from the with-skill Eval 1 output:

Practical value: this directly addresses the maintenance pain point where the README gradually drifts away from the codebase, because contributors can see exactly which README sections must be updated when the code changes.

4.3 CLI End-to-End Examples and No-Fabrication¶

The skill's End-to-End Example Rule requires CLI tools to provide a complete "input command -> output description" example, and it explicitly forbids inventing JSON / YAML output bodies when there is no evidence.

With skill (Eval 2):

schema-gen generate --format json --output ./schemas ./internal/models
# -> writes schema file(s) to ./schemas/

schema-gen validate ./schemas/models.json
# -> prints validation result to stdout

Without skill (Eval 2): it only showed command examples, without the input-to-output description. The Examples subsection under Usage showed command variants, but readers could not tell what output to expect.

4.4 Defense Against Fabricated Content¶

This is the most important failure in the without-skill runs.

In Eval 3, while fixing existing fabricated content such as fake Travis CI badges and wrong DB config, the without-skill run introduced new fabricated content:

## Installation
docker pull acme/notification-svc:latest

The skill's Evidence Completeness Gate explicitly requires "base every statement on repository evidence", and no new fabrication appeared in any of the 3 with-skill scenarios.

4.5 Badge Strategy¶

The skill's Badge Detection Gate requires scanning in the order CI -> Coverage -> Language version -> License. As a result, the three-badge combination (CI + Go + License) was produced consistently in all three scenarios. Without the skill, the model only added the CI badge proactively. The Go-version and License badges need explicit rules to appear consistently.

4.6 ToC Navigation Quality (CLI Scenario)¶

The with-skill Eval 2 ToC:

- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Commands & Flags](#commands--flags)
- [End-to-End Example](#end-to-end-example)
- [Project Structure](#project-structure)
- [Development Commands](#development-commands)
- [Contributing](#contributing)
- [License](#license)
- [Documentation Maintenance](#documentation-maintenance)

4.7 Boundary with Claude's Base Model¶

Capabilities the Base Model Already Has (No Skill Gain)¶

Capability Gaps in the Base Model (Filled by the Skill)¶

5. Token Cost-Effectiveness Analysis¶

5.1 Skill Size¶

readme-generator is a multi-file skill. SKILL.md contains the core rules, and references are loaded on demand.

Typical loading scenarios (following the "Load References Selectively" rule):

5.2 Quality Gains per Token¶

5.3 Cost-Effectiveness by Token Segment¶

Breaking SKILL.md into functional modules:

5.4 High-Leverage vs Low-Leverage Instructions¶

High leverage (~1,620 tokens -> directly contributes 11+ assertion deltas): - Documentation Maintenance (200 tok -> 3 assertions) - Evidence Mapping (150 tok -> 3 assertions) - Output Contract + Scorecard (600 tok -> 3 assertions) - End-to-End Example + no-fabrication (220 tok -> 1 assertion + defensive value) - Badge Detection (250 tok -> 2 assertions) - Command Verifiability Gate (250 tok -> 1 assertion + defensive value)

Medium leverage (~750 tokens -> indirect contribution): - README Navigation Rule / ToC (200 tok -> 1 assertion) - Anti-Example 1 (200 tok -> defensive guarantee) - Structure Policy (350 tok -> section completeness)

Low leverage (~550 tokens -> 0 direct deltas in untested scenarios): - Chinese / Bilingual Guidelines (bilingual-guidelines.md, ~271 tok) - on demand, not triggered - Monorepo Rules (monorepo-rules.md, ~421 tok) - on demand, not triggered

Reference materials (~2,500-5,200 tokens depending on scenario): - golden-*.md provides README structure templates (indirectly improves section order and completeness) - templates.md provides the full skeleton (indirectly improves consistency in project-type routing) - discover_readme_needs.sh provides deterministic scanning (indirectly improves evidence completeness)

5.5 Token Efficiency Rating¶

5.6 Cost-Effectiveness Compared with go-makefile-writer¶

The readme-generator SKILL.md is about 2.4x the size of go-makefile-writer, and its token cost per 1% improvement is about 2.0x higher. Given that readme-generator has to cover 5 project-type routes, multilingual support, both refactor and generation modes, and a much more complex evidence-driven constraint system than Makefile generation, this gap is a reasonable reflection of task complexity rather than poor efficiency.

6. Overall Score¶

6.1 Dimension Scores¶

6.2 Weighted Total Score¶

7. Improvement Suggestions¶

7.1 [P1] Minimum Coverage Constraint for Project Structure¶

Issue: in the with-skill README for Eval 3, the Project Structure section had only one line:

cmd/server/     # server entry point

It omitted directories such as internal/api/, internal/db/, and pkg/cache/, even though these were clearly evidenced by the import paths in cmd/server/main.go.

Suggestion: in Generation Workflow Step 1 (Discover), add a rule to scan the entrypoint's import paths and use them to supplement internal/ and pkg/ directories. Also enforce a minimum threshold such as "Project Structure must list at least 3 meaningful directories."

7.2 [P2] Clarify Priority Between License Section and License Badge¶

Issue: under Community and Governance Files, SKILL.md says "LICENSE -> Add License section or badge", but the priority is unclear, which leads to inconsistent output across scenarios (sometimes only a badge, sometimes only a section).

Suggestion: define an explicit priority rule: - README > 80 lines: a License badge is enough; no separate License section required - README <= 80 lines or public-facing repository: keep both the badge and a dedicated License section

7.3 [P3] Add More Evaluation Scenarios¶

8. Evaluation Materials¶