go-makefile-writer Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-03-11 Subject: go-makefile-writer

go-makefile-writer is a skill for creating or refactoring Makefiles in Go repositories, suitable for unifying build, test, lint, run, and CI entry points, and for converging existing Makefiles of varying quality with minimal effort. Its three main strengths are: automatically planning target sets and naming rules from repository structure for more stable, readable Makefiles; consistent version pinning and normative constraints for key targets like install-tools, ci, and tidy to reduce drift; and in Refactor mode, emphasis on minimal-diff and backward compatibility—fixing issues without breaking existing usage patterns.

1. Evaluation Overview¶

This evaluation assesses the go-makefile-writer skill along two dimensions: actual task performance and Token cost-effectiveness. It uses 3 Makefile generation/refactoring scenarios of increasing complexity (single-binary creation, multi-binary+Docker creation, defective Makefile refactoring). Each scenario runs with both with-skill and without-skill configurations, for 3 scenarios × 2 configurations = 6 independent subagent runs, scored against 42 assertions.

2. Test Methodology¶

2.1 Scenario Design¶

2.2 Execution¶

• Each scenario uses an independent Git repo with pre-seeded code and go.mod
• With-skill runs first read SKILL.md and its referenced materials (golden template, quality guide)
• Without-skill runs read no skill; Makefile is generated by model default behavior
• All runs execute in independent subagents in parallel

3. Assertion Pass Rate¶

3.1 Summary¶

3.2 Classification of 13 Without-Skill Failed Assertions¶

3.3 Trend: Skill Advantage Decreases with Scenario Complexity¶

This is expected: Eval 3’s user prompt explicitly listed all 6 defects, effectively embedding the skill’s knowledge in the prompt. Eval 1’s prompt was minimal and most dependent on the skill’s conventions.

4. Dimension-by-Dimension Comparison¶

4.1 Naming Convention (cmd/-path semantics)¶

This is the largest single-item delta, contributing 2 assertion failures in Eval 1.

Analysis: Without-skill naturally used per-binary naming in multi-binary scenarios (Eval 2/3), but in the single-binary scenario defaulted to generic names. The skill’s rule "Map target names to cmd/ path semantics: cmd/<name> → build-<name>" ensures consistency.

Practical value: Consistent naming enables: - No target renaming when scaling from single to multi-binary - Unified Makefile style across teams - Predictable target names in CI scripts

4.2 install-tools and Version Pinning¶

Analysis: Without-skill in Eval 2 embedded golangci-lint installation in the lint target (@latest auto-install). This works locally but in CI causes: - Non-deterministic builds (different versions at different times) - Re-installing tools on every CI run (slow)

The skill explicitly requires "Pin tool versions in install-tools for CI reproducibility".

4.3 Output Contract (Structured Report)¶

This is a skill-only differentiated output. With-skill produces a report after each run containing:

Without-skill produced brief task summaries but no structured Output Contract.

Practical value: The Output Contract enables: - Auditable Makefile changes (PR reviewers know what changed and why) - Traceable backward compatibility in Refactor mode - Documented CI validation results

4.4 ci Target Naming¶

The skill specifies "CI target: ci (fmt-check + lint + test + cover-check in one pass)". Without-skill in Eval 2 used check with content fmt-check vet test (missing cover-check), not fully aligned with the standard CI pipeline.

4.5 Impact of Golden Template¶

With-skill Makefiles closely follow the golden template structure (variables → build → run → quality → ci → version → tools → clean → phony → help), while without-skill structures varied.

Key Eval 2 difference: Without-skill used $(eval $(call build-template,...)) dynamic metaprogramming for build targets; with-skill used explicit per-binary targets per the golden template. The skill’s Anti-Patterns section explicitly flags "Overly dynamic Make metaprogramming (eval/call/define) that reduces readability when explicit targets would be clearer".

4.6 Actual Makefile Quality Comparison¶

Using Eval 2 (most complex scenario) as an example:

5. Token Cost-Effectiveness Analysis¶

5.1 Skill Size¶

go-makefile-writer is a multi-file skill (SKILL.md + references + scripts). What is loaded into context depends on which files the subagent reads.

Typical load scenarios:

5.2 Token Cost for Quality Gain¶

5.3 Token Segment Cost-Effectiveness¶

SKILL.md content split by functional module:

5.4 High-Leverage vs Low-Leverage Instructions¶

High leverage (~360 tokens SKILL.md → 12 assertion delta): - Naming convention cmd/<name> → build-<name> (100 tok → 2) - Output Contract definition (300 tok → 3) — template portion contributes most - install-tools version pinning (80 tok → 3) - ci target specification (50 tok → 2) - tidy target (30 tok → 1) - lint tool check (40 tok → 1)

Medium leverage (~310 tokens → indirect contribution): - Anti-Patterns section (250 tok) — avoided eval/call metaprogramming in Eval 2 - docker-build variable spec (60 tok → 1)

Low leverage (~350 tokens → 0 delta): - Go Version Awareness (150 tok) — not tested - Monorepo Support (200 tok) — not tested

References (~2,150–2,660 tokens → indirect contribution): - Golden templates drive overall Makefile structure consistency - Quality guide provides concrete recipe implementations

5.5 Token Efficiency Rating¶

5.6 Comparison with git-commit Skill Cost-Effectiveness¶

go-makefile-writer’s SKILL.md cost-effectiveness is close to git-commit, but references add significant token overhead. Reference value mainly shows in Makefile structure consistency and avoiding anti-patterns—quality dimensions that are hard to quantify with assertions.

6. Boundary Analysis vs Claude Base Model Capabilities¶

6.1 Base Model Capabilities (No Skill Increment)¶

6.2 Base Model Gaps (Skill Fills)¶

7. Overall Score¶

7.1 Dimension Scores¶

7.2 Weighted Total¶

8. Evaluation Artifacts¶

9. Addendum — Real-Repo Task Verification (2026-09-10)¶

The evaluation above measures a with/without-skill A/B on three constructed scenarios. It does not answer a different question: on an unfamiliar repository of a shape the skill claims to handle, does the agent actually finish the job and leave the repo working? This addendum answers that, and its most useful result is a negative one about the rest of the assertion set.

9.1 Method¶

Five genuine Go repositories were built — each compiles, each has passing tests — one per shape in SKILL.md's project-shape table:

Each ran as a real nested agent session with the skill installed and invoked by name. Grading is by execution — make is run and its exit status and artifacts are read. Nothing is graded by matching text in the Makefile; three successive versions of the grader had to be corrected for exactly that (details in §9.4).

library and monorepo were also run without the skill, same prompt minus the invocation, graded by the same assertions.

9.2 Result¶

All five shapes were handled correctly: the library got no build-*/run-* targets and no -ldflags; the cgo project got a linux target that does not set CGO_ENABLED=0; the monorepo built the program hidden in entry.go and skipped the package queue decoy; the refactor kept make unit, make vet and make compile working.

9.3 The honest part: 9 of the 10 assertions do not discriminate¶

The no-skill arm scored identically on every assertion but one. Base Claude already writes a parseable, .PHONY-declaring, self-documenting Makefile, already avoids inventing build targets for a library, already finds a program whose file is not called main.go, and already skips a package queue file named main.go. On monorepo it did so with a dynamic pattern rule that is arguably a better design than the skill's explicit per-binary targets.

One assertion separated the arms, in both scenarios that had a baseline:

Both no-skill Makefiles wrote the defect verbatim:

@out="$(gofmt -l .)"; if [ -n "$out" ]; then ... exit 1; fi

gofmt -l on unparsable source prints to stderr, prints nothing to stdout and exits 2, so the check passes. Both with-skill runs branched on the status instead — and notably, one of them wrote if ! out=$(...) rather than the template's status=$? form. The skill transmitted the rule, and the agent implemented it its own way.

So the measured value of this skill on these five repos is narrow and real: it is the difference between a Makefile that reports success on a broken tree and one that does not. Everything else in the output, base Claude produces unaided. That cost 1.4×–2.9× the tokens and 1.1×–2.3× the wall time.

9.4 Three grader defects, recorded because they nearly became findings¶

Each was caught only by asking "would a correct answer fail this?":

1. Grading the agent's own explanation. The library Makefile's comments say "there is deliberately no -ldflags here"; a substring check read that as -ldflags being present. Comments are now stripped before matching.
2. GOFLAGS=-mod=mod in the harness environment. It is rejected in workspace mode, so go list -m failed, the module list came back empty, and the monorepo run looked like a skill failure. Worth recording what the generated Makefile did with it: the empty-list guard fired and the target failed loudly with a diagnostic instead of looping zero times and reporting success — the exact accident that guard exists for, arriving by accident.
3. Assertions on target names. test-all and build-all were required by name; both arms had named them test and build with per-module targets beside them. Now graded by artifacts and by which modules a run visits.

9.5 What this still does not establish¶

• Five repos, one run each, one grader. No repetition, so run-to-run variance is unmeasured.
• Three of the five have no no-skill baseline, so their 10/10 is evidence the skill works, not evidence it was needed.
• No fixture has a build that must fail, a vendored tree, a Go version below 1.21, or a Windows toolchain.
• The assertion set does not cover tool-version pinning or Output Contract conformance — dimensions §3 and §4 measured and where the skill's advantage was largest.

9.6 Artifacts¶