create-pr Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-03-11 Evaluation target: create-pr

create-pr is a structured PR-creation skill for main-branch pull requests. It runs branch hygiene checks, quality verification, security scanning, and PR content generation before submission, aiming for reviewable, traceable, and safely mergeable PRs. Its three main strengths are: an 8-gate mandatory preflight flow that prevents unverified changes from entering review; a three-tier confidence model (confirmed / likely / suspected) that drives draft vs ready decisions and reduces misclassification; and a fixed-section PR Body template that makes test evidence, risk notes, and uncovered items more complete and consistent.

1. Skill Overview¶

create-pr is a structured PR-creation skill that defines 8 mandatory Gates (A–H), a 3-tier confidence model, non-negotiable rules, and a PR Body template with 8 required sections. Its goal is to ensure every PR passes full preflight, quality checks, security scanning, and commit-format validation before push.

Core components:

2. Test Design¶

2.1 Scenario Definition¶

2.2 Assertion Matrix (34 items)¶

Scenario 1 — Clean Feature (13 items)

Scenario 2 — Poor Commit Hygiene (10 items)

Scenario 3 — Security-Sensitive Change (11 items)

3. Pass Rate Comparison¶

3.1 Overall Pass Rate¶

Pass-rate gain: +71 pp (with PARTIAL: +63 pp)

3.2 Pass Rate by Scenario¶

3.3 Substantive Dimensions (Core Capabilities Independent of Flow Structure)¶

To control for "flow-assertion bias", 12 substantive checks unrelated to flow structure were evaluated:

Substantive pass rate: With-Skill 12/12 (100%) vs Without-Skill 7/12 (58%), gain +42 pp.

4. Key Differences¶

4.1 With-Skill-Only Behaviors (Baseline Never Shows)¶

4.2 Behaviors Baseline Can Do but at Lower Quality¶

4.3 Scenario-Level Findings¶

Scenario 1 (clean feature): - With-Skill: All 7 Gates pass, confidence = confirmed, recommend ready. Ran full toolchain: gosec, govulncheck, golangci-lint. - Without-Skill: Only make test + go vet, no security scan. Incorrectly recommended draft (based on YAGNI, not Gate failures).

Scenario 2 (poor commits): - With-Skill: Gate B warns on branch name, Gate D detects lint failure (missing GoDoc), Gate G detects non-CC commit. Confidence = suspected, recommend draft. Provides 6-step fix plan. - Without-Skill: Identified commit message and GoDoc issues but not branch naming; did not run lint; no structured Gate verdict.

Scenario 3 (security-sensitive): - With-Skill: Gate E detects ghp_ token via rg/gosec/golangci-lint, produces detailed security report with CWE ID, severity, fix steps, token revocation steps. Explicitly blocks push/create. - Without-Skill: Found token via code review, correctly marked CRITICAL, but no tool evidence chain, no CWE reference, fix advice less specific.

5. Token Cost-Effectiveness¶

5.1 Skill Context Token Cost¶

Note: scripts/create_pr.py (1449 lines, ~10,000 tokens) is only loaded in script mode and is not part of default context.

5.2 Cost-Effectiveness¶

5.3 Comparison with Other Skills¶

create-pr has the best tokens/1% among these skills, mainly because its pass-rate delta is very large (+71 pp)—the baseline is weak in structured PR creation, so the skill’s marginal value is high.

5.4 Token Return Curve¶

Token investment vs. return:

~2,500 tokens (SKILL.md only):
  → Gains: Gate flow, Non-Negotiables, Confidence model, Command Playbook
  → Estimated coverage: ~90% of pass-rate gain

+400 tokens (pr-body-template.md):
  → Gains: 8-section PR Body template
  → Estimated coverage: +8% pass-rate gain (PR Body structure assertions)

+500 tokens (checklists):
  → Gains: Stage-specific checklists
  → Estimated coverage: +2% pass-rate gain (low marginal value)

SKILL.md alone provides ~90% of the value; reference files add the remaining 10%.

6. Overall Score¶

6.1 Dimension Scores¶

Dimension notes:

• Gate execution completeness: With-Skill ran all 7 Gates systematically in all 3 scenarios, with exact commands and output evidence. Without-Skill only ran make test (occasionally go vet), with no auth check, branch naming check, risk classification, or security scan tools.
• PR Body structure quality: With-Skill always produced 8 required sections (Problem/Context, What Changed, Why, Risk/Rollback, Test Evidence, Security Notes, Breaking Changes, Reviewer Checklist). Without-Skill produced free-form bodies, often missing Risk/Rollback, Security Notes, Breaking Changes.
• Security scan capability: With-Skill used rg regex + gosec + govulncheck triple cross-validation. In Scenario 3, all three tools independently detected the ghp_ token (including CWE-798). Without-Skill found the token via code review only, with no tool evidence chain.
• Confidence/Draft decision: With-Skill was correct in 3/3 scenarios (1: confirmed→ready; 2: suspected→draft; 3: suspected→draft). Without-Skill incorrectly recommended draft in Scenario 1 (YAGNI, not Gate results); Scenarios 2/3 recommended draft correctly but without formal reasoning.
• Commit format compliance: With-Skill Gate G validates CC format, char count, and tone. In Scenario 2 it identified 3 violations (missing type(scope):, past tense, no structured format). Without-Skill identified issues in Scenario 2 but no char count or precise format check.
• Structured output report: With-Skill strictly followed Output Contract (Gate verdict → Uncovered Risk → PR metadata → Next Actions). Without-Skill had no structured output or Gate verdict summary table.

6.2 Weighted Total¶

6.3 Comparison with Other Skills¶

create-pr has the highest overall score among these skills because:

1. Very large pass-rate delta (+71 pp): PR creation is a weak area for the baseline model
2. No weak dimensions: 5 of 6 dimensions at full score; only Token cost-effectiveness below full due to ~30% content redundancy
3. Best token cost-effectiveness (~48 tok/1%): Despite higher absolute token count (~3,400), the large pass-rate delta makes unit cost lowest

Deductions: Token cost-effectiveness (7.5/10) is the only clearly below-full dimension, mainly because: - Bundled Script section (~500 tokens) is 14% of SKILL.md but unused by agents - Merge Strategy Guidance (~200 tokens) has no value in non-Squash scenarios - Monorepo Support (~80 tokens) is useless for single-module repos

7. Conclusion¶

The create-pr skill has the largest pass-rate delta in this evaluation (+71 pp) and the best tokens/1% (~48 tokens/1%). This indicates that PR creation is an area where the baseline model lacks structured capability, so the skill’s marginal value is very high.

Core value: 1. 8-Gate mandatory flow: Ensures security scan, lint, auth, etc. are not skipped 2. Confidence model: Turns draft/ready from subjective judgment into formal reasoning 3. Multi-tool cross-validation: Gate E’s rg + gosec + govulncheck triple detection stood out in Scenario 3 4. 8-section PR Body: Standardized output gives reviewers a consistent experience

Main risk: ~30% of SKILL.md (script description, Monorepo, merge strategy) is unused in typical scenarios; token cost-effectiveness could be improved with modular trimming.