git-commit Skill Evaluation Report¶

Evaluation Framework: skill-creator Evaluation Date: 2026-03-25 Subject: git-commit

git-commit is a safety-enhanced commit workflow skill that runs repository state checks, staging analysis, secret scanning, ecosystem-aware quality gates, and Angular (Conventional Commits) message generation before executing git commit. Its three standout strengths: a mandatory 7-step workflow (Preflight → Staging → Secret Gate → Quality Gate → Compose → Commit → Report) that replaces the baseline model's habit of staging and committing directly; regex-based secret scanning with layered triage that precisely blocked a hardcoded API key in Scenario 2; and ecosystem-aware quality gates (Go/Node/Python/Java/Rust) that ensure vet/test/lint runs before every commit — steps the baseline skipped in all three scenarios.

1. Skill Overview¶

git-commit is a structured commit safety skill defining a mandatory 7-step workflow, Hard Rules, five ecosystem quality gates, secret-scanning regexes, and a scope-discovery mechanism. Its goal: every commit passes complete safety preflight, logical staging, secret scanning, quality verification, and message normalization before execution.

Core Components:

2. Test Design¶

2.1 Scenario Definitions¶

2.2 Assertion Matrix (35 items)¶

Scenario 1 — Clean Go Feature (13 items)

Scenario 2 — Python Multi-Concern + Secret (12 items)

Scenario 3 — Node.js >8 Files, Messy History (10 items)

3. Pass Rate Comparison¶

3.1 Overall Pass Rate¶

Pass rate improvement: +77 percentage points (strict) / +69pp (with PARTIAL)

3.2 Pass Rate by Scenario¶

3.3 Substantive Dimension (Capability-Focused, Structure-Independent)¶

To remove "workflow-structure bias," 15 additional checks were evaluated independently of workflow steps:

Substantive pass rate: With-Skill 15/15 (100%) vs Without-Skill 5.5/15 (37%), improvement +63pp.

4. Key Difference Analysis¶

4.1 Behaviors Unique to With-Skill (Completely Absent in Baseline)¶

4.2 Behaviors the Baseline Does, But with Lower Quality¶

4.3 Scenario-Level Key Findings¶

Scenario 1 (Clean Go feature): - With-Skill: All 7 steps completed. Full 7-item preflight; secret scan clean; go vet + go test passed; scope calc confirmed from history frequency; message feat(calc): add multiply operation (35 chars) precise and concise; post-commit report complete. - Without-Skill: Only ran git status / git diff / git log (3 steps). No go vet or go test. No secret scan. No preflight checks. Message feat(calc): add Multiply function (33 chars) — correct format but included a system-default Co-Authored-By line. No post-commit report.

Scenario 2 (Python secret): - With-Skill: After passing preflight, precisely identified 3 logical concerns. Secret scan matched both sk-[A-Za-z0-9]{20,} and api[_-]?key\s*= on src/config.py:5. Triage confirmed non-test/non-doc/non-comment → BLOCKED. Report included exact filename, line number, matched pattern names, and remediation (os.environ["API_KEY"] + key rotation). Proposed 3 split commits, each subject ≤ 50 chars (49/44/45). - Without-Skill: Caught the sk-proj- key via manual diff review (PASS), correctly blocked and suggested env var replacement. But no regex evidence chain, only mentioned file and line — no matched pattern name reported. Tended toward a single commit (two at most); did not identify the logger as an independent concern.

Scenario 3 (Node.js messy history): - With-Skill: Detected 10 files > 8 threshold, listed all files and requested confirmation. Identified 6 logical groups (config/middleware, auth+test, task+test, user+test, index wiring, README). git log showed no CC history → omit scope → use type: subject format. All 6 subject lines ≤ 50 chars (42/44/42/38/etc.). Ran npm test + npm run lint (both exit 0). - Without-Skill: Committed all 10 files as one commit — no file count threshold, no logical grouping. Message feat: add auth, users, and tasks modules with tests (51 characters, exceeds the 50-char limit). No npm test or npm run lint. Noticed the non-CC history but chose to ignore it (kept CC format — correct behavior).

5. Token Cost-Effectiveness¶

5.1 Skill Context Token Cost¶

Note: Only one ecosystem's quality gate reference is loaded per commit.

5.2 Actual Token Usage (6 Evaluation Agents)¶

With-Skill average: ~30,547 tokens, ~143s, ~31 tool calls Without-Skill average: ~26,221 tokens, ~117s, ~17 tool calls

With-Skill agents consumed on average +17% more tokens and +22% more time, spent on the additional preflight checks, secret scanning, and quality gate steps. Scenario 3's Without-Skill agent anomalously consumed more tokens (33,290 vs 30,068) — without structural guidance, it made more exploratory file reads.

5.3 Cost-Effectiveness Calculation¶

Note: "Context cost" counts only SKILL.md + reference loading; "runtime overhead" includes extra tool calls from preflight, secret scan, and quality gate execution.

5.4 Comparison with Other Skills¶

git-commit leads on context tokens per 1% improvement (~17), for three reasons: 1. SKILL.md is extremely lean (184 lines) — progressive reference loading prevents context bloat 2. Per-file quality gate design is highly efficient — only one ecosystem reference is ever loaded, adding just ~150–320 tokens per commit 3. Large pass rate delta (+77pp) — git commit is a domain where baseline models critically lack structured safety workflows

Even including runtime overhead (~73 tok/1%), git-commit still outperforms go-makefile-writer on context cost alone. The additional tool calls guided by the skill (quality gates, secret scanning) consume more tokens, but their safety output far exceeds the cost.

5.5 Token Return Curve¶

Token investment → return mapping:

~1,150 tokens (SKILL.md only):
  → Gets: 7-step workflow, Hard Rules, secret regexes, staging threshold, scope discovery
  → Estimated coverage: ~85% of pass rate improvement

+150–320 tokens (1 quality-gate reference):
  → Gets: Ecosystem-specific vet/test/lint commands and thresholds
  → Estimated coverage: +12% of pass rate improvement (Quality Gate assertions)

+0 tokens (edge cases / examples already inlined):
  → Gets: Empty commit, post-merge residuals, submodule handling
  → Estimated coverage: +3% (edge case coverage)

SKILL.md alone delivers ~85% of the value. The progressive reference loading design achieves optimal token efficiency.

6. Overall Scoring¶

6.1 Dimension Scores¶

Dimension notes:

• Preflight completeness: With-Skill ran 6 preflight checks systematically across all 3 scenarios (work tree, status, conflicts, branch, rebase, merge/cherry-pick), plus submodule awareness. Without-Skill only ran git status/git diff/git log — no conflict detection, no detached HEAD check, no rebase/merge state check.
• Secret scanning: With-Skill used 13 regex patterns + filename patterns in a dual-scan, with 4-level triage to filter false positives. In Scenario 2, precisely matched sk-[A-Za-z0-9]{20,} and api[_-]?key\s*=. Without-Skill caught the obvious sk-proj- key via manual diff review but had no tool evidence chain and reported no pattern name.
• Quality gate execution: With-Skill ran go vet + go test, pytest + ruff (deferred), and npm test + npm run lint across the 3 scenarios. Without-Skill ran zero tests or lint tools in all 3 scenarios — the largest capability gap.
• Staging logic: With-Skill identified 3 logical concerns in Scenario 2 and proposed 3 split commits; in Scenario 3 triggered the >8 file threshold, listed all files, and identified 6 logical groups. Without-Skill defaulted to a single commit in Scenario 2 and committed all 10 files together in Scenario 3.
• CC message quality: With-Skill produced subjects ≤ 50 chars across all scenarios (35/49/44/45/42/38, etc.), using git log frequency analysis for scope decisions. Without-Skill's Scenario 3 subject was 51 characters — over the limit — and no scope frequency analysis was performed.
• Structured output report: With-Skill strictly followed the 7-step workflow output order; post-commit report included hash, files, and gate status. Without-Skill only ran git status to confirm success — no structured report.

6.2 Weighted Score¶

6.3 Comparison with Other Skills¶

git-commit earns the highest overall score of the three skills, primarily because:

1. Token efficiency is significantly ahead (~17 tok/1% vs ~48 and ~63): progressive reference loading keeps SKILL.md at just 184 lines while covering 5 ecosystems
2. Largest pass rate delta (+77pp): git commit is the domain where baseline models are weakest in structured safety workflows — baseline skipped tests in every scenario
3. No weak dimensions: all 6 dimensions scored ≥ 9.0

Point deductions: - Secret scanning (9.5/10): Current regexes cover mainstream secret types but lack patterns for JWT, Twilio (SK[0-9a-fA-F]{32}), Mailgun, and other newer SaaS platforms - Token cost-effectiveness (9.0/10): Despite having the best absolute efficiency, the Edge Cases section (~100 tokens) sees low utilization in typical scenarios

7. Conclusion¶

git-commit is the skill with the largest pass rate delta (+77pp) and best token efficiency (~17 tok/1%) in this evaluation. This indicates that git commit is a domain where baseline models critically lack structured safety processes — baseline never ran any tests or lint tools across all 3 test scenarios, making the skill's marginal value exceptionally high.

Core value: 1. Quality gate: zero to one — Baseline never runs vet/test/lint; the skill guarantees "every commit passes a quality gate" 2. Regex secret scanning — 13 patterns + 4-level triage precisely blocked a hardcoded API key in Scenario 2, providing an evidence chain that manual review cannot match 3. Staging safety net — >8 file confirmation + logical split proposals prevented 10 mixed files from being committed as one blob in Scenario 3 4. Progressive reference loading — 5 ecosystem gates stored in separate files, loaded on demand; typical token cost is just ~1,300 (SKILL.md + 1 gate)

Design strengths: - SKILL.md strictly held to 184 lines (target ≤ 200) with very high information density - Quality gate per-file design achieves "one SKILL.md, five ecosystems" — a model example of progressive loading - Hard Rules first + precise thresholds (8 files, 50 chars, 3-commit scope frequency) make rules verifiable and unambiguous

Improvement suggestions: 1. Extend secret regexes to cover JWT, Twilio (SK[0-9a-fA-F]{32}), Mailgun, and other emerging platforms 2. Add guidance on creating a .commit-secret-allowlist file to reduce first-time setup friction 3. Consider moving the Edge Cases section into a reference file to further trim SKILL.md (~80 tokens saved)