git-commit Skill Evaluation Report¶

Evaluation Framework: skill-creator Evaluation Date: 2026-03-25 Regression Addendum: 2026-04-10 Subject: git-commit

git-commit is a safety-enhanced commit workflow skill that runs repository state checks, staging analysis, secret scanning, ecosystem-aware quality gates, and Angular (Conventional Commits) message generation before executing git commit. Its three standout strengths: a mandatory 7-step workflow (Preflight → Staging → Secret Gate → Quality Gate → Compose → Commit → Report) that replaces the baseline model's habit of staging and committing directly; regex-based secret scanning with layered triage that precisely blocked a hardcoded API key in Scenario 2; and ecosystem-aware quality gates (Go/Node/Python/Java/Rust) that ensure vet/test/lint runs before every commit — steps the baseline skipped in all three scenarios.

1. Skill Overview¶

git-commit is a structured commit safety skill defining a mandatory 7-step workflow, Hard Rules, five ecosystem quality gates, secret-scanning regexes, and a scope-discovery mechanism. Its goal: every commit passes complete safety preflight, logical staging, secret scanning, quality verification, and message normalization before execution. The April 2026 addendum extends regression coverage to young-repo scope bootstrap, executable subject guards, and timeout overrides.

Core Components:

2. Test Design¶

2.1 Scenario Definitions¶

Regression addendum: the original 3 evaluated scenarios remain unchanged for comparability. The current repository version adds 7 deterministic golden fixtures that specifically cover commit-message generation failure modes that the original matrix did not isolate well: new-repo scope bootstrap, mixed-root scope omission, executable 50-character guarding, and timeout override precedence. Regression addendum (September 2026): the scenario results above are the record of Round-2 addendum (September 2026): a second review round found that round 1 had Round-3 addendum (September 2026): round 3 found a false-block defect that round 2 Round-4 addendum (September 2026): the standing objection was evidence scope, not Round-5 addendum (September 2026): three defects, two of them regressions from Round-6 addendum (September 2026): round 5's claim that the cancellation defect was fixed was wrong. A retest showed the executor still returning while the gate ran on: SIGHUP gave exit 129 in 0s with the gate alive, and a TERM-ignoring gate survived a cancelled executor by 12s and then continued. Cause: the perl watcher's child called setpgrp(0, 0) — measured perl pgid 60922 vs gate pgid 60923 — so the supervisor added in round 5 was signalling and escalating on a group that held only perl; and no $SIG{HUP} was set at all. Fixed by removing the child's setpgrp (isolation already comes from set -m one level up), so one addressable group holds perl, the gate and grandchildren. Grandchild cleanup on expiry and all exit codes (124 timeout, passthrough, 143 early signal) re-verified, on macOS/perl and on Debian/GNU-timeout and Alpine/BusyBox.

Why round 5 believed itself is the part worth keeping: its test drove a timeout shim that left its child in the same process group, so it never reproduced the topology that breaks. A stand-in that is tidier than the real thing confirms fixes against a system that does not exist. The tests now drive the real watcher (SIGHUP, TERM-ignoring gate) and assert the invariant separately — the gate's group must be addressable and be neither the supervisor's nor the caller's. Restoring the exact pre-fix watcher is now caught.

Two mutations survive and are recorded, not chased: re-adding the watcher's old pid-targeted handlers is equivalent once the group is shared; and deleting the verification behind the cancellation message cannot be caught behaviourally, because its branch needs a process that survives SIGKILL — it is pinned structurally and labelled as structural. round 4, one of them invalidating round 4's own eval evidence. (1) if docker run … | sed made the matrix test sed, so a container exiting 42 was summarised as "all exercised implementations pass" with exit 0; a second pytest … | tail hid failing suites in full mode. Both now capture status before formatting. (2) Normalising the timeout code replaced exec with a supervisor that forwarded nothing: SIGTERM to the executor returned 143 while the gate ran on and wrote its side effect 2s later. Cancellation now goes to the tool's own process group, waits for it to empty, escalates to KILL, then exits. (3) The eval's skill arm had been running with no skill loaded — SKILL_DIR came from $0, so copying the runner voided it while keeping the skill label. Four cells reported earlier as wins were void; they are deleted, not annotated.

The eval now refuses to start without SKILL.md, stamps skill path/lines/hash into every transcript so validity is auditable per cell, and takes EVAL_SKILL_DIR/EVAL_SCEN_DIR. Standing eval summary: no measured decision benefit. Where both arms have valid data (scenarios 01–06) they agree, so those cells are evidence about the scenarios, not the skill. Scenario 10 is the one known discriminator, and only after its grader was tightened from length alone to format and length.

Six fault-injection mutations now cover these fixes and all are killed. Two survived first and are worth recording: an assertion that cancellation produces "no side effect at all" passed by luck (graceful TERM exists so a gate may clean up), and an assertion that the group kill makes cancellation prompt failed on correct code — what the group form actually buys is that the escalation reaches the whole tree. Suite: 191 pass / 5 skip on macOS, Ubuntu 24.04 and Alpine; 160 pass / 36 skip on interval-less Debian 12. rules, so this round produced evidence — and the evidence changed the design twice. Five environments were exercised with the shipped scripts via ephemeral containers: macOS 15.6 (BSD awk), Debian 12 (mawk 1.3.4 20200120), Debian 12 + gawk 5.2.1, Ubuntu 24.04 (mawk 1.3.4 20240123) and Alpine 3.24 (BusyBox awk, no perl). Two facts only a real matrix could surface: Debian 12's default awk has no regex interval support, so twelve credential patterns match nothing there and the capability probe must refuse rather than report clean; and BusyBox timeout does not implement the 124 convention at all (143 plain, 137 with -k), making the previous "124 or 137" contract wrong on Alpine. The enforcer now normalises on elapsed time to a single code, so the contract became shorter by becoming correct. A real gitleaks was exercised for the first time: it confirmed the exit-code design from the binary's own --help, showed v8.18.4 has no git subcommand (correctly failed closed), and invalidated one of this repository's own fixtures — gitleaks allowlists AKIAIOSFODNN7EXAMPLE, so tests needing the real scanner to fire must use a non-allowlisted value.

The suite itself was wrong about its applicability: on interval-less Debian it produced 31 failures while the script behaved correctly. Tests now gate on the capability and skip with a reason naming the awk and the fix; installing gawk lifts the gate and all 172 run.

scripts/eval/ adds decision-branch probes — the one thing no deterministic suite reaches. Its README states the limits plainly: it grades a stated decision in plan mode, is non-deterministic, reports per repetition instead of averaging, carries a no-skill control arm, and treats a missing answer as INCOMPLETE rather than as a score. Still unverified: Windows/WSL, original-awk, gitleaks on macOS, and execution-fidelity (as opposed to branch selection) under an agent. The scenario results earlier in this report predate all four rounds and were not re-run. had introduced — pipefail plus an early-exiting context extractor plus a pipe writer produce SIGPIPE (exit 141) on any blob past the pipe buffer, so a completed context render was reported as a scanner failure. Clean at 3 lines, exit 2 at 20k; every small fixture missed it, including the ones written for the pipefail fix. Also fixed: SKILL.md's own §7 report command escaped a deleted CJK path ("\346\272\220…"), which the path-escaping fix had covered in all three scripts and missed in the documented commands.

Decision-probe result (2026-09-07, 1 repetition): skill arm 6/6 — notably STOP when the secret gate exits 2, and ASK for a real-looking key under testdata/. The control arm covers only scenarios 01-02, which the base model also gets right; 04-06 therefore have no control data and their wins are unattributed, and one repetition shows reachability, not reliability.

Verification scope was extended and, where it could not be, made explicit: scripts/run_portability_matrix.sh re-runs the suite per available awk and exits non-zero if nothing was exercised (an empty matrix is not a green one); a real gitleaks contract test runs where the binary exists and is a visible skip where it does not (run_regression.sh -rs); and a whole-workflow test executes §1-§7 in order and asserts a real commit lands. That last one is what caught the §7 defect.

Honest limits of the current evidence. This host has one awk (BSD 20200816, intervals supported), no GNU coreutils timeout, and no gitleaks — so the GNU --kill-after branch is proven by argv assertion against a shim rather than the real binary, the interval-refusal path is proven by shim rather than a real interval-less awk, and 5 of 6 awk rows are UNVERIFIED. The runner reports all of this rather than implying coverage. The scenario results above still predate every round and were not re-run. fixed the reported instances rather than the underlying classes. Four further defects were reproduced and fixed: paths read in git's escaped display form (core.quotePath), so CJK-named sources selected no gate and *.pem stopped matching; stage-failure checking that covered only the git read, so a broken classifier still exited 0 with no output; a redaction rule that let a long token preceding a sensitive key print in full on the finding line; and a timeout that silently degraded to a tool unable to force-kill. Verification moved from "did the new tests fail against the old code" to mutation testing — each fixed defect is reintroduced one at a time and the suite must catch it. That found one fix (pipefail in the secret scanner) whose test passed either way; the assertion was tightened to require every failed stage to report itself. The scenario results above still predate all of this and were not re-run. the original A/B evaluation and are left unchanged. Those scenarios were not re-run after the September 2026 hardening round, so no score in this report reflects it. That round acted on an external review which reproduced four script defects — deletions excluded from ecosystem detection, git read failures exiting 0 with empty output (fail-open) in both the secret scanner and the ecosystem detector, JSON/uppercase keyed secrets neither detected nor masked in context lines, and a GNU timeout branch with no --kill-after — plus two test-quality defects: a SKILL.md line floor that penalised concision, and golden fixtures whose expected scope was produced by the test file's own Python re-implementation of the rules. Scope resolution now lives in scripts/resolve-scope.sh and the golden fixtures drive that script against real git repositories; the file inventory in §1 predates it. 35 of the current 145 tests fail against the pre-hardening scripts, which is the evidence that they pin the defects rather than merely raising the count.

2.2 Assertion Matrix (35 items)¶

Scenario 1 — Clean Go Feature (13 items)

Scenario 2 — Python Multi-Concern + Secret (12 items)

Scenario 3 — Node.js >8 Files, Messy History (10 items)

3. Pass Rate Comparison¶

3.1 Overall Pass Rate¶

Pass rate improvement: +77 percentage points (strict) / +69pp (with PARTIAL)

3.2 Pass Rate by Scenario¶

3.3 Substantive Dimension (Capability-Focused, Structure-Independent)¶

To remove "workflow-structure bias," 15 additional checks were evaluated independently of workflow steps:

Substantive pass rate: With-Skill 15/15 (100%) vs Without-Skill 5.5/15 (37%), improvement +63pp.

4. Key Difference Analysis¶

4.1 Behaviors Unique to With-Skill (Completely Absent in Baseline)¶

4.2 Behaviors the Baseline Does, But with Lower Quality¶

4.3 Scenario-Level Key Findings¶

Scenario 1 (Clean Go feature): - With-Skill: All 7 steps completed. Full 7-item preflight; secret scan clean; go vet + go test passed; scope calc confirmed from history frequency; message feat(calc): add multiply operation (35 chars) precise and concise; post-commit report complete. - Without-Skill: Only ran git status / git diff / git log (3 steps). No go vet or go test. No secret scan. No preflight checks. Message feat(calc): add Multiply function (33 chars) — correct format but included a system-default Co-Authored-By line. No post-commit report.

Scenario 2 (Python secret): - With-Skill: After passing preflight, precisely identified 3 logical concerns. Secret scan matched both sk-[A-Za-z0-9]{20,} and api[_-]?key\s*= on src/config.py:5. Triage confirmed non-test/non-doc/non-comment → BLOCKED. Report included exact filename, line number, matched pattern names, and remediation (os.environ["API_KEY"] + key rotation). Proposed 3 split commits, each subject ≤ 50 chars (49/44/45). - Without-Skill: Caught the sk-proj- key via manual diff review (PASS), correctly blocked and suggested env var replacement. But no regex evidence chain, only mentioned file and line — no matched pattern name reported. Tended toward a single commit (two at most); did not identify the logger as an independent concern.

Scenario 3 (Node.js messy history): - With-Skill: Detected 10 files > 8 threshold, listed all files and requested confirmation. Identified 6 logical groups (config/middleware, auth+test, task+test, user+test, index wiring, README). git log showed no CC history → omit scope → use type: subject format. All 6 subject lines ≤ 50 chars (42/44/42/38/etc.). Ran npm test + npm run lint (both exit 0). - Without-Skill: Committed all 10 files as one commit — no file count threshold, no logical grouping. Message feat: add auth, users, and tasks modules with tests (51 characters, exceeds the 50-char limit). No npm test or npm run lint. Noticed the non-CC history but chose to ignore it (kept CC format — correct behavior).

5. Token Cost-Effectiveness¶

5.1 Skill Context Token Cost¶

Note: Only one ecosystem's quality gate reference is loaded per commit.

5.2 Actual Token Usage (6 Evaluation Agents)¶

With-Skill average: ~30,547 tokens, ~143s, ~31 tool calls Without-Skill average: ~26,221 tokens, ~117s, ~17 tool calls

With-Skill agents consumed on average +17% more tokens and +22% more time, spent on the additional preflight checks, secret scanning, and quality gate steps. Scenario 3's Without-Skill agent anomalously consumed more tokens (33,290 vs 30,068) — without structural guidance, it made more exploratory file reads.

5.3 Cost-Effectiveness Calculation¶

Note: "Context cost" counts only SKILL.md + reference loading; "runtime overhead" includes extra tool calls from preflight, secret scan, and quality gate execution.

5.4 Comparison with Other Skills¶

git-commit leads on context tokens per 1% improvement (~17), for three reasons: 1. SKILL.md is extremely lean (169 lines) — progressive reference loading prevents context bloat while leaving room for executable guardrails 2. Per-file quality gate design is highly efficient — only one ecosystem reference is ever loaded, adding just ~150–320 tokens per commit 3. Large pass rate delta (+77pp) — git commit is a domain where baseline models critically lack structured safety workflows

Even including runtime overhead (~73 tok/1%), git-commit still outperforms go-makefile-writer on context cost alone. The additional tool calls guided by the skill (quality gates, secret scanning) consume more tokens, but their safety output far exceeds the cost.

5.5 Token Return Curve¶

Token investment → return mapping:

~1,150 tokens (SKILL.md only):
  → Gets: 7-step workflow, Hard Rules, secret regexes, staging threshold, scope discovery
  → Estimated coverage: ~85% of pass rate improvement

+150–320 tokens (1 quality-gate reference):
  → Gets: Ecosystem-specific vet/test/lint commands and thresholds
  → Estimated coverage: +12% of pass rate improvement (Quality Gate assertions)

+0 tokens (edge cases / examples already inlined):
  → Gets: Empty commit, post-merge residuals, submodule handling
  → Estimated coverage: +3% (edge case coverage)

SKILL.md alone delivers ~85% of the value. The progressive reference loading design achieves optimal token efficiency.

6. Overall Scoring¶

6.1 Dimension Scores¶

Dimension notes:

• Preflight completeness: With-Skill ran 6 preflight checks systematically across all 3 scenarios (work tree, status, conflicts, branch, rebase, merge/cherry-pick), plus submodule awareness. Without-Skill only ran git status/git diff/git log — no conflict detection, no detached HEAD check, no rebase/merge state check.
• Secret scanning: With-Skill used 13 regex patterns + filename patterns in a dual-scan, with 4-level triage to filter false positives. In Scenario 2, precisely matched sk-[A-Za-z0-9]{20,} and api[_-]?key\s*=. Without-Skill caught the obvious sk-proj- key via manual diff review but had no tool evidence chain and reported no pattern name.
• Quality gate execution: With-Skill ran go vet + go test, pytest + ruff (deferred), and npm test + npm run lint across the 3 scenarios. Without-Skill ran zero tests or lint tools in all 3 scenarios — the largest capability gap.
• Staging logic: With-Skill identified 3 logical concerns in Scenario 2 and proposed 3 split commits; in Scenario 3 triggered the >8 file threshold, listed all files, and identified 6 logical groups. Without-Skill defaulted to a single commit in Scenario 2 and committed all 10 files together in Scenario 3.
• CC message quality: With-Skill produced subjects ≤ 50 chars across all scenarios (35/49/44/45/42/38, etc.), using git log frequency analysis for scope decisions. Without-Skill's Scenario 3 subject was 51 characters — over the limit — and no scope frequency analysis was performed.
• Structured output report: With-Skill strictly followed the 7-step workflow output order; post-commit report included hash, files, and gate status. Without-Skill only ran git status to confirm success — no structured report.

6.2 Weighted Score¶

6.3 Comparison with Other Skills¶

git-commit earns the highest overall score of the three skills, primarily because:

1. Token efficiency is significantly ahead (~17 tok/1% vs ~48 and ~63): progressive reference loading keeps SKILL.md at just 169 lines while covering 5 ecosystems
2. Largest pass rate delta (+77pp): git commit is the domain where baseline models are weakest in structured safety workflows — baseline skipped tests in every scenario
3. No weak dimensions: all 6 dimensions scored ≥ 9.0

Point deductions: - Secret scanning (9.5/10): Current regexes cover mainstream secret types but lack patterns for JWT, Twilio (SK[0-9a-fA-F]{32}), Mailgun, and other newer SaaS platforms - Token cost-effectiveness (9.0/10): Despite having the best absolute efficiency, the Edge Cases section (~100 tokens) sees low utilization in typical scenarios

7. Conclusion¶

git-commit is the skill with the largest pass rate delta (+77pp) and best token efficiency (~17 tok/1%) in this evaluation. This indicates that git commit is a domain where baseline models critically lack structured safety processes — baseline never ran any tests or lint tools across all 3 test scenarios, making the skill's marginal value exceptionally high.

Core value: 1. Quality gate: zero to one — Baseline never runs vet/test/lint; the skill guarantees "every commit passes a quality gate" 2. Regex secret scanning — 13 patterns + 4-level triage precisely blocked a hardcoded API key in Scenario 2, providing an evidence chain that manual review cannot match 3. Staging safety net — >8 file confirmation + logical split proposals prevented 10 mixed files from being committed as one blob in Scenario 3 4. Progressive reference loading — 5 ecosystem gates stored in separate files, loaded on demand; typical token cost is just ~1,300 (SKILL.md + 1 gate)

Design strengths: - SKILL.md strictly held to 169 lines (target ≤ 200) with very high information density - Quality gate per-file design achieves "one SKILL.md, five ecosystems" — a model example of progressive loading - Hard Rules first + precise thresholds (8 files, 50 chars, 3-commit scope frequency) make rules verifiable and unambiguous

Improvement suggestions: 1. Extend secret regexes to cover JWT, Twilio (SK[0-9a-fA-F]{32}), Mailgun, and other emerging platforms 2. Add guidance on creating a .commit-secret-allowlist file to reduce first-time setup friction 3. Consider moving the Edge Cases section into a reference file to further trim SKILL.md (~80 tokens saved)