api-design Skill Evaluation Report¶

Method: skill-creator A/B testing Date: 2026-04-18 Subject: skills/api-design/ — REST API contract designer and reviewer

The api-design skill shows a striking pattern: the baseline performs near-perfectly on standard multi-defect reviews and minimal-context scenarios, but drops to just 66.7% on the breaking-change scenario (public API with 12 active partner integrations) — the largest single-scenario gap we've seen across all evaluated skills. The skill's Gate 3 STOP mechanism and the §8.7 Compatibility Assessment structure are responsible for most of the difference.

1. Skill Overview¶

Core files:

Key safety rules the skill enforces: - AE-1: Verbs in URLs (/createUser) break REST semantics and tooling - AE-2: HTTP 200 for errors tricks CDNs, clients, and monitoring - AE-3: Unstructured error messages force clients to string-match, which breaks on rewording - AE-5: Missing object-level authorization (IDOR) — OWASP API Security Top 1 - Breaking Change GUARDRAIL: Any field removal, rename, or type change on a public API must trigger Gate 3 STOP and require a migration plan before proceeding

2. Test Design¶

2.1 Scenarios¶

2.2 Assertion Matrix (24 total)¶

Scenario 1 — Multi-defect REST review (9 assertions)

Scenario 1 result: With Skill 9/9, Without Skill 8.5/9 — the only gap is that the baseline mentions AE-5 once by accident but never systematically cites AE numbers.

Scenario 2 — Public API breaking changes (9 assertions)

Scenario 2 result: With Skill 9/9, Without Skill 6/9 — three failures: Gate STOP framework, §8.9 table format, and Data basis label.

Scenario 3 — Minimal context, degraded mode (6 assertions)

Scenario 3 result: With Skill 6/6, Without Skill 6/6 — no gap at all.

3. Pass Rate Summary¶

3.1 Overall¶

Delta: +14.6 percentage points (strict PASS)

3.2 By scenario¶

Why S2 is the outlier. A public API breaking-change assessment needs three things the baseline skipped entirely:

1. Gate 3 STOP: the baseline went straight to narrative recommendations without formally classifying each change as UNSAFE and requiring a migration plan
2. §8.9 table format: the baseline used freeform headers ("Compatibility Summary", "What Must Be Done") rather than the structured 4-column risk table
3. Data basis label: simply absent in the baseline output

S1 performs well at baseline (94.4%) because REST anti-patterns are widely trained into the model. S3 is perfect at baseline for the same reason we've seen across previous skill evaluations.

4. Key Differences¶

4.1 Behaviors unique to the With-Skill group¶

4.2 Technical knowledge comparison¶

The baseline knows the domain well — Sunset headers, v2 migration, IDOR-safe 404, all come out correctly. The gap is purely about whether the structured framework is followed.

5. Token Cost Analysis¶

5.1 Skill context overhead¶

5.2 Actual token consumption¶

5.3 Efficiency summary¶

The S2 reversal is the most interesting finding in this evaluation. The Without-Skill agent spent only 14,053 tokens — the cheapest run in the entire evaluation — yet produced the lowest-quality output. The With-Skill agent spent 18,577 tokens and produced the highest quality gain (+33.3 pp).

This points to a principle worth stating explicitly: when the structured output is the value (a breaking-change assessment is only useful if every change is classified individually and traceable), token cost going up is fine. The extra ~4,500 tokens bought the Gate 3 STOP, the §8.7 compatibility table, and the Data basis label — all three of which the baseline skipped entirely.

For S1 and S3, the skill saves about 47% of tokens while matching or exceeding baseline quality, so the overall average is still a 33% saving.

6. Weighted Scores¶

6.1 Dimension scores (out of 5)¶

6.2 Weighted total (out of 10)¶

7. Findings and Recommendations¶

Finding 1: Breaking-change assessment is where the skill adds the most value¶

Unlike the migration and caching skills — where the quality gap is fairly uniform across scenarios — api-design has an uneven distribution:

If your team manages a public API with external partners, the skill's ROI is substantially higher than for internal API reviews.

Finding 2: "Framework overhead = value" for structured assessments¶

In S2, the baseline chose to give a concise narrative (14,053 tokens, lowest in the test) and missed all three framework-specific behaviors. This illustrates a general pattern:

A structured framework doesn't just impose a format — it forces a thinking process. Gate 3's STOP condition compels the reviewer to classify every change before offering any recommendations. Without it, a capable model will jump straight to "here's what to do" and skip the classification entirely.

Finding 3: Comparison across evaluated skills¶

api-design has the most scenario-to-scenario variation of any evaluated skill. Basic REST design knowledge (plural nouns, 4xx status codes, IDOR awareness) is thoroughly trained into the model. Public API versioning and breaking-change governance are not — at least not at the level of structured, traceable, gate-controlled analysis.

Improvement suggestions¶

1. Make compatibility-rules.md trigger earlier: currently it only loads at Deep depth, but any request mentioning "breaking", "deprecation", or "versioning" should load it regardless of depth. This would prevent the S2 scenario from slipping through without the full framework.

2. Add a multi-breaking-change golden fixture: the existing API-006 tests a single breaking change. There is no fixture testing Gate 3 STOP when multiple breaking changes appear simultaneously. This is a coverage gap given how common that situation is.

3. Add a webhook API fixture: webhooks appear in the consumer type list but have no dedicated golden scenario (noted as Medium priority in COVERAGE.md).

8. Conclusion¶

Rating: Production-ready. Highest ROI when used for public API and partner integration scenarios.

Three things the skill does well:

1. Breaking-change governance: the Gate 3 STOP mechanism and §8.7 Compatibility Assessment add 33.3 percentage points of quality in the scenario that matters most — public API changes with active integrations. This is the largest single-scenario gain across all skills evaluated so far.

2. Auditable output: the §8.1–§8.9 section structure makes every API review traceable. For compliance-sensitive contexts (partner SLAs, external contracts), a structured scorecard with a Data basis: label is more than a formatting preference — it's a record.

3. Efficient for standard reviews: S1 and S3 save around 47% of tokens while matching or improving on baseline quality.

Recommended usage priority:

1. Highest ROI — public or partner-facing API breaking-change assessments
2. Standard use — internal API full-surface reviews with multiple defects to catch
3. Optional — minimal-context scenarios where the baseline already performs perfectly and the skill mainly contributes structural consistency