e2e-test Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-03-11 Evaluation target: e2e-test

e2e-test is an end-to-end testing practice skill for critical user journeys. It supports designing E2E coverage strategy, handling flaky tests, defining CI gates, and turning exploratory verification into maintainable automated tests. Its three main strengths are: preferring Agent Browser for exploration and reproduction, then Playwright or the project’s native test framework for code, with a clear tool path; built-in environment gates, runner selection, and result-strength control for honest degradation across tech stacks instead of rigid templates; and structured output plus machine-readable JSON for test governance, triage, and CI integration.

1. Evaluation Overview¶

This evaluation reviews the e2e-test skill along two axes: actual task performance and token cost-effectiveness. Three scenarios were designed (E2E journey coverage, flaky test triage, CI gate design). Each scenario was run with both with-skill and without-skill configurations, for 3 scenarios × 2 configs = 6 independent subagent runs, scored against 39 assertions.

Special challenge: issue2md is a pure Go web app with no Node.js/Playwright/package.json, while e2e-test favors Playwright. This tests the skill’s environment adaptation and degradation strategy.

2. Test Methodology¶

2.1 Scenario Design¶

2.2 Special Challenge: Playwright Skill vs. Go Project¶

issue2md’s characteristics make it a boundary test scenario:

This tests the skill’s degradation ability—when the preferred toolchain does not apply, can it correctly identify and choose an alternative?

2.3 Execution¶

• With-skill runs load SKILL.md and selectively load reference files
• Without-skill runs load no skill; model default behavior
• All runs execute in independent subagents in parallel

3. Assertion Pass Rate¶

3.1 Summary¶

3.2 Per-Assertion Details¶

Eval 1: E2E Journey Coverage (15 assertions)¶

Eval 2: Flaky Test Triage (12 assertions)¶

Eval 3: CI Gate Design (12 assertions)¶

3.3 Classification of 19 Without-Skill Failures¶

3.4 Trend: Skill Advantage by Task Type¶

Flaky triage is where the skill adds the most value—the baseline can find root causes and suggest fixes but lacks triage methodology (reproduce → classify → fix/quarantine) and stability proof requirements (-count=20 verification).

4. Dimension-by-Dimension Comparison¶

4.1 Environment Adaptation (Core Differentiator)¶

This is the most distinctive dimension in this evaluation. The skill is designed for Playwright first, but when faced with a pure Go project:

Analysis: With-skill’s Operating Model §5 ("Produce only the strongest deliverable the environment can actually support") correctly guided the degradation decision. The skill did not blindly generate Playwright code; after the Environment Gate confirmed the toolchain was missing, it chose the Go HTTP path. The degradation rationale was explicitly recorded, which matters for PR review and team alignment.

4.2 Five Mandatory Gates Coverage¶

This is the highest-value dimension of the skill—with-skill covered all 5 Gates in all 3 scenarios; without-skill missed multiple Gates in all 3.

Practical value: The Gate system prevents three common errors: 1. False execution claims — Execution Integrity Gate ensures "Not run" is explicitly labeled 2. Single pass = fix — Stability Gate requires -count=20 verification 3. Missing config dependencies — Configuration Gate lists all variables and their available/missing/unknown status

4.3 Output Contract and Machine-Readable JSON¶

With-skill outputs included:

Without-skill reports were not low quality (Eval 3’s CI strategy was thorough), but lacked standardized structure. This means: - Report format varies by task type - CI/tooling cannot consume results programmatically - Results from multiple runs are hard to compare

4.4 Flaky Triage Methodology (Eval 2 Deep Dive)¶

This is where with-skill advantage was largest (+66.7%).

Analysis: Root-cause quality was comparable (both found go run compile + 3s timeout). Without-skill lacked a triage methodology framework. The skill’s Flaky Test Policy ("reproduce with repeat runs → classify → fix → quarantine only with owner, issue, and removal deadline") provides a complete process guarantee.

4.5 CI Strategy Design (Eval 3 Deep Dive)¶

This is where without-skill was closest (+33.3%).

Analysis: Without-skill showed strong baseline ability in CI design—it designed a tiered strategy, found the swagger generation bug, and provided a detailed Rollout Plan. The skill’s increment is mainly in structured Gate validation and machine-readable output.

5. Token Cost-Effectiveness¶

5.1 Skill Size¶

5.2 Actual Load Scenarios¶

5.3 Token Cost vs. Quality Gain¶

5.4 Comparison with Other Skills¶

Analysis:

• Highest absolute gain (+48.7%) — e2e-test assertion delta (19) is the largest in the series
• SKILL.md cost-effectiveness good (~57 tok/1%) — similar to git-commit (~51 tok) and api-integration-test (~49 tok)
• Typical load cost-effectiveness high (~176 tok/1%) — reference volume is large (6 files ~11,710 tokens), much of it Playwright-specific

5.5 Token Segment Cost-Effectiveness¶

5.6 Token Efficiency Rating¶

6. Boundary with Base Model Capabilities¶

6.1 Capabilities Base Model Already Has (No Skill Increment)¶

6.2 Base Model Gaps (Skill Fills)¶

7. Overall Score¶

7.1 Dimension Scores¶

7.2 Weighted Total¶

Token cost-effectiveness lowers the total—SKILL.md cost-effectiveness is good, but Playwright-specific reference content has no value for non-JS projects.

8. Evaluation Materials¶