mongo-migration Skill Evaluation Report¶

Method: skill-creator A/B testing Date: 2026-04-18 Subject: skills/mongo-migration/ — MongoDB schema migration safety reviewer (MongoDB 4.4–7.0+)

MongoDB migration safety is a specialized domain, and the base model handles the core defects well. The measurable gaps are in output structure consistency and token efficiency. The evaluation used three A/B test scenarios with 24 scored assertions covering the skill's three primary working modes.

1. Skill Overview¶

Core files:

Key MongoDB safety rules the skill enforces: - _id-range batched updates (never unbounded updateMany) - Explicit write concern (w: "majority") - Validator progression (moderate → strict, never jump straight to strict) - New-field pattern for type migrations (amount_v2 + dual-read + backfill + cleanup) - Rolling index builds for collections >50M documents

2. Test Design¶

2.1 Scenarios¶

2.2 Assertion Matrix (24 total)¶

Scenario 1 — Index safety + write concern (9 assertions)

Scenario 1 result: With Skill 9/9, Without Skill 8/9 — the only gap is the §9.9 table format.

Scenario 2 — Large-collection field type migration (9 assertions)

Scenario 2 result: With Skill 9/9, Without Skill 8/9 — again, the only gap is the §9.9 table format.

Scenario 3 — Degraded mode, no context (6 assertions)

Scenario 3 result: With Skill 6/6, Without Skill 5/6 — the §9.9 table format fails again.

3. Pass Rate Summary¶

3.1 Overall¶

Delta: +12.5 percentage points

3.2 By scenario¶

The pattern is consistent: all three lost points come from the same root cause — the §9.9 Uncovered Risks output format. Without the skill, the baseline uses numbered lists or prose paragraphs. With the skill, §9.9 is always a 4-column table (Area | Reason | Impact | Follow-up). This is a structural output contract requirement, not a knowledge gap.

4. Key Differences¶

4.1 Behaviors unique to the With-Skill group¶

4.2 Technical knowledge comparison¶

Conclusion: MongoDB migration safety knowledge — WiredTiger, _id-range batching, validator progression — is well-trained in the base model. The skill's value is about enforcing structure, not transferring knowledge.

4.3 Scenario 3 anomaly: With Skill costs more tokens¶

Scenario 3 (degraded mode) produced a reversal: With Skill (36,706 tokens, 3 tool calls) was more expensive than Without Skill (31,986 tokens, 2 tool calls).

Why: SKILL.md §3 specifies "unknown collection size → assume Large → Deep depth → load both reference files." The skill followed this rule and loaded large-collection-migration.md and mongo-ddl-lock-matrix.md, generating extra input tokens and tool calls. The baseline stayed at Standard depth with a leaner output.

Implication: this reflects the skill's conservative design philosophy — loading more context rather than risk missing something. But in a fully context-free scenario, it increases cost unnecessarily.

5. Token Cost Analysis¶

5.1 Skill context overhead¶

5.2 Actual token consumption¶

5.3 Efficiency¶

S1 and S2 are highly efficient: when context is available (version and collection size known), the skill saves nearly 50% of tokens while maintaining higher quality. The savings come from structured output replacing exploratory reasoning, plus zero extra tool calls.

S3 is in the negative range: when context is completely absent, the skill's conservative depth trigger (Deep + all reference files) increases cost by ~15%. Compare with pg-migration, which handles the same situation at Standard depth without this anomaly. This is a known design issue — see §7.

6. Weighted Scores¶

6.1 Dimension scores (out of 5)¶

6.2 Weighted total (out of 10)¶

7. Findings and Recommendations¶

Finding 1: mongo-migration baseline is strong, similar to pg-migration¶

MongoDB migration safety — WiredTiger exhaustion, _id-range batching, validator progression — is thoroughly trained into the base model. The skill's value is concentrated in format enforcement, not knowledge injection.

Finding 2: The §9.9 table format is the single most consistent differentiator¶

Across all three scenarios, the baseline's content was nearly identical to the skill's. The only consistent failure was the §9.9 output format. The baseline's numbered lists and prose paragraphs: - Cannot be machine-parsed in a CI/CD pipeline - Lack the Impact and Follow-up columns needed for team-level action tracking

The skill's 4-column table can be copied directly into a JIRA or Linear ticket description.

Finding 3: Scenario 3 triggers Deep depth too aggressively¶

SKILL.md specifies "unknown collection size → assume Large → Deep depth → load all reference files," which produces ~14.8% extra token cost in a Minimal context. Suggested fix: add a rule to §3 Depth Selection:

If context is Minimal (script only, no size or version information), confirm collection size with the user before triggering Deep depth, or default to Standard depth in Degraded mode to avoid over-consuming when information is absent.

Finding 4: Token efficiency depends heavily on available context¶

Collect MongoDB version and collection size before invoking the skill — this is the single best way to maximize token efficiency.

8. Conclusion¶

Rating: production-ready. Strongly recommended for all MongoDB migration reviews.

Three things the skill does well:

1. Structural enforcement: the §9.9 Uncovered Risks 4-column table makes risk items trackable and machine-parseable. Without the skill, 100% of baseline outputs used freeform formats that can't be parsed or directly actioned.

2. Evaluation framework consistency: Gate analysis, AE cross-references, and Data basis: labels ensure that every review is reproducible and comparable across engineers and over time.

3. Token efficiency when context is available: S1 and S2 save 47% of tokens by replacing exploratory web searches and tool calls with structured framework output.

Three improvement suggestions:

1. Add a rule in §3 Depth Selection to avoid triggering Deep depth in Minimal context — stay at Standard depth with degraded mode instead.

2. Add notes in §9 Output Contract about transaction wrapping behavior in common MongoDB migration tools (golang-migrate, mongomigrate) — analogous to the golang-migrate notes already in the pg-migration skill.

3. Consider adding a fourth test scenario covering reshardCollection (shard key migration), which is the most operationally complex MongoDB 5.0+ feature and currently has no dedicated golden fixture.