yt-dlp-downloader Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-03-12 Evaluation subject: yt-dlp-downloader

yt-dlp-downloader is a skill for generating and running yt-dlp download commands. It is suited for single videos, playlists, audio extraction, subtitle downloads, SponsorBlock, resolution limits, and authenticated download scenarios. Its three standout strengths are: probe-first, using format lists and subtitle info to decide command combinations instead of guessing parameters; safe defaults including --no-playlist, retries, output naming, and archive to reduce accidental full-playlist downloads, re-downloads, and runaway commands; and structured execution reports, especially useful for complex combined requests to reuse, review, and adjust.

1. Evaluation Overview¶

This evaluation reviews the yt-dlp-downloader skill along two axes: actual task performance and token cost-effectiveness. Three yt-dlp command-generation scenarios of increasing complexity were designed (single video download, audio extraction + subtitles, playlist + resolution + SponsorBlock + subtitles). Each scenario was run with both with-skill and without-skill configurations, for 3 scenarios × 2 configs = 6 independent subagent runs, scored against 40 assertions.

2. Test Methodology¶

2.1 Scenario Design¶

2.2 Execution¶

• With-skill runs first read SKILL.md and its referenced materials
• Without-skill runs read no skill, using model default yt-dlp knowledge
• All runs in Degraded mode (no yt-dlp installed); evaluates command recommendation quality, not actual execution
• 6 subagents run in parallel

3. Assertion Pass Rate¶

3.1 Overview¶

3.2 Per-Assertion Details¶

Eval 1: Single Video Download¶

Eval 2: Audio Extraction + Subtitles¶

Eval 3: Playlist + 720p + SponsorBlock + Subtitles¶

3.3 Classification of 22 Failed Assertions (Without-Skill)¶

3.4 Trend: Skill Advantage Increases with Scenario Complexity¶

Unlike the go-makefile-writer evaluation where "Skill advantage decreases with complexity", this skill is strongest in the most complex scenario. Reason: yt-dlp command combinations have many implicit rules (--write-subs with --embed-subs, playlist template zero-padding, SponsorBlock ffmpeg dependency, etc.); the base model omits more details when stacking multiple scenarios.

4. Dimension-by-Dimension Comparison¶

4.1 Output Contract (Structured Report)¶

This is a Skill-only differentiator, contributing 3 assertion deltas.

Practical value: Output Contract enables: - Auditable command recommendations (know why specific flags were chosen) - Transparent Probe decisions (whether probe was skipped and why) - Clear next steps for users (no guessing)

4.2 Probe Decision Framework¶

This is the skill’s core design advantage, contributing 3 assertion deltas.

Without-skill’s key issue: assumes subtitles exist and adds --write-subs or --embed-subs; if subtitles don’t exist, silent failure. The skill’s Probe Gate forces verify-before-download.

4.3 Safety Guard Flags¶

--no-playlist is the highest-risk safety gap. When a YouTube watch URL includes &list=, omitting --no-playlist downloads the entire playlist instead of one video, potentially causing tens of GB of accidental downloads. This is explicitly addressed in Skill Anti-Example #3.

4.4 Command Technical Correctness¶

With-skill’s bv* selector is better than bestvideo because * includes pre-merged video streams (some sites only offer pre-merged format). Without-skill’s bestvideo does not include pre-merged streams.

4.5 Ambiguity Resolution Quality¶

In Eval 3, "Chinese subtitles" is ambiguous:

Both cover three language codes, but With-skill’s ambiguity resolution is more transparent — users know why these codes were chosen and how to adjust.

5. Token Cost-Effectiveness Analysis¶

5.1 Skill Size¶

5.2 Typical Load Scenarios¶

SKILL.md’s "Load References Selectively" section guides on-demand loading:

5.3 Token Cost for Quality Gain¶

5.4 Token Segment Cost-Effectiveness¶

5.5 High-Leverage vs Low-Leverage Instructions¶

High leverage (~820 tokens SKILL.md → 20 assertion deltas): - Safe defaults (150 tok → 7 assertions) - Probe Gate (250 tok → 3 assertions) - Output Contract (200 tok → 3 assertions) - --no-playlist rule (80 tok → 2 assertions) - Other single-rule items (140 tok → 5 assertions)

Medium leverage (~770 tokens → indirect): - Anti-Examples (350 tok) — avoid hardcoded format ID - Gate pipeline (300 tok) — drive structured thinking flow - Scope classification (120 tok) — correct multi-scenario overlay identification

Low leverage (~150 tokens → 0 deltas): - Auth Safety (100 tok) — no auth scenario in this eval - Live Stream (50 tok) — no live stream in this eval

References (~3,090–4,090 tokens → indirect): - scenario-templates.md drives command completeness and flag selection - golden-examples.md drives answer format consistency - decision-rules.md drives format selection technical correctness

5.6 Token Efficiency Rating¶

5.7 Comparison with Other Skills’ Cost-Effectiveness¶

yt-dlp-downloader has best token cost-effectiveness among the three skills because: 1. Base model has weaker grasp of yt-dlp’s implicit rules (45% baseline vs go-makefile 69%), more room for improvement 2. Skill’s high-leverage rules are compact (safe defaults, probe gate, output contract only ~820 tokens) 3. Reference conditional loading is well designed; simple scenarios don’t load everything

6. Boundary Analysis vs Base Model Capabilities¶

6.1 Capabilities Base Model Already Has (No Skill Increment)¶

6.2 Base Model Gaps (Skill Fills)¶

7. Overall Score¶

7.1 Dimension Scores¶

7.2 Weighted Total Score¶

Command technical correctness increment is scored lower because Without-skill’s core commands are technically sound — the base model has good grasp of basic yt-dlp usage; the skill’s core value is safety guards, Probe discipline, and structured reports.

8. Evaluation Materials¶