google-search Skill Evaluation Report¶

Evaluation framework: skill-creator Evaluation date: 2026-03-12 Subject: google-search Revised 2026-08-13: added § 3.5 Validity Limitations. The A/B experiment was not re-run; all measured numbers are from the 2026-03-12 run.

google-search is a research/search skill that turns "help me search for this" into a verifiable search workflow. It suits fact lookups, error debugging, official docs retrieval, technology comparisons, and public-information gathering that needs source support. Its three main strengths are: classifying the question, defining the evidence chain, and choosing the mode first—elevating search from "finding links" to "finding evidence for conclusions"; outputs include confidence, source tier, budget status, and reusable queries so the search process is reviewable and continuable; and it emphasizes execution completeness and degradation declarations, clearly distinguishing "verified conclusions" from "partial results with insufficient evidence".

1. Evaluation Overview¶

This evaluation assesses the google-search skill along two dimensions: actual task performance and Token cost-effectiveness. It uses 3 search scenarios of increasing complexity (Quick-mode fact lookup, Standard-mode error debugging, Deep-mode framework comparison). Each scenario runs with both with-skill and without-skill configurations, for 3 scenarios × 2 configurations = 6 independent subagent runs, scored against 27 assertions.

Key finding: The core value of the google-search skill is search discipline and report structure, not search content quality. The base model already has strong search and synthesis ability (answer correctness, source coverage, code example quality all good), but lacks metadata for the search process (mode choice, budget control, evidence chain tracking, degradation declaration, confidence labels, reusable queries). The skill fills this "search operation discipline" gap.

Read the +74.1 point figure together with § 3.5 Validity Limitations. It measures compliance with the output contract this skill itself defines, it was scored on assertions that mostly test skill-mandated fields, and the two arms did not run on the same model. Answer correctness scored 3/3 in both arms.

2. Test Methodology¶

2.1 Scenario Design¶

2.2 Execution¶

• With-skill runs first read SKILL.md and related references (query-patterns, programmer-search-patterns, source-evaluation, etc.)
• Without-skill runs read no skill; search follows model default behavior
• All runs may use WebSearch and WebFetch tools
• 6 subagents run in parallel (with-skill uses default model, without-skill uses fast model)

2.3 Skill Characteristics¶

google-search is a multi-file skill (1 SKILL.md + 6 reference files) with conditional loading.

Sizes below are as measured on 2026-03-12, the day this A/B ran. They no longer describe the current skill — the 2026-08-13 revision grew SKILL.md and programmer-search-patterns.md substantially. Re-derive rather than trusting this table:

cd skills/google-search && for f in SKILL.md references/*.md; do echo "$(wc -w < "$f")  $f"; done

Two of the March figures were already wrong when written: chinese-search-ecosystem.md was recorded as 279 words against 904 today, and worked-examples.md was left out of the table entirely. Token estimates in this report use ≈1.5 tokens per word; they are derived, not counted.

Actual load per scenario:

3. Assertion Pass Rate¶

3.1 Summary¶

3.2 Item-by-Item Scoring¶

Eval 1: Go database/sql default pool size (Quick mode)¶

Eval 2: gRPC context deadline exceeded (Standard mode)¶

Eval 3: Go HTTP framework comparison (Deep mode)¶

3.3 Classification of 20 Without-Skill Failed Assertions¶

Note: As with the deep-research evaluation, all 20 failures are search discipline / report format failures, not content quality failures. Without-skill passed on answer correctness, source coverage, and code examples.

3.4 Comparison with deep-research Skill¶

google-search has a larger assertion delta because it requires not only a report template (deep-research’s 7-section) but also search process metadata (mode, budget, evidence chain, degradation level, reusable queries, precise query strategies). The base model does not produce these concepts at all.

3.5 Validity Limitations¶

Three limits bound what the +74.1 percentage-point figure supports. They are recorded here because the number otherwise reads as a claim about search quality, which it is not.

1. The assertions mostly test fields the skill mandates. Execution mode, degradation level, evidence-chain status, confidence and source-tier labels, reusable queries — these are artifacts this skill defines. A run with no skill loaded has no reason to emit any of them and scores zero on those items by construction. § 3.3 already classifies all 20 without-skill failures as search-discipline and report-format items rather than wrong answers. The delta therefore measures compliance with a format the skill itself introduces. It is not evidence that answers got better.

2. The two arms did not use the same model. Per § 2.2, with-skill runs used the default model while without-skill runs used the fast model. Model capability is confounded with skill presence, so no portion of the delta can be attributed cleanly to the skill. Only re-running both arms on one model can separate the two.

3. Content quality showed no measurable difference. § 4.5 and the overview table both record 3/3 versus 3/3 on answer correctness. On the dimension users care about most, this evaluation found no gain — which the headline number does not convey.

What the evaluation does support: with the skill loaded, the model reliably produces an auditable search report, at a cost of roughly 3,100 tokens for SKILL.md. What it does not support: that the skill makes searches more accurate, or that any 74-point gain exists on an axis other than conformance to this skill's own output contract.

4. Every token figure in § 2.3 and § 5 predates the 2026-08-13 revision. SKILL.md grew from 2,085 to 3,016 words and programmer-search-patterns.md from 1,031 to 1,771. The cost-effectiveness ratios (tokens per 1% pass-rate gain, tokens per assertion) divide an old numerator by an old denominator and must not be recomputed against current sizes — the pass-rate gain was produced by the March content, not by today's. Treat § 5 as a record of the March version.

A future revision should hold the model constant across arms; score at least some assertions against externally verifiable ground truth (a known version number, a known release date) instead of the presence of a field; and include one scenario whose correct answer is a refusal to conclude, so honest degradation is measured rather than assumed.

That harness now exists but has not been run. skills/google-search/scripts/eval_forward.py implements all three corrections: one --model for both arms, criteria that check facts verified against primary sources (go doc database/sql defaults, GitHub's code-search qualifier set) rather than field presence, and two scenarios whose correct answer is a labelled refusal — an official recommendation no vendor publishes, and content inside a walled garden. It deliberately has no "did it print a degradation line" criterion. Its grader is tested offline (--self-test plus scripts/tests/test_forward_eval.py, which also verifies the harness can report the skill losing), and a harness failure exits INCOMPLETE rather than grading. Executing it needs an authenticated CLI in an unsandboxed shell:

cd skills/google-search && python3 scripts/eval_forward.py --run --model sonnet --out eval_out

Until that runs, no number in this report describes the current skill's behaviour.

Offline regression tests have the mirror-image limitation. They now check values, structure, and scope instead of keyword presence, and they cover query-syntax validity and output-contract conformance — but they cannot observe mode selection, whether a listed query was executed, or whether a cited page was opened. See skills/google-search/scripts/tests/COVERAGE.md § "What these tests still cannot prove".

4. Dimension-by-Dimension Comparison¶

4.1 Output Contract (8 Fields)¶

Practical value: - Degradation level showed highest value in Eval 3—With-skill honestly declared "Partial" (TechEmpower data from third-party interpretation, no named company production cases), while Without-skill gave conclusions without marking uncertainty - Evidence chain status lets readers track "which evidence is satisfied, which is missing", avoiding treating partial data as complete conclusions - Reusable queries give readers the ability to "continue searching"—5 well-designed Google queries are more lasting value than a single answer

4.2 Search Strategy Discipline¶

4.3 Confidence + Source-Tier Labels¶

Eval 3 With-skill output labeled all 14 key numbers with dual labels:

| Fiber real-world RPS | ~36,000 | May 2024 | Medium | Primary (independent benchmark) |
| Fiber JSON RPS (TechEmpower R23) | ~735,000 | March 2025 | Low | Third-party interpretation of Official |

This distinguishes "Medium confidence from Primary source" from "Low confidence from Third-party interpretation", so readers know TechEmpower data is secondhand and downgraded. Without-skill Eval 3 cited 16 sources and many numbers but no number had credibility or source-tier labels.

4.4 Honest Degradation¶

Eval 3 With-skill output best illustrates this mechanism:

Degradation Level: Partial — Strong benchmark data and ecosystem analysis available. However: TechEmpower Round 23 Go-specific per-framework numbers could not be directly verified from TechEmpower's own site... Large-scale production experience reports... were not found from named companies with disclosed architectures.

This degradation statement clearly informs readers of two specific uncertainties, avoiding treating the comparison as fully confirmed fact. Without-skill Eval 3 also found no named company cases but did not declare this limitation.

4.5 Content Quality Comparison¶

Key conclusion: Consistent with the deep-research skill evaluation—the base model is already strong on content; the skill’s increment is entirely in search discipline and report metadata.

5. Token Cost-Effectiveness Analysis¶

5.1 Skill Size¶

5.2 Token Cost for Quality Gain¶

5.3 Token Segment Cost-Effectiveness¶

5.4 High-Leverage vs Low-Leverage Instructions¶

High leverage (~1,050 tokens → 18 assertion delta): - Output Contract 8-field definition (~300 tok → 6) - Confidence + Source-tier dual-label rules (~200 tok → 3) - Reusable Queries requirement (~100 tok → 3) - Evidence Chain Gate (~300 tok → 3) - Source Assessment requirement (~150 tok → 3)

Medium leverage (~5,150 tokens → 2 direct + indirect): - query-patterns.md (~1,800 tok → 2 + indirect search quality) - programmer-search-patterns.md (~1,500 tok → indirect) - source-evaluation.md (~1,400 tok → indirect) - Other Gates (~450 tok → indirect)

Low leverage (~800 tokens → 0 direct delta): - Worked Examples (~500 tok) - Anti-Examples (~300 tok)

5.5 Comparison with Other Skills’ Cost-Effectiveness¶

google-search has the highest absolute pass-rate gain (+74.1%), but SKILL.md-level unit cost-effectiveness (~42 tok/1%) is similar to yt-dlp-downloader (~43) and tdd-workflow (~45). Typical-load cost-effectiveness (~105 tok/1%) is higher due to more reference files.

6. Boundary Analysis vs Base Model Capabilities¶

6.1 Base Model Capabilities (No Skill Increment)¶

6.2 Base Model Gaps (Skill Fills)¶

Core finding: The base model’s "search results → answer" ability is strong, but "search process auditability" and "conclusion credibility labeling" are zero. The google-search skill’s value lies in the latter two.

7. Overall Score¶

7.1 Dimension Scores¶

7.2 Weighted Total¶

The lower Token cost-effectiveness score (7.0/10) reflects higher typical load (~7,800 tok) from more reference files, even though SKILL.md cost-effectiveness (~42 tok/1%) is comparable to peer skills.

8. Evaluation Artifacts¶