redis-cache-strategy Skill Evaluation Report¶

Method: skill-creator A/B testing Date: 2026-04-18 Subject: skills/redis-cache-strategy/ — Redis caching strategy design and review skill

Redis cache safety rules enjoy exceptionally high training coverage in the base model, and baseline quality in this evaluation reached 89.6%. The skill's core value manifests in two dimensions: framework reference consistency (AE number cross-referencing, explicit Gate analysis) and token efficiency (an average of 49.7% savings across three scenarios — the most stable efficiency advantage of any skill evaluated to date).

§1 Skill Overview¶

Core components:

Key safety rules enforced by the skill: - AE-1: TTL=0 (immortal key) → data never expires - AE-2: write-behind without a durable queue → data loss on process crash - AE-3: cache-aside without singleflight → stampede breaks through to DB - AE-5: distributed lock without TTL or token check → deadlock + lock theft - GUARDRAIL: write-behind is prohibited for financial / audit-critical data

§2 Test Design¶

2.1 Scenario Definitions¶

2.2 Assertion Matrix (24 assertions)¶

Scenario S1 — Cache-Aside Defects (9 assertions)

S1 summary: With Skill 9/9, Without Skill 8/9 (lost point: AE number references absent)

Scenario S2 — Distributed Lock + Write-Behind (9 assertions)

S2 summary: With Skill 9/9, Without Skill 8/9 (lost point: explicit Gate analysis absent)

Scenario S3 — Minimal Context / Degraded Mode (6 assertions)

S3 summary: With Skill 6/6, Without Skill 5.5/6 (PARTIAL: § prefix format not consistently applied)

§3 Pass Rate Summary¶

3.1 Overall Assertion Pass Rate¶

Delta: +10.4 percentage points (strict PASS basis)

3.2 Pass Rate by Scenario¶

Pattern: All three lost points belong to a single category — framework reference consistency (AE numbers, Gate declarations, § prefix). Core safety knowledge (TTL jitter, singleflight, Lua CAS, write-behind guardrail) scored 100% in both groups. This indicates that Redis cache safety rules are deeply embedded in the base model; the skill's value lies in reference traceability and token efficiency, not knowledge transfer.

§4 Key Difference Analysis¶

4.1 Behaviors Exclusive to With-Skill¶

4.2 Core Technical Knowledge Comparison¶

All critical Redis safety checks were correctly identified by both groups:

Conclusion: Redis cache safety knowledge is one of the most thoroughly trained domains in the base model. The skill adds no extra value in technical content, but provides measurable advantages in framework consistency and token efficiency.

4.3 Baseline Comparison Across Skills¶

Trend: As domain knowledge matures in the base model, the skill's delta narrows and its value shifts from knowledge delivery to structural constraint. redis-cache-strategy represents the extreme end of this trend — the skill contributes almost no new knowledge but provides a consistent 49.7% token saving.

§5 Token Cost Analysis¶

5.1 Skill Context Token Cost¶

5.2 Actual Token Consumption¶

5.3 Cost-Efficiency Metrics¶

Structural finding: Token savings are exceptionally consistent across all three scenarios (variance ±2%), with no S3 anomaly (contrast: mongo-migration S3 ran +15% over baseline). The reason: redis-cache-strategy's §3 Depth Selection correctly handles minimal context — unknown scale does not trigger Deep depth; the skill stays at Standard depth with conservative assumptions, avoiding unnecessary reference file loading.

Without-Skill tool call breakdown: Each scenario incurred 3 tool calls (likely web searches for Redis docs / Go code examples). This not only inflated token consumption but introduced network dependency and non-determinism. The With-Skill group inlines all knowledge, resulting in zero tool calls and more stable responses.

§6 Weighted Scores¶

6.1 Dimension Scores (5-point scale)¶

6.2 Weighted Total Score (out of 10)¶

§7 Findings and Recommendations¶

Finding 1: redis-cache-strategy has the strongest baseline of any skill evaluated¶

A baseline of 89.6% indicates that Redis cache safety rules (singleflight, TTL jitter, Lua CAS, write-behind prohibition) have become built-in knowledge in the base model. This stands in sharp contrast to mysql-migration (52% baseline), where the skill's primary value was knowledge injection. For redis-cache-strategy, value comes almost entirely from structural constraint, not knowledge delivery.

Implication for skill design: In mature, well-covered domains, skills should focus more on output structure standardisation (§9 contract, AE numbering, Gate framework) and less on knowledge documentation.

Finding 2: Token efficiency is the most consistent differentiator (−49.7%)¶

All three scenarios consistently saved approximately 50% in tokens, with no anomalous outliers (contrast: mongo-migration S3). This stability comes from two factors: - With Skill: Framework guidance drives direct structured output generation — 0 tool calls - Without Skill: Exploratory reasoning + 3 web searches per scenario — more output but with duplication

For high-frequency Redis cache reviews (e.g., PR review in CI/CD), running 100 scenarios per month under this skill yields roughly 50% token savings, translating to approximately 2× cost efficiency.

Finding 3: S3 Minimal Context shows no anomaly — Depth Selection trigger logic is correct¶

Under minimal context (version / scale / SLA unknown), redis-cache-strategy correctly selects Standard depth rather than Deep, avoiding the token overrun seen in mongo-migration's S3 due to a Deep depth trigger. This validates the conservative trigger design in §3 Depth Selection.

Finding 4: §9.9 table format is already widely adopted at baseline¶

As observed in previous evaluations, the Without-Skill group spontaneously used the | Area | Reason | Impact | Follow-up | 4-column format. This format appears to have become the base model's default output pattern, likely due to training data coverage from skill documentation.

Recommendation: Given that the format is already broadly covered, skill maintenance should focus on rules that are harder for the baseline to execute correctly: 1. Pattern Selection Matrix for complex scenarios (e.g., mixed read/write ratio pattern selection) 2. Isolation design for multi-service shared caches 3. Distinguishing Redlock from single-node lock applicability

§8 Conclusion¶

redis-cache-strategy is rated production-ready and recommended for all Redis caching layer design and review workflows.

Core value propositions: 1. Token efficiency lead: Average savings of 49.7% across three scenarios — the most stable efficiency advantage of any evaluated skill; well-suited for high-frequency CI/PR workflows 2. Traceable framework references: AE numbers, Gate declarations, and §9.x prefix ensure every review can be traced back to the governing specification 3. Zero web-search dependency: Inlined knowledge means the With-Skill group requires no external tool calls, providing a clear advantage in network-constrained or latency-sensitive environments

Improvement recommendations: 1. Add a golden fixture for multi-service shared cache scenarios (CACHE-015), covering tenant isolation and keyspace separation 2. Add AE-14 (Lua script atomicity loss under Redis Cluster) to §7 Anti-Examples, addressing a common misconception in cluster deployments 3. Consider making the Minimal context depth rule explicit in §4 Degradation Modes: state that unknown scale does not trigger Deep depth (currently implicit — recommend making it explicit)