kafka-event-driven-design Skill Evaluation Report¶

Framework: skill-creator Evaluation date: 2026-04-18 Subject: kafka-event-driven-design

kafka-event-driven-design is a Kafka architecture design and review skill covering topic design, partition strategy, event schema definition (Avro/Protobuf), idempotent producers, consumer deduplication, dead letter queues (DLQ), exactly-once semantics, Schema Registry compatibility, backpressure handling, and consumer lag monitoring. This evaluation ran 3 A/B test scenarios (6 real model calls) and graded 23 assertions comparing responses with and without the skill.

The headline finding: the baseline model has solid Kafka knowledge overall — scenario 1 (fan-out design) scored 75% weighted without the skill. The skill's differentiated value concentrates on three things: correctly classifying enable.idempotence=false as a Critical defect (the baseline called it "acceptable"), enforcing BACKWARD_TRANSITIVE schema compatibility (the baseline defaulted to the weaker BACKWARD), and flagging missing DLQ as Critical during producer reviews (the baseline skipped it entirely).

1. Skill Overview¶

kafka-event-driven-design defines 4 mandatory gates (Context Collection → Scope Classification → Risk Classification → Output Completeness), 3 depth levels (Lite / Standard / Deep), 4 degradation modes (Full / Degraded / Minimal / Planning), and a 14-item design checklist. The §9 output contract ensures every response includes architecture design, risk assessment, implementation patterns, monitoring alerts, and an uncovered risks section.

Core components:

2. Test Design¶

2.1 Scenarios¶

2.2 Assertion Matrix (23 assertions)¶

Scenario 0 — Producer Config Review (9 assertions)

Scenario 0: Without-Skill = 2 pass + 1 partial + 6 fail (weighted 2.5/9 = 28%) | With-Skill = 9/9

Scenario 1 — Multi-Consumer Fan-out Design (8 assertions)

Scenario 1: Without-Skill = 5 pass + 2 partial + 1 fail (weighted 6/8 = 75%) | With-Skill = 8/8

Scenario 2 — Degraded: Topic Design Question (6 assertions)

Scenario 2: Without-Skill = 3 pass + 0 partial + 3 fail (weighted 3/6 = 50%) | With-Skill = 6/6

3. Pass Rate Comparison¶

3.1 Overall¶

Improvement: +57 pp (strict) / +50 pp (weighted)

3.2 By Scenario¶

3.3 Key Differentiating Dimensions¶

4. Key Difference Analysis¶

4.1 Behaviors unique to With-Skill¶

A3 — Idempotent=false classified as Critical

This is the sharpest knowledge divergence in the evaluation. The baseline response explicitly stated:

"Idempotent=false Is Acceptable Here, But Note the Trade-Off — With retries enabled and Idempotent=false, a retry after a broker ack-but-network-drop produces a duplicate message. For at-least-once this is allowed by definition..."

This is technically defensible in isolation, but misses the point in an order processing context. The skill classifies it as a Critical FAIL and specifies that enable.idempotence=true with MaxOpenRequests=1 is part of the minimum safe configuration — the direct application of AE-1. The baseline lacks a classification framework that makes this distinction automatic.

A4 — Missing DLQ flagged as Critical during a producer review

The baseline said nothing about DLQ when reviewing producer code. The skill flags it as Critical 0/3 FAIL in the scorecard and explains in §9.9: no DLQ means a single poison message causes infinite redelivery, stalling the entire partition. This is AE-3 directly applied. The baseline, when looking at producer code, doesn't naturally connect to the consumer-side DLQ requirement.

B7 — BACKWARD_TRANSITIVE vs. BACKWARD

The baseline recommended Schema Registry with BACKWARD compatibility. The skill chose BACKWARD_TRANSITIVE and explained why it matters:

"BACKWARD_TRANSITIVE checks compatibility against ALL previous schema versions, not just the immediately preceding one — critical when multiple consumers may be deployed at different schema versions simultaneously during rolling deploys."

BACKWARD only guarantees the new schema can read data written by the previous version. During rolling deployments, consumers may be running several schema versions apart. BACKWARD_TRANSITIVE closes that gap. This precision comes from event-schema-patterns.md §2.1.

4.2 Areas where the baseline already performs well¶

B2/B3/B4/B5/B8 — Scenario 1 Kafka architecture knowledge

The baseline did well in scenario 1: it correctly recommended order_id as the partition key, three separate Consumer Groups, DB-level idempotent processing for payment-service (INSERT ... ON CONFLICT), per-consumer lag alerts, and DLQs.

This confirms that the baseline has internalized common Kafka architecture patterns. The skill's incremental contribution in scenario 1 is narrower: complete event envelope fields (notably correlation_id), the BACKWARD_TRANSITIVE precision, and the structured §9 output — plus a §9.9 section covering 9 non-obvious risks such as "does payment-service actually need to subscribe to orders.shipped?" and "does the external payment gateway support idempotency keys?"

C2/C3/C4 — Scenario 2 technical judgments

The baseline correctly identified all three concrete problems (generic topic name, single partition, 1-day retention) and gave actionable recommendations with sensible target values. The skill's additional value here is structural: formally naming the degradation mode, triggering the Gate 1 context request, converting unknowns into §9.9 blocking gaps, and producing a scorecard (0/12 FAIL) that makes the overall verdict actionable.

4.3 Scenario-level takeaways¶

Scenario 0 shows the largest gap (+72 pp). The baseline understands acks, but misclassifies idempotence, misses DLQ entirely when reviewing producer code, and doesn't check for event envelope metadata. For business-critical order events, these gaps carry real production risk.

Scenario 1 shows the baseline at its strongest (75% weighted). This is consistent with the oracle-migration evaluation: modern LLMs have solid internalized knowledge of mainstream distributed systems patterns. The skill's value here is about precision and completeness, not filling knowledge gaps.

Scenario 2 illustrates the skill's degradation protocol value. The baseline gave useful, technically correct advice on three specific issues — but never triggered Gate 1, never asked for delivery guarantee or throughput, and never identified the design parameters that would fundamentally change the answer (e.g., if the requirement is exactly-once, the whole design shifts).

5. Token Cost Analysis¶

5.1 Skill context token cost¶

5.2 Actual token consumption (6 real model calls)¶

Note: Token counts are full session totals (input + tool calls + tool results + output), as reported by the Agent tool's usage field. Scenario 1's 17 tool calls (the highest) reflects reading all 3 reference files plus SKILL.md. In production usage where the skill is injected as a system prompt rather than read via tools, the overhead falls to SKILL.md (~9,500 tokens) plus on-demand reference files — closer to 40–60% per call.

5.3 Cost-benefit perspective¶

Scenario 0 has the highest business value density. The baseline's misclassification of enable.idempotence=false as "acceptable" means a network glitch or broker restart in a production order system produces duplicate events. For financial events, the cost of a single double-charge incident — investigation, reversal, customer support, potential chargeback — dwarfs any token cost. Scenario 1 shows that where the baseline already has solid knowledge, the skill's marginal cost-benefit ratio is lower, but it still adds precision (BACKWARD_TRANSITIVE) and structural guarantees (§9.9) that the baseline never produces on its own.

6. Scoring Summary¶

6.1 By dimension¶

6.2 Weighted total¶

7. Conclusion¶

kafka-event-driven-design achieved 100% assertion coverage across 3 scenarios and 23 assertions in 6 real model calls, lifting the weighted pass rate from 50% to 100% (+50 pp).

Like the oracle-migration evaluation, this one found the baseline stronger than expected — scenario 1 hit 75% weighted, confirming that mainstream Kafka architecture patterns are well internalized by modern LLMs. The skill's core value falls into three categories:

1. Correct classification of Critical defects — The baseline treats enable.idempotence=false as an acceptable trade-off for at-least-once delivery. The skill classifies it as Critical FAIL. For financial event pipelines, that misclassification is the direct path to duplicate charges in production. The skill's AE-1 and AE-3 rules make this automatic and non-negotiable.

2. Precision knowledge, not just general knowledge — BACKWARD_TRANSITIVE vs. BACKWARD, the full event envelope (including correlation_id and source_service), and the Outbox Pattern recommendation aren't things the baseline doesn't know — they're things the baseline doesn't surface at the right moment with the right classification. The reference files give the skill a citable specification to work from.

3. Structured output contract — The §9 scorecard and §9.9 Uncovered Risks turn a review or design session into an engineering decision artifact that can be used as a CI/CD gate. No matter how strong the baseline's knowledge, it won't produce this format unprompted.

Recommendation: production-ready. Recommended for all Kafka architecture design and code review workflows. For financial event pipelines (payment, order), the enable.idempotence classification and DLQ enforcement are the single highest-value contributions.