Kafka Event-Driven Design Review¶

Quick Reference¶

§1 Scope¶

In scope — Kafka event-driven architecture for production backend services:

• Topic design (naming, partition count, replication factor, retention)
• Partition key strategy (ordering guarantees, hot partition avoidance)
• Event schema design (Avro/Protobuf/JSON Schema, schema evolution, compatibility)
• Producer configuration (acks, retries, idempotence, transactional producers)
• Consumer group design (assignment strategy, rebalance handling, commit strategy)
• Idempotent consumption (deduplication, idempotency keys, exactly-once semantics)
• Dead letter queue (DLQ) and retry patterns
• Backpressure and consumer lag management
• Schema Registry integration and compatibility modes

Out of scope — delegate to dedicated skills:

• Kafka cluster operations, broker config, ZooKeeper/KRaft migration → Kafka ops
• Application code changes unrelated to Kafka → go-code-reviewer
• General API design → api-design

§2 Mandatory Gates¶

Execute gates sequentially. Each gate has a STOP condition.

Gate 1: Context Collection¶

STOP: Cannot determine what events are being produced/consumed (no domain context). Clarify before proceeding.

PROCEED: At least event type, ordering requirement, and delivery guarantee are known.

Gate 2: Scope Classification¶

STOP: Not Kafka-related (e.g., REST API design). Redirect to appropriate skill.

PROCEED: Kafka event-driven intent confirmed.

Gate 3: Risk Classification¶

STOP: Any UNSAFE item without mitigation plan.

PROCEED: Every component has risk level and mitigation.

Gate 4: Output Completeness¶

Before delivering output, verify all §9 Output Contract sections present. §9.9 Uncovered Risks must never be empty.

§3 Depth Selection¶

Force Standard or higher when any signal appears: schema evolution requirement, exactly-once semantics, multi-consumer-group fan-out, partition key redesign, consumer group migration, compacted topics for event sourcing.

§4 Degradation Modes¶

When context is incomplete, degrade gracefully — never fabricate assumptions about ordering requirements.

Hard rule: Never claim "exactly-once" without verifying transactional producer + consumer read-committed isolation + idempotent processing. In Degraded/Minimal mode, flag "delivery guarantee unverified" in §9.9.

§5 Design Checklist¶

Execute every item. Mark PASS / WARN / FAIL with evidence.

5.1 Topic Design¶

1. Topic naming convention — follow {domain}.{entity}.{event-type} or {team}.{service}.{event} pattern. Names should be greppable, meaningful, and avoid generic names like "events" or "messages".

2. Partition count sized for throughput — each partition is the unit of parallelism. Rule of thumb: partitions ≥ max expected consumer instances. Over-partitioning wastes resources; under-partitioning caps throughput. Partition count cannot be decreased (only increased, which breaks key-based ordering).

3. Replication factor ≥ 3 for production topics — min.insync.replicas = 2 with acks = all ensures no data loss on single broker failure. RF=1 is unacceptable for any non-ephemeral data.

4. Retention policy matches use case — time-based (default 7 days) for event streaming; compacted for entity-state topics (latest value per key retained indefinitely); infinite retention for event sourcing.

5.2 Producer Design¶

1. acks=all + retries + enable.idempotence=true — this is the minimum safe producer config. acks=1 risks data loss on leader failure. acks=0 is fire-and-forget. Idempotent producer (Kafka 0.11+) prevents duplicate writes from retries.

2. Partition key chosen for ordering + distribution — key determines which partition an event goes to. Events with the same key are ordered. Common keys: entity ID (order_id, user_id). Avoid: timestamp (hot partition), null (round-robin = no ordering), high-cardinality unbounded (too many partitions).

3. Event schema includes metadata — every event should carry: event_id (UUID), event_type, timestamp, source_service, correlation_id, schema_version. Without these, debugging and deduplication become impossible.

4. Transactional producer for multi-topic atomicity — if producing to multiple topics must be atomic (e.g., event + outbox), use Kafka transactions (initTransactions, beginTransaction, commitTransaction). Without transactions, partial writes create inconsistency.

5.3 Consumer Design¶

1. Idempotent consumption — at-least-once delivery means consumers WILL receive duplicates (after rebalance, retry, or producer retry). Every consumer must handle duplicates: deduplication by event_id, idempotent database operations (INSERT ON CONFLICT), or exactly-once via transactional consumer.

2. Commit strategy explicit — enable.auto.commit=true (default) commits offsets periodically, risking reprocessing on crash. For at-least-once: manual commit after processing. For exactly-once: commit offsets within the transaction that processes the event.

3. Dead letter queue (DLQ) for poison messages — messages that repeatedly fail processing must be routed to a DLQ topic instead of blocking the partition. Without DLQ, a single bad message blocks all subsequent messages in that partition indefinitely.

4. Consumer lag monitoring — consumer_lag (latest offset - committed offset) per partition is the primary health metric. Alert on sustained lag > threshold. Use kafka-consumer-groups.sh or metrics exporter. Lag = events waiting to be processed.

5.4 Schema Evolution & Operations¶

1. Schema compatibility mode set — Schema Registry supports: BACKWARD (new reader, old data), FORWARD (old reader, new data), FULL (both). Choose based on deployment strategy. BACKWARD_TRANSITIVE is safest for most cases. Breaking changes require a new topic.

2. Backpressure handling defined — what happens when consumers can't keep up? Options: scale consumers (up to partition count), increase batch size, apply rate limiting on producer, drop low-priority events. "Consumer crashes under load" is not a strategy.

§6 Partition Design (Standard + Deep)¶

Quick decision guide — for schema patterns load references/event-schema-patterns.md.

Hot partition detection and mitigation¶

• Symptom: one partition has 10x the events of others; consumer for that partition lags
• Cause: skewed partition key (e.g., one tenant produces 90% of events)
• Fix: composite key (tenant_id + entity_id), or custom partitioner that spreads hot keys across N partitions while maintaining per-entity ordering

§7 Anti-Examples¶

AE-1: Producer with acks=1 for critical business events¶

// WRONG: acks=1 — leader acknowledges before replication; data lost on leader failure
producer, _ := sarama.NewSyncProducer(brokers, config)
// config.Producer.RequiredAcks = sarama.WaitForLocal  // acks=1
// RIGHT: acks=all + idempotent
config.Producer.RequiredAcks = sarama.WaitForAll
config.Producer.Idempotent = true
config.Net.MaxOpenRequests = 1

AE-2: Consumer without idempotency handling¶

// WRONG: processes event and assumes it won't be delivered again
func handleOrderCreated(event OrderCreated) {
    db.Insert(event.Order)  // duplicate delivery → duplicate insert → constraint violation or double-charge
}
// RIGHT: idempotent processing
func handleOrderCreated(event OrderCreated) {
    db.Exec("INSERT INTO orders ... ON CONFLICT (id) DO NOTHING", event.Order)
}

AE-3: No dead letter queue — poison message blocks partition¶

// WRONG: bad message causes infinite retry loop, blocking all events behind it
func consume(msg *sarama.ConsumerMessage) {
    if err := process(msg); err != nil {
        log.Error(err)
        // message is not committed → redelivered forever
    }
}
// RIGHT: route to DLQ after N retries
if retryCount >= maxRetries {
    producer.Send(dlqTopic, msg)
    consumer.CommitMessage(msg)  // advance past the poison message
}

AE-4: Null partition key — loses ordering guarantee¶

// WRONG: null key → round-robin across partitions → order events scattered
producer.SendMessage(&sarama.ProducerMessage{
    Topic: "order.events",
    Value: sarama.ByteEncoder(data),
    // Key is nil → no ordering guarantee
})
// RIGHT: use entity ID as partition key
producer.SendMessage(&sarama.ProducerMessage{
    Topic: "order.events",
    Key:   sarama.StringEncoder(order.ID),
    Value: sarama.ByteEncoder(data),
})

AE-5: Schema change without compatibility check¶

// WRONG: removed required field — breaks all existing consumers
// v1: {"order_id": "123", "amount": 99.50, "currency": "USD"}
// v2: {"order_id": "123", "amount": 99.50}  // removed currency!
// RIGHT: add new optional fields, never remove required ones (BACKWARD compatible)

AE-6: Kafka issue reported as application logic bug¶

-- WRONG: "Bug: some orders processed twice causing double charge"
-- This is a consumer idempotency issue, not application logic.
-- RIGHT: "Consumer lacks idempotency: duplicate Kafka delivery causes double processing"

Extended anti-examples (AE-7 through AE-13) in references/consumer-anti-examples.md.

§8 Kafka Design Scorecard¶

Critical — any FAIL means overall FAIL¶

• [ ] Producer uses acks=all + enable.idempotence=true for non-ephemeral events
• [ ] Consumer handles duplicate delivery (idempotent processing or exactly-once)
• [ ] Dead letter queue exists for poison messages (no infinite retry loops)

Standard — 4 of 5 must pass¶

• [ ] Partition key matches ordering requirement (not null for ordered events)
• [ ] Schema includes event metadata (event_id, event_type, timestamp, source)
• [ ] Schema compatibility mode configured in Schema Registry
• [ ] Consumer lag monitoring with alert threshold defined
• [ ] Commit strategy explicitly chosen (auto-commit disabled for at-least-once)

Hygiene — 3 of 4 must pass¶

• [ ] Topic naming follows convention ({domain}.{entity}.{event-type})
• [ ] Replication factor ≥ 3 with min.insync.replicas = 2
• [ ] Backpressure strategy defined (scale / rate-limit / drop)
• [ ] Retention policy matches use case (time-based / compacted / infinite)

Verdict: X/12; Critical: Y/3; Standard: Z/5; Hygiene: W/4. PASS requires: Critical 3/3 AND Standard ≥4/5 AND Hygiene ≥3/4.

§9 Output Contract¶

Every design review MUST produce these sections. Write "N/A — [reason]" if inapplicable.

### 9.1 Context Gate
| Item | Value | Source |

### 9.2 Depth & Mode
[Lite/Standard/Deep] × [review/design/troubleshoot] — [rationale]

### 9.3 Risk Assessment
| Component | Risk | Notes |

### 9.4 Architecture Design (Standard/Deep; "N/A — Lite" for Lite)
- Topic topology + partition strategy
- Producer + consumer configuration
- Schema design + evolution strategy

### 9.5 Implementation (topic config, producer/consumer code patterns)

### 9.6 Validation Plan
- End-to-end event flow test
- Duplicate delivery test
- Schema evolution test (produce v2, consume with v1 reader)

### 9.7 Failure Handling
- DLQ routing + monitoring
- Consumer rebalance behavior
- Broker failure recovery

### 9.8 Monitoring & Alerts
- Consumer lag per partition
- DLQ message rate
- Producer error rate, batch size, latency

### 9.9 Uncovered Risks (MANDATORY — never empty)
| Area | Reason | Impact | Follow-up |

Volume rules: - FAIL: always fully detailed - WARN: up to 10; overflow to §9.9 - PASS: summary only - §9.9 minimum: document all assumptions (especially delivery guarantee if unverified)

Scorecard summary (append after §9.9):

Scorecard: X/12 — Critical Y/3, Standard Z/5, Hygiene W/4 — PASS/FAIL
Data basis: [full context | degraded | minimal | planning]

§10 Reference Loading Guide¶