Monitoring & Alerting Design Review¶

Quick Reference¶

§1 Scope¶

In scope — monitoring and alerting for production backend services:

• SLI (Service Level Indicator) definition and measurement
• SLO (Service Level Objective) target setting and error budget
• Prometheus alerting rules (PromQL, for duration, severity labels)
• Burn-rate alerting (multi-window, multi-burn-rate SLO alerts)
• Grafana dashboard design (layout, variable templating, panel types)
• Alert routing configuration (PagerDuty/OpsGenie/Slack, severity-based routing)
• Alert fatigue audit (noise ratio, actionability, deduplication)
• Label cardinality management (high-cardinality label detection)
• On-call runbook integration (alert → runbook → action mapping)

Out of scope — delegate to dedicated skills:

• Metrics/tracing instrumentation in code → go-observability-review
• Application performance profiling → go-benchmark
• Infrastructure provisioning (Prometheus/Grafana setup) → ops tooling
• Log aggregation pipeline design → separate skill

§2 Mandatory Gates¶

Execute gates sequentially. Each gate has a STOP condition.

Gate 1: Context Collection¶

STOP: Cannot determine what the service does (no SLI candidates identifiable). Clarify before proceeding.

PROCEED: At least service type and traffic pattern known.

Gate 2: Scope Classification¶

STOP: Request is about code instrumentation (not alert/dashboard design). Redirect to go-observability-review.

PROCEED: Monitoring/alerting design intent confirmed.

Gate 3: Risk Classification¶

STOP: Any UNSAFE change without impact assessment.

PROCEED: Every change has risk level and mitigation.

Gate 4: Output Completeness¶

Before delivering output, verify all §8 Output Contract sections present. §8.9 Uncovered Risks must never be empty.

§3 Depth Selection¶

Force Standard or higher when any signal appears: SLO definition, burn-rate alerting, PagerDuty/OpsGenie routing, multi-service dashboard, alert fatigue investigation, label cardinality concern.

§4 Degradation Modes¶

When context is incomplete, degrade gracefully — never guess traffic patterns.

Hard rule: Never set alert thresholds without knowing the traffic pattern. A 1% error rate alert on a 10-QPS service fires on a single error; on a 10K-QPS service it means 100 errors/sec. In Degraded mode, flag all thresholds as "requires traffic data to validate" in §8.9.

§5 Design Checklist¶

Execute every item. Mark PASS / WARN / FAIL with evidence.

5.1 SLI/SLO Foundation¶

1. SLIs defined for the service — every production service needs measurable SLIs. Standard SLIs by service type:
2. API service: availability (success ratio), latency (p50/p95/p99), error rate
3. Worker/consumer: processing rate, lag, error rate, processing latency
4. Batch job: completion rate, duration, data quality

5. Data pipeline: throughput, freshness, correctness

6. SLOs set with error budget — each SLI has a target (e.g., "99.9% availability over 30 days"). Error budget = 1 - SLO target (e.g., 0.1% = ~43 minutes/month of allowed downtime). SLOs must be agreed with stakeholders, not invented by engineers.

7. Burn-rate alerting for SLOs — instead of raw threshold alerts, use multi-window burn-rate alerts that fire when error budget is being consumed too fast. This dramatically reduces false positives. Load references/sli-slo-patterns.md for patterns.

5.2 Alert Rule Quality¶

1. Every alert is actionable — when this alert fires, is there a concrete action the on-call can take? If the answer is "look at it and hope it resolves," the alert should be a dashboard graph, not a page. Non-actionable alerts cause alert fatigue.

2. for duration prevents flapping — Prometheus for clause should be set to absorb transient spikes. Too short (e.g., for: 1m on a noisy metric) → flapping alerts. Too long (e.g., for: 30m) → delayed notification. Typical: for: 5m for warning, for: 2m for critical.

3. Severity labels match routing — alerts must have severity: critical|warning|info labels that map to routing rules. Critical → PagerDuty page. Warning → Slack channel. Info → dashboard only. Mislabeled severity causes either missed pages or unnecessary wake-ups.

4. Alert includes runbook link — every alerting rule should include an annotations.runbook_url pointing to a runbook with: what the alert means, how to diagnose, how to mitigate. Without runbooks, on-call responders waste time Googling their own alerts.

5.3 Dashboard Design¶

1. Dashboard follows USE/RED method — organize dashboards by signal type:
2. USE (infrastructure): Utilization, Saturation, Errors (CPU, memory, disk, network)
3. RED (services): Rate, Errors, Duration (request rate, error rate, latency)

4. Top row: golden signals overview. Detail rows: drill-down by endpoint/consumer/partition.

5. Dashboard uses variables for templating — Grafana variables ($service, $namespace, $instance) allow one dashboard to serve multiple instances. Avoid hardcoded label values in queries.

6. No high-cardinality labels in dashboard queries — labels like user_id, request_id, trace_id in PromQL queries explode time series count and crash Prometheus. Use bounded labels: method, status_code, endpoint (allowlisted).

5.4 Operations & Routing¶

1. Alert routing matches severity — critical alerts page on-call (PagerDuty/OpsGenie with escalation). Warning alerts go to team Slack channel. Info alerts are dashboard-only. No unrouted alerts.

2. Deduplication and grouping configured — Alertmanager group_by prevents firing 100 instances of the same alert. group_wait and group_interval control batch notification timing. Without grouping, a single incident generates N alerts for N instances.

3. Inhibition rules prevent alert cascade — if the database is down, suppress all "elevated error rate" alerts from services that depend on it. Without inhibition, one root cause generates dozens of symptomatic alerts.

4. Alert fatigue metrics tracked — measure: total alerts/week, alerts-per-on-call-shift, % of alerts that required action, MTTA (mean time to acknowledge). Target: <5 pages/week per on-call, >80% actionability rate.

5.5 Rule Validation (Eat Your Own Dog Food)¶

A skill that preaches "no false positives, no silent gaps" must validate its rules mechanically, not by eyeball:

1. Rules pass promtool check rules — syntax + PromQL validation before any rule ships:

   promtool check rules alerts.yml
   

2. SLO-critical alerts have promtool test rules unit tests — Prometheus natively supports asserting "given this input series, this alert fires (or stays silent)". This is the executable form of the sensitivity/specificity tradeoff:

   # alerts_test.yml — run with: promtool test rules alerts_test.yml
   rule_files:
     - alerts.yml
   evaluation_interval: 1m
   tests:
     - interval: 1m
       input_series:
         - series: 'http_errors_total{job="api"}'
           values: '0+10x10'      # 10 errors/min
         - series: 'http_requests_total{job="api"}'
           values: '0+100x10'     # 100 req/min → 10% error rate
       alert_rule_test:
         - eval_time: 10m
           alertname: HighErrorRate
           exp_alerts:
             - exp_labels: {severity: critical, job: api}
   

   Write at least two cases per SLO-critical alert: one where it MUST fire, one where it MUST stay silent (transient spike absorbed by for).

3. Routing config passes amtool check-config — Alertmanager routing/inhibition changes are validated before deploy:

   amtool check-config alertmanager.yml
   

If promtool/amtool are unavailable in the environment, state Not run — <tool> unavailable in §8.4 and list the exact commands for the user; never claim rules are validated without running them.

§6 Anti-Examples¶

AE-1: Alert on absolute count instead of rate¶

# WRONG: fires when 10 errors exist (even over 24 hours = normal)
- alert: HighErrorCount
  expr: http_errors_total > 10
# RIGHT: alert on error rate relative to traffic
- alert: HighErrorRate
  expr: rate(http_errors_total[5m]) / rate(http_requests_total[5m]) > 0.01
  for: 5m

AE-2: No for duration — flapping on transient spike¶

# WRONG: fires immediately on any spike, resolves in seconds, pages at 3AM
- alert: HighLatency
  expr: histogram_quantile(0.99, rate(http_duration_seconds_bucket[5m])) > 1
# RIGHT: require sustained condition
- alert: HighLatency
  expr: histogram_quantile(0.99, rate(http_duration_seconds_bucket[5m])) > 1
  for: 5m

AE-3: Alert without runbook — on-call doesn't know what to do¶

# WRONG: no annotations, no runbook
- alert: DatabaseConnectionPoolExhausted
  expr: db_pool_active >= db_pool_max
# RIGHT: include runbook and summary
- alert: DatabaseConnectionPoolExhausted
  expr: db_pool_active >= db_pool_max
  for: 2m
  annotations:
    summary: "DB connection pool exhausted on {{ $labels.instance }}"
    runbook_url: "https://wiki.example.com/runbooks/db-pool-exhausted"

AE-4: user_id in PromQL label — cardinality explosion¶

# WRONG: unique label per user → millions of time series
- record: user_request_duration
  expr: histogram_quantile(0.99, rate(http_duration_seconds_bucket{user_id!=""}[5m]))
# RIGHT: use bounded labels only (method, status_code, endpoint)

AE-5: Critical alert routed to Slack only — no page¶

# WRONG: critical alert goes to Slack where it drowns in messages
route:
  receiver: slack-team
  routes:
    - match: {severity: critical}
      receiver: slack-team  # should be pagerduty!
# RIGHT: critical → PagerDuty, warning → Slack

AE-6: Monitoring gap reported as "system is stable"¶

WRONG: "No alerts fired this month, so the system is healthy"
RIGHT: "No alerts fired — verify alert coverage: are SLIs measured? Are thresholds correct? Absence of alerts ≠ absence of problems"

Extended anti-examples (AE-7 through AE-13) in references/alert-anti-patterns.md.

§7 Monitoring Scorecard¶

Critical — any FAIL means overall FAIL¶

• [ ] SLIs defined and measured for the service (availability, latency, error rate minimum)
• [ ] Every alert is actionable (clear action when it fires, not just "investigate")
• [ ] Alert severity labels match routing (critical → page, warning → Slack, info → dashboard)

Standard — 4 of 5 must pass¶

• [ ] SLOs set with error budget and stakeholder agreement
• [ ] for duration set on all alerts to prevent flapping
• [ ] Runbook link included in alert annotations
• [ ] Dashboard follows USE/RED method with variable templating
• [ ] No high-cardinality labels in alert rules or dashboard queries

Hygiene — 3 of 4 must pass¶

• [ ] Alert grouping and deduplication configured (Alertmanager group_by)
• [ ] Inhibition rules prevent alert cascade from single root cause
• [ ] Alert fatigue metrics tracked (alerts/week, actionability rate)
• [ ] Burn-rate alerting for SLO-critical services

Verdict: X/12; Critical: Y/3; Standard: Z/5; Hygiene: W/4. PASS requires: Critical 3/3 AND Standard ≥4/5 AND Hygiene ≥3/4.

§8 Output Contract¶

Every monitoring review MUST produce these sections. Write "N/A — [reason]" if inapplicable.

### 8.1 Context Gate
| Item | Value | Source |

### 8.2 Depth & Mode
[Lite/Standard/Deep] × [review/design/audit] — [rationale]

### 8.3 SLI/SLO Definition (Standard/Deep)
| SLI | Measurement | SLO Target | Error Budget |

### 8.4 Alert Rules
- Per alert: name, expr, for, severity, summary, runbook_url
- Validation evidence: `promtool check rules` / `promtool test rules` output,
  or `Not run — <tool> unavailable` + exact commands for the user

### 8.5 Dashboard Spec (Standard/Deep)
- Panel layout, queries, variables, drill-down structure

### 8.6 Routing Configuration
- Severity → destination mapping
- Grouping, deduplication, inhibition rules

### 8.7 Alert Fatigue Assessment (audit mode)
- Total alerts/week, actionability %, recommendations

### 8.8 Runbook Mapping
| Alert | Runbook URL | Last Updated |

### 8.9 Uncovered Risks (MANDATORY — never empty)
| Area | Reason | Impact | Follow-up |

Volume rules: - FAIL: always fully detailed - WARN: up to 10; overflow to §8.9 - PASS: summary only - §8.9 minimum: document all assumptions (especially traffic pattern if unknown)

Scorecard summary (append after §8.9):

Scorecard: X/12 — Critical Y/3, Standard Z/5, Hygiene W/4 — PASS/FAIL
Data basis: [full context | degraded | minimal | planning]

§9 Reference Loading Guide¶