Building an Effective On-Call System: From Postmortems to Proactive Defense

1. On-Call System Design

An effective On-Call system goes beyond merely scheduling shifts. It requires well-designed alert severity levels, escalation policies, and incident response workflows. The goal is to reduce alert fatigue while ensuring critical incidents are never missed.

2. PagerDuty Configuration and Automation

Below is a sample PagerDuty service configuration and its accompanying automation script.

PagerDuty Service Configuration

yaml
# PagerDuty service and escalation policy configuration
service:
  name: "production-critical"
  escalation_policy: "sre-primary"

  alert_grouping:
    type: "time"
    timeout: 300  # Group similar alerts within 5 minutes

  auto_resolve_timeout: 600  # Auto-resolve after 10 minutes

  urgency_rule:
    type: "use_support_hours"
    support_hours:
      timezone: "Asia/Shanghai"
      start_time: "09:00"
      end_time: "18:00"
      days_of_week: [1, 2, 3, 4, 5]

  escalation_rules:
    - targets: ["on-call-primary"]
      escalation_timeout: 15  # 15 minutes to acknowledge
    - targets: ["on-call-secondary"]
      escalation_timeout: 10
    - targets: ["sre-manager"]
      escalation_timeout: 5

On-Call Automation Shell Script

bash
#!/bin/bash
# On-Call Handover Check Script
set -euo pipefail

echo "=== On-Call Handover Checklist ==="

# Check pending alerts
ALERTS=$(curl -s -H "Authorization: Token ${PD_API_TOKEN}" \
  "https://api.pagerduty.com/incidents?statuses[]=triggered&statuses[]=acknowledged" \
  | jq '.incidents | length')

echo "Pending alerts: ${ALERTS}"

if [ "${ALERTS}" -gt 0 ]; then
  echo "Warning: There are unresolved alerts. Please handle before handover!"
  exit 1
fi

# Check ongoing changes
echo "Checking ongoing changes..."
kubectl get deployments --all-namespaces \
  -o json | jq -r '.items[] | select(.metadata.annotations["change/in-progress"]=="true") | .metadata.namespace + "/" + .metadata.name' \
  || echo "No ongoing changes"

echo "=== Handover check complete ==="

3. Postmortems and Proactive Defense

Every incident should be followed by a blameless postmortem documenting the timeline, root cause, and improvement actions. By establishing chaos engineering experiments and continuous performance load testing, shift On-Call from reactive firefighting to proactive defense.