SRE on SRE & AI Field Notes

Deep Learning in SRE Intelligent Operations Practice

Mon, 20 Jul 2026 00:00:00 +0000

1. Pain Points of Traditional SRE Operations and the AI Introduction

In modern large-scale distributed systems, SRE teams face thousands of monitoring metrics and massive amounts of log data every day. Traditional alerting rules rely on manually set static thresholds, which can easily cause “alert storms” or missed detections.

2. Time Series-Based Anomaly Detection Model

For the periodicity and burstiness of KPI time series, we adopted a hybrid neural network model combining Transformer and LSTM.

Core Code: LSTM-Based Anomaly Detection

 
 python
 
import torch
import torch.nn as nn

class AnomalyDetector(nn.Module):
 def __init__(self, input_dim, hidden_dim, num_layers):
 super().__init__()
 self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)
 self.fc = nn.Linear(hidden_dim, 1)

 def forward(self, x):
 lstm_out, _ = self.lstm(x)
 last_hidden = lstm_out[:, -1, :]
 return torch.sigmoid(self.fc(last_hidden))

model = AnomalyDetector(input_dim=64, hidden_dim=128, num_layers=2)
threshold = 0.85

3. Summary and Future Outlook

Introducing AI technology is not about replacing SREs, but about freeing engineers from heavy repetitive on-call labor.

Cloud-Native Chaos Engineering Practice Guide

Thu, 18 Jun 2026 00:00:00 +0000

1. Introduction to Chaos Engineering

Chaos engineering is an experimental methodology for distributed systems that aims to build confidence in a system’s ability to withstand unpredictable production conditions.

2. Chaos Mesh Practice

We built a complete chaos engineering platform based on Chaos Mesh, covering fault injection across network, Pod, and node dimensions.

Core Example: Network Delay Fault Injection

 
 
 
 

 yaml
 
 
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
 name: network-delay
spec:
 action: delay
 mode: one
 selector:
 namespaces: ["production"]
 labelSelectors:
 app: payment-service
 delay:
 latency: "500ms"
 jitter: "100ms"
 duration: "5m"

Kubernetes High Availability and Fault Auto-Healing Practice

Thu, 25 Jun 2026 00:00:00 +0000

1. Control Plane Multi-Active Deployment

Kubernetes control plane high availability is the foundation of all upper-layer service stability. We adopt a three-node multi-active deployment pattern, distributing API Server requests through a load balancer, with the etcd cluster ensuring data consistency via the Raft protocol.

2. etcd Cluster Tuning

etcd is the core of cluster state storage, and its performance directly affects cluster recovery speed. In practice, we deploy etcd data directories on NVMe SSDs, set --quota-backend-bytes to 8GB, and enable client-side compression.

3. Pod Anti-Affinity and Topology Distribution

To avoid single points of failure, we use Pod anti-affinity policies to ensure multiple replicas of the same service are distributed across different nodes and availability zones.

Deployment Anti-Affinity Configuration

 
 
 
 

 yaml
 
 
apiVersion: apps/v1
kind: Deployment
metadata:
 name: api-gateway
spec:
 replicas: 3
 selector:
 matchLabels:
 app: api-gateway
 template:
 metadata:
 labels:
 app: api-gateway
 spec:
 affinity:
 podAntiAffinity:
 preferredDuringSchedulingIgnoredDuringExecution:
 - weight: 100
 podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - api-gateway
 topologyKey: "kubernetes.io/hostname"
 topologySpreadConstraints:
 - maxSkew: 1
 topologyKey: "topology.kubernetes.io/zone"
 whenUnsatisfiable: DoNotSchedule
 labelSelector:
 matchLabels:
 app: api-gateway
 containers:
 - name: api-gateway
 image: api-gateway:latest
 resources:
 requests:
 cpu: "500m"
 memory: "512Mi"
 limits:
 cpu: "2"
 memory: "2Gi"
 readinessProbe:
 httpGet:
 path: /health
 port: 8080

Custom Self-Healing Controller

 
 
 
 

 go
 
 
package controller

import (
 "context"
 "log"

 corev1 "k8s.io/api/core/v1"
 "sigs.k8s.io/controller-runtime/pkg/client"
 "sigs.k8s.io/controller-runtime/pkg/reconcile"
)

type PodReconciler struct {
 client.Client
}

func (r *PodReconciler) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {
 var pod corev1.Pod
 if err := r.Get(ctx, req.NamespacedName, &pod); err != nil {
 return reconcile.Result{}, client.IgnoreNotFound(err)
 }

 if pod.Status.Phase == corev1.PodPending {
 for _, cond := range pod.Status.Conditions {
 if cond.Reason == "Unschedulable" {
 log.Printf("Detected unschedulable Pod: %s/%s, attempting eviction", pod.Namespace, pod.Name)
 }
 }
 }
 return reconcile.Result{}, nil
}

4. Summary

High availability is not a one-time configuration but a continuous evolution process. Through the combination of control plane multi-active, etcd tuning, anti-affinity policies, and self-healing Controllers, we reduced the cluster’s annualized failure time from 12 hours to under 30 minutes.

Building an Effective On-Call System: From Postmortems to Proactive Defense

Sat, 25 Jul 2026 00:00:00 +0000

1. On-Call System Design

An effective On-Call system goes beyond merely scheduling shifts. It requires well-designed alert severity levels, escalation policies, and incident response workflows. The goal is to reduce alert fatigue while ensuring critical incidents are never missed.

2. PagerDuty Configuration and Automation

Below is a sample PagerDuty service configuration and its accompanying automation script.

PagerDuty Service Configuration

 
 
 
 

 yaml
 
 
# PagerDuty service and escalation policy configuration
service:
 name: "production-critical"
 escalation_policy: "sre-primary"

 alert_grouping:
 type: "time"
 timeout: 300 # Group similar alerts within 5 minutes

 auto_resolve_timeout: 600 # Auto-resolve after 10 minutes

 urgency_rule:
 type: "use_support_hours"
 support_hours:
 timezone: "Asia/Shanghai"
 start_time: "09:00"
 end_time: "18:00"
 days_of_week: [1, 2, 3, 4, 5]

 escalation_rules:
 - targets: ["on-call-primary"]
 escalation_timeout: 15 # 15 minutes to acknowledge
 - targets: ["on-call-secondary"]
 escalation_timeout: 10
 - targets: ["sre-manager"]
 escalation_timeout: 5

On-Call Automation Shell Script

 
 
 
 

 bash
 
 
#!/bin/bash
# On-Call Handover Check Script
set -euo pipefail

echo "=== On-Call Handover Checklist ==="

# Check pending alerts
ALERTS=$(curl -s -H "Authorization: Token ${PD_API_TOKEN}" \
 "https://api.pagerduty.com/incidents?statuses[]=triggered&statuses[]=acknowledged" \
 | jq '.incidents | length')

echo "Pending alerts: ${ALERTS}"

if [ "${ALERTS}" -gt 0 ]; then
 echo "Warning: There are unresolved alerts. Please handle before handover!"
 exit 1
fi

# Check ongoing changes
echo "Checking ongoing changes..."
kubectl get deployments --all-namespaces \
 -o json | jq -r '.items[] | select(.metadata.annotations["change/in-progress"]=="true") | .metadata.namespace + "/" + .metadata.name' \
 || echo "No ongoing changes"

echo "=== Handover check complete ==="

3. Postmortems and Proactive Defense

Every incident should be followed by a blameless postmortem documenting the timeline, root cause, and improvement actions. By establishing chaos engineering experiments and continuous performance load testing, shift On-Call from reactive firefighting to proactive defense.