Posts on SRE & AI Field Notes

Deep Learning in SRE Intelligent Operations Practice

Mon, 20 Jul 2026 00:00:00 +0000

1. Pain Points of Traditional SRE Operations and the AI Introduction

In modern large-scale distributed systems, SRE teams face thousands of monitoring metrics and massive amounts of log data every day. Traditional alerting rules rely on manually set static thresholds, which can easily cause “alert storms” or missed detections.

2. Time Series-Based Anomaly Detection Model

For the periodicity and burstiness of KPI time series, we adopted a hybrid neural network model combining Transformer and LSTM.

Core Code: LSTM-Based Anomaly Detection

 
 python
 
import torch
import torch.nn as nn

class AnomalyDetector(nn.Module):
 def __init__(self, input_dim, hidden_dim, num_layers):
 super().__init__()
 self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)
 self.fc = nn.Linear(hidden_dim, 1)

 def forward(self, x):
 lstm_out, _ = self.lstm(x)
 last_hidden = lstm_out[:, -1, :]
 return torch.sigmoid(self.fc(last_hidden))

model = AnomalyDetector(input_dim=64, hidden_dim=128, num_layers=2)
threshold = 0.85

3. Summary and Future Outlook

Introducing AI technology is not about replacing SREs, but about freeing engineers from heavy repetitive on-call labor.

Go Microservices Architecture Practice and Performance Tuning

Fri, 15 May 2026 00:00:00 +0000

1. Evolution of Microservice Architecture

Migrating from monolithic applications to microservice architecture has been a hot topic in recent years. Go, with its lightweight goroutine model and excellent concurrency support, has become a popular choice for building microservices.

2. API Contract Design

In microservice architecture, APIs are the core of inter-service communication. We use Protocol Buffers for interface definition.

Core Code: gRPC Service Definition

 
 
 
 

 go
 
 
package service

import (
 "context"
 "log"
 "net"

 "google.golang.org/grpc"
 pb "github.com/example/proto"
)

type UserServer struct {
 pb.UnimplementedUserServiceServer
}

func (s *UserServer) GetUser(ctx context.Context, req *pb.GetUserRequest) (*pb.User, error) {
 return &pb.User{
 Id: req.Id,
 Name: "alice",
 Email: "alice@example.com",
 }, nil
}

func main() {
 lis, _ := net.Listen("tcp", ":8080")
 srv := grpc.NewServer()
 pb.RegisterUserServiceServer(srv, &UserServer{})
 log.Fatal(srv.Serve(lis))
}

Cloud-Native Chaos Engineering Practice Guide

Thu, 18 Jun 2026 00:00:00 +0000

1. Introduction to Chaos Engineering

Chaos engineering is an experimental methodology for distributed systems that aims to build confidence in a system’s ability to withstand unpredictable production conditions.

2. Chaos Mesh Practice

We built a complete chaos engineering platform based on Chaos Mesh, covering fault injection across network, Pod, and node dimensions.

Core Example: Network Delay Fault Injection

 
 
 
 

 yaml
 
 
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
 name: network-delay
spec:
 action: delay
 mode: one
 selector:
 namespaces: ["production"]
 labelSelectors:
 app: payment-service
 delay:
 latency: "500ms"
 jitter: "100ms"
 duration: "5m"

Building Millisecond-Level Observability with Prometheus + eBPF

Fri, 10 Jul 2026 00:00:00 +0000

1. Why eBPF Observability

Traditional observability solutions rely on application instrumentation and sidecar proxies, introducing significant performance overhead and blind spots. eBPF allows safe execution of sandboxed programs within the kernel, capturing system calls, network packets, and file operations without modifying application code or introducing additional latency.

2. Three-Layer Monitoring with eBPF

We split observability into network, storage, and application layers. The network layer tracks TCP connection latency and retransmission rates via eBPF; the storage layer monitors file I/O and page cache hit rates; the application layer uses eBPF uprobes to capture key function call chains.

3. Prometheus Federation Architecture

For multi-cluster scenarios, we adopted the Prometheus federation model: each cluster runs an independent Prometheus Server collecting eBPF-exported metrics, with a central Prometheus aggregating them for a global cross-cluster view.

Kubernetes High Availability and Fault Auto-Healing Practice

Thu, 25 Jun 2026 00:00:00 +0000

1. Control Plane Multi-Active Deployment

Kubernetes control plane high availability is the foundation of all upper-layer service stability. We adopt a three-node multi-active deployment pattern, distributing API Server requests through a load balancer, with the etcd cluster ensuring data consistency via the Raft protocol.

2. etcd Cluster Tuning

etcd is the core of cluster state storage, and its performance directly affects cluster recovery speed. In practice, we deploy etcd data directories on NVMe SSDs, set --quota-backend-bytes to 8GB, and enable client-side compression.

3. Pod Anti-Affinity and Topology Distribution

To avoid single points of failure, we use Pod anti-affinity policies to ensure multiple replicas of the same service are distributed across different nodes and availability zones.

Deployment Anti-Affinity Configuration

 
 
 
 

 yaml
 
 
apiVersion: apps/v1
kind: Deployment
metadata:
 name: api-gateway
spec:
 replicas: 3
 selector:
 matchLabels:
 app: api-gateway
 template:
 metadata:
 labels:
 app: api-gateway
 spec:
 affinity:
 podAntiAffinity:
 preferredDuringSchedulingIgnoredDuringExecution:
 - weight: 100
 podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - api-gateway
 topologyKey: "kubernetes.io/hostname"
 topologySpreadConstraints:
 - maxSkew: 1
 topologyKey: "topology.kubernetes.io/zone"
 whenUnsatisfiable: DoNotSchedule
 labelSelector:
 matchLabels:
 app: api-gateway
 containers:
 - name: api-gateway
 image: api-gateway:latest
 resources:
 requests:
 cpu: "500m"
 memory: "512Mi"
 limits:
 cpu: "2"
 memory: "2Gi"
 readinessProbe:
 httpGet:
 path: /health
 port: 8080

Custom Self-Healing Controller

 
 
 
 

 go
 
 
package controller

import (
 "context"
 "log"

 corev1 "k8s.io/api/core/v1"
 "sigs.k8s.io/controller-runtime/pkg/client"
 "sigs.k8s.io/controller-runtime/pkg/reconcile"
)

type PodReconciler struct {
 client.Client
}

func (r *PodReconciler) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {
 var pod corev1.Pod
 if err := r.Get(ctx, req.NamespacedName, &pod); err != nil {
 return reconcile.Result{}, client.IgnoreNotFound(err)
 }

 if pod.Status.Phase == corev1.PodPending {
 for _, cond := range pod.Status.Conditions {
 if cond.Reason == "Unschedulable" {
 log.Printf("Detected unschedulable Pod: %s/%s, attempting eviction", pod.Namespace, pod.Name)
 }
 }
 }
 return reconcile.Result{}, nil
}

4. Summary

High availability is not a one-time configuration but a continuous evolution process. Through the combination of control plane multi-active, etcd tuning, anti-affinity policies, and self-healing Controllers, we reduced the cluster’s annualized failure time from 12 hours to under 30 minutes.

Building Enterprise Intelligent Operations Knowledge Base with RAG

Wed, 15 Jul 2026 00:00:00 +0000

1. The Pain of Operations Knowledge Management

In large organizations, operations knowledge is often scattered across Wikis, Runbooks, Slack records, and incident reports. On-call engineers must navigate multiple systems to locate critical information. RAG technology offers a new approach to solving this problem.

2. System Architecture

We built a complete RAG pipeline based on the LangChain framework: documents are parsed, split, and stored in Chroma vector database; when an engineer asks a question, the system first retrieves the most relevant document fragments, then generates an answer via an LLM.

3. Key Technologies and Implementation

We chose text-embedding-3-small as the embedding model, with a chunk size of 512 tokens and 64 token overlap. The retrieval uses a hybrid strategy: vector similarity + BM25 keyword weighting.

LLM Infrastructure Setup and GPU Cluster Performance Tuning

Wed, 22 Jul 2026 00:00:00 +0000

1. Inference Infrastructure Selection

When deploying large language model inference services, performance and cost are the primary considerations. vLLM, with its PagedAttention memory management mechanism, achieves 2-4x throughput improvement over traditional solutions, making it our inference engine of choice.

2. vLLM Distributed Deployment

For models with over 70B parameters, single-GPU memory is insufficient. We use Tensor Parallelism to split the model across 4 A100-80G GPUs, with Ray for elastic scaling.

3. GPU Memory Optimization

Beyond vLLM’s own optimizations, we enabled FlashAttention-2, quantization inference (FP8/INT4), and Continuous Batching, increasing per-GPU throughput from 1200 tokens/s to 4800 tokens/s.

vLLM Deployment Script

 
 
 
 

 python
 
 
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

model_name = "meta-llama/Llama-3.1-70B-Instruct"

llm = LLM(
 model=model_name,
 tensor_parallel_size=4,
 gpu_memory_utilization=0.90,
 max_model_len=8192,
 enable_prefix_caching=True,
 enforce_eager=False,
)

tokenizer = AutoTokenizer.from_pretrained(model_name)

sampling_params = SamplingParams(
 temperature=0.7,
 top_p=0.9,
 max_tokens=2048,
)

prompts = ["Explain distributed systems architecture in detail."]
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
 print(output.outputs[0].text)

GPU Monitoring Script

 
 
 
 

 bash
 
 
#!/bin/bash
INTERVAL=5

echo "Timestamp,GPU_ID,GPU_Util,Memory_Used(MB),Memory_Free(MB),Temperature(C),Power(W)"

while true; do
 TIMESTAMP=$(date +%Y-%m-%dT%H:%M:%S)
 nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.free,temperature.gpu,power.draw \
 --format=csv,noheader,nounits \
 | while IFS=, read -r gpu_id util mem_used mem_free temp power; do
 echo "$TIMESTAMP,$gpu_id,$util,$mem_used,$mem_free,$temp,$power"
 done
 sleep $INTERVAL
done

4. NVIDIA MIG Partitioning

For smaller models or development environments, we use MIG to split A100 GPUs into multiple independent instances, improving GPU utilization and tenant isolation.

Architecture Evolution Reflections for 2026

Thu, 23 Jul 2026 00:00:00 +0000

1. From Cloud-Native to AI-Native

The most significant change from 2024 to 2026 has been the full shift from “Cloud-Native” to “AI-Native.” Kubernetes is no longer just a microservice orchestration platform but has become the core infrastructure for GPU resource scheduling, model service deployment, and inference pipeline orchestration.

2. From Microservices to Agent Collaboration

Traditional microservice architecture centers on API contracts, emphasizing communication through well-defined interfaces. With the maturation of LLMs, a new architectural pattern is emerging: Agent collaboration – each service is no longer a passive “endpoint” waiting for requests, but an “intelligent agent” with autonomous decision-making capabilities.

3. The Reshaping of the SRE Role

AI is not only changing architecture but also reshaping the daily work of SREs. Alert analysis, fault diagnosis, and capacity planning are being progressively taken over by AI-assisted tools. New skills SREs need to master include Prompt Engineering, RAG knowledge base construction, and LLM inference tuning.

GitOps Workflow Best Practices for Kubernetes Cluster Management

Sat, 25 Jul 2026 00:00:00 +0000

1. GitOps Overview

GitOps is an operational model that uses a Git repository as the single source of truth, leveraging declarative configuration and automated sync mechanisms to ensure that the cluster state always matches the desired state in the repository. In Kubernetes environments, GitOps has become the de facto standard for managing multi-cluster, multi-environment deployments.

2. ArgoCD Best Practices

Using ApplicationSet for multi-cluster deployment management is recommended, combined with Kustomize for environment-specific configuration differentiation. ArgoCD’s automated sync, self-healing, and drift detection capabilities significantly reduce operational overhead.

ArgoCD Application YAML

 
 
 
 

 yaml
 
 
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: production-app
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://github.com/team/config-repo.git
 targetRevision: main
 path: overlays/production
 destination:
 server: https://kubernetes.default.svc
 namespace: production
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true

CI/CD Shell Script

 
 
 
 

 bash
 
 
#!/bin/bash
# GitOps CI/CD Pipeline
set -euo pipefail

REPO_URL="https://github.com/team/app-repo.git"
CONFIG_REPO="https://github.com/team/config-repo.git"
COMMIT_SHA="${CI_COMMIT_SHA:?commit sha required}"

# Build image
docker build -t app:"${COMMIT_SHA}" .

# Push image
docker tag app:"${COMMIT_SHA}" registry.example.com/app:"${COMMIT_SHA}"
docker push registry.example.com/app:"${COMMIT_SHA}"

# Update image version in config repo
git clone "${CONFIG_REPO}" config-tmp
cd config-tmp
sed -i "s|image: app:.*|image: registry.example.com/app:${COMMIT_SHA}|" overlays/production/deployment.yaml
git add -A
git commit -m "chore: bump app image to ${COMMIT_SHA}"
git push origin main

3. Summary

GitOps brings deployment configurations under version control, enabling auditable and revertible release processes. Combined with the automation capabilities of tools like ArgoCD, teams can increase deployment frequency multiple times over while reducing the risk of human error.

Python Async Patterns: Deep Dive into Coroutines and Event Loops

Sat, 25 Jul 2026 00:00:00 +0000

1. Async Programming Basics

Python’s asyncio library provides an event-loop-based coroutine concurrency model, ideal for I/O-bound and high-concurrency applications. Understanding coroutines, the event loop, and task scheduling is key to mastering asynchronous programming.

2. Core Example

The following example demonstrates writing concurrent API requests using async/await, and executing multiple coroutines in parallel with asyncio.gather.

Python Async Example

 
 
 
 

 python
 
 
import asyncio
import aiohttp
from typing import List

async def fetch_url(session: aiohttp.ClientSession, url: str) -> dict:
 """Fetch URL content asynchronously"""
 async with session.get(url) as response:
 return await response.json()

async def fetch_all(urls: List[str]) -> List[dict]:
 """Fetch all URLs concurrently"""
 async with aiohttp.ClientSession() as session:
 tasks = [fetch_url(session, url) for url in urls]
 return await asyncio.gather(*tasks, return_exceptions=True)

async def main():
 urls = [
 "https://api.example.com/users",
 "https://api.example.com/posts",
 "https://api.example.com/comments",
 ]
 results = await fetch_all(urls)
 for i, result in enumerate(results):
 status = "Success" if not isinstance(result, Exception) else "Failed"
 print(f"Request {i}: {status}")

if __name__ == "__main__":
 asyncio.run(main())

Run Script

 
 bash
 
#!/bin/bash
# Async program runner
python3 -m pip install aiohttp # Install dependencies
python3 async_fetch.py # Run async program

3. Best Practices

Use asyncio.create_task for background tasks, asyncio.Semaphore to control concurrency, and asyncio.timeout for timeout management. Avoid CPU-bound operations inside coroutines; use concurrent.futures for parallel computation instead.

MLOps Pipeline in Practice: Automating from Model Training to Deployment

Sat, 25 Jul 2026 00:00:00 +0000

1. MLOps Pipeline Overview

MLOps brings DevOps principles to the machine learning lifecycle, automating model training, evaluation, deployment, and monitoring. A well-designed MLOps pipeline can significantly shorten the time from experimentation to production deployment.

2. Training Pipeline Implementation

Below is a training pipeline implementation in Python, covering the complete workflow of data loading, model training, and evaluation.

Python Training Pipeline

 
 
 
 

 python
 
 
import mlflow
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import accuracy_score, precision_score

def train_pipeline(data_path: str) -> float:
 """End-to-end training pipeline"""
 mlflow.set_experiment("production-model")

 with mlflow.start_run():
 # Data loading
 df = pd.read_parquet(data_path)
 X = df.drop("target", axis=1)
 y = df["target"]

 # Train-test split
 X_train, X_test, y_train, y_test = train_test_split(
 X, y, test_size=0.2, random_state=42
 )

 # Model training
 model = GradientBoostingClassifier(
 n_estimators=300,
 max_depth=6,
 learning_rate=0.1,
 )
 model.fit(X_train, y_train)

 # Model evaluation
 y_pred = model.predict(X_test)
 acc = accuracy_score(y_test, y_pred)
 prec = precision_score(y_test, y_pred, average="weighted")

 mlflow.log_metrics({"accuracy": acc, "precision": prec})
 mlflow.sklearn.log_model(model, "model")

 return acc

if __name__ == "__main__":
 accuracy = train_pipeline("s3://data/features/train.parquet")
 print(f"Model accuracy: {accuracy:.4f}")

Kubernetes Deployment Configuration

 
 
 
 

 yaml
 
 
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: production-model
spec:
 predictor:
 sklearn:
 storageUri: s3://mlflow-artifacts/1/model
 resources:
 requests:
 cpu: "1"
 memory: 2Gi
 limits:
 cpu: "2"
 memory: 4Gi

3. Summary

The core of an MLOps pipeline lies in automation and reproducibility. By standardizing the training, evaluation, and deployment processes, teams can iterate on models more rapidly while maintaining production stability.

Distributed Database Sharding Strategies and Consistency Guarantees

Sat, 25 Jul 2026 00:00:00 +0000

1. Sharding Strategies

When a single database reaches the terabyte scale, sharding becomes inevitable. Common partitioning strategies include hash sharding, range sharding, and consistent hashing. Choosing the right shard key is the core of the design.

2. Go Sharding Implementation

The following example demonstrates a consistent hash sharding router implemented in Go, responsible for routing requests to the correct database shard.

Go Sharding Logic

 
 
 
 

 go
 
 
package shard

import (
	"crypto/sha256"
	"fmt"
	"hash/crc32"
)

type Router struct {
	shards []string
}

func NewRouter(nodes []string) *Router {
	return &Router{shards: nodes}
}

// ShardKey calculates the target shard based on user ID
func (r *Router) ShardKey(userID string) string {
	h := crc32.ChecksumIEEE([]byte(userID))
	idx := int(h) % len(r.shards)
	return r.shards[idx]
}

// ShardKeyConsistent uses consistent hashing
func (r *Router) ShardKeyConsistent(userID string) string {
	hash := sha256.Sum256([]byte(userID))
	value := int(hash[0])<<24 | int(hash[1])<<16 |
		int(hash[2])<<8 | int(hash[3])
	idx := value % len(r.shards)
	// Virtual node support (simplified)
	return r.shards[idx]
}

func main() {
	nodes := []string{"shard-0", "shard-1", "shard-2", "shard-3"}
	router := NewRouter(nodes)

	userIDs := []string{"user_1001", "user_1002", "user_1003"}
	for _, uid := range userIDs {
		shard := router.ShardKeyConsistent(uid)
		fmt.Printf("User %s -> %s\n", uid, shard)
	}
}

SQL Query Example

 
 
 
 

 sql
 
 
-- Create shard table
CREATE TABLE IF NOT EXISTS orders_0000 (
 id BIGINT PRIMARY KEY,
 user_id VARCHAR(64) NOT NULL,
 amount DECIMAL(12,2) NOT NULL,
 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
 INDEX idx_user_id (user_id)
);

-- Cross-shard query (results merged at application layer)
-- The query runs independently on each shard
SELECT user_id, SUM(amount) as total
FROM orders_0000
WHERE created_at >= '2026-01-01'
GROUP BY user_id;

3. Consistency Guarantees

For distributed transactions, TCC or Saga patterns are recommended, combined with eventual consistency compensation mechanisms. For read-heavy scenarios, use read-write separation and cache warming strategies to improve performance.

Building an Effective On-Call System: From Postmortems to Proactive Defense

Sat, 25 Jul 2026 00:00:00 +0000

1. On-Call System Design

An effective On-Call system goes beyond merely scheduling shifts. It requires well-designed alert severity levels, escalation policies, and incident response workflows. The goal is to reduce alert fatigue while ensuring critical incidents are never missed.

2. PagerDuty Configuration and Automation

Below is a sample PagerDuty service configuration and its accompanying automation script.

PagerDuty Service Configuration

 
 
 
 

 yaml
 
 
# PagerDuty service and escalation policy configuration
service:
 name: "production-critical"
 escalation_policy: "sre-primary"

 alert_grouping:
 type: "time"
 timeout: 300 # Group similar alerts within 5 minutes

 auto_resolve_timeout: 600 # Auto-resolve after 10 minutes

 urgency_rule:
 type: "use_support_hours"
 support_hours:
 timezone: "Asia/Shanghai"
 start_time: "09:00"
 end_time: "18:00"
 days_of_week: [1, 2, 3, 4, 5]

 escalation_rules:
 - targets: ["on-call-primary"]
 escalation_timeout: 15 # 15 minutes to acknowledge
 - targets: ["on-call-secondary"]
 escalation_timeout: 10
 - targets: ["sre-manager"]
 escalation_timeout: 5

On-Call Automation Shell Script

 
 
 
 

 bash
 
 
#!/bin/bash
# On-Call Handover Check Script
set -euo pipefail

echo "=== On-Call Handover Checklist ==="

# Check pending alerts
ALERTS=$(curl -s -H "Authorization: Token ${PD_API_TOKEN}" \
 "https://api.pagerduty.com/incidents?statuses[]=triggered&statuses[]=acknowledged" \
 | jq '.incidents | length')

echo "Pending alerts: ${ALERTS}"

if [ "${ALERTS}" -gt 0 ]; then
 echo "Warning: There are unresolved alerts. Please handle before handover!"
 exit 1
fi

# Check ongoing changes
echo "Checking ongoing changes..."
kubectl get deployments --all-namespaces \
 -o json | jq -r '.items[] | select(.metadata.annotations["change/in-progress"]=="true") | .metadata.namespace + "/" + .metadata.name' \
 || echo "No ongoing changes"

echo "=== Handover check complete ==="

3. Postmortems and Proactive Defense

Every incident should be followed by a blameless postmortem documenting the timeline, root cause, and improvement actions. By establishing chaos engineering experiments and continuous performance load testing, shift On-Call from reactive firefighting to proactive defense.