Kubernetes on SRE & AI Field Notes

Cloud-Native Chaos Engineering Practice Guide

Thu, 18 Jun 2026 00:00:00 +0000

1. Introduction to Chaos Engineering

Chaos engineering is an experimental methodology for distributed systems that aims to build confidence in a system’s ability to withstand unpredictable production conditions.

2. Chaos Mesh Practice

We built a complete chaos engineering platform based on Chaos Mesh, covering fault injection across network, Pod, and node dimensions.

Core Example: Network Delay Fault Injection

 
 
 
 

 yaml
 
 
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
 name: network-delay
spec:
 action: delay
 mode: one
 selector:
 namespaces: ["production"]
 labelSelectors:
 app: payment-service
 delay:
 latency: "500ms"
 jitter: "100ms"
 duration: "5m"

Kubernetes High Availability and Fault Auto-Healing Practice

Thu, 25 Jun 2026 00:00:00 +0000

1. Control Plane Multi-Active Deployment

Kubernetes control plane high availability is the foundation of all upper-layer service stability. We adopt a three-node multi-active deployment pattern, distributing API Server requests through a load balancer, with the etcd cluster ensuring data consistency via the Raft protocol.

2. etcd Cluster Tuning

etcd is the core of cluster state storage, and its performance directly affects cluster recovery speed. In practice, we deploy etcd data directories on NVMe SSDs, set --quota-backend-bytes to 8GB, and enable client-side compression.

3. Pod Anti-Affinity and Topology Distribution

To avoid single points of failure, we use Pod anti-affinity policies to ensure multiple replicas of the same service are distributed across different nodes and availability zones.

Deployment Anti-Affinity Configuration

 
 
 
 

 yaml
 
 
apiVersion: apps/v1
kind: Deployment
metadata:
 name: api-gateway
spec:
 replicas: 3
 selector:
 matchLabels:
 app: api-gateway
 template:
 metadata:
 labels:
 app: api-gateway
 spec:
 affinity:
 podAntiAffinity:
 preferredDuringSchedulingIgnoredDuringExecution:
 - weight: 100
 podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - api-gateway
 topologyKey: "kubernetes.io/hostname"
 topologySpreadConstraints:
 - maxSkew: 1
 topologyKey: "topology.kubernetes.io/zone"
 whenUnsatisfiable: DoNotSchedule
 labelSelector:
 matchLabels:
 app: api-gateway
 containers:
 - name: api-gateway
 image: api-gateway:latest
 resources:
 requests:
 cpu: "500m"
 memory: "512Mi"
 limits:
 cpu: "2"
 memory: "2Gi"
 readinessProbe:
 httpGet:
 path: /health
 port: 8080

Custom Self-Healing Controller

 
 
 
 

 go
 
 
package controller

import (
 "context"
 "log"

 corev1 "k8s.io/api/core/v1"
 "sigs.k8s.io/controller-runtime/pkg/client"
 "sigs.k8s.io/controller-runtime/pkg/reconcile"
)

type PodReconciler struct {
 client.Client
}

func (r *PodReconciler) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {
 var pod corev1.Pod
 if err := r.Get(ctx, req.NamespacedName, &pod); err != nil {
 return reconcile.Result{}, client.IgnoreNotFound(err)
 }

 if pod.Status.Phase == corev1.PodPending {
 for _, cond := range pod.Status.Conditions {
 if cond.Reason == "Unschedulable" {
 log.Printf("Detected unschedulable Pod: %s/%s, attempting eviction", pod.Namespace, pod.Name)
 }
 }
 }
 return reconcile.Result{}, nil
}

4. Summary

High availability is not a one-time configuration but a continuous evolution process. Through the combination of control plane multi-active, etcd tuning, anti-affinity policies, and self-healing Controllers, we reduced the cluster’s annualized failure time from 12 hours to under 30 minutes.

GitOps Workflow Best Practices for Kubernetes Cluster Management

Sat, 25 Jul 2026 00:00:00 +0000

1. GitOps Overview

GitOps is an operational model that uses a Git repository as the single source of truth, leveraging declarative configuration and automated sync mechanisms to ensure that the cluster state always matches the desired state in the repository. In Kubernetes environments, GitOps has become the de facto standard for managing multi-cluster, multi-environment deployments.

2. ArgoCD Best Practices

Using ApplicationSet for multi-cluster deployment management is recommended, combined with Kustomize for environment-specific configuration differentiation. ArgoCD’s automated sync, self-healing, and drift detection capabilities significantly reduce operational overhead.

ArgoCD Application YAML

 
 
 
 

 yaml
 
 
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: production-app
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://github.com/team/config-repo.git
 targetRevision: main
 path: overlays/production
 destination:
 server: https://kubernetes.default.svc
 namespace: production
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true

CI/CD Shell Script

 
 
 
 

 bash
 
 
#!/bin/bash
# GitOps CI/CD Pipeline
set -euo pipefail

REPO_URL="https://github.com/team/app-repo.git"
CONFIG_REPO="https://github.com/team/config-repo.git"
COMMIT_SHA="${CI_COMMIT_SHA:?commit sha required}"

# Build image
docker build -t app:"${COMMIT_SHA}" .

# Push image
docker tag app:"${COMMIT_SHA}" registry.example.com/app:"${COMMIT_SHA}"
docker push registry.example.com/app:"${COMMIT_SHA}"

# Update image version in config repo
git clone "${CONFIG_REPO}" config-tmp
cd config-tmp
sed -i "s|image: app:.*|image: registry.example.com/app:${COMMIT_SHA}|" overlays/production/deployment.yaml
git add -A
git commit -m "chore: bump app image to ${COMMIT_SHA}"
git push origin main

3. Summary

GitOps brings deployment configurations under version control, enabling auditable and revertible release processes. Combined with the automation capabilities of tools like ArgoCD, teams can increase deployment frequency multiple times over while reducing the risk of human error.