Kubernetes High Availability and Fault Auto-Healing Practice

Thu, 25 Jun 2026 00:00:00 +0000

1. Control Plane Multi-Active Deployment

Kubernetes control plane high availability is the foundation of all upper-layer service stability. We adopt a three-node multi-active deployment pattern, distributing API Server requests through a load balancer, with the etcd cluster ensuring data consistency via the Raft protocol.

2. etcd Cluster Tuning

etcd is the core of cluster state storage, and its performance directly affects cluster recovery speed. In practice, we deploy etcd data directories on NVMe SSDs, set --quota-backend-bytes to 8GB, and enable client-side compression.

3. Pod Anti-Affinity and Topology Distribution

To avoid single points of failure, we use Pod anti-affinity policies to ensure multiple replicas of the same service are distributed across different nodes and availability zones.

Deployment Anti-Affinity Configuration

 
 
 
 

 yaml
 
 
apiVersion: apps/v1
kind: Deployment
metadata:
 name: api-gateway
spec:
 replicas: 3
 selector:
 matchLabels:
 app: api-gateway
 template:
 metadata:
 labels:
 app: api-gateway
 spec:
 affinity:
 podAntiAffinity:
 preferredDuringSchedulingIgnoredDuringExecution:
 - weight: 100
 podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - api-gateway
 topologyKey: "kubernetes.io/hostname"
 topologySpreadConstraints:
 - maxSkew: 1
 topologyKey: "topology.kubernetes.io/zone"
 whenUnsatisfiable: DoNotSchedule
 labelSelector:
 matchLabels:
 app: api-gateway
 containers:
 - name: api-gateway
 image: api-gateway:latest
 resources:
 requests:
 cpu: "500m"
 memory: "512Mi"
 limits:
 cpu: "2"
 memory: "2Gi"
 readinessProbe:
 httpGet:
 path: /health
 port: 8080

Custom Self-Healing Controller

 
 
 
 

 go
 
 
package controller

import (
 "context"
 "log"

 corev1 "k8s.io/api/core/v1"
 "sigs.k8s.io/controller-runtime/pkg/client"
 "sigs.k8s.io/controller-runtime/pkg/reconcile"
)

type PodReconciler struct {
 client.Client
}

func (r *PodReconciler) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {
 var pod corev1.Pod
 if err := r.Get(ctx, req.NamespacedName, &pod); err != nil {
 return reconcile.Result{}, client.IgnoreNotFound(err)
 }

 if pod.Status.Phase == corev1.PodPending {
 for _, cond := range pod.Status.Conditions {
 if cond.Reason == "Unschedulable" {
 log.Printf("Detected unschedulable Pod: %s/%s, attempting eviction", pod.Namespace, pod.Name)
 }
 }
 }
 return reconcile.Result{}, nil
}

4. Summary

High availability is not a one-time configuration but a continuous evolution process. Through the combination of control plane multi-active, etcd tuning, anti-affinity policies, and self-healing Controllers, we reduced the cluster’s annualized failure time from 12 hours to under 30 minutes.