Kubernetes High Availability and Fault Auto-Healing Practice

1. Control Plane Multi-Active Deployment

Kubernetes control plane high availability is the foundation of all upper-layer service stability. We adopt a three-node multi-active deployment pattern, distributing API Server requests through a load balancer, with the etcd cluster ensuring data consistency via the Raft protocol.

2. etcd Cluster Tuning

etcd is the core of cluster state storage, and its performance directly affects cluster recovery speed. In practice, we deploy etcd data directories on NVMe SSDs, set --quota-backend-bytes to 8GB, and enable client-side compression.

3. Pod Anti-Affinity and Topology Distribution

To avoid single points of failure, we use Pod anti-affinity policies to ensure multiple replicas of the same service are distributed across different nodes and availability zones.

Deployment Anti-Affinity Configuration

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-gateway
  template:
    metadata:
      labels:
        app: api-gateway
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - api-gateway
              topologyKey: "kubernetes.io/hostname"
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: "topology.kubernetes.io/zone"
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: api-gateway
      containers:
      - name: api-gateway
        image: api-gateway:latest
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "2"
            memory: "2Gi"
        readinessProbe:
          httpGet:
            path: /health
            port: 8080

Custom Self-Healing Controller

go
package controller

import (
    "context"
    "log"

    corev1 "k8s.io/api/core/v1"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/reconcile"
)

type PodReconciler struct {
    client.Client
}

func (r *PodReconciler) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {
    var pod corev1.Pod
    if err := r.Get(ctx, req.NamespacedName, &pod); err != nil {
        return reconcile.Result{}, client.IgnoreNotFound(err)
    }

    if pod.Status.Phase == corev1.PodPending {
        for _, cond := range pod.Status.Conditions {
            if cond.Reason == "Unschedulable" {
                log.Printf("Detected unschedulable Pod: %s/%s, attempting eviction", pod.Namespace, pod.Name)
            }
        }
    }
    return reconcile.Result{}, nil
}

4. Summary

High availability is not a one-time configuration but a continuous evolution process. Through the combination of control plane multi-active, etcd tuning, anti-affinity policies, and self-healing Controllers, we reduced the cluster’s annualized failure time from 12 hours to under 30 minutes.