Every microservices architecture eventually confronts the same question: when a service instance degrades but does not crash - responding slowly, returning errors for a subset of requests, exhausting its connection pool - how quickly can the system detect it and remove it from the load-balanced pool? Manual detection means someone’s pager goes off, they log in, diagnose, and drain the instance. In a 50-service mesh with 5 instances per service, that is not a process that scales. You need the mesh itself to notice, react, and recover - before the degraded instance cascades failures upstream.

Think of a service mesh like the nervous system in a body. Individual neurons (sidecar proxies) sense local conditions - response latency, error rates, connection drops - and report to the brain (control plane). The brain synthesizes signals across the entire body, makes decisions (reroute, remove, restart), and sends commands back without the person (the developer) having to do anything. The autonomic nervous system handles 99% of the body’s maintenance without conscious thought.

The challenge is distinguishing a sick instance from a slow external dependency. If payment-service-3 is slow, it might be degraded - or the database it talks to might be degraded. Blindly evicting every slow instance can cascade-evict the entire fleet if a shared dependency is slow. The self-healing logic needs to correlate symptoms across multiple instances to attribute fault correctly.

At scale, we need to solve for: sub-minute detection of degraded instances, automatic traffic draining without dropping in-flight requests, remediation actions that do not cause more disruption than the original fault, and enough human override capability that operators can stop automated healing when the automation itself is wrong.

Requirements and Constraints

Functional Requirements

• Continuously probe health of every service instance (liveness, readiness, deep health)
• Detect degradation via multiple signals: error rate, latency percentile, connection saturation
• Automatically reroute traffic away from degraded instances within 30 seconds of detection
• Trigger remediation actions (restart, scale-out, alert) based on configurable policies
• Support canary health gating - block a canary from receiving more traffic if its error rate exceeds threshold
• Provide a circuit breaker per upstream service at the sidecar level

Non-Functional Requirements

• Health check latency: probes complete in <5 seconds
• Reroute propagation: traffic weights updated across all sidecars within 10 seconds
• False positive rate: <1% of healthy instances incorrectly evicted per day
• Availability: control plane failure must not prevent sidecars from enforcing known-good state
• Scale: support 10,000 service instances across 200 services

Constraints

• Application code must not be modified - all mesh logic lives in the sidecar
• Out of scope: service discovery (we assume a service registry exists), mutual TLS (separate concern)
• Automated restarts are rate-limited to 3 per hour per instance to prevent restart storms

High-Level Architecture

The system has two planes: a data plane of sidecar proxies co-located with each service instance, and a control plane that aggregates health signals and pushes routing decisions.

Each service pod runs a sidecar proxy (think Envoy or a custom implementation) that intercepts all inbound and outbound traffic. The sidecar maintains local state: a circuit breaker per upstream service, an error rate counter over a rolling window, and a health probe agent that periodically checks the local application.

The control plane has three components: a health aggregator that collects signals from all sidecars, a remediation engine that applies policies to decide what action to take, and a configuration distributor that pushes updated route weights back to sidecars.

The Sidecar Proxy

The sidecar is the workhorse of the mesh. It wraps every service instance and owns three responsibilities: traffic interception, local circuit breaking, and health probe reporting.

Traffic interception is achieved via iptables rules set at pod startup. All inbound and outbound TCP traffic is redirected to the sidecar’s listener ports. The application sees neither the interception nor the proxy - from its perspective, connections behave normally.

The circuit breaker watches the error rate to each upstream service over a rolling 10-second window. When the error rate crosses a threshold (default 50%), the circuit opens: all requests to that upstream immediately fail fast with a synthetic error. After a configured timeout (default 30 seconds), one probe request is allowed through. If it succeeds, the circuit closes. If it fails, the timeout resets.

// Circuit breaker state machine per upstream service
type CircuitState int

const (
    StateClosed   CircuitState = iota // normal operation
    StateOpen                         // failing fast
    StateHalfOpen                     // probing
)

type CircuitBreaker struct {
    upstream     string
    state        CircuitState
    errorCount   atomic.Int64
    totalCount   atomic.Int64
    lastStateChange time.Time
    threshold    float64       // e.g. 0.5 = 50% error rate
    cooldown     time.Duration // e.g. 30s
    mu           sync.Mutex
}

func (cb *CircuitBreaker) Allow() bool {
    cb.mu.Lock()
    defer cb.mu.Unlock()

    switch cb.state {
    case StateClosed:
        return true
    case StateOpen:
        if time.Since(cb.lastStateChange) > cb.cooldown {
            cb.state = StateHalfOpen
            cb.lastStateChange = time.Now()
            return true // allow the probe request
        }
        return false
    case StateHalfOpen:
        return false // only the single probe request allowed
    }
    return true
}

func (cb *CircuitBreaker) Record(err bool) {
    cb.totalCount.Add(1)
    if err {
        cb.errorCount.Add(1)
    }
    cb.mu.Lock()
    defer cb.mu.Unlock()

    total := cb.totalCount.Load()
    if total < 10 { // minimum sample size
        return
    }
    errorRate := float64(cb.errorCount.Load()) / float64(total)

    switch cb.state {
    case StateClosed:
        if errorRate > cb.threshold {
            cb.state = StateOpen
            cb.lastStateChange = time.Now()
            cb.errorCount.Store(0)
            cb.totalCount.Store(0)
        }
    case StateHalfOpen:
        if !err {
            cb.state = StateClosed
        } else {
            cb.state = StateOpen
            cb.lastStateChange = time.Now()
        }
    }
}

Health Probe Types

A service that returns 200 OK on /health may still be degraded. A naive liveness probe catches only crashes. A real health probe strategy uses three layers, each probing different failure modes.

Liveness probes answer: is the process alive? An HTTP probe to /healthz or a TCP connect is sufficient. If this fails, the process is dead and should be restarted. These run every 10 seconds with a 2-second timeout. Three consecutive failures trigger a restart.

Readiness probes answer: is this instance ready to serve traffic? A deeper check that validates database connectivity, cache reachability, and any warm-up requirements. If this fails, the instance should be removed from the load-balanced pool but not restarted. These run every 5 seconds.

Deep health probes answer: is this instance performing well? A synthetic transaction probe - make a real test request through the full stack and measure latency and correctness. A slow but technically successful deep probe indicates degradation even without errors. These run every 30 seconds.

# Deep health probe: executes a synthetic test request
import httpx
import asyncio
from dataclasses import dataclass

@dataclass
class ProbeResult:
    probe_type: str
    success: bool
    latency_ms: float
    error: str | None = None

async def deep_health_probe(instance_addr: str, probe_config: dict) -> ProbeResult:
    start = asyncio.get_event_loop().time()
    timeout = probe_config.get("timeout_sec", 5.0)

    try:
        async with httpx.AsyncClient(timeout=timeout) as client:
            resp = await client.get(
                f"http://{instance_addr}{probe_config['path']}",
                headers={"X-Health-Probe": "deep", "X-Probe-ID": probe_config['probe_id']}
            )
            latency_ms = (asyncio.get_event_loop().time() - start) * 1000

            # Check response validity
            if resp.status_code != 200:
                return ProbeResult("deep", False, latency_ms, f"HTTP {resp.status_code}")

            # Check latency SLA
            sla_ms = probe_config.get("latency_sla_ms", 500)
            if latency_ms > sla_ms:
                return ProbeResult("deep", False, latency_ms,
                                   f"Latency {latency_ms:.0f}ms exceeds SLA {sla_ms}ms")

            return ProbeResult("deep", True, latency_ms)

    except httpx.TimeoutException:
        latency_ms = timeout * 1000
        return ProbeResult("deep", False, latency_ms, "timeout")
    except Exception as e:
        latency_ms = (asyncio.get_event_loop().time() - start) * 1000
        return ProbeResult("deep", False, latency_ms, str(e))

Traffic Weight Shifting

Removing an unhealthy instance from the pool is not a binary operation in a well-designed mesh. Binary removal (0% or 100%) can cause traffic spikes as load redistributes instantly. Traffic weight shifting gradually reduces an instance’s share while the remaining instances absorb the load.

The control plane maintains a weight vector for each service: [instance_1: 0.33, instance_2: 0.33, instance_3: 0.33]. When instance_2 is detected as degraded, the control plane initiates a drain sequence:

1. Reduce instance_2’s weight to 0.20, increase others proportionally
2. Wait 10 seconds, check if instance_2 is recovering
3. If still degraded, reduce to 0.05 (minimal - just enough for probes)
4. After full drain timeout (60 seconds), remove from pool entirely

# Weight shifting protocol
from typing import Dict
import asyncio

class TrafficWeightController:
    def __init__(self, xds_client):
        self.xds_client = xds_client  # Envoy xDS API client
        self.weights: Dict[str, Dict[str, float]] = {}  # service -> {instance -> weight}

    async def drain_instance(self, service: str, instance_id: str,
                              drain_steps: list[float] = [0.2, 0.05, 0.0]):
        """Gradually drain traffic from a degraded instance."""
        original_weights = self.weights.get(service, {}).copy()

        for target_weight in drain_steps:
            current = self.weights[service]
            remaining_instances = {k: v for k, v in current.items() if k != instance_id}

            # Redistribute the drained weight proportionally to healthy instances
            drained = current.get(instance_id, 0) - target_weight
            total_remaining = sum(remaining_instances.values())

            new_weights = {
                k: v + (v / total_remaining) * drained
                for k, v in remaining_instances.items()
            }
            new_weights[instance_id] = target_weight

            self.weights[service] = new_weights
            await self._push_weights(service, new_weights)
            await asyncio.sleep(10)  # wait between steps

        # Final removal
        del self.weights[service][instance_id]
        await self._push_weights(service, self.weights[service])

    async def _push_weights(self, service: str, weights: Dict[str, float]):
        """Push updated weights to all sidecars via xDS."""
        await self.xds_client.update_cluster_weights(service, weights)

Control Plane vs Data Plane

The separation between control plane and data plane is the most important architectural decision in a service mesh - and the one most often confused.

The data plane handles every request. It intercepts, routes, load-balances, retries, and enforces circuit breakers. It is in the critical path. Latency must be sub-millisecond. It cannot make network calls to the control plane on the hot path. It works from a local copy of routing state.

The control plane makes decisions. It aggregates health signals, runs policy engines, updates routing configuration, and manages the lifecycle of instances. It is asynchronous. A 500ms decision latency is acceptable. It communicates with the data plane via a configuration push protocol (like Envoy’s xDS API) over a long-lived gRPC stream.

// xDS ClusterLoadAssignment - control plane pushes this to sidecars
// to update routing weights after health state changes
message ClusterLoadAssignment {
  string cluster_name = 1;
  repeated LocalityLbEndpoints endpoints = 2;
}

message LocalityLbEndpoints {
  Locality locality = 1;
  repeated LbEndpoint lb_endpoints = 2;
  uint32 load_balancing_weight = 3;  // relative weight
}

message LbEndpoint {
  Endpoint endpoint = 1;
  HealthStatus health_status = 2;  // HEALTHY, DEGRADED, UNHEALTHY
  google.protobuf.UInt32Value load_balancing_weight = 3;
}

enum HealthStatus {
  UNKNOWN = 0;
  HEALTHY = 1;
  UNHEALTHY = 2;
  DRAINING = 3;
  DEGRADED = 4;
}

Automatic Remediation Triggers

Detecting a degraded instance and rerouting traffic buys time. Remediating the root cause closes the loop. The remediation engine maps health signals to actions, with configurable thresholds and rate limits to prevent automation from making things worse.

# Remediation policy configuration
remediation_policies:
  - name: restart-on-oom
    trigger:
      signal: memory_utilization
      threshold: 0.95
      window_seconds: 60
      consecutive_samples: 3
    actions:
      - type: graceful_restart
        drain_connections_sec: 30
    rate_limit:
      max_per_hour: 3
      min_interval_sec: 600  # 10 minutes between restarts

  - name: scale-out-on-sustained-load
    trigger:
      signal: cpu_utilization
      threshold: 0.80
      window_seconds: 300   # 5 minutes sustained
      min_instances_above: 2  # majority of instances affected
    actions:
      - type: scale_out
        increment: 2        # add 2 instances
    rate_limit:
      max_per_hour: 2

  - name: evict-high-error-rate
    trigger:
      signal: error_rate_5xx
      threshold: 0.30       # 30% error rate
      window_seconds: 120
    actions:
      - type: drain_instance
        target: triggering_instance
      - type: alert
        channel: pagerduty
        severity: warning
    rate_limit:
      max_evictions_per_service: 0.50  # never evict more than 50% of a service

The max_evictions_per_service guard is critical. Without it, a shared database slowdown can cause all instances of a service to appear degraded simultaneously, and the remediation engine will try to evict them all, taking the service completely offline.

Canary Health Gating

Canary deployments - routing 5% of traffic to a new version before full rollout - require health gating: automatically halting the canary promotion if the new version degrades.

The canary gate compares key metrics between the canary and stable versions over a configurable analysis window. If the canary’s error rate is more than 1% higher than stable, or P99 latency is 20% worse, the gate blocks promotion and optionally rolls back.

# Canary health gate analysis
from dataclasses import dataclass
from typing import Optional
import statistics

@dataclass
class CanaryMetrics:
    error_rate: float      # fraction 0.0-1.0
    p50_latency_ms: float
    p99_latency_ms: float
    request_count: int

@dataclass
class GateDecision:
    action: str   # "promote", "hold", "rollback"
    reason: str
    confidence: float

def evaluate_canary_gate(
    canary: CanaryMetrics,
    stable: CanaryMetrics,
    config: dict
) -> GateDecision:
    """Compare canary vs stable, decide whether to promote."""

    # Minimum sample size for statistical validity
    if canary.request_count < config.get("min_requests", 100):
        return GateDecision("hold", "insufficient canary traffic", 0.0)

    # Error rate comparison
    error_delta = canary.error_rate - stable.error_rate
    error_threshold = config.get("max_error_rate_delta", 0.01)
    if error_delta > error_threshold:
        return GateDecision(
            "rollback",
            f"canary error rate {canary.error_rate:.1%} exceeds stable "
            f"{stable.error_rate:.1%} by {error_delta:.1%} (threshold {error_threshold:.1%})",
            0.95
        )

    # P99 latency comparison
    latency_ratio = canary.p99_latency_ms / max(stable.p99_latency_ms, 1)
    latency_threshold = config.get("max_p99_ratio", 1.20)
    if latency_ratio > latency_threshold:
        return GateDecision(
            "rollback",
            f"canary P99 {canary.p99_latency_ms:.0f}ms is {latency_ratio:.1f}x stable "
            f"{stable.p99_latency_ms:.0f}ms (threshold {latency_threshold:.1f}x)",
            0.90
        )

    return GateDecision("promote", "all metrics within thresholds", 0.85)

Data Model

-- Service registry: all known services and instances
CREATE TABLE service_instances (
    id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    service_name VARCHAR(128) NOT NULL,
    instance_id  VARCHAR(128) NOT NULL,
    address      VARCHAR(256) NOT NULL,
    port         INTEGER NOT NULL,
    zone         VARCHAR(64),
    version      VARCHAR(64),
    registered_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    last_seen_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    status        VARCHAR(20) NOT NULL DEFAULT 'healthy'
                  CHECK (status IN ('healthy', 'degraded', 'draining', 'unhealthy', 'removed')),
    UNIQUE (service_name, instance_id)
);

CREATE INDEX idx_instance_service ON service_instances (service_name, status);

-- Health check results: time-series of probe outcomes
CREATE TABLE health_check_results (
    id             BIGSERIAL PRIMARY KEY,
    instance_id    VARCHAR(128) NOT NULL,
    probe_type     VARCHAR(20) NOT NULL,  -- liveness, readiness, deep
    success        BOOLEAN NOT NULL,
    latency_ms     NUMERIC(8,2),
    error_detail   TEXT,
    checked_at     TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_health_instance_time ON health_check_results (instance_id, checked_at DESC);

-- Partition by day: SELECT create_time_partitions('health_check_results', 'checked_at', '1 day');

-- Remediation actions taken: full audit log
CREATE TABLE remediation_events (
    id            BIGSERIAL PRIMARY KEY,
    service_name  VARCHAR(128) NOT NULL,
    instance_id   VARCHAR(128),
    action_type   VARCHAR(64) NOT NULL,  -- restart, drain, scale_out, alert
    trigger_signal VARCHAR(64) NOT NULL,
    trigger_value  NUMERIC(10,4),
    outcome        VARCHAR(20),          -- success, failed, in_progress
    initiated_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    completed_at   TIMESTAMPTZ,
    initiated_by   VARCHAR(128) NOT NULL DEFAULT 'auto'
);

CREATE INDEX idx_remediation_service ON remediation_events (service_name, initiated_at DESC);

-- Traffic weights: current routing state per service
CREATE TABLE routing_weights (
    service_name VARCHAR(128) NOT NULL,
    instance_id  VARCHAR(128) NOT NULL,
    weight       NUMERIC(6,4) NOT NULL DEFAULT 1.0,
    updated_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (service_name, instance_id)
);

Key Algorithms and Protocols

Outlier Detection Algorithm

The control plane uses an outlier detection algorithm to identify statistically anomalous instances without requiring manual thresholds per service.

import statistics
from typing import List

def detect_outliers(
    instance_error_rates: dict[str, float],
    min_instances: int = 3
) -> list[str]:
    """
    Identify instances whose error rate is an outlier using
    modified Z-score (robust to non-normal distributions).
    Returns instance IDs to consider for ejection.
    """
    if len(instance_error_rates) < min_instances:
        return []  # not enough data to compare

    rates = list(instance_error_rates.values())
    median = statistics.median(rates)
    # Median absolute deviation (MAD) - robust to outliers
    mad = statistics.median([abs(r - median) for r in rates])

    if mad == 0:
        return []  # all instances identical, no outliers

    outliers = []
    for instance_id, rate in instance_error_rates.items():
        # Modified Z-score: values above 3.5 are conventionally outliers
        modified_z = 0.6745 * (rate - median) / mad
        if modified_z > 3.5:
            outliers.append(instance_id)

    return outliers

The MAD-based outlier detection is more robust than mean/standard-deviation approaches. A single extremely bad instance inflates the mean and standard deviation, potentially hiding other moderately bad instances. MAD is insensitive to extreme values.

Scaling and Performance

Capacity estimation:
  Services: 200
  Instances per service: 50 average
  Total instances: 10,000

Health probe load (per sidecar agent):
  - Liveness: 1 probe/10s
  - Readiness: 1 probe/5s
  - Deep: 1 probe/30s
  - Total: ~0.37 probes/s per instance

Control plane ingest rate:
  - 10,000 instances x 0.37 = 3,700 probe results/s
  - Each result: 200 bytes
  - Total: 740 KB/s ingest bandwidth

xDS push rate (weight updates):
  - Worst case: 10 degradation events/min
  - Each event: push to all 10,000 sidecars
  - xDS stream, delta protocol: only changed endpoints
  - Bandwidth: 10 events/min x 1 KB delta x 10,000 sidecars = 1.7 MB/s

Control plane memory:
  - 10,000 instances x 1KB state = 10 MB base
  - 3,700 probe results/s x 60s window = 222,000 rows = 44 MB
  - Total: ~100 MB with indexes (very manageable)

The control plane is a single-region service, but it must not be a single point of failure. Run 3 replicas behind a leader election (using etcd or Raft). Only the leader makes remediation decisions and pushes xDS updates. Follower replicas ingest health signals so they can take over immediately on leader failure.

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
Control plane unavailable	Sidecar xDS stream disconnect	No new routing updates; existing state enforced	Sidecars hold last-known routing; leader election restores control plane in <30s
Sidecar crash	Application pod loses proxy	Pod’s traffic bypasses mesh	Kubernetes restarts sidecar container; traffic rejected during restart window
False positive eviction	Multiple healthy instances evicted	Service capacity reduced	Eviction cap (50% max) limits blast radius; manual override API restores weights
Probe endpoint overloaded	Deep probes cause extra load	Probe results show elevated latency	Probe rate backoff when probe latency >2x normal; jitter added to probe schedules
Remediation loop	Restart triggers more restarts	Restart storm	Per-instance restart rate limit (3/hour); exponential backoff on consecutive failures
Network partition between zones	Zone-specific health signals lost	Cross-zone routing decisions delayed	Each zone’s sidecar agents send signals independently; control plane uses last-known for partitioned zones

Comparison of Approaches

Approach	Detection Time	Blast Radius	Complexity	Best Fit
Manual ops on-call	5-30 minutes	Controlled by human	Low	Small teams, infrequent incidents
Kubernetes liveness probes only	30s (3 failed probes x 10s)	Controlled by k8s	Low	Simple services, crash-only failures
Sidecar circuit breaker (no control plane)	10s (rolling window)	Per-sidecar, no coordination	Medium	Services with independent upstreams
Full service mesh (Istio/Linkerd)	<30s end-to-end	Fleet-wide coordination	High	Microservices at scale, multi-team
Custom self-healing mesh (this design)	<30s + ML outlier detection	Capped at 50% per service	Very high	Large orgs needing custom policies

The right choice depends on organizational scale. A 3-service application does not need a full service mesh - Kubernetes readiness probes and a single circuit breaker library are sufficient. The full self-healing mesh design in this post is appropriate when you have 50+ services, multiple independent teams, and the operational cost of manual incident response exceeds the engineering cost of building and operating the mesh.

Key Takeaways

• Sidecar proxies intercept all traffic without application code changes; they are the universal enforcement point for health-based routing decisions.
• Circuit breakers protect upstreams from overload cascades by failing fast locally rather than queueing requests to an unhealthy service.
• Health probe layering - liveness, readiness, deep - separates “is the process alive” from “is this instance performing well enough to serve traffic.”
• Traffic weight shifting gives degraded instances a recovery window by gradually draining rather than instantly removing them from the pool.
• Control plane / data plane separation ensures that control plane failures do not disrupt request handling - sidecars enforce last-known-good state autonomously.
• Automatic remediation triggers must be rate-limited and capped - automation that acts too aggressively on correlated signals can cascade a single instance failure into a full service outage.
• Canary health gating uses ratio-based comparison against stable, adapting to the service’s baseline performance rather than fixed thresholds.
• Outlier detection using MAD-based Z-scores identifies the worst-performing instance relative to its peers, correctly handling services with inherently high absolute error rates.

The counter-intuitive lesson: the control plane should be conservative and slow. It has a global view of the mesh, which means its mistakes are also global. The data plane should be fast and local - the sidecar’s circuit breaker reacts in seconds because it sees only its own upstream, and a wrong local decision affects only that one service relationship. Build your fast-path safety mechanisms into the sidecar, and use the control plane for coordination and policy - not for emergency response.

Frequently Asked Questions

Q: Why build a custom service mesh instead of using Istio or Linkerd? A: Istio and Linkerd are the right choice for most organizations. They provide battle-tested implementations of everything described in this post plus mTLS, observability, and policy enforcement. The reasons to build custom are: need for deeply integrated custom remediation actions (e.g., triggering your proprietary deployment system), performance requirements that Envoy’s overhead cannot meet (very high-throughput, low-latency services), or organizational constraints (regulated industries where open-source control plane is problematic). If you can use a managed service mesh, do.

Q: How do you prevent the circuit breaker from opening during a planned maintenance window? A: Introduce a “maintenance mode” flag per service instance in the control plane. During planned maintenance, set the flag before draining traffic. The circuit breaker and outlier detection logic treat maintenance-mode instances as expected-unhealthy and exclude them from statistical comparisons. The flag prevents automated remediation actions (restarts, alerts) on the draining instance.

Q: What happens when the upstream service is healthy but its database is slow? A: The circuit breaker at the caller’s sidecar will open if error responses cross the threshold, but slow responses without errors won’t trigger it. The deep health probe is the right detection mechanism: if service-B’s deep probe latency degrades because its database is slow, the control plane increases its drain weight, slowing the rate of new requests. Simultaneously, the control plane should correlate the signal across all service-B instances - if all of them degrade simultaneously, the cause is a shared dependency, not instance-level failure. The remediation engine suppresses per-instance eviction in this case and fires a different alert class.

Q: How does the mesh handle a “slow poison” bug - a new deployment that is correct initially but degrades over time (memory leak, connection pool exhaustion)? A: The deep health probe is the key mechanism. Because it runs on a 30-second schedule, it will catch gradual degradation even when the instance appears healthy at the API level. The remediation engine looks for increasing probe latency trends, not just threshold crossings. If an instance’s deep probe latency has been growing 5% per minute for the last 10 minutes, that is a pattern that warrants a proactive restart even before the latency threshold is crossed.

Q: How do you validate that a new remediation policy won’t cause harm before enabling it in production? A: Dry-run mode: run the policy engine against real health signals but emit only audit log entries instead of taking actions. After 1-2 weeks, analyze what the policy would have done and validate the outcomes against incident reports. Shadow mode is the service mesh equivalent of chaos engineering’s steady-state hypothesis validation - prove the automation behaves correctly before giving it authority to act.

Q: How does this design interact with zero-downtime rolling deployments? A: The deployment system sets new instances to “draining” (weight=0) until they pass their readiness probe. This integrates directly with the mesh’s weight management - new instances are invisible to the load balancer until the sidecar reports them as ready. The mesh also ensures that old instances complete their in-flight requests during the drain window before being shut down, maintaining zero-downtime semantics.

Interview Questions

Q: How does the circuit breaker at the sidecar level differ from retry logic in the application, and when would you use each? Expected depth: Sidecar circuit breaker operates at the network level, applying to all traffic regardless of retry configuration; it opens globally for all callers of a service when error rate exceeds threshold. Application-level retry logic is per-request and per-client. Circuit breaker prevents retry storms by short-circuiting before retries happen. The combination: application retries on transient errors, circuit breaker opens after sustained failures to prevent retry amplification. Discuss the half-open state as the mechanism that prevents a circuit breaker from staying open after the upstream recovers.

Q: How would you detect whether a degraded instance is causing failures or a shared downstream dependency? Expected depth: Correlation analysis: if a single instance degrades while others are healthy, the instance is the cause. If all instances of a service degrade simultaneously, look downstream. The control plane should track per-service upstream error rates - if service-B degrades when database-1 error rates rise, attribute fault to the database. Discuss how to suppress per-instance eviction when the signal is correlated, and how to surface the correct alert (database issue, not service-B issue).

Q: Walk through a canary rollout that goes wrong and explain exactly how the health gate stops it. Expected depth: Describe the canary getting 5% of traffic, health gate collecting metrics over the analysis window, comparing error rate and P99 latency ratios against stable, detecting a regression (e.g., new version has a memory leak causing P99 to grow), gate firing a “rollback” decision, traffic weight for canary set to 0% and stable restored to 100%, rollback event emitted to deployment system. Discuss the analysis window length tradeoff: too short produces false positives from variance, too long means more users see the bad version.

Q: How do you prevent the automated remediation system from taking an action that makes an incident worse? Expected depth: Cover the key safety valves: maximum eviction percentage (50%), per-instance restart rate limit (3/hour), minimum inter-action interval, correlated-fault detection suppressing eviction, dry-run mode for policy validation, and a manual circuit breaker that ops teams can use to halt all automated actions during an active incident. Discuss the concept of “blast radius” - every automated action should have a bounded worst-case impact.

Q: How does the control plane push routing updates to 10,000 sidecars efficiently without overwhelming the network? Expected depth: Discuss the xDS protocol’s delta variant (Delta ADS) - instead of pushing the full route table, only push changes. A single instance eviction generates a delta of 1 endpoint change, not a 10,000-endpoint table. Sidecars maintain their own copy of the routing table and apply deltas. Discuss fan-out infrastructure (gRPC streaming to each sidecar), possible use of gossip-based propagation for large meshes, and the tradeoff between push (low latency, high bandwidth) and pull (high latency, low bandwidth) for routing updates.