Restart on CrashRestart-on-crash is an operational strategy and configuration pattern used to automatically restart software processes, services, or containers when they terminate unexpectedly. The goal is to improve availability and reduce downtime by quickly returning failed components to a working state. While simple in concept, a robust restart-on-crash strategy requires careful configuration, observability, and an understanding of failure modes so restarts don’t mask underlying problems or cause cascading failures.
Why use restart-on-crash?
Improved availability. Automatic restarts shorten the time a service is unavailable after an unexpected failure.
Reduced manual intervention. Operators don’t need to intervene for every transient or sporadic crash.
Graceful recovery from transient faults. Many crashes stem from temporary resource exhaustion, network glitches, or brief dependency failures that a restart resolves.
However, restart-on-crash is not a cure-all. Blindly restarting a repeatedly failing process can hide bugs, cause resource thrashing, or amplify downstream outages. Use restarts as part of an overall reliability strategy that includes monitoring, alerting, rate-limiting restarts, and root-cause analysis.
Common environments and how they implement restart-on-crash
systemd (Linux)
systemd unit files support several restart policies via the Restart= option:
- Restart=no — (default) don’t restart.
- Restart=on-failure — restart on non-zero exit codes or signals.
- Restart=always — restart regardless of exit status.
- Restart=on-abnormal / on-success / on-watchdog — finer-grained control.
Useful complementary options:
- RestartSec= — delay before restarting.
- StartLimitBurst= and StartLimitIntervalSec= — limit rapid restart cycles.
- ExecStartPre/ExecStartPost — run pre/post hooks.
Example unit snippet:
[Service] ExecStart=/usr/local/bin/myapp Restart=on-failure RestartSec=5s StartLimitBurst=5 StartLimitIntervalSec=60 Docker
Docker supports restart policies on containers:
- no — default, no restart
- on-failure[:max-retries] — restart only on non-zero exit codes (optional max retries)
- always — always restart regardless of the exit code
- unless-stopped — like always, but don’t restart when the container was manually stopped
CLI example:
docker run --restart=on-failure:5 my-image Compose example:
services: web: image: my-image restart: on-failure:5 Kubernetes
Kubernetes uses pod restart policies and controllers:
- Pod-level restartPolicy supports Always, OnFailure, and Never (but is rarely used directly for production).
- Deployments, ReplicaSets, StatefulSets, and DaemonSets ensure a desired number of pod replicas are running. If a pod crashes, the controller creates a replacement.
- Liveness and readiness probes allow Kubernetes to detect stuck processes and restart unhealthy containers proactively.
Kubernetes focuses on maintaining desired state rather than per-process restart settings, and leverages scheduling, affinity, and resource requests/limits to avoid repeated restarts.
Design considerations and best practices
- Rate limiting and backoff
- Avoid tight restart loops; use exponential backoff or increasing RestartSec delays.
- systemd StartLimit and Docker max-retries help; orchestration layers should support backoff strategies.
- Distinguish transient vs. persistent failures
- Configure restart policies to handle transient crashes, but route persistent failures to alerts and human investigation.
- Use exit codes and signals to differentiate fatal errors from recoverable ones.
- Observe and alert
- Track restart counts, timestamps, and crash reasons in metrics and logs.
- Alert when restarts exceed thresholds within a time window (e.g., >3 restarts in 10 minutes).
- Collect diagnostic data on crash
- Capture core dumps, logs, stack traces, and environment state on failure to enable postmortem analysis.
- Use centralized logging and structured logs to speed triage.
- Graceful shutdown and startup
- Ensure processes handle termination signals cleanly so restarts don’t leave resources in inconsistent states (locks, files, external sessions).
- Use readiness probes (Kubernetes) or health checks (load balancers) to prevent traffic routing to a container still starting up.
- Resource limits and health
- Crash loops may be caused by resource exhaustion; set memory/cpu limits and requests, and ensure the scheduler places pods where capacity exists.
- Monitor OOM (out-of-memory) kills and tune memory usage or limits accordingly.
- Avoid masking root causes
- Restart policies should not be a substitute for fixing bugs. Use restarts to maintain availability while you diagnose and resolve the underlying fault.
- Safe defaults
- For most production services: Restart=on-failure (systemd) or on-failure with limited retries (Docker) plus alerting; Kubernetes controllers for higher-level resilience.
Common failure patterns and how restarts help or hurt
- Transient dependency outage (database briefly unreachable): Restart often helps if the service cannot gracefully reconnect; better approach—implement reconnection logic and exponential backoff.
- Memory leak: Restart hides the symptom temporarily but doesn’t fix the leak; add monitoring for memory growth and set memory limits to trigger restarts only as a mitigation.
- Configuration error: Repeated restarts will fail; treat this as a persistent failure, alert immediately, and prevent continuous restart loops.
- Deadlocks or hung threads: Liveness probes that restart on stuckness can improve availability, assuming the restart clears the hung state.
- Crash on startup due to missing environment variable: Repeated restart may spam logs and make debugging harder; prefer failing fast with clear errors and alerting.
Example workflows
Small service (systemd)
- Use Restart=on-failure, RestartSec=10s, StartLimitBurst=4, StartLimitIntervalSec=300.
- Configure centralized logging and set an alert when Restart counter increments >3 within 15 minutes.
Containerized service (Docker Compose)
- restart: on-failure:3
- Include a healthcheck so the container only receives traffic after it’s ready.
- Collect exit codes and write core dumps to a mounted volume for analysis.
Kubernetes microservice
- Deployment with resource requests/limits; readiness and liveness probes.
- Use Horizontal Pod Autoscaler for load spikes rather than relying on restarts.
- Configure Prometheus alert: pod restart count > 2 per 10 minutes → PagerDuty.
Troubleshooting restart loops
- Inspect logs and recent config changes.
- Check exit codes and system signals (OOM killer, SIGABRT).
- Increase RestartSec/backoff to slow loops while preserving availability.
- Disable automatic restart temporarily to debug start-up in isolation.
- Reproduce locally with same env vars and inputs.
- Gather core dumps and enable more verbose logging during diagnostic window.
Security and operational concerns
- Restarting processes that manage secrets or tokens should ensure secrets are reloaded securely and not logged.
- Ensure restarted processes don’t inadvertently reinitialize sensitive state (e.g., reset counters or recreate admin credentials).
- When restarts cause rapid re-registration with external services (e.g., service discovery), implement jitter to avoid thundering-herd problems.
Summary
Restart-on-crash is a powerful tool for improving availability, but it must be used thoughtfully: couple restart policies with rate limiting, robust observability, clear alerting thresholds, and cause analysis processes. Properly implemented, restart-on-crash reduces downtime for transient issues while giving teams the data they need to fix persistent problems rather than hide them.
Leave a Reply