Graceful Shutdown & Signal Handling
Handle SIGTERM and SIGINT so HTTP servers finish in-flight requests, background workers stop cleanly, and Kubernetes rolling deploys do not drop active connections.
Summary
Orchestrators send SIGTERM before killing a container.
http.Server.Shutdown stops accepting new connections and waits for active requests until a context deadline.
Background goroutines need explicit cancel via context.Context or close channels.
golang.org/x/sync/errgroup coordinates multiple listeners and workers shutting down together.
Recipe
Quick-reference recipe card - copy-paste ready.
quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
<-quit
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
_ = server.Shutdown(ctx)When to reach for this:
- Kubernetes Deployments with rolling updates
- Autoscaling scale-in events
- Local development when Ctrl+C should not corrupt in-flight writes
- Services with background consumers that must commit offsets before exit
- Multi-server processes (HTTP + metrics + gRPC) shutting down in order
Working Example
package main
import (
"context"
"errors"
"log/slog"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"golang.org/x/sync/errgroup"
)
func main() {
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
mux := http.NewServeMux()
mux.HandleFunc("GET /work", func(w http.ResponseWriter, r *http.Request) {
select {
case <-time.After(2 * time.Second):
w.Write([]byte("done"))
case <-r.Context().Done():
return
}
})
api := &http.Server{Addr: ":8080", Handler: mux, ReadHeaderTimeout: 5 * time.Second}
admin := &http.Server{Addr: ":9090", Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("metrics"))
})}
g, gctx := errgroup.WithContext(ctx)
g.Go(func() error {
slog.Info("api listening", "addr", api.Addr)
if err := api.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
return err
}
return nil
})
g.Go(func() error {
slog.Info("admin listening", "addr", admin.Addr)
if err := admin.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
return err
}
return nil
})
g.Go(func() error {
<-gctx.Done()
slog.Info("shutdown started")
shutdownCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
if err := api.Shutdown(shutdownCtx); err != nil {
slog.Error("api shutdown", "err", err)
}
if err := admin.Shutdown(shutdownCtx); err != nil {
slog.Error("admin shutdown", "err", err)
}
slog.Info("shutdown complete")
return nil
})
if err := g.Wait(); err != nil {
slog.Error("server error", "err", err)
os.Exit(1)
}
}What this demonstrates:
signal.NotifyContextcancels root context on SIGTERM- Two HTTP servers shut down with shared timeout budget
- errgroup goroutine waits for signal then calls
Shutdownon each server - Handlers respect
r.Context()cancellation during drain
Deep Dive
How It Works
ListenAndServeblocks untilShutdowncloses the listener.Shutdownmarks server closed, closes idle connections, and waits for active handlers.- Handlers should honor
r.Context().Done()to exit promptly when clients disconnect. - After HTTP drain, close database pools, flush logs, and stop trace exporters.
Shutdown Order
| Step | Action |
|---|---|
| 1 | Stop accepting new HTTP/gRPC work (Shutdown) |
| 2 | Cancel worker context so consumers stop polling |
| 3 | Wait for in-flight handler and worker completion |
| 4 | Close DB pools and clients |
| 5 | Flush telemetry (TracerProvider.Shutdown) |
Timeout Alignment
| Source | Typical value |
|---|---|
| App shutdown context | 10-30s |
Kubernetes terminationGracePeriodSeconds | 30s default |
| Load balancer deregistration delay | Account in total budget |
Go Notes
// Workers tied to shutdown context
func runWorker(ctx context.Context) error {
for {
select {
case <-ctx.Done():
return ctx.Err()
default:
processOneJob(ctx)
}
}
}Gotchas
- Calling
os.ExitwithoutShutdown- Drops active requests on deploy; always drain first. - Shutdown timeout too short - Long uploads fail mid-stream; align with P99 request duration.
- Background goroutines ignore context - Process hangs past grace period and gets SIGKILL.
- Only shutting down one server - Metrics or debug listeners keep process alive unexpectedly.
- Blocking
Shutdownon main without goroutine - Cannot handle signal on same thread if not structured correctly. - Ignoring
http.ErrServerClosed- Normal afterShutdown; treat as success not failure. - Readiness still true during shutdown - Flip readiness false before drain so LB stops new traffic early.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
signal.NotifyContext | Modern Go 1.16+ services | You need legacy signal channel patterns |
| Manual signal channel | Fine-grained control over signals | Simpler NotifyContext suffices |
errgroup | Multiple servers or workers | Single ListenAndServe in tiny binaries |
Close instead of Shutdown | Immediate hard stop (tests only) | Production deploys |
FAQs
What is the difference between Shutdown and Close?
Shutdown drains active requests gracefully.
Close closes connections immediately.
How do I stop receiving new Kubernetes traffic first?
Set readiness to false (or implement preStop hook sleep) before calling Shutdown so endpoints drain.
Should workers use the same context as HTTP?
Often yes - cancel a shared root context on SIGTERM so all subsystems stop together.
What if Shutdown times out?
Log remaining work, optionally call Close on the server, and exit non-zero for observability.
Does Shutdown wait for WebSockets?
Long-lived connections need explicit close handling in handlers; default drain may not end them quickly.
How do I test graceful shutdown?
Start server in a goroutine, send SIGTERM to self in test, assert handler completes within timeout.
What about gRPC GracefulStop?
Use grpc.Server.GracefulStop() with a timeout fallback to Stop() for forced termination.
Should I handle SIGHUP?
Common for log rotation reload configs; separate from deploy SIGTERM handling.
Can I nest Shutdown calls?
Call once per server instance; duplicate calls return ErrServerClosed.
How does errgroup help?
It propagates the first error and waits for all goroutines, useful when admin and API servers run together.
What logs should I emit on shutdown?
Log signal received, drain start, per-subsystem completion, and final exit for deploy correlation.
Do I need shutdown for CLI tools?
Short CLIs often exit on context cancel only.
Long-running daemons and servers need full drain patterns.
Related
- Observability Basics - minimal SIGTERM example
- Health, Readiness & Liveness Probes - readiness before drain
- net/http Best Practices - shutdown tier rules
- OpenTelemetry Tracing in Go Services - flush spans on exit
- Production Go: Logs, Metrics, Traces, and Signals - operational signals
Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).