Production Go: Logs, Metrics, Traces, and Signals
Running a Go service in production means you can answer what happened, how often, and how long it took - without SSHing into a pod to read unstructured stdout.
//
Search across all documentation pages
Running a Go service in production means you can answer what happened, how often, and how long it took - without SSHing into a pod to read unstructured stdout.
Observability Basics collects runnable snippets; sibling articles cover slog, Prometheus, OpenTelemetry, probes, shutdown, and configuration.
Observability rests on three pillars that complement each other.
Logs record individual events: a handler started, a payment failed, a retry exhausted.
Structured logs (key-value fields) let you filter user_id=… or trace_id=… in Loki or CloudWatch without regex archaeology.
Go's log/slog package (stdlib since Go 1.21) is the default choice for new services.
Metrics compress behavior into numbers over time: request rate, error ratio, duration percentiles.
Prometheus pulls counters and histograms from /metrics endpoints; Grafana charts RED (Rate, Errors, Duration) per route or dependency.
Traces stitch spans across services so you see that a 2 s API call spent 1.8 s in PostgreSQL and 150 ms in a downstream HTTP call.
OpenTelemetry SDKs emit spans to Jaeger, Tempo, or vendor backends.
Beyond the three pillars, production Go services expose operational signals: /healthz and /readyz for orchestrators, SIGTERM handling for rolling deploys, and environment-driven configuration so the same binary runs in staging and prod.
Request
|
v
middleware -----> trace span (OTel)
| |
v v
handler -----> slog fields (trace_id, route)
|
v
metrics <----- counter/histogram (status, latency)
Instrumentation attaches at stable boundaries.
HTTP middleware wraps http.Handler to start spans, increment metrics, and attach a request ID to context.Context.
That context flows into database and RPC calls so logs and child spans share the same correlation key.
| Signal | Question it answers | Typical Go tooling | Cardinality risk |
|---|---|---|---|
| Logs | What happened on this request? | log/slog, zap (optional) | High if you log unique IDs per line |
| Metrics | How many? How fast? What error rate? | prometheus/client_golang | High if labels include unbounded user IDs |
| Traces | Where did time go across services? | OpenTelemetry Go SDK | Medium; sampling mitigates |
| Probes | Is the process alive? Can it serve traffic? | net/http handlers | Low |
| Shutdown | Can deploys drain safely? | signal.Notify, server.Shutdown | N/A |
Logs and traces link through shared trace_id fields.
Metrics stay aggregate: never put raw email addresses in Prometheus labels.
Configuration loads once at startup (flags + env) or watches files for hot reload; observability endpoints should reflect config errors in readiness, not liveness.
Cardinality is the silent killer of metrics stacks.
Prefer http_route="/users/{id}" template labels over per-user labels.
Histogram buckets should match SLO thresholds (for example 50 ms, 200 ms, 1 s).
Sampling keeps trace volume manageable: head-based sampling at the edge or tail-based sampling for errors only.
Always record errors and high-latency spans even when baseline traffic is sampled down.
Log volume grows with QPS.
Set levels per environment (INFO in prod, DEBUG in dev), redact secrets at the handler, and avoid logging full request bodies by default.
Go's slog supports LogValuer for lazy expensive fields.
Kubernetes distinguishes liveness (restart if dead) from readiness (remove from load balancer if dependencies fail).
A database outage should fail readiness, not liveness, or you restart pods that cannot fix the problem.
Graceful shutdown pairs SIGTERM with a bounded context.Context passed to http.Server.Shutdown and background worker drains.
| Approach | Strength | Weakness | Best Fit |
|---|---|---|---|
| slog only | Zero deps, stdlib | No ecosystem of appenders | New microservices |
| zap | Very fast, rich ecosystem | Extra dependency | High-QPS logging |
| Prometheus pull | Simple scrape model | Cardinality discipline required | Kubernetes services |
| OpenTelemetry | Vendor-neutral traces | SDK setup overhead | Multi-service meshes |
| Env-only config | 12-factor simple | No file-based defaults | Containers, serverless |
Structured logs with request ID, a /metrics endpoint with request duration histogram, /healthz and /readyz, and SIGTERM shutdown cover the baseline.
Add tracing when you have more than one downstream dependency or service.
Start with log/slog in the standard library.
Reach for zap when profiling shows logging allocations on the hot path or you need features slog does not provide yet.
Inject trace_id and span_id from the active OpenTelemetry span into slog attributes on each request.
Log backends can link to trace UIs when IDs match.
Rate (requests per second), Errors (failed requests), Duration (latency distribution).
Instrument at the middleware layer with route template labels, not raw paths with IDs.
Fail readiness when critical dependencies (database, cache, required upstream API) are unreachable.
Keep liveness cheap so Kubernetes does not restart healthy processes during dependency outages.
Well-designed metrics and sampled traces add low single-digit percent overhead.
Unbounded debug logging and high-cardinality labels hurt more than histogram updates.
Mount /metrics on a separate listener or mux path, restrict network access, and scrape with Prometheus or a compatible agent.
Misconfiguration causes outages that look like code bugs.
Structured startup logs, readiness checks for required env vars, and documented defaults reduce mean time to recovery.
They still use net/http handlers.
Wrap the router with the same middleware for logging, metrics, and tracing regardless of framework.
Log error type, operation, correlation IDs, and safe context.
Avoid passwords, tokens, and full credit card numbers even in error paths.
Workers benefit from spans around job processing and external calls.
Link job ID in logs even when HTTP request IDs do not exist.
Use httptest to hit handlers and assert metric increments or log output with slog test handlers.
Verify probe endpoints return expected status codes under mocked dependency failures.
Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).
Reviewed by Chris St. John·Last updated Jul 16, 2026