Production Go: Logs, Metrics, Traces, and Signals
Running a Go service in production means you can answer what happened, how often, and how long it took - without SSHing into a pod to read unstructured stdout.
Observability Basics collects runnable snippets; sibling articles cover slog, Prometheus, OpenTelemetry, probes, shutdown, and configuration.
Summary
- Observability is the ability to infer internal system state from external outputs: logs (discrete events), metrics (aggregated counters and histograms), and traces (request-scoped latency chains), plus operational signals like health endpoints and shutdown behavior.
- Insight: Go binaries are small and fast, but that does not make failures visible. Without shared request IDs, RED dashboards, and bounded probe handlers, on-call engineers guess from log grep and CPU graphs.
- Key Concepts: structured logging, RED/USE metrics, distributed tracing, correlation IDs, liveness/readiness, graceful shutdown, 12-factor config.
- When to Use: Every HTTP API, gRPC service, worker, and operator that runs outside a developer laptop needs this stack before the first production deploy.
- Limitations/Trade-offs: More signals mean storage cost and cardinality risk; tracing adds overhead; verbose logs can leak secrets; hot-reload config trades simplicity for complexity.
- Related Topics: net/http middleware, context propagation, Kubernetes probes, Prometheus scraping, OpenTelemetry exporters, Viper and flag parsing.
Foundations
Observability rests on three pillars that complement each other.
Logs record individual events: a handler started, a payment failed, a retry exhausted.
Structured logs (key-value fields) let you filter user_id=… or trace_id=… in Loki or CloudWatch without regex archaeology.
Go's log/slog package (stdlib since Go 1.21) is the default choice for new services.
Metrics compress behavior into numbers over time: request rate, error ratio, duration percentiles.
Prometheus pulls counters and histograms from /metrics endpoints; Grafana charts RED (Rate, Errors, Duration) per route or dependency.
Traces stitch spans across services so you see that a 2 s API call spent 1.8 s in PostgreSQL and 150 ms in a downstream HTTP call.
OpenTelemetry SDKs emit spans to Jaeger, Tempo, or vendor backends.
Beyond the three pillars, production Go services expose operational signals: /healthz and /readyz for orchestrators, SIGTERM handling for rolling deploys, and environment-driven configuration so the same binary runs in staging and prod.
Request
|
v
middleware -----> trace span (OTel)
| |
v v
handler -----> slog fields (trace_id, route)
|
v
metrics <----- counter/histogram (status, latency)
Mechanics & Interactions
Instrumentation attaches at stable boundaries.
HTTP middleware wraps http.Handler to start spans, increment metrics, and attach a request ID to context.Context.
That context flows into database and RPC calls so logs and child spans share the same correlation key.
| Signal | Question it answers | Typical Go tooling | Cardinality risk |
|---|---|---|---|
| Logs | What happened on this request? | log/slog, zap (optional) | High if you log unique IDs per line |
| Metrics | How many? How fast? What error rate? | prometheus/client_golang | High if labels include unbounded user IDs |
| Traces | Where did time go across services? | OpenTelemetry Go SDK | Medium; sampling mitigates |
| Probes | Is the process alive? Can it serve traffic? | net/http handlers | Low |
| Shutdown | Can deploys drain safely? | signal.Notify, server.Shutdown | N/A |
Logs and traces link through shared trace_id fields.
Metrics stay aggregate: never put raw email addresses in Prometheus labels.
Configuration loads once at startup (flags + env) or watches files for hot reload; observability endpoints should reflect config errors in readiness, not liveness.
Advanced Considerations & Applications
Cardinality is the silent killer of metrics stacks.
Prefer http_route="/users/{id}" template labels over per-user labels.
Histogram buckets should match SLO thresholds (for example 50 ms, 200 ms, 1 s).
Sampling keeps trace volume manageable: head-based sampling at the edge or tail-based sampling for errors only.
Always record errors and high-latency spans even when baseline traffic is sampled down.
Log volume grows with QPS.
Set levels per environment (INFO in prod, DEBUG in dev), redact secrets at the handler, and avoid logging full request bodies by default.
Go's slog supports LogValuer for lazy expensive fields.
Kubernetes distinguishes liveness (restart if dead) from readiness (remove from load balancer if dependencies fail).
A database outage should fail readiness, not liveness, or you restart pods that cannot fix the problem.
Graceful shutdown pairs SIGTERM with a bounded context.Context passed to http.Server.Shutdown and background worker drains.
| Approach | Strength | Weakness | Best Fit |
|---|---|---|---|
| slog only | Zero deps, stdlib | No ecosystem of appenders | New microservices |
| zap | Very fast, rich ecosystem | Extra dependency | High-QPS logging |
| Prometheus pull | Simple scrape model | Cardinality discipline required | Kubernetes services |
| OpenTelemetry | Vendor-neutral traces | SDK setup overhead | Multi-service meshes |
| Env-only config | 12-factor simple | No file-based defaults | Containers, serverless |
Common Misconceptions
- Stdout logging is enough for production - Unstructured lines are hard to query; you need JSON or key-value fields and consistent attribute names.
- Metrics replace logs - Metrics show aggregates; they cannot tell you which user hit the bug. Use both.
- Tracing is only for microservices - Even monoliths benefit when one handler calls five dependencies; a single trace explains slow requests.
- Health check equals readiness - Liveness should be cheap; readiness should verify databases and feature flags.
- More labels make better dashboards - Unbounded label cardinality crashes Prometheus and raises cost.
FAQs
What is the minimum observability stack for a new Go API?
Structured logs with request ID, a /metrics endpoint with request duration histogram, /healthz and /readyz, and SIGTERM shutdown cover the baseline.
Add tracing when you have more than one downstream dependency or service.
Should I use slog or zap?
Start with log/slog in the standard library.
Reach for zap when profiling shows logging allocations on the hot path or you need features slog does not provide yet.
How do logs and traces connect?
Inject trace_id and span_id from the active OpenTelemetry span into slog attributes on each request.
Log backends can link to trace UIs when IDs match.
What is RED for HTTP services?
Rate (requests per second), Errors (failed requests), Duration (latency distribution).
Instrument at the middleware layer with route template labels, not raw paths with IDs.
When should readiness fail?
Fail readiness when critical dependencies (database, cache, required upstream API) are unreachable.
Keep liveness cheap so Kubernetes does not restart healthy processes during dependency outages.
Does observability slow down Go services?
Well-designed metrics and sampled traces add low single-digit percent overhead.
Unbounded debug logging and high-cardinality labels hurt more than histogram updates.
Where do I expose metrics?
Mount /metrics on a separate listener or mux path, restrict network access, and scrape with Prometheus or a compatible agent.
Is configuration part of observability?
Misconfiguration causes outages that look like code bugs.
Structured startup logs, readiness checks for required env vars, and documented defaults reduce mean time to recovery.
How do frameworks like gin or chi fit in?
They still use net/http handlers.
Wrap the router with the same middleware for logging, metrics, and tracing regardless of framework.
What should I log on errors?
Log error type, operation, correlation IDs, and safe context.
Avoid passwords, tokens, and full credit card numbers even in error paths.
Can I skip tracing for batch workers?
Workers benefit from spans around job processing and external calls.
Link job ID in logs even when HTTP request IDs do not exist.
How do I test observability hooks?
Use httptest to hit handlers and assert metric increments or log output with slog test handlers.
Verify probe endpoints return expected status codes under mocked dependency failures.
Related
- Observability Basics - runnable starting examples
- Structured Logging with slog - handlers and levels
- Prometheus Metrics & RED/USE Dashboards - counters and histograms
- OpenTelemetry Tracing in Go Services - span propagation
- Health, Readiness & Liveness Probes - Kubernetes probes
- Graceful Shutdown & Signal Handling - SIGTERM drains
Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).