Observability Best Practices

Production Go services stay debuggable when logs, metrics, traces, probes, and shutdown behavior follow shared rules that protect on-call engineers and observability budgets.

How to Use This List

Adopt Tier A rules in service templates before the first production deploy.
Encode label allowlists and probe paths in code review checklists.
Pair metrics rules with Grafana dashboards and alert runbooks.
Revisit after major traffic growth or new dependencies.

A - Logs and slog

Emit JSON logs in production with a single default handler. Aggregators index fields reliably; text is for local dev only.
Include request_id and trace_id on every request-scoped log line. On-call can jump from log line to trace without grep gymnastics.
Use stable attribute names across services (err, http.route, user_id). Consistent schemas make cross-service queries possible.
Set log level from environment; default to Info in prod. Debug volume scales with QPS and raises cost.
Redact secrets and tokens in a custom handler or middleware. Passwords must never reach log storage.
Log errors once at the boundary with wrapped context. Avoid duplicate stack traces at every layer.

B - Metrics and cardinality

Label HTTP metrics with route templates, not raw paths. /users/{id} prevents unbounded series from IDs.
Never put user IDs, emails, or order IDs in Prometheus labels. Cardinality explosions take down metrics backends.
Register metrics once in main, not per request. Duplicate registration panics and duplicates series.
Tune histogram buckets to SLO thresholds. Default buckets hide latency cliffs near your p99 target.
Expose /metrics on an admin listener or restricted network. Public scrape leaks internal operational detail.
Exclude health check traffic from RED charts or label probe=true. Probe QPS skews rate and duration aggregates.

C - Tracing and sampling

Propagate context.Context through HTTP handlers, gRPC, and database calls. Broken context breaks trace trees.
Use ParentBased or ratio sampling in production. Full tracing at high QPS overwhelms collectors.
Record errors on spans with span.RecordError. Failed traces must remain visible under sampling.
Call TracerProvider.Shutdown on exit. Buffered spans drop on SIGKILL without flush.
Set service.name and deployment environment resource attributes. Backends group traces per team and stage.
Span at I/O boundaries, not every helper function. Noise hides the spans that explain latency.

D - Health, readiness, and probes

Keep liveness cheap with no external I/O. Dependency checks belong on readiness only.
Return 503 on readiness when critical dependencies fail. Load balancers stop traffic without pod restart loops.
Allowlist /healthz and /readyz in auth middleware. Kubelet probes do not send bearer tokens.
Bound readiness checks with sub-second timeouts. Slow probes flap endpoints during partial outages.
Delay readiness until migrations and warm-up finish. Early traffic causes schema and cache errors.
Document startup vs readiness vs liveness in the service README. New operators must know which endpoint means what.

E - Shutdown and lifecycle

Handle SIGTERM and call http.Server.Shutdown with a bounded context. Rolling deploys drain in-flight work.
Align shutdown timeout with Kubernetes terminationGracePeriodSeconds. Avoid SIGKILL mid-request.
Cancel worker contexts when HTTP drain starts. Background pollers must not outlive the grace window.
Set readiness false before shutdown begins. Stops new traffic while active requests finish.
Log shutdown phases: signal received, drain start, complete. Deploy logs correlate with user-facing errors.
Close database pools and flush exporters after HTTP drain. Telemetry and connections must not leak on exit.

F - Configuration and operability

Validate required env vars at startup and exit non-zero on failure. Misconfig should not become a running pod.
Log a safe config snapshot at Info on boot. Operators verify port, environment, and feature flags without secrets.
Inject configuration structs; do not read os.Getenv in handlers. Tests and clarity suffer from hidden env access.
Treat observability endpoints as part of the public ops contract. Document paths, ports, and scrape intervals.
Run go test with slog test handlers for critical middleware. Regressions in request logging are caught in CI.
Review observability in every incident postmortem. Missing labels or IDs becomes a tracked action item.

FAQs

Which rules block merge in review?

Route template labels, readiness vs liveness separation, secret redaction, and SIGTERM shutdown are non-negotiable for HTTP services.

Do these apply to workers without HTTP?

Yes for logs, metrics, tracing, and shutdown.

Replace probe rules with queue lag metrics and consumer health checks.

How do I enforce cardinality limits?

Code review plus lint rules that ban WithLabelValues using dynamic path segments or user identifiers.

When is Debug logging acceptable in prod?

Briefly during targeted incidents with explicit toggles and time-bounded feature flags, never as a default.

How do best practices relate to the explainer article?

The explainer teaches why the pillars exist.

This list is the enforceable checklist for teams shipping services.

Should I standardize on slog or zap?

Prefer slog for new code unless profiling proves zap is required on your hot path.

What is the minimum metrics set?

Request total counter, duration histogram, and process Go collector metrics cover most API SLO dashboards.

How often should I revisit this list?

After doubling traffic, adding a dependency, or any observability-related incident.

Do gin and chi services follow the same rules?

Yes.

Framework choice does not change probe, shutdown, or cardinality discipline.

How do I onboard new services?

Copy a service template with middleware, metrics registration, probes, and shutdown already wired.

Production Go: Logs, Metrics, Traces, and Signals - conceptual overview
Structured Logging with slog - logging detail
Prometheus Metrics & RED/USE Dashboards - RED metrics
OpenTelemetry Tracing in Go Services - tracing setup
Graceful Shutdown & Signal Handling - drain patterns
net/http Best Practices - HTTP-specific companion rules

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Observability Best Practices

Production Go services stay debuggable when logs, metrics, traces, probes, and shutdown behavior follow shared rules that protect on-call engineers and observability budgets.

How to Use This List

Adopt Tier A rules in service templates before the first production deploy.
Encode label allowlists and probe paths in code review checklists.
Pair metrics rules with Grafana dashboards and alert runbooks.
Revisit after major traffic growth or new dependencies.

A - Logs and slog

Emit JSON logs in production with a single default handler. Aggregators index fields reliably; text is for local dev only.
Include request_id and trace_id on every request-scoped log line. On-call can jump from log line to trace without grep gymnastics.
Use stable attribute names across services (err, http.route, user_id). Consistent schemas make cross-service queries possible.
Set log level from environment; default to Info in prod. Debug volume scales with QPS and raises cost.
Redact secrets and tokens in a custom handler or middleware. Passwords must never reach log storage.
Log errors once at the boundary with wrapped context. Avoid duplicate stack traces at every layer.

B - Metrics and cardinality

Label HTTP metrics with route templates, not raw paths. /users/{id} prevents unbounded series from IDs.
Never put user IDs, emails, or order IDs in Prometheus labels. Cardinality explosions take down metrics backends.
Register metrics once in main, not per request. Duplicate registration panics and duplicates series.
Tune histogram buckets to SLO thresholds. Default buckets hide latency cliffs near your p99 target.
Expose /metrics on an admin listener or restricted network. Public scrape leaks internal operational detail.
Exclude health check traffic from RED charts or label probe=true. Probe QPS skews rate and duration aggregates.

C - Tracing and sampling

Propagate context.Context through HTTP handlers, gRPC, and database calls. Broken context breaks trace trees.
Use ParentBased or ratio sampling in production. Full tracing at high QPS overwhelms collectors.
Record errors on spans with span.RecordError. Failed traces must remain visible under sampling.
Call TracerProvider.Shutdown on exit. Buffered spans drop on SIGKILL without flush.
Set service.name and deployment environment resource attributes. Backends group traces per team and stage.
Span at I/O boundaries, not every helper function. Noise hides the spans that explain latency.

D - Health, readiness, and probes

Keep liveness cheap with no external I/O. Dependency checks belong on readiness only.
Return 503 on readiness when critical dependencies fail. Load balancers stop traffic without pod restart loops.
Allowlist /healthz and /readyz in auth middleware. Kubelet probes do not send bearer tokens.
Bound readiness checks with sub-second timeouts. Slow probes flap endpoints during partial outages.
Delay readiness until migrations and warm-up finish. Early traffic causes schema and cache errors.
Document startup vs readiness vs liveness in the service README. New operators must know which endpoint means what.

E - Shutdown and lifecycle

Handle SIGTERM and call http.Server.Shutdown with a bounded context. Rolling deploys drain in-flight work.
Align shutdown timeout with Kubernetes terminationGracePeriodSeconds. Avoid SIGKILL mid-request.
Cancel worker contexts when HTTP drain starts. Background pollers must not outlive the grace window.
Set readiness false before shutdown begins. Stops new traffic while active requests finish.
Log shutdown phases: signal received, drain start, complete. Deploy logs correlate with user-facing errors.
Close database pools and flush exporters after HTTP drain. Telemetry and connections must not leak on exit.

F - Configuration and operability

Validate required env vars at startup and exit non-zero on failure. Misconfig should not become a running pod.
Log a safe config snapshot at Info on boot. Operators verify port, environment, and feature flags without secrets.
Inject configuration structs; do not read os.Getenv in handlers. Tests and clarity suffer from hidden env access.
Treat observability endpoints as part of the public ops contract. Document paths, ports, and scrape intervals.
Run go test with slog test handlers for critical middleware. Regressions in request logging are caught in CI.
Review observability in every incident postmortem. Missing labels or IDs becomes a tracked action item.

FAQs

Which rules block merge in review?

Route template labels, readiness vs liveness separation, secret redaction, and SIGTERM shutdown are non-negotiable for HTTP services.

Do these apply to workers without HTTP?

Yes for logs, metrics, tracing, and shutdown.

Replace probe rules with queue lag metrics and consumer health checks.

How do I enforce cardinality limits?

Code review plus lint rules that ban WithLabelValues using dynamic path segments or user identifiers.

When is Debug logging acceptable in prod?

Briefly during targeted incidents with explicit toggles and time-bounded feature flags, never as a default.

How do best practices relate to the explainer article?

The explainer teaches why the pillars exist.

This list is the enforceable checklist for teams shipping services.

Should I standardize on slog or zap?

Prefer slog for new code unless profiling proves zap is required on your hot path.

What is the minimum metrics set?

Request total counter, duration histogram, and process Go collector metrics cover most API SLO dashboards.

How often should I revisit this list?

After doubling traffic, adding a dependency, or any observability-related incident.

Do gin and chi services follow the same rules?

Yes.

Framework choice does not change probe, shutdown, or cardinality discipline.

How do I onboard new services?

Copy a service template with middleware, metrics registration, probes, and shutdown already wired.

Production Go: Logs, Metrics, Traces, and Signals - conceptual overview
Structured Logging with slog - logging detail
Prometheus Metrics & RED/USE Dashboards - RED metrics
OpenTelemetry Tracing in Go Services - tracing setup
Graceful Shutdown & Signal Handling - drain patterns
net/http Best Practices - HTTP-specific companion rules

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Observability Best Practices

Search Documentation

Observability Best Practices