Production Troubleshooting Best Practices

Q: Can lint enforce any of this?

CI can ban sql.Query without context in new code, require ReadHeaderTimeout, and block public pprof routes in config review.

Prevention habits that shrink incident duration for Go platforms: SLOs, dashboards, runbooks, and rehearsed response.

Apply as design-review rubric and quarterly platform hygiene, not only after outages.

How to Use This List

Tick items as established platform standards, not aspirational someday notes.
Use in service template PRs and launch checklists for new Go binaries.
Pair with game days; untested runbooks are fiction.
Tie checkbox rules to alerts or CI gates where possible.
Revisit after every SEV-1 post-mortem; add one new guardrail per incident class.

A - Observability before incidents

Expose RED metrics per route template. Rate, errors, and duration histograms power first-minute triage.
Export sql.DB.Stats() as Prometheus gauges. WaitCount alerts catch pool exhaustion before total stall.
Track goroutine count and GC pause percentiles. Climbing goroutines and STW time precede many memory incidents.
Structured logs with request_id and trace_id. Correlation cuts log search time during SEV-1.
Embed build SHA and Go version in /healthz. Deploy correlation without digging in CI.

B - Runtime diagnostics ready day one

Mount pprof on localhost or authenticated admin ports only. Never public /debug/pprof.
Document port-forward and token steps in the runbook. On-call should not discover auth during pages.
Save profile capture commands in the incident bundle script. CPU, heap, and goroutine in one place.
Rehearse profile under load in staging quarterly. Idle profiles train bad habits.
Evaluate Go 1.26 leak profile experiment in staging CI. Optional signal for goroutine growth regressions.

C - Capacity and runtime config

Set MaxOpenConns from DB limit / replica count formula. Document arithmetic in the service README.
Align GOMEMLIMIT to ~90% of container memory limit. Reduce OOMKilled before blind limit raises.
Set GOMAXPROCS to match CPU quota (or use automaxprocs). Runnable queue latency shows mismatch in traces.
Use QueryContext with deadlines on every DB call. Pool waits must not outlive client timeouts.
Propagate context.Context to all background goroutines. Cuts leak class that survives restarts.

D - Deploy and mitigation discipline

Maintain one-command rollback to last known-good SHA. Go binaries make rollback fast when artifacts exist.
Canary deploys on state-changing services. Watch p99 and pool wait, not only error rate.
Freeze risky deploys during error budget burn. Policy beats heroics.
Feature flags for heavy endpoints. Shed load without emergency deploy.
Graceful shutdown with Server.Shutdown timeout below K8s grace period. Avoid false 5xx during rollouts.

E - Process and learning

Define SEV levels and escalation in writing. Everyone pages the same playbook.
Blameless post-mortem within 5 business days for SEV-1/2. Template includes Go runtime snapshot table.
On-call handoff checklist every shift change. Go-specific signals, not only ticket queue.
Annual game day: OOM, pool exhaustion, bad deploy. Muscle memory beats documentation alone.
Track post-mortem actions to closure in the same sprint system. Open actions guarantee recurrence.

When You Are Done

Tiers A and B should be true before any production launch.

Tiers C through E mature over quarters; prioritize items that burned error budget in the last two incidents.

If all tiers pass review and game day succeeds, defer new tooling until the next measured gap.

FAQs

Which items are launch blockers?

RED metrics, bounded connection pool, context on DB calls, non-public pprof, health metadata, and rollback path.

How do best practices differ from runbooks?

Best practices prevent classes of failure.

Runbooks execute steps during active incidents.

Both are required.

Should small teams skip game days?

Run a tabletop at minimum once a year.

Thirty minutes of walkthrough beats first real OOM alone.

How do SLOs relate to troubleshooting?

SLO burn triggers incident severity and deploy freeze policy.

Alerts should map to SLO windows, not only static thresholds.

What is the minimum dashboard set?

RED per service, goroutines, heap, GC pause, pool wait, CPU/memory vs limits, deploy markers.

Do best practices apply to batch workers?

Yes.

Workers need pool limits, ctx cancel, metrics, and profile access too.

How often update runbooks?

After every SEV-1 and when Go minor version upgrades change GC behavior.

Can lint enforce any of this?

CI can ban sql.Query without context in new code, require ReadHeaderTimeout, and block public pprof routes in config review.

What about multi-region Go fleets?

Handoff and dashboards note per-region SLO burn.

Profiles may be region-specific during partial outages.

When is enough prevention enough?

When SLOs are green, game days pass, and last two incidents had alerts fire before customer-wide impact.

Optimize features until the next measured regression.

Incident Response for Go Services - triage mental model
On-Call Handoff Checklist for Go Platforms - shift transitions
Post-Mortem Template for Go Incidents - RCA doc
Production Troubleshooting Basics - hands-on examples
Observability Best Practices - signal foundation
Performance Best Practices - profiling discipline

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Production Troubleshooting Best Practices

Prevention habits that shrink incident duration for Go platforms: SLOs, dashboards, runbooks, and rehearsed response.

Apply as design-review rubric and quarterly platform hygiene, not only after outages.

How to Use This List

Tick items as established platform standards, not aspirational someday notes.
Use in service template PRs and launch checklists for new Go binaries.
Pair with game days; untested runbooks are fiction.
Tie checkbox rules to alerts or CI gates where possible.
Revisit after every SEV-1 post-mortem; add one new guardrail per incident class.

A - Observability before incidents

Expose RED metrics per route template. Rate, errors, and duration histograms power first-minute triage.
Export sql.DB.Stats() as Prometheus gauges. WaitCount alerts catch pool exhaustion before total stall.
Track goroutine count and GC pause percentiles. Climbing goroutines and STW time precede many memory incidents.
Structured logs with request_id and trace_id. Correlation cuts log search time during SEV-1.
Embed build SHA and Go version in /healthz. Deploy correlation without digging in CI.

B - Runtime diagnostics ready day one

Mount pprof on localhost or authenticated admin ports only. Never public /debug/pprof.
Document port-forward and token steps in the runbook. On-call should not discover auth during pages.
Save profile capture commands in the incident bundle script. CPU, heap, and goroutine in one place.
Rehearse profile under load in staging quarterly. Idle profiles train bad habits.
Evaluate Go 1.26 leak profile experiment in staging CI. Optional signal for goroutine growth regressions.

C - Capacity and runtime config

Set MaxOpenConns from DB limit / replica count formula. Document arithmetic in the service README.
Align GOMEMLIMIT to ~90% of container memory limit. Reduce OOMKilled before blind limit raises.
Set GOMAXPROCS to match CPU quota (or use automaxprocs). Runnable queue latency shows mismatch in traces.
Use QueryContext with deadlines on every DB call. Pool waits must not outlive client timeouts.
Propagate context.Context to all background goroutines. Cuts leak class that survives restarts.

D - Deploy and mitigation discipline

Maintain one-command rollback to last known-good SHA. Go binaries make rollback fast when artifacts exist.
Canary deploys on state-changing services. Watch p99 and pool wait, not only error rate.
Freeze risky deploys during error budget burn. Policy beats heroics.
Feature flags for heavy endpoints. Shed load without emergency deploy.
Graceful shutdown with Server.Shutdown timeout below K8s grace period. Avoid false 5xx during rollouts.

E - Process and learning

Define SEV levels and escalation in writing. Everyone pages the same playbook.
Blameless post-mortem within 5 business days for SEV-1/2. Template includes Go runtime snapshot table.
On-call handoff checklist every shift change. Go-specific signals, not only ticket queue.
Annual game day: OOM, pool exhaustion, bad deploy. Muscle memory beats documentation alone.
Track post-mortem actions to closure in the same sprint system. Open actions guarantee recurrence.

When You Are Done

Tiers A and B should be true before any production launch.

Tiers C through E mature over quarters; prioritize items that burned error budget in the last two incidents.

If all tiers pass review and game day succeeds, defer new tooling until the next measured gap.

FAQs

Which items are launch blockers?

RED metrics, bounded connection pool, context on DB calls, non-public pprof, health metadata, and rollback path.

How do best practices differ from runbooks?

Best practices prevent classes of failure.

Runbooks execute steps during active incidents.

Both are required.

Should small teams skip game days?

Run a tabletop at minimum once a year.

Thirty minutes of walkthrough beats first real OOM alone.

How do SLOs relate to troubleshooting?

SLO burn triggers incident severity and deploy freeze policy.

Alerts should map to SLO windows, not only static thresholds.

What is the minimum dashboard set?

RED per service, goroutines, heap, GC pause, pool wait, CPU/memory vs limits, deploy markers.

Do best practices apply to batch workers?

Yes.

Workers need pool limits, ctx cancel, metrics, and profile access too.

How often update runbooks?

After every SEV-1 and when Go minor version upgrades change GC behavior.

Can lint enforce any of this?

CI can ban sql.Query without context in new code, require ReadHeaderTimeout, and block public pprof routes in config review.

What about multi-region Go fleets?

Handoff and dashboards note per-region SLO burn.

Profiles may be region-specific during partial outages.

When is enough prevention enough?

When SLOs are green, game days pass, and last two incidents had alerts fire before customer-wide impact.

Optimize features until the next measured regression.

Incident Response for Go Services - triage mental model
On-Call Handoff Checklist for Go Platforms - shift transitions
Post-Mortem Template for Go Incidents - RCA doc
Production Troubleshooting Basics - hands-on examples
Observability Best Practices - signal foundation
Performance Best Practices - profiling discipline

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Production Troubleshooting Best Practices

Search Documentation

Production Troubleshooting Best Practices