Incident Response for Go Services

Q: How do deploys cause incidents without code bugs?

Changed GOMAXPROCS, new default timeouts, migration locks, feature flags, or different GOMEMLIMIT env vars alter runtime behavior without logic changes. Diff config and artifact metadata, not only source.

Q: How do I know if the database or the Go service is at fault?

Compare app sql.DB.Stats().WaitCount with DB server connection counts and slow-query logs. If the app waits on db.Conn while DB CPU is idle, suspect pool misconfiguration or missing context deadlines in handlers.

Production Go services fail in recognizable patterns.

On-call engineers who know those patterns spend minutes on triage instead of hours guessing.

This page builds the mental model for incident response on compiled, garbage-collected binaries: what to check first, how Go runtime signals differ from interpreted stacks, and when to stabilize traffic before profiling.

Summary

Incident response for Go is a time-boxed loop: confirm customer impact, scan SLIs and deploy history, apply safe mitigations, then collect Go-specific evidence (pprof, goroutine stacks, database/sql pool stats) to find root cause.
Insight: Go's low per-request overhead hides problems until saturation.

A connection pool at MaxOpenConns, a GC storm after a deploy, or a goroutine leak can look like "mysterious latency" until you know which dashboard and profile to open.

Key Concepts: SEV level, error budget, RED metrics, mitigation vs diagnosis, pprof, goroutine profile, sql.DB.Stats, deploy correlation, runbook, post-mortem.
When to Use: Paging alerts, elevated 5xx or p99 latency, OOMKilled pods, stuck rollouts, and handoffs between on-call shifts for Go platforms.
Limitations/Trade-offs: Not every outage is a Go bug - DNS, upstream SaaS, and network partitions dominate some incidents.

Profiling under extreme load can worsen symptoms if sampling is misconfigured.

Related Topics: Observability signals, CPU/heap profiling, GC tuning, connection pooling, goroutine leaks, graceful shutdown.

Foundations

An incident is unplanned degradation that threatens SLOs or user trust: elevated errors, timeouts, data unavailability, or security exposure.

Incident response is the coordinated work to restore service and learn what broke.

For Go services, the runtime gives you cheap concurrency and automatic memory management.

Those strengths become liabilities when goroutines block forever, allocations spike GC pause time, or sql.DB waits queue behind exhausted pools.

Experienced responders treat Go incidents like a symptom tree, not a single bug hunt:

User-visible pain          First questions
────────────────────────────────────────────────────────────
5xx / timeouts spike       Recent deploy? Upstream healthy?
p99 up, errors flat        GC, locks, pool wait, slow IO?
Memory climbing            Heap leak, goroutine leak, cache?
Pods OOMKilled             Limit vs working set, GOMEMLIMIT
DB timeouts everywhere     sql.DB stats, server connections

Mitigation reduces customer pain: rollback, scale replicas, shed load, disable a feature flag, or fail open on non-critical paths.

Diagnosis explains why mitigation worked and what to fix permanently.

During SEV-1, do both in parallel - but never let perfect root-cause analysis block a known-good rollback.

Mechanics & Interactions

Go incident triage follows a repeatable sequence that maps to tooling you should already expose in staging and production.

1. Confirm scope and severity.

Check error rate, latency percentiles, and saturation (CPU, memory, open connections) per service and region.

Correlate with deploy events, config pushes, certificate expiry, and dependency status pages.

2. Stabilize if the error budget is burning.

Rollback the last binary if correlation is strong.

Go rollbacks are fast when artifacts are versioned and migrations are backward compatible.

Scale horizontally only when the bottleneck is CPU-bound handler work, not DB pool or lock contention.

3. Read structured logs with request IDs.

slog or zap JSON lines should include trace_id, request_id, handler name, and error chains.

Panic stacks in Go are single-threaded snapshots - capture them from centralized logging, not only kubectl logs tail.

4. Open runtime diagnostics.

Signal	Where	What it tells you
CPU profile	`/debug/pprof/profile`	Hot functions under load
Heap profile	`/debug/pprof/heap`	Alloc sites, retained memory
Goroutine dump	`/debug/pprof/goroutine?debug=2`	Blocked stacks, leak patterns
`sql.DB.Stats`	Metrics exporter	Wait count, idle vs in-use
`runtime/metrics`	`/debug/vars` or OTel	GC pause, heap live

Profiles must be taken during representative load.

An idle pod's CPU profile is noise.

5. Narrow the failure class.

Deploy-correlated: diff binaries, flags, and schema migrations.
Saturation: GOMAXPROCS vs CPU limit, thread pool, epoll limits.
Memory/GC: heap growth, GOMEMLIMIT, allocation churn in hot paths.
Concurrency: mutex profiles, goroutine count monotonic increase.
Data layer: pool exhaustion, slow queries, missing context deadlines.

// Illustrative: export pool pressure during an incident
stats := db.Stats()
log.Printf("db open=%d inUse=%d idle=%d wait=%d waitDur=%s",
    stats.OpenConnections, stats.InUse, stats.Idle,
    stats.WaitCount, stats.WaitDuration)

6. Communicate and document.

Incident channel updates: impact, current hypothesis, mitigation in flight, next check time.

Save profile files, dashboard links, and deploy SHAs for the post-mortem.

Advanced Considerations & Applications

Kubernetes and Go add platform-specific wrinkles.

OOMKilled containers often mean heap plus goroutine stacks exceeded limits.memory without GOMEMLIMIT alignment.

Liveness probes that hit handlers doing real work can amplify load during incidents.

Readiness should fail when dependencies are unhealthy so load balancers stop sending traffic.

Multi-service incidents need blast-radius thinking.

A shared Redis or Postgres tier can make every Go microservice look sick simultaneously.

Trace exemplars from OpenTelemetry help prove whether latency is local handler work or downstream grpc waits.

Game days and runbooks convert this mental model into muscle memory.

Rehearse rollback, profile collection under load, and on-call handoff templates before production pages.

Approach	Strength	Weakness	Best Fit
Rollback first	Fast user relief	May hide latent bug	Strong deploy correlation
Scale out	Simple knob	Worsens pool/lock issues	Stateless CPU-bound work
Profile in prod	Ground-truth stacks	Needs auth, adds CPU sample cost	Latency mystery after stable deploy
Traffic shed	Protects core path	Reduced functionality	Partial outage, flag-ready

Regulatory and enterprise contexts require immutable incident timelines.

Automate capture of Go version (runtime.Version()), GOMAXPROCS, and build metadata in /healthz responses for support bundles.

Common Misconceptions

"Restarting the pod fixes it, so we are done" - Restarts clear memory leaks temporarily but guarantees recurrence.

The fix is cancellation, pool limits, or code change backed by profiles.

"Go has garbage collection, so memory incidents are impossible" - GC reclaims unreachable heap, not goroutines blocked on channels or unbounded caches.

OOM is common when limits ignore stack and off-heap cost.

"High CPU means bad algorithm" - Often it is JSON allocation, excessive logging, or lock contention visible only under concurrency.
"We need a debugger attached in production" - Most Go production incidents resolve with metrics, logs, and pprof.

Delve is for reproducible dev/staging failures.

"Post-mortems are blame assignment" - Blameless post-mortems focus on system gaps: missing alerts, unclear runbooks, absent pool metrics.

FAQs

What should I check in the first five minutes?

Confirm user impact and SLO burn.

Scan error rate, p99 latency, and recent deploys.

Rollback or scale if correlation is obvious.

Open service logs for panic stacks and upstream timeout messages.

How is Go incident response different from Node or Java?

Go binaries are single-process with lightweight goroutines.

Goroutine dumps and pprof are standard.

There is no separate JVM heap dump format - use heap profiles and GOMEMLIMIT instead.

Connection pooling is explicit via database/sql, not ORM-managed by default.

When should I collect a CPU profile during an outage?

After mitigation stabilizes errors or on a canary pod receiving production traffic.

Sample 20-30 seconds while load continues.

Avoid full heap profiling at MemProfileRate = 1 on every replica simultaneously.

What metrics belong on the incident dashboard?

RED: request rate, errors, duration histograms.

Plus Go-specific: goroutine count, GC pause, heap in use, sql.DB wait count, and process CPU/memory vs limits.

Should I page the whole team for every alert?

No.

Use SEV definitions: SEV-1 customer-wide impact gets immediate escalation; SEV-3 internal-only issues can wait for business hours with a documented owner.

How do deploys cause incidents without code bugs?

Changed GOMAXPROCS, new default timeouts, migration locks, feature flags, or different GOMEMLIMIT env vars alter runtime behavior without logic changes.

Diff config and artifact metadata, not only source.

What is the difference between mitigation and fix?

Mitigation restores SLOs temporarily (rollback, scale, shed load).

Fix prevents recurrence (pool sizing, context cancellation, query index).

Both belong in the timeline; only the fix closes the engineering ticket.

How do I know if the database or the Go service is at fault?

Compare app sql.DB.Stats().WaitCount with DB server connection counts and slow-query logs.

If the app waits on db.Conn while DB CPU is idle, suspect pool misconfiguration or missing context deadlines in handlers.

Is it safe to expose pprof during incidents?

Yes on localhost or authenticated admin ports.

Never expose /debug/pprof on the public internet.

CPU profiling adds small overhead; coordinate heap profiling to one pod at a time.

What evidence should a Go post-mortem include?

Timeline, deploy SHA, Go version, mitigation steps, profile links or files, sql.DB stats graphs, goroutine count trend, and concrete follow-up tickets with owners.

When should I stop investigating and hand off?

Hand off when mitigations are stable, evidence is saved, and the next check time is documented.

Incoming on-call needs open questions, not a live debugging session without context.

Do race detector results apply to production incidents?

The race detector is for CI and local reproduction.

Production symptoms that match races (nondeterministic wrong values) should trigger a staging repro with go test -race, not -race in prod.

Production Troubleshooting Basics - first-step examples for metrics, logs, and pprof
Live CPU & Heap Profiling Under Incident Load - safe profile capture during outages
OOM, GC Storm & Latency Spike Playbooks - memory incident runbooks
Database Connection Pool Exhaustion - diagnosing sql.DB saturation
Production Go: Logs, Metrics, Traces, and Signals - observability foundation
Debugging Go: Observable Failures and Sharp Edges - symptom families behind many incidents

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

On-call engineers who know those patterns spend minutes on triage instead of hours guessing.

Summary

Incident response for Go is a time-boxed loop: confirm customer impact, scan SLIs and deploy history, apply safe mitigations, then collect Go-specific evidence (pprof, goroutine stacks, database/sql pool stats) to find root cause.
Insight: Go's low per-request overhead hides problems until saturation.

A connection pool at MaxOpenConns, a GC storm after a deploy, or a goroutine leak can look like "mysterious latency" until you know which dashboard and profile to open.

Key Concepts: SEV level, error budget, RED metrics, mitigation vs diagnosis, pprof, goroutine profile, sql.DB.Stats, deploy correlation, runbook, post-mortem.
When to Use: Paging alerts, elevated 5xx or p99 latency, OOMKilled pods, stuck rollouts, and handoffs between on-call shifts for Go platforms.
Limitations/Trade-offs: Not every outage is a Go bug - DNS, upstream SaaS, and network partitions dominate some incidents.

Profiling under extreme load can worsen symptoms if sampling is misconfigured.

Related Topics: Observability signals, CPU/heap profiling, GC tuning, connection pooling, goroutine leaks, graceful shutdown.

Foundations

An incident is unplanned degradation that threatens SLOs or user trust: elevated errors, timeouts, data unavailability, or security exposure.

Incident response is the coordinated work to restore service and learn what broke.

For Go services, the runtime gives you cheap concurrency and automatic memory management.

Those strengths become liabilities when goroutines block forever, allocations spike GC pause time, or sql.DB waits queue behind exhausted pools.

Experienced responders treat Go incidents like a symptom tree, not a single bug hunt:

User-visible pain          First questions
────────────────────────────────────────────────────────────
5xx / timeouts spike       Recent deploy? Upstream healthy?
p99 up, errors flat        GC, locks, pool wait, slow IO?
Memory climbing            Heap leak, goroutine leak, cache?
Pods OOMKilled             Limit vs working set, GOMEMLIMIT
DB timeouts everywhere     sql.DB stats, server connections

Mitigation reduces customer pain: rollback, scale replicas, shed load, disable a feature flag, or fail open on non-critical paths.

Diagnosis explains why mitigation worked and what to fix permanently.

During SEV-1, do both in parallel - but never let perfect root-cause analysis block a known-good rollback.

Mechanics & Interactions

Go incident triage follows a repeatable sequence that maps to tooling you should already expose in staging and production.

1. Confirm scope and severity.

Check error rate, latency percentiles, and saturation (CPU, memory, open connections) per service and region.

Correlate with deploy events, config pushes, certificate expiry, and dependency status pages.

2. Stabilize if the error budget is burning.

Rollback the last binary if correlation is strong.

Go rollbacks are fast when artifacts are versioned and migrations are backward compatible.

Scale horizontally only when the bottleneck is CPU-bound handler work, not DB pool or lock contention.

3. Read structured logs with request IDs.

slog or zap JSON lines should include trace_id, request_id, handler name, and error chains.

Panic stacks in Go are single-threaded snapshots - capture them from centralized logging, not only kubectl logs tail.

4. Open runtime diagnostics.

Signal	Where	What it tells you
CPU profile	`/debug/pprof/profile`	Hot functions under load
Heap profile	`/debug/pprof/heap`	Alloc sites, retained memory
Goroutine dump	`/debug/pprof/goroutine?debug=2`	Blocked stacks, leak patterns
`sql.DB.Stats`	Metrics exporter	Wait count, idle vs in-use
`runtime/metrics`	`/debug/vars` or OTel	GC pause, heap live

Profiles must be taken during representative load.

An idle pod's CPU profile is noise.

5. Narrow the failure class.

Deploy-correlated: diff binaries, flags, and schema migrations.
Saturation: GOMAXPROCS vs CPU limit, thread pool, epoll limits.
Memory/GC: heap growth, GOMEMLIMIT, allocation churn in hot paths.
Concurrency: mutex profiles, goroutine count monotonic increase.
Data layer: pool exhaustion, slow queries, missing context deadlines.

// Illustrative: export pool pressure during an incident
stats := db.Stats()
log.Printf("db open=%d inUse=%d idle=%d wait=%d waitDur=%s",
    stats.OpenConnections, stats.InUse, stats.Idle,
    stats.WaitCount, stats.WaitDuration)

6. Communicate and document.

Incident channel updates: impact, current hypothesis, mitigation in flight, next check time.

Save profile files, dashboard links, and deploy SHAs for the post-mortem.

Advanced Considerations & Applications

Kubernetes and Go add platform-specific wrinkles.

OOMKilled containers often mean heap plus goroutine stacks exceeded limits.memory without GOMEMLIMIT alignment.

Liveness probes that hit handlers doing real work can amplify load during incidents.

Readiness should fail when dependencies are unhealthy so load balancers stop sending traffic.

Multi-service incidents need blast-radius thinking.

A shared Redis or Postgres tier can make every Go microservice look sick simultaneously.

Trace exemplars from OpenTelemetry help prove whether latency is local handler work or downstream grpc waits.

Game days and runbooks convert this mental model into muscle memory.

Rehearse rollback, profile collection under load, and on-call handoff templates before production pages.

Approach	Strength	Weakness	Best Fit
Rollback first	Fast user relief	May hide latent bug	Strong deploy correlation
Scale out	Simple knob	Worsens pool/lock issues	Stateless CPU-bound work
Profile in prod	Ground-truth stacks	Needs auth, adds CPU sample cost	Latency mystery after stable deploy
Traffic shed	Protects core path	Reduced functionality	Partial outage, flag-ready

Regulatory and enterprise contexts require immutable incident timelines.

Automate capture of Go version (runtime.Version()), GOMAXPROCS, and build metadata in /healthz responses for support bundles.

Common Misconceptions

"Restarting the pod fixes it, so we are done" - Restarts clear memory leaks temporarily but guarantees recurrence.

The fix is cancellation, pool limits, or code change backed by profiles.

"Go has garbage collection, so memory incidents are impossible" - GC reclaims unreachable heap, not goroutines blocked on channels or unbounded caches.

OOM is common when limits ignore stack and off-heap cost.

"High CPU means bad algorithm" - Often it is JSON allocation, excessive logging, or lock contention visible only under concurrency.
"We need a debugger attached in production" - Most Go production incidents resolve with metrics, logs, and pprof.

Delve is for reproducible dev/staging failures.

"Post-mortems are blame assignment" - Blameless post-mortems focus on system gaps: missing alerts, unclear runbooks, absent pool metrics.

FAQs

What should I check in the first five minutes?

Confirm user impact and SLO burn.

Scan error rate, p99 latency, and recent deploys.

Rollback or scale if correlation is obvious.

Open service logs for panic stacks and upstream timeout messages.

How is Go incident response different from Node or Java?

Go binaries are single-process with lightweight goroutines.

Goroutine dumps and pprof are standard.

There is no separate JVM heap dump format - use heap profiles and GOMEMLIMIT instead.

Connection pooling is explicit via database/sql, not ORM-managed by default.

When should I collect a CPU profile during an outage?

After mitigation stabilizes errors or on a canary pod receiving production traffic.

Sample 20-30 seconds while load continues.

Avoid full heap profiling at MemProfileRate = 1 on every replica simultaneously.

What metrics belong on the incident dashboard?

RED: request rate, errors, duration histograms.

Plus Go-specific: goroutine count, GC pause, heap in use, sql.DB wait count, and process CPU/memory vs limits.

Should I page the whole team for every alert?

No.

Use SEV definitions: SEV-1 customer-wide impact gets immediate escalation; SEV-3 internal-only issues can wait for business hours with a documented owner.

How do deploys cause incidents without code bugs?

Changed GOMAXPROCS, new default timeouts, migration locks, feature flags, or different GOMEMLIMIT env vars alter runtime behavior without logic changes.

Diff config and artifact metadata, not only source.

What is the difference between mitigation and fix?

Mitigation restores SLOs temporarily (rollback, scale, shed load).

Fix prevents recurrence (pool sizing, context cancellation, query index).

Both belong in the timeline; only the fix closes the engineering ticket.

How do I know if the database or the Go service is at fault?

Compare app sql.DB.Stats().WaitCount with DB server connection counts and slow-query logs.

If the app waits on db.Conn while DB CPU is idle, suspect pool misconfiguration or missing context deadlines in handlers.

Is it safe to expose pprof during incidents?

Yes on localhost or authenticated admin ports.

Never expose /debug/pprof on the public internet.

CPU profiling adds small overhead; coordinate heap profiling to one pod at a time.

What evidence should a Go post-mortem include?

Timeline, deploy SHA, Go version, mitigation steps, profile links or files, sql.DB stats graphs, goroutine count trend, and concrete follow-up tickets with owners.

When should I stop investigating and hand off?

Hand off when mitigations are stable, evidence is saved, and the next check time is documented.

Incoming on-call needs open questions, not a live debugging session without context.

Do race detector results apply to production incidents?

The race detector is for CI and local reproduction.

Production symptoms that match races (nondeterministic wrong values) should trigger a staging repro with go test -race, not -race in prod.

Production Troubleshooting Basics - first-step examples for metrics, logs, and pprof
Live CPU & Heap Profiling Under Incident Load - safe profile capture during outages
OOM, GC Storm & Latency Spike Playbooks - memory incident runbooks
Database Connection Pool Exhaustion - diagnosing sql.DB saturation
Production Go: Logs, Metrics, Traces, and Signals - observability foundation
Debugging Go: Observable Failures and Sharp Edges - symptom families behind many incidents

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Incident Response for Go Services

Search Documentation

Incident Response for Go Services