Post-Mortem Template for Go Incidents
Dense reference for blameless post-mortems on Go service outages.
Copy sections into your incident doc.
Collect Go-specific runtime evidence so follow-ups are actionable, not narrative-only.
How to Use This List
- Fill metadata and timeline within 48 hours while memory is fresh.
- Paste tables as-is; replace placeholders in angle brackets.
- Attach profile files and dashboard links in an evidence appendix.
- Every follow-up needs an owner and due date.
- Share read-only with engineering; avoid individual blame language.
Document Header
| Field | Value |
|---|---|
| Incident ID | <INC-1234> |
| Title | <Short customer-facing description> |
| SEV | <1 / 2 / 3> |
| Date (UTC) | <YYYY-MM-DD> |
| Duration | <start - end> |
| Author | <name> |
| Status | <Draft / Final> |
Executive Summary (3-5 sentences)
| Prompt | Your answer |
|---|---|
| Customer impact | <who, what broke, error/latency magnitude> |
| Root cause (one line) | <technical cause in plain language> |
| Trigger | <deploy, traffic, dependency, config> |
| Fix shipped | <rollback, patch, scale, flag> |
| Recurrence risk | <low/med/high until follow-ups done> |
Timeline
| Time (UTC) | Event | Actor |
|---|---|---|
<t0> | Alert fired: <alert name> | PagerDuty |
<t1> | On-call acknowledged | <name> |
<t2> | Mitigation: <rollback/scale> | <name> |
<t3> | Root cause identified | <name> |
<t4> | Incident resolved | <name> |
<t5> | Post-mortem scheduled | <name> |
Go Runtime Snapshot (collect at incident time)
| Item | Value |
|---|---|
| Go version | <runtime.Version() / image tag> |
| Build SHA | <git sha from /healthz or CI> |
GOMAXPROCS | <env or automaxprocs> |
GOMEMLIMIT | <value or unset> |
GOGC | <default 100 or override> |
| Container memory limit | <K8s limit> |
| Replica count | <at peak> |
| GC / heap metrics link | <dashboard URL> |
Evidence Checklist
| Artifact | Collected? | Location |
|---|---|---|
| CPU profile (20-30s) | <Y/N> | <path or link> |
Heap profile (inuse + alloc) | <Y/N> | <path> |
Goroutine dump debug=2 | <Y/N> | <path> |
sql.DB.Stats() graph | <Y/N> | <link> |
| Trace ID exemplars | <Y/N> | <link> |
| Deploy diff / config diff | <Y/N> | <link> |
| Panic stack logs | <Y/N> | <link> |
Five Whys (blameless)
| # | Why? |
|---|---|
| 1 | <symptom> because <immediate cause> |
| 2 | Because <deeper cause> |
| 3 | Because <process or design gap> |
| 4 | Because <missing guardrail> |
| 5 | Because <root organizational/technical gap> |
Contributing Factors
| Category | Factor |
|---|---|
| Code | <e.g. missing ctx deadline> |
| Config | <e.g. MaxOpenConns too high x replicas> |
| Observability | <e.g. no WaitCount alert> |
| Process | <e.g. deploy during freeze> |
| Dependency | <e.g. DB failover> |
What Went Well
<e.g. rollback runbook under 5 minutes><e.g. profiles saved before restart>
What Went Poorly
<e.g. pprof token expired><e.g. no GOMEMLIMIT on new service>
Follow-Up Actions
| ID | Action | Owner | Priority | Due |
|---|---|---|---|---|
| 1 | <code fix> | <team> | P0 | <date> |
| 2 | <alert> | <team> | P1 | <date> |
| 3 | <runbook update> | <team> | P2 | <date> |
| 4 | <game day scenario> | <team> | P2 | <date> |
Go-Specific Follow-Up Ideas
| Symptom family | Typical action |
|---|---|
| OOMKilled | Set GOMEMLIMIT, fix leak, right-size limit |
| GC / latency | Reduce alloc hot path, canary GOGC |
| Pool exhaustion | Resize MaxOpenConns, add WaitCount alert |
| Goroutine leak | ctx cancel, goleak CI, leak profile in staging |
| Panic | Nil interface audit, add -race test |
FAQs
How long should the post-mortem be?
Two to four pages including tables.
Depth matters in timeline and follow-ups, not prose length.
Is individual blame ever appropriate?
Focus on systems and guardrails.
HR-sensitive conduct is out of band from the engineering doc.
What if we never found root cause?
Document hypotheses ruled out, evidence gaps, and monitoring added to catch the next occurrence.
Should customer impact include internal-only outages?
Yes for SEV definitions.
Mark customer-visible vs internal clearly in executive summary.
Who approves follow-up tickets?
Service owner and EM for P0/P1.
Track in the same system as normal engineering work.
When is a post-mortem required?
Team policy varies.
Common rule: any SEV-1/2 or SLO budget burn above threshold.
Do we need profiles if we rolled back?
Yes.
Pre-rollback profiles prove the regression; post-rollback confirms fix.
How do we store large .prof files?
Object storage with incident ID prefix.
Link from evidence table; do not paste binary in wiki.
Should we mention Go version upgrades?
Always record runtime.Version().
GC behavior differs across releases, especially 1.26 Green Tea.
Can this template cover security incidents?
Use the same timeline structure.
Redact sensitive details; involve security team separately.
Related
- Incident Response for Go Services - triage mental model
- On-Call Handoff Checklist for Go Platforms - shift transition
- OOM, GC Storm & Latency Spike Playbooks - memory runbook
- Database Connection Pool Exhaustion - pool incidents
- Production Troubleshooting Best Practices - prevention habits
- Observability Best Practices - signal quality
Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).