Debugging & Defect Scenarios Best Practices

Q: What is the fastest win for flaky tests?

Run go test -race -count=50 on the flaky package. Most flakiness in Go services is races or time assumptions.

Prevention checklists and RCA templates for Go incidents.

These rules turn recurring sharp edges into review habits, CI gates, and post-incident notes that stop the same defect class from shipping twice.

How to Use This List

Walk sections A-D during code review for concurrent handlers, parsers, and repository layers.
Enable go test -race, govet, and golangci-lint on packages that share mutable state.
On incidents, classify symptoms using the explainer taxonomy before rewriting business logic.
Store RCA outcomes as ADRs or linter enablements, not only Slack threads.

A - Triage and observability

Classify the symptom family first (nil, slice, race, leak, time). Match stacks and metrics to a defect bucket before deep code changes.
Capture full panic stacks and goroutine counts at incident start. GOTRACEBACK=all and pprof endpoints accelerate root cause.
Log request IDs, RPC metadata, and RFC3339 timestamps on errors. Correlate user reports with single request traces.
Reproduce under -race and realistic parallelism. Flaky local passes hide data races that production load exposes.
Distinguish dependency outages from language defects. DNS, DB, and third-party 5xx mimic application bugs until boundaries are checked.

B - Nil, interfaces, and errors

Never return (nil, nil) for "not found" on pointer types. Return ErrNotFound or a typed sentinel callers handle with errors.Is.
Check pointers after nil errors when APIs allow absent values. err == nil does not guarantee non-nil *T.
Avoid typed nil pointers in error interfaces. Return untyped nil error or a concrete error value, not (*MyError)(nil) unless callers expect it.
Validate JSON-decoded pointers before use. null unmarshals to nil without error.
Enable nil static analysis (nilaway, staticcheck) on service boundaries. Catch return paths analyzers flag before runtime panic.

Copy or cap-limit before handing subslices to callers. Use s[low:high:high] or append([]T(nil), s...) when consumers may append.
Log len, cap, and backing mutations when debugging append bugs. Silent corruption rarely panics.
Do not share maps between goroutines without synchronization. Use mutex, sync.Map, or shard locks for caches.
Treat bytes.Buffer.Bytes() views as ephemeral. Copy if retaining beyond the next buffer mutation.
Run -race when slices or structs are mutated from handlers and background refresh loops.

D - Concurrency, context, and loops

Pass r.Context() through HTTP handlers to all blocking I/O. Client disconnect and Shutdown must stop work.
Always call cancel() from WithCancel/WithTimeout. defer cancel() immediately after creation.
Add select on ctx.Done() for goroutines blocked on channels. Prevent leaks when producers never send.
Copy loop variables into goroutines on libraries targeting older Go. v := v or parameter pass remains defensive after Go 1.22.
Avoid defer inside loops without a per-iteration function. Defers stack until function exit and leak file descriptors.
Use sync.Once for lazy singletons, not unsynchronized double-checked init. Race detector validates Once paths cheaply.

E - Time, shutdown, and production hygiene

Parse timestamps with explicit zones or ParseInLocation. Zone-less layouts use UTC, not the developer laptop zone.
Store instants in UTC; convert with In(loc) for display. Compare with Equal, not formatted strings.
Embed time/tzdata or install tzdata in minimal containers. LoadLocation fails silently in CI if never called.
Bound graceful shutdown with http.Server.Shutdown timeout. Cancel root context when signal arrives.
Document RCA template fields: symptom family, first bad frame, missing guard, permanent fix test. Close incidents with a regression test or linter rule.

FAQs

Which rules matter most for new Go services?

Context propagation, no shared mutable maps without locks, explicit error returns instead of (nil, nil), and -race in CI on handler packages.

Should every PR run the full checklist?

Use A for incidents, B-D for concurrent or parser changes, E for API and deploy changes.

Full pass monthly on hot packages.

How does this relate to the walkthrough articles?

Each letter maps to articles in this section.

Use the list for prevention; use walkthroughs when teaching reproduction steps.

What belongs in an RCA template?

Symptom family, timeline, goroutine/panic evidence, root cause frame, fix PR, and guard (test, lint, or monitor) proving recurrence is detected.

When is printf debugging enough?

Single-goroutine logic bugs and small repro programs.

Move to Delve and pprof when concurrency, leaks, or interface nil semantics appear.

Do best practices differ for TinyGo or WASM?

Core nil, slice, and race rules apply.

Some stdlib (tzdata, parts of reflection) differ - verify board targets per manifest pins.

How often should we run leak drills?

Quarterly on staging: load, stop traffic, confirm goroutine count returns to baseline within minutes.

Can linters replace review?

No - linters catch patterns reviewers miss.

Review catches wrong semantics linters cannot model.

What is the fastest win for flaky tests?

Run go test -race -count=50 on the flaky package.

Most flakiness in Go services is races or time assumptions.

Should we block merge on race detector failures?

Yes for packages that mutate shared state or spawn goroutines in handlers.

Pure packages may be excluded explicitly in CI config.

Debugging Go: Observable Failures and Sharp Edges - symptom taxonomy
Go Debugging Basics - tooling entry point
Nil Interface vs Nil Pointer Defect Walkthrough - nil rules in depth
Data Race Scenarios & race Detector Fixes - race fixes
Goroutine Leaks & Missing Context Cancellation - shutdown and leaks
Staticcheck and golangci-lint in CI - automate guards

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Debugging & Defect Scenarios Best Practices

Prevention checklists and RCA templates for Go incidents.

These rules turn recurring sharp edges into review habits, CI gates, and post-incident notes that stop the same defect class from shipping twice.

How to Use This List

Walk sections A-D during code review for concurrent handlers, parsers, and repository layers.
Enable go test -race, govet, and golangci-lint on packages that share mutable state.
On incidents, classify symptoms using the explainer taxonomy before rewriting business logic.
Store RCA outcomes as ADRs or linter enablements, not only Slack threads.

A - Triage and observability

Classify the symptom family first (nil, slice, race, leak, time). Match stacks and metrics to a defect bucket before deep code changes.
Capture full panic stacks and goroutine counts at incident start. GOTRACEBACK=all and pprof endpoints accelerate root cause.
Log request IDs, RPC metadata, and RFC3339 timestamps on errors. Correlate user reports with single request traces.
Reproduce under -race and realistic parallelism. Flaky local passes hide data races that production load exposes.
Distinguish dependency outages from language defects. DNS, DB, and third-party 5xx mimic application bugs until boundaries are checked.

B - Nil, interfaces, and errors

Never return (nil, nil) for "not found" on pointer types. Return ErrNotFound or a typed sentinel callers handle with errors.Is.
Check pointers after nil errors when APIs allow absent values. err == nil does not guarantee non-nil *T.
Avoid typed nil pointers in error interfaces. Return untyped nil error or a concrete error value, not (*MyError)(nil) unless callers expect it.
Validate JSON-decoded pointers before use. null unmarshals to nil without error.
Enable nil static analysis (nilaway, staticcheck) on service boundaries. Catch return paths analyzers flag before runtime panic.

Copy or cap-limit before handing subslices to callers. Use s[low:high:high] or append([]T(nil), s...) when consumers may append.
Log len, cap, and backing mutations when debugging append bugs. Silent corruption rarely panics.
Do not share maps between goroutines without synchronization. Use mutex, sync.Map, or shard locks for caches.
Treat bytes.Buffer.Bytes() views as ephemeral. Copy if retaining beyond the next buffer mutation.
Run -race when slices or structs are mutated from handlers and background refresh loops.

D - Concurrency, context, and loops

Pass r.Context() through HTTP handlers to all blocking I/O. Client disconnect and Shutdown must stop work.
Always call cancel() from WithCancel/WithTimeout. defer cancel() immediately after creation.
Add select on ctx.Done() for goroutines blocked on channels. Prevent leaks when producers never send.
Copy loop variables into goroutines on libraries targeting older Go. v := v or parameter pass remains defensive after Go 1.22.
Avoid defer inside loops without a per-iteration function. Defers stack until function exit and leak file descriptors.
Use sync.Once for lazy singletons, not unsynchronized double-checked init. Race detector validates Once paths cheaply.

E - Time, shutdown, and production hygiene

Parse timestamps with explicit zones or ParseInLocation. Zone-less layouts use UTC, not the developer laptop zone.
Store instants in UTC; convert with In(loc) for display. Compare with Equal, not formatted strings.
Embed time/tzdata or install tzdata in minimal containers. LoadLocation fails silently in CI if never called.
Bound graceful shutdown with http.Server.Shutdown timeout. Cancel root context when signal arrives.
Document RCA template fields: symptom family, first bad frame, missing guard, permanent fix test. Close incidents with a regression test or linter rule.

FAQs

Which rules matter most for new Go services?

Context propagation, no shared mutable maps without locks, explicit error returns instead of (nil, nil), and -race in CI on handler packages.

Should every PR run the full checklist?

Use A for incidents, B-D for concurrent or parser changes, E for API and deploy changes.

Full pass monthly on hot packages.

How does this relate to the walkthrough articles?

Each letter maps to articles in this section.

Use the list for prevention; use walkthroughs when teaching reproduction steps.

What belongs in an RCA template?

Symptom family, timeline, goroutine/panic evidence, root cause frame, fix PR, and guard (test, lint, or monitor) proving recurrence is detected.

When is printf debugging enough?

Single-goroutine logic bugs and small repro programs.

Move to Delve and pprof when concurrency, leaks, or interface nil semantics appear.

Do best practices differ for TinyGo or WASM?

Core nil, slice, and race rules apply.

Some stdlib (tzdata, parts of reflection) differ - verify board targets per manifest pins.

How often should we run leak drills?

Quarterly on staging: load, stop traffic, confirm goroutine count returns to baseline within minutes.

Can linters replace review?

No - linters catch patterns reviewers miss.

Review catches wrong semantics linters cannot model.

What is the fastest win for flaky tests?

Run go test -race -count=50 on the flaky package.

Most flakiness in Go services is races or time assumptions.

Should we block merge on race detector failures?

Yes for packages that mutate shared state or spawn goroutines in handlers.

Pure packages may be excluded explicitly in CI config.

Debugging Go: Observable Failures and Sharp Edges - symptom taxonomy
Go Debugging Basics - tooling entry point
Nil Interface vs Nil Pointer Defect Walkthrough - nil rules in depth
Data Race Scenarios & race Detector Fixes - race fixes
Goroutine Leaks & Missing Context Cancellation - shutdown and leaks
Staticcheck and golangci-lint in CI - automate guards

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Debugging & Defect Scenarios Best Practices

Search Documentation

Debugging & Defect Scenarios Best Practices