Runtime & GC Best Practices

Q: What belongs in an incident runbook?

Steps to capture heap and goroutine profiles, NumGoroutine graph, recent deploy diff, cache config, and current GOGC/GOMEMLIMIT env.

Q: What about gRPC and controller-runtime operators?

Long-lived controllers leak badly without ctx cancel and Stop on informers. Apply goroutine and cache caps aggressively in operators.

Q: Can golangci-lint catch leak patterns?

Partially. Combine bodyclose, noctx, govet, and race detector CI with runtime metrics. Linters do not replace profiles.

Production tuning workflow with metrics and load tests.

These rules distill the Runtime & GC section: measure first, tune knobs with headroom, and fix retainers before chasing micro-optimizations.

How to Use This List

Apply during service design, Go minor upgrades, and memory incident postmortems.
Encode Tier A items in runbooks and deployment templates.
Pair with load tests that export gctrace and pprof artifacts.
Change one variable (GOGC, GOMEMLIMIT, alloc hot spots) per experiment.
Re-baseline after Go 1.26 Green Tea GC becomes your default collector.

A - Measure before tuning

Profile alloc_space and inuse_space under real load. Know top allocators before changing GOGC.
Export /gc/cpu:fraction-of-cpu-time and go_memstats_heap_alloc_bytes. Alert on sustained drift, not single spikes.
Track NumGoroutine and StackInuse with request rate. Separate heap leaks from goroutine stack growth.
Capture GODEBUG=gctrace=1 during load tests. Store output with deploy tags for comparison.
Run go test -bench with benchmem on hot packages. Catch alloc regressions in CI.

B - Container and cgroup sizing

Set GOMEMLIMIT to ~85-90% of pod memory limit. Leave headroom for stacks and off-heap memory.
Document limit vs request in README. Operators need both for capacity planning.
Match GOMAXPROCS to CPU limit when not automatic. Avoid invisible throttle with excess Ps.
Size limits from load test RSS peaks, not dev laptops. Include warmup and steady-state phases.
Plan for post-spike FreeOSMemory only after rare batch jobs. Not a substitute for leak fixes.

C - GC knob discipline

Prefer GOMEMLIMIT over aggressive low GOGC in Kubernetes. CPU starvation hurts latency more than soft caps help.
Change GOGC only with before/after p99 and gc_cpu metrics. One knob per canary.
Ban GOGC=off outside isolated benchmarks. Document any exception with time bounds.
Log debug.SetGCPercent changes if used dynamically. Rare, observable, reversible.
Re-benchmark after Go upgrades (especially 1.26 Green Tea). Default collector changes shift CPU profiles.

D - Allocation and escape hygiene

Fix hot make loops before tuning GC. Allocation rate drives mark CPU.
Run go build -gcflags="-m" on proven hot paths. Verify escapes match expectations.
Use sync.Pool only for short-lived scratch buffers. Never pool objects with hidden state.
Cap caches with TTL or LRU. No unbounded package-level map retainers.
Replace time.After in loops with reusable timers. Prevent timer heap leaks.

E - Goroutines, stacks, and shutdown

Bound fan-out with worker pools or semaphores. Goroutine count must plateau under sustained load.
Propagate context.Context and select on Done() in loops. Prevent parked goroutine leaks.
Close channels from producer side; close subscriptions on disconnect. Document ownership.
Use Shutdown(ctx) with bounded timeout on HTTP/gRPC stop. Drain workers before exit.
Prefer explicit Close() over runtime.SetFinalizer for handles. Finalizers are diagnostic-only.

FAQs

What is the minimum production GC observability?

Heap alloc gauge, goroutine count, GC CPU fraction, and process RSS.

Add gctrace capture in load tests before changing knobs.

Should every service set GOMEMLIMIT?

Yes for cgroup-limited deployments (Kubernetes, Cloud Run, ECS).

Bare-metal with ample RAM may rely on defaults initially but should still measure.

When is lowering GOGC correct?

When you have CPU headroom and must shrink peak heap for colocation or latency outliers tied to large heaps.

Validate with load tests.

How do Green Tea changes affect this list?

Green Tea lowers mark CPU for many workloads in Go 1.26.

Re-profile after upgrade; knob strategy stays the same.

What belongs in an incident runbook?

Steps to capture heap and goroutine profiles, NumGoroutine graph, recent deploy diff, cache config, and current GOGC/GOMEMLIMIT env.

Should we call runtime.GC in health checks?

No.

It adds latency noise and hides leaks temporarily.

Use dedicated diagnostic jobs instead.

How do chi, gin, and echo fit in?

Frameworks do not change GC mechanics.

Their middleware patterns affect goroutine lifetime and request-scoped allocations - profile your handlers.

What about gRPC and controller-runtime operators?

Long-lived controllers leak badly without ctx cancel and Stop on informers.

Apply goroutine and cache caps aggressively in operators.

How often should we rerun load tests?

Every Go minor upgrade, monthly for tier-1 services, and after any change to caches or concurrency limits.

Can golangci-lint catch leak patterns?

Partially.

Combine bodyclose, noctx, govet, and race detector CI with runtime metrics.

Linters do not replace profiles.

The Go Runtime: Scheduler, GC, and Memory - conceptual foundation
GOGC, GOMEMLIMIT & Finalizers - knob reference
Memory Leaks in Go: Common Causes - leak cheatsheet
GC Tuning for Latency-Sensitive Services - latency playbook
CPU & Heap Profiling with pprof - profiling how-to

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Runtime & GC Best Practices

Production tuning workflow with metrics and load tests.

These rules distill the Runtime & GC section: measure first, tune knobs with headroom, and fix retainers before chasing micro-optimizations.

How to Use This List

Apply during service design, Go minor upgrades, and memory incident postmortems.
Encode Tier A items in runbooks and deployment templates.
Pair with load tests that export gctrace and pprof artifacts.
Change one variable (GOGC, GOMEMLIMIT, alloc hot spots) per experiment.
Re-baseline after Go 1.26 Green Tea GC becomes your default collector.

A - Measure before tuning

Profile alloc_space and inuse_space under real load. Know top allocators before changing GOGC.
Export /gc/cpu:fraction-of-cpu-time and go_memstats_heap_alloc_bytes. Alert on sustained drift, not single spikes.
Track NumGoroutine and StackInuse with request rate. Separate heap leaks from goroutine stack growth.
Capture GODEBUG=gctrace=1 during load tests. Store output with deploy tags for comparison.
Run go test -bench with benchmem on hot packages. Catch alloc regressions in CI.

B - Container and cgroup sizing

Set GOMEMLIMIT to ~85-90% of pod memory limit. Leave headroom for stacks and off-heap memory.
Document limit vs request in README. Operators need both for capacity planning.
Match GOMAXPROCS to CPU limit when not automatic. Avoid invisible throttle with excess Ps.
Size limits from load test RSS peaks, not dev laptops. Include warmup and steady-state phases.
Plan for post-spike FreeOSMemory only after rare batch jobs. Not a substitute for leak fixes.

C - GC knob discipline

Prefer GOMEMLIMIT over aggressive low GOGC in Kubernetes. CPU starvation hurts latency more than soft caps help.
Change GOGC only with before/after p99 and gc_cpu metrics. One knob per canary.
Ban GOGC=off outside isolated benchmarks. Document any exception with time bounds.
Log debug.SetGCPercent changes if used dynamically. Rare, observable, reversible.
Re-benchmark after Go upgrades (especially 1.26 Green Tea). Default collector changes shift CPU profiles.

D - Allocation and escape hygiene

Fix hot make loops before tuning GC. Allocation rate drives mark CPU.
Run go build -gcflags="-m" on proven hot paths. Verify escapes match expectations.
Use sync.Pool only for short-lived scratch buffers. Never pool objects with hidden state.
Cap caches with TTL or LRU. No unbounded package-level map retainers.
Replace time.After in loops with reusable timers. Prevent timer heap leaks.

E - Goroutines, stacks, and shutdown

Bound fan-out with worker pools or semaphores. Goroutine count must plateau under sustained load.
Propagate context.Context and select on Done() in loops. Prevent parked goroutine leaks.
Close channels from producer side; close subscriptions on disconnect. Document ownership.
Use Shutdown(ctx) with bounded timeout on HTTP/gRPC stop. Drain workers before exit.
Prefer explicit Close() over runtime.SetFinalizer for handles. Finalizers are diagnostic-only.

FAQs

What is the minimum production GC observability?

Heap alloc gauge, goroutine count, GC CPU fraction, and process RSS.

Add gctrace capture in load tests before changing knobs.

Should every service set GOMEMLIMIT?

Yes for cgroup-limited deployments (Kubernetes, Cloud Run, ECS).

Bare-metal with ample RAM may rely on defaults initially but should still measure.

When is lowering GOGC correct?

When you have CPU headroom and must shrink peak heap for colocation or latency outliers tied to large heaps.

Validate with load tests.

How do Green Tea changes affect this list?

Green Tea lowers mark CPU for many workloads in Go 1.26.

Re-profile after upgrade; knob strategy stays the same.

What belongs in an incident runbook?

Steps to capture heap and goroutine profiles, NumGoroutine graph, recent deploy diff, cache config, and current GOGC/GOMEMLIMIT env.

Should we call runtime.GC in health checks?

No.

It adds latency noise and hides leaks temporarily.

Use dedicated diagnostic jobs instead.

How do chi, gin, and echo fit in?

Frameworks do not change GC mechanics.

Their middleware patterns affect goroutine lifetime and request-scoped allocations - profile your handlers.

What about gRPC and controller-runtime operators?

Long-lived controllers leak badly without ctx cancel and Stop on informers.

Apply goroutine and cache caps aggressively in operators.

How often should we rerun load tests?

Every Go minor upgrade, monthly for tier-1 services, and after any change to caches or concurrency limits.

Can golangci-lint catch leak patterns?

Partially.

Combine bodyclose, noctx, govet, and race detector CI with runtime metrics.

Linters do not replace profiles.

The Go Runtime: Scheduler, GC, and Memory - conceptual foundation
GOGC, GOMEMLIMIT & Finalizers - knob reference
Memory Leaks in Go: Common Causes - leak cheatsheet
GC Tuning for Latency-Sensitive Services - latency playbook
CPU & Heap Profiling with pprof - profiling how-to

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Runtime & GC Best Practices

Search Documentation

Runtime & GC Best Practices