Advanced Concurrency Best Practices

Patterns for resilient concurrent APIs and services.

These rules distill the advanced-concurrency section: bound parallelism, propagate cancellation, shut down cleanly, and prove safety with tests and profiles.

How to Use This List

Apply when designing a new concurrent service or reviewing a hot path before launch
Use Tier A on every PR touching goroutines or channels
Promote to Tier B after the first production incident or load test
Pair with Avoiding Goroutine Leaks & Channel Deadlocks during incidents
Revisit when Go minor releases change scheduler or x/sync behavior

A - Bound and protect

Cap parallel outbound I/O with semaphore or pool. Match limits to DB MaxOpenConns and partner API quotas.
Rate-limit request starts at the edge. Use token bucket before work enters expensive fan-out.
Shed load with explicit errors when queues are full. Return 503 or retryable errors instead of unbounded buffering.
Size channel buffers deliberately. Document why each buffer depth exists; avoid "9999" defaults.
Fan-out only when downstream capacity supports it. Measure 429s and latency before raising parallelism.

B - Context and cancellation

Pass context.Context as the first parameter after the receiver. Cancel propagates through QueryContext, gRPC, and HTTP clients.
Use r.Context() in HTTP handlers. Tie handler work to client disconnect and server shutdown.
Derive errgroup context with WithContext. First failure cancels siblings still doing I/O.
Separate background job context from request context. Cron and outbox workers cancel on SIGTERM, not client idle.
Honor ctx.Done() in every long loop or blocking retry. No goroutine should ignore cancel for seconds.

C - Structure and ownership

Document who closes each channel on the team wiki or code comment. One sender closes; receivers never close shared inputs.
Use worker pools for steady streams; errgroup for one-shot batches. Do not mix metaphors without clear boundaries.
Propagate errors with errgroup or typed Result structs. Do not panic from worker goroutines.
Prefer immutable config swaps via atomic.Pointer. Readers load snapshots; writers publish new structs.
Keep pipeline stages small and testable. Feed closed channels in unit tests for each stage.

D - Shutdown and lifecycle

Implement graceful HTTP shutdown with timeout. Server.Shutdown then cancel root context.
Close job channels only after producers finish. Use WaitGroup on producers before close.
Wait on workers before process exit. WaitGroup, errgroup Wait, or pool Shutdown in main.
Wire SIGTERM to the same path as intentional shutdown. Kubernetes expects drain within terminationGracePeriodSeconds.
Assert goroutine count returns to baseline in integration tests. Catch leaks before production.

E - Observability and verification

Run go test -race on concurrent packages in CI. Block merge on data races.
Export in-flight and queue depth metrics. Alert on sustained growth without traffic rise.
Profile goroutines under load in staging. pprof goroutine endpoint before major releases.
Load test with realistic downstream latency. Unbounded mocks hide backpressure bugs.
Log shed and rate-limit events at info with counters. Operators need visibility when protecting dependencies.

FAQs

What is the minimum bar for a concurrent HTTP service?

Request context propagation, outbound semaphore, graceful shutdown, and -race in CI on handler packages.

Channels or mutex for shared cache?

Mutex around a map is usually clearer.

Channels when passing ownership between stages, not for every shared read.

How many goroutines is too many?

No fixed number - too many is when memory, scheduling latency, or downstream errors rise without throughput gain.

Should every service use errgroup?

Use it for parallel fetches with shared cancel.

Worker pools and simple WaitGroup joins remain valid.

When are atomics enough?

Single-word metrics and flags updated at high frequency.

Multi-field invariants need mutex or transactions.

How do I review concurrent code?

Trace startup, steady state, error path, and shutdown on paper.

Check every go has exit and every channel has one closer.

Is unbuffered channel safer?

It synchronizes more aggressively but does not prevent leaks or deadlocks by itself.

What belongs in ADRs?

Pool sizes, rate limits, shed policy, and shutdown ordering - numbers reviewers can challenge.

Third-party pool libraries?

Acceptable when metrics needs justify dependency.

Stdlib channels plus errgroup cover most services.

How often rerun load tests?

After concurrency refactors and before peak season.

Automate a smoke load on every release candidate.

Production Concurrency: Patterns Beyond Hello Goroutine - conceptual overview
Advanced Concurrency Basics - runnable patterns
Avoiding Goroutine Leaks & Channel Deadlocks - incident cheatsheet
Concurrency Fundamentals Best Practices - channel and mutex basics
context Package Best Practices - cancellation habits

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Advanced Concurrency Best Practices

Patterns for resilient concurrent APIs and services.

These rules distill the advanced-concurrency section: bound parallelism, propagate cancellation, shut down cleanly, and prove safety with tests and profiles.

How to Use This List

Apply when designing a new concurrent service or reviewing a hot path before launch
Use Tier A on every PR touching goroutines or channels
Promote to Tier B after the first production incident or load test
Pair with Avoiding Goroutine Leaks & Channel Deadlocks during incidents
Revisit when Go minor releases change scheduler or x/sync behavior

A - Bound and protect

Cap parallel outbound I/O with semaphore or pool. Match limits to DB MaxOpenConns and partner API quotas.
Rate-limit request starts at the edge. Use token bucket before work enters expensive fan-out.
Shed load with explicit errors when queues are full. Return 503 or retryable errors instead of unbounded buffering.
Size channel buffers deliberately. Document why each buffer depth exists; avoid "9999" defaults.
Fan-out only when downstream capacity supports it. Measure 429s and latency before raising parallelism.

B - Context and cancellation

Pass context.Context as the first parameter after the receiver. Cancel propagates through QueryContext, gRPC, and HTTP clients.
Use r.Context() in HTTP handlers. Tie handler work to client disconnect and server shutdown.
Derive errgroup context with WithContext. First failure cancels siblings still doing I/O.
Separate background job context from request context. Cron and outbox workers cancel on SIGTERM, not client idle.
Honor ctx.Done() in every long loop or blocking retry. No goroutine should ignore cancel for seconds.

C - Structure and ownership

Document who closes each channel on the team wiki or code comment. One sender closes; receivers never close shared inputs.
Use worker pools for steady streams; errgroup for one-shot batches. Do not mix metaphors without clear boundaries.
Propagate errors with errgroup or typed Result structs. Do not panic from worker goroutines.
Prefer immutable config swaps via atomic.Pointer. Readers load snapshots; writers publish new structs.
Keep pipeline stages small and testable. Feed closed channels in unit tests for each stage.

D - Shutdown and lifecycle

Implement graceful HTTP shutdown with timeout. Server.Shutdown then cancel root context.
Close job channels only after producers finish. Use WaitGroup on producers before close.
Wait on workers before process exit. WaitGroup, errgroup Wait, or pool Shutdown in main.
Wire SIGTERM to the same path as intentional shutdown. Kubernetes expects drain within terminationGracePeriodSeconds.
Assert goroutine count returns to baseline in integration tests. Catch leaks before production.

E - Observability and verification

Run go test -race on concurrent packages in CI. Block merge on data races.
Export in-flight and queue depth metrics. Alert on sustained growth without traffic rise.
Profile goroutines under load in staging. pprof goroutine endpoint before major releases.
Load test with realistic downstream latency. Unbounded mocks hide backpressure bugs.
Log shed and rate-limit events at info with counters. Operators need visibility when protecting dependencies.

FAQs

What is the minimum bar for a concurrent HTTP service?

Request context propagation, outbound semaphore, graceful shutdown, and -race in CI on handler packages.

Channels or mutex for shared cache?

Mutex around a map is usually clearer.

Channels when passing ownership between stages, not for every shared read.

How many goroutines is too many?

No fixed number - too many is when memory, scheduling latency, or downstream errors rise without throughput gain.

Should every service use errgroup?

Use it for parallel fetches with shared cancel.

Worker pools and simple WaitGroup joins remain valid.

When are atomics enough?

Single-word metrics and flags updated at high frequency.

Multi-field invariants need mutex or transactions.

How do I review concurrent code?

Trace startup, steady state, error path, and shutdown on paper.

Check every go has exit and every channel has one closer.

Is unbuffered channel safer?

It synchronizes more aggressively but does not prevent leaks or deadlocks by itself.

What belongs in ADRs?

Pool sizes, rate limits, shed policy, and shutdown ordering - numbers reviewers can challenge.

Third-party pool libraries?

Acceptable when metrics needs justify dependency.

Stdlib channels plus errgroup cover most services.

How often rerun load tests?

After concurrency refactors and before peak season.

Automate a smoke load on every release candidate.

Production Concurrency: Patterns Beyond Hello Goroutine - conceptual overview
Advanced Concurrency Basics - runnable patterns
Avoiding Goroutine Leaks & Channel Deadlocks - incident cheatsheet
Concurrency Fundamentals Best Practices - channel and mutex basics
context Package Best Practices - cancellation habits

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Advanced Concurrency Best Practices

Search Documentation

Advanced Concurrency Best Practices