gRPC Best Practices

Q: How often should we run buf breaking?

On every PR touching .proto files. Treat failures like failing unit tests.

Q: How do we test gRPC handlers?

Use bufconn for in-process integration tests. Cover status codes, metadata, and stream EOF behavior.

Q: How do we version RPCs without duplicating logic?

Implement v2 handlers that delegate to shared domain packages. Avoid copy-pasting business rules across generated interfaces.

Versioning protos, deadlines, and load-balancing gRPC in K8s.

These rules keep contracts stable, RPCs observable, and rollouts safe from laptop through production clusters.

How to Use This List

Apply during schema changes, service scaffolding, and platform reviews.
Run buf breaking (or equivalent) in CI before merging .proto edits.
Pair with golangci-lint and integration tests that dial real servers via bufconn.
Revisit when ingress, mesh, or client platforms change.

A - Schema and versioning

Never reuse or renumber protobuf fields. Reserve deleted tags and names in .proto files.
Version packages (v1, v2) for incompatible API changes. Run dual servers or dual RPCs during migration windows.
Run breaking-change detection in CI against main. Block merges that change field types or wire numbers.
Keep go_package and Java package options stable. Moving generated import paths breaks downstream modules silently.
Use optional or FieldMask for partial updates. Distinguish unset fields from intentional zero values.

B - Server implementation

Embed Unimplemented<Service>Server on every service struct. Forward compatibility when RPCs are added.
Return status.Error with the closest standard code. Reserve Internal for true bugs, not validation failures.
Attach structured details (ErrorInfo, BadRequest) for machine-readable errors. Gateways and clients can branch without string parsing.
Honor context cancellation in long handlers and stream loops. Stop work when clients disconnect.
Enable reflection only in non-production or authenticated debug environments. Reflection exposes your full API surface.

C - Client calls and resilience

Set a deadline or timeout on every client context. Default "no deadline" hangs forever behind slow backends.
Retry only idempotent unary RPCs on Unavailable with backoff and jitter. Avoid retry storms during outages.
Reuse grpc.ClientConn across requests. Dialing per RPC wastes TCP and TLS handshakes.
Close connections on process shutdown. Release HTTP/2 resources cleanly alongside GracefulStop on servers.
Map retriable vs fatal codes explicitly in client libraries. Document behavior for application teams.

D - Interceptors, metadata, and security

Centralize JWT/mTLS validation in unary and stream interceptors. Handlers assume identity already verified.
Propagate trace context via metadata (traceparent or OTel carriers). Pair with otelgrpc handlers for spans.
Keep secrets in metadata or TLS, not in protobuf log fields. Redact authorization headers in logging interceptors.
Use TLS credentials in production for clients and servers. insecure credentials are for local dev only.
Register panic recovery interceptors at the server boundary. Panics become Internal with logged stack traces.

E - Streaming and performance

Default to unary RPCs until streaming is proven necessary. Simpler operations and load balancers.
Handle io.EOF as normal stream end, not always as error. Return appropriate status on failure paths only.
Bound in-memory buffering on client and server streams. Process or spill chunks instead of unbounded slices.
Tune max message sizes consistently on clients, servers, and gateways. Align limits end to end.
Configure keepalive for NATs and idle L4 load balancers. Prevent silent TCP drops on long-lived HTTP/2 connections.

F - Kubernetes and operations

Register grpc.health.v1 and wire liveness/readiness probes. Set NOT_SERVING until dependencies are warm.
Use connection-aware balancing (headless Service, mesh, or xDS). Naive L4 rotation breaks HTTP/2 streams.
Call GracefulStop on SIGTERM with a bounded Stop fallback. Match pod terminationGracePeriodSeconds.
Expose grpc-gateway or BFF for JSON clients instead of punching gRPC through public ingress blindly.
Document grpcurl debugging steps for on-call. Include required metadata headers and staging endpoints.

FAQs

What is the single highest-impact practice?

Client deadlines on every RPC.

They prevent cascading latency and make cancellation observable.

How often should we run buf breaking?

On every PR touching .proto files.

Treat failures like failing unit tests.

Should we compress all messages?

Enable compression for large repetitive payloads.

Skip it for tiny high-frequency messages where CPU dominates.

One server or split gateway?

Split when scale and blast radius differ.

Single binary is fine for small services if ports are separated clearly.

How do we test gRPC handlers?

Use bufconn for in-process integration tests.

Cover status codes, metadata, and stream EOF behavior.

When is server reflection acceptable?

Staging, local dev, or prod behind strong auth and network policy.

Never expose unauthenticated reflection on the public internet.

How do we version RPCs without duplicating logic?

Implement v2 handlers that delegate to shared domain packages.

Avoid copy-pasting business rules across generated interfaces.

What metrics matter?

RPC rate, latency histograms by method, error code counters, and in-flight stream gauges.

OpenTelemetry gRPC instrumentation covers most needs.

gRPC in Go: Contracts, Streaming, and Performance - section overview
gRPC Basics - setup patterns
Protocol Buffers Schema Design - evolution rules
gRPC Middleware, Interceptors & Metadata - cross-cutting concerns
REST vs gRPC Trade-off Guide - transport decisions

pkg.go.dev - Official package documentation and discovery site.
Effective Go - Official guide to writing clear, idiomatic Go.
Go Blog (Official) - Official announcements, deep dives, and release notes.
Learning Go (2nd Edition) - Jon Bodner.
The Go Programming Language (ebook) - Donovan & Kernighan.

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

gRPC Best Practices

Versioning protos, deadlines, and load-balancing gRPC in K8s.

These rules keep contracts stable, RPCs observable, and rollouts safe from laptop through production clusters.

How to Use This List

Apply during schema changes, service scaffolding, and platform reviews.
Run buf breaking (or equivalent) in CI before merging .proto edits.
Pair with golangci-lint and integration tests that dial real servers via bufconn.
Revisit when ingress, mesh, or client platforms change.

A - Schema and versioning

Never reuse or renumber protobuf fields. Reserve deleted tags and names in .proto files.
Version packages (v1, v2) for incompatible API changes. Run dual servers or dual RPCs during migration windows.
Run breaking-change detection in CI against main. Block merges that change field types or wire numbers.
Keep go_package and Java package options stable. Moving generated import paths breaks downstream modules silently.
Use optional or FieldMask for partial updates. Distinguish unset fields from intentional zero values.

B - Server implementation

Embed Unimplemented<Service>Server on every service struct. Forward compatibility when RPCs are added.
Return status.Error with the closest standard code. Reserve Internal for true bugs, not validation failures.
Attach structured details (ErrorInfo, BadRequest) for machine-readable errors. Gateways and clients can branch without string parsing.
Honor context cancellation in long handlers and stream loops. Stop work when clients disconnect.
Enable reflection only in non-production or authenticated debug environments. Reflection exposes your full API surface.

C - Client calls and resilience

Set a deadline or timeout on every client context. Default "no deadline" hangs forever behind slow backends.
Retry only idempotent unary RPCs on Unavailable with backoff and jitter. Avoid retry storms during outages.
Reuse grpc.ClientConn across requests. Dialing per RPC wastes TCP and TLS handshakes.
Close connections on process shutdown. Release HTTP/2 resources cleanly alongside GracefulStop on servers.
Map retriable vs fatal codes explicitly in client libraries. Document behavior for application teams.

D - Interceptors, metadata, and security

Centralize JWT/mTLS validation in unary and stream interceptors. Handlers assume identity already verified.
Propagate trace context via metadata (traceparent or OTel carriers). Pair with otelgrpc handlers for spans.
Keep secrets in metadata or TLS, not in protobuf log fields. Redact authorization headers in logging interceptors.
Use TLS credentials in production for clients and servers. insecure credentials are for local dev only.
Register panic recovery interceptors at the server boundary. Panics become Internal with logged stack traces.

E - Streaming and performance

Default to unary RPCs until streaming is proven necessary. Simpler operations and load balancers.
Handle io.EOF as normal stream end, not always as error. Return appropriate status on failure paths only.
Bound in-memory buffering on client and server streams. Process or spill chunks instead of unbounded slices.
Tune max message sizes consistently on clients, servers, and gateways. Align limits end to end.
Configure keepalive for NATs and idle L4 load balancers. Prevent silent TCP drops on long-lived HTTP/2 connections.

F - Kubernetes and operations

Register grpc.health.v1 and wire liveness/readiness probes. Set NOT_SERVING until dependencies are warm.
Use connection-aware balancing (headless Service, mesh, or xDS). Naive L4 rotation breaks HTTP/2 streams.
Call GracefulStop on SIGTERM with a bounded Stop fallback. Match pod terminationGracePeriodSeconds.
Expose grpc-gateway or BFF for JSON clients instead of punching gRPC through public ingress blindly.
Document grpcurl debugging steps for on-call. Include required metadata headers and staging endpoints.

FAQs

What is the single highest-impact practice?

Client deadlines on every RPC.

They prevent cascading latency and make cancellation observable.

How often should we run buf breaking?

On every PR touching .proto files.

Treat failures like failing unit tests.

Should we compress all messages?

Enable compression for large repetitive payloads.

Skip it for tiny high-frequency messages where CPU dominates.

One server or split gateway?

Split when scale and blast radius differ.

Single binary is fine for small services if ports are separated clearly.

How do we test gRPC handlers?

Use bufconn for in-process integration tests.

Cover status codes, metadata, and stream EOF behavior.

When is server reflection acceptable?

Staging, local dev, or prod behind strong auth and network policy.

Never expose unauthenticated reflection on the public internet.

How do we version RPCs without duplicating logic?

Implement v2 handlers that delegate to shared domain packages.

Avoid copy-pasting business rules across generated interfaces.

What metrics matter?

RPC rate, latency histograms by method, error code counters, and in-flight stream gauges.

OpenTelemetry gRPC instrumentation covers most needs.

gRPC in Go: Contracts, Streaming, and Performance - section overview
gRPC Basics - setup patterns
Protocol Buffers Schema Design - evolution rules
gRPC Middleware, Interceptors & Metadata - cross-cutting concerns
REST vs gRPC Trade-off Guide - transport decisions

pkg.go.dev - Official package documentation and discovery site.
Effective Go - Official guide to writing clear, idiomatic Go.
Go Blog (Official) - Official announcements, deep dives, and release notes.
Learning Go (2nd Edition) - Jon Bodner.
The Go Programming Language (ebook) - Donovan & Kernighan.

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

gRPC Best Practices

Search Documentation

gRPC Best Practices