Enterprise Delivery Best Practices

A condensed checklist of delivery rules for Go microservices: artifact discipline, staged promotion, traffic safety, verification, flags, rollbacks, metrics, and compliance.

How to Use This List

Embed Tier A rules in golden pipeline templates before service teams fork repos.
Review Tier B at architecture review for new services.
Use Tier C for regulated or high-blast-radius systems.
Revisit quarterly against DORA trends and incident post-mortems.

A - Artifacts and CI

Build immutable images tagged with git SHA. Auditors and rollbacks need exact commit identity, not :latest.
Run go test ./... with -race on every PR. Data races caught in CI are cheaper than prod incidents.
Scan images and run govulncheck before promotion. CVE gates belong in pipeline, not ticket comments.
Use multi-stage Dockerfiles with distroless or minimal bases. Smaller attack surface; faster pulls during rollouts.
Pin Go toolchain in CI and Dockerfile. go 1.26 in go.mod must match builder image.
Generate SBOM or attach go version -m output to registry. Supply-chain audits ask what was in prod.

B - Promotion and environments

Promote the same digest staging → prod. Never rebuild on the deploy host.
Config via environment only. Same binary in every environment; secrets from manager, not image layers.
Block prod deploy without staging smoke pass. Smoke must hit business routes, not only /healthz.
Record deploy events for DORA metrics. Lead time, CFR, and MTTR require structured deploy logs.
Keep last-known-good SHA one command away. Document in service README and incident template.
Freeze deploys during active SEV-1 unless rollback. Prevents stacking failures during recovery.

C - Traffic shifting and runtime

Separate liveness and readiness probes. Readiness checks dependencies; liveness stays cheap.
Use canary or blue-green for user-facing APIs. Rolling update alone is weak for high CFR targets.
Set maxUnavailable: 0 for critical Deployments. No capacity drop during rollouts.
Implement graceful shutdown on SIGTERM. Match terminationGracePeriodSeconds to longest request.
Export build version in metrics and logs. Correlate incidents to SHA without guessing.
Warm connection pools before marking ready. Avoid 5xx spike when canary receives first traffic.

D - Schema, flags, and data

Follow expand-contract for database migrations. Deploy binary only after expand phase is live.
Default new feature flags to off in prod. Dark launch; enable via targeting after verification.
Evaluate flags once per request. Avoid provider storms in hot loops.
Retire flags within 90 days of full rollout. Permanent if flag branches become unmaintainable.
Plan data fixes separately from binary rollback. Bad writes may survive image revert.
Never run destructive down migrations in incidents. Forward-fix or restore from snapshot with DBA.

E - Verification and rollback

Wire progressive delivery analysis to Prometheus. Auto-promote on green; auto-rollback on threshold breach.
Run external synthetics on prod paths. In-VPC smoke misses DNS, TLS, and auth edge cases.
Define rollback triggers before deploy day. 5xx and p99 multipliers documented per service.
Practice rollback game days quarterly. Measure MTTR; update runbook when mesh or CD changes.
Disable flags before binary rollback when isolated. Faster recovery when canary binary is otherwise fine.
Capture rollback actor and digest in audit log. SOX and post-mortems need evidence.

F - Compliance and governance

Require change ticket ID for prod pipeline. Fail job if CHANGE_TICKET empty in regulated services.
Enforce separation of duties on prod approval. Merger ≠ sole approver.
Ship audit JSON to immutable log storage. WORM or SIEM retention per policy.
Document break-glass with post-hoc review SLA. Emergencies happen; evidence must still exist.
Sign artifacts with cosign or equivalent. Provenance links digest to pipeline and git ref.
Align compliance gates with DORA tracking. Approvals add lead time - automate everything else.

FAQs

Which tier should a new Go microservice implement first?

Tier A and B minimum before first prod customer.

Add C when external SLAs exist; D-Gates when regulated.

How do best practices interact with monorepos?

Per-service image and deploy events even when one pipeline builds many targets.

DORA and rollback docs are per service, not per repo.

Are daily prod deploys compatible with SOX?

Yes with automated attestations, approvals, and audit logs.

Manual CAB per deploy does not scale for Go microservices.

When can we skip canary?

Internal workers with no customer impact and strong staging parity may use guarded rolling updates.

Revisit when blast radius grows.

Platform team vs service team ownership?

Platform owns golden pipeline, mesh, and audit sinks.

Service owns smoke paths, flags, migrations, and SLO thresholds.

How many feature flags is healthy?

Track count and age in catalog.

Retire aggressively; tens of active flags per service is a smell.

Distroless and readiness probes?

No shell does not block HTTP/gRPC probes.

Binary must expose probe endpoints directly.

Link best practices to incident review?

Map each SEV post-mortem action to a checklist item.

Missing item becomes a pipeline or template fix.

Shipping Go Services Safely at Enterprise Scale - conceptual overview
Enterprise Delivery Basics - hands-on examples
Rollback Runbooks for Go Deployments - incident checklist
DORA Metrics for Go Teams - measure delivery health
Change Management & Compliance Gates - audit and approvals

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Enterprise Delivery Best Practices

A condensed checklist of delivery rules for Go microservices: artifact discipline, staged promotion, traffic safety, verification, flags, rollbacks, metrics, and compliance.

How to Use This List

Embed Tier A rules in golden pipeline templates before service teams fork repos.
Review Tier B at architecture review for new services.
Use Tier C for regulated or high-blast-radius systems.
Revisit quarterly against DORA trends and incident post-mortems.

A - Artifacts and CI

Build immutable images tagged with git SHA. Auditors and rollbacks need exact commit identity, not :latest.
Run go test ./... with -race on every PR. Data races caught in CI are cheaper than prod incidents.
Scan images and run govulncheck before promotion. CVE gates belong in pipeline, not ticket comments.
Use multi-stage Dockerfiles with distroless or minimal bases. Smaller attack surface; faster pulls during rollouts.
Pin Go toolchain in CI and Dockerfile. go 1.26 in go.mod must match builder image.
Generate SBOM or attach go version -m output to registry. Supply-chain audits ask what was in prod.

B - Promotion and environments

Promote the same digest staging → prod. Never rebuild on the deploy host.
Config via environment only. Same binary in every environment; secrets from manager, not image layers.
Block prod deploy without staging smoke pass. Smoke must hit business routes, not only /healthz.
Record deploy events for DORA metrics. Lead time, CFR, and MTTR require structured deploy logs.
Keep last-known-good SHA one command away. Document in service README and incident template.
Freeze deploys during active SEV-1 unless rollback. Prevents stacking failures during recovery.

C - Traffic shifting and runtime

Separate liveness and readiness probes. Readiness checks dependencies; liveness stays cheap.
Use canary or blue-green for user-facing APIs. Rolling update alone is weak for high CFR targets.
Set maxUnavailable: 0 for critical Deployments. No capacity drop during rollouts.
Implement graceful shutdown on SIGTERM. Match terminationGracePeriodSeconds to longest request.
Export build version in metrics and logs. Correlate incidents to SHA without guessing.
Warm connection pools before marking ready. Avoid 5xx spike when canary receives first traffic.

D - Schema, flags, and data

Follow expand-contract for database migrations. Deploy binary only after expand phase is live.
Default new feature flags to off in prod. Dark launch; enable via targeting after verification.
Evaluate flags once per request. Avoid provider storms in hot loops.
Retire flags within 90 days of full rollout. Permanent if flag branches become unmaintainable.
Plan data fixes separately from binary rollback. Bad writes may survive image revert.
Never run destructive down migrations in incidents. Forward-fix or restore from snapshot with DBA.

E - Verification and rollback

Wire progressive delivery analysis to Prometheus. Auto-promote on green; auto-rollback on threshold breach.
Run external synthetics on prod paths. In-VPC smoke misses DNS, TLS, and auth edge cases.
Define rollback triggers before deploy day. 5xx and p99 multipliers documented per service.
Practice rollback game days quarterly. Measure MTTR; update runbook when mesh or CD changes.
Disable flags before binary rollback when isolated. Faster recovery when canary binary is otherwise fine.
Capture rollback actor and digest in audit log. SOX and post-mortems need evidence.

F - Compliance and governance

Require change ticket ID for prod pipeline. Fail job if CHANGE_TICKET empty in regulated services.
Enforce separation of duties on prod approval. Merger ≠ sole approver.
Ship audit JSON to immutable log storage. WORM or SIEM retention per policy.
Document break-glass with post-hoc review SLA. Emergencies happen; evidence must still exist.
Sign artifacts with cosign or equivalent. Provenance links digest to pipeline and git ref.
Align compliance gates with DORA tracking. Approvals add lead time - automate everything else.

FAQs

Which tier should a new Go microservice implement first?

Tier A and B minimum before first prod customer.

Add C when external SLAs exist; D-Gates when regulated.

How do best practices interact with monorepos?

Per-service image and deploy events even when one pipeline builds many targets.

DORA and rollback docs are per service, not per repo.

Are daily prod deploys compatible with SOX?

Yes with automated attestations, approvals, and audit logs.

Manual CAB per deploy does not scale for Go microservices.

When can we skip canary?

Internal workers with no customer impact and strong staging parity may use guarded rolling updates.

Revisit when blast radius grows.

Platform team vs service team ownership?

Platform owns golden pipeline, mesh, and audit sinks.

Service owns smoke paths, flags, migrations, and SLO thresholds.

How many feature flags is healthy?

Track count and age in catalog.

Retire aggressively; tens of active flags per service is a smell.

Distroless and readiness probes?

No shell does not block HTTP/gRPC probes.

Binary must expose probe endpoints directly.

Link best practices to incident review?

Map each SEV post-mortem action to a checklist item.

Missing item becomes a pipeline or template fix.

Shipping Go Services Safely at Enterprise Scale - conceptual overview
Enterprise Delivery Basics - hands-on examples
Rollback Runbooks for Go Deployments - incident checklist
DORA Metrics for Go Teams - measure delivery health
Change Management & Compliance Gates - audit and approvals

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Enterprise Delivery Best Practices

Search Documentation

Enterprise Delivery Best Practices