Cloud & Serverless Best Practices

Cold start tuning, observability, and cost guardrails.

Apply these rules in CI templates, service README files, and platform reviews so every Go deployment gets predictable latency, bills, and incident response.

How to Use This List

Adopt Tier A items before production traffic on any new platform.
Wire Tier B observability into dashboards and alerts during the first deploy week.
Revisit cost rules monthly - serverless bills surprise teams when concurrency or memory defaults drift.

A - Cold start and packaging

Cross-compile with CGO_ENABLED=0 for Lambda and minimal containers. Avoids dynamic linker surprises on provided runtimes.
Strip binaries with -ldflags="-s -w" unless you need Delve symbols in dev. Smaller zips load faster on cold start.
Name Lambda artifacts bootstrap at zip root for provided.al2023. Wrong names fail at runtime with opaque errors.
Lazy-init AWS/GCP clients with sync.Once, not in init(). Keeps unused code paths off the cold path.
Benchmark arm64 vs amd64 for your dependency tree. Graviton often wins for Go but verify before fleet-wide cutover.
Keep container images under a few hundred MB. Large layers slow Cloud Run and ACA cold starts.

B - Handlers and correctness

Design for at-least-once delivery on queues and webhooks. Idempotent keys and dedupe stores are mandatory.
Pass context.Context from the platform into all SDK calls. Honors API Gateway, Cloud Run, and client disconnect deadlines.
Bound work with context.WithTimeout slightly below platform limits. Return structured errors before hard kills.
Avoid mutable local disk except /tmp on Lambda. Treat instance storage as ephemeral.
Reuse DB pools across warm invocations; size pools to concurrency. Prevents connection storms after scale-out events.
Return distinct errors for retryable vs terminal failures. Drives DLQ behavior and alert noise quality.

C - Security and identity

Never commit long-lived cloud access keys to git or images. Use execution roles, workload identity, managed identity.
Scope IAM to resource ARN prefixes per service. Reject * admin policies on runtime identities.
Fetch secrets from Secrets Manager / Secret Manager at runtime. Cache in memory with rotation awareness.
Enable encryption at rest on buckets holding user data. Default bucket policies or SSE-KMS on sensitive prefixes.
Require authentication on public HTTP services unless explicitly a health demo. Lock down --allow-unauthenticated defaults after spikes.
Run govulncheck on modules touching cloud SDKs. Supply-chain issues land in credential and HTTP paths first.

D - Observability

Emit JSON logs with log/slog to stdout/stderr. CloudWatch, Cloud Logging, and App Insights parse structured fields.
Include request ID, trace ID, and cold-start flag in log attributes. Separates warm vs cold latency incidents.
Export RED metrics: rate, errors, duration per route and per dependency. Prometheus or vendor agents on containers; Lambda extensions where needed.
Alert on error rate and p99 duration, not CPU alone. Serverless autoscaler hides CPU saturation until timeouts spike.
Trace outbound SDK calls when latency SLOs tighten. OpenTelemetry wrappers exist for aws-sdk-go-v2 and gRPC.
Log platform metadata (function version, revision) on startup once. Speeds rollback decisions during incidents.

E - Networking and resilience

Set ReadHeaderTimeout on every http.Server in containers. Blocks slowloris before handlers run.
Handle SIGTERM with bounded http.Server.Shutdown. Cloud Run and ECS send terminate signals before SIGKILL.
Separate liveness and readiness health endpoints. Stop traffic without restarting tasks when dependencies fail.
Document VPC connector / security group requirements in README. On-call should not grep Terraform during outages.
Retry idempotent SDK operations only; use jittered backoff for app-level retries. Prevents amplify loops on throttling.
Configure DLQs for async triggers with alarm on depth. Poison messages must surface within minutes.

F - Cost guardrails

Right-size Lambda memory after load tests - memory scales CPU. Over-provisioning doubles bills without latency wins.
Set Cloud Run max instances and concurrency caps in prod. Protects databases from unbounded scale events.
Use scale-to-zero only when cold starts fit SLOs; otherwise set min instances. Economic vs latency trade-off made explicit.
Delete unused preview environments and stale function versions. Orphaned resources accumulate quietly.
Monitor egress to object storage and cross-AZ traffic. Data transfer often exceeds compute on export workloads.
Schedule non-urgent batch as jobs, not always-on pods. Fargate hours add up for nightly-only work.

FAQs

What is the first metric to watch after launch?

p99 end-to-end latency split by cold vs warm path.

If cold dominates, fix packaging and init before raising memory spend.

How strict should IAM be on dev accounts?

Dev can be looser for velocity but must not share prod roles or secrets.

Use separate accounts or namespaces with parallel policies.

Should I use Lambda layers for shared Go dependencies?

Layers help polyglot teams share agents; pure Go binaries rarely need layers unless distributing observability extensions.

Prefer smaller single binaries first.

How do I test idempotency?

Chaos-test duplicate SQS deliveries and HTTP retries in integration tests; assert one visible side effect.

When do I leave serverless for Kubernetes?

When you need operators, custom CRDs, service mesh features, or uniform multi-team cluster standards that managed functions cannot host.

Does Go's GC affect Lambda memory sizing?

Yes - leave headroom above heap live size; OOM kills are harder to debug than slightly higher memory settings.

Go in the Cloud: Lambda, Cloud Run, and Beyond - conceptual foundation
Cloud Deploy Basics - hands-on deploy examples
AWS Lambda Custom Runtime & provided.al2023 - packaging details
Secrets, IAM & Workload Identity - credential posture
Serverless Go Architecture Decision Guide - platform choice
Incident Response for Go Services - when alerts fire

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Cloud & Serverless Best Practices

Cold start tuning, observability, and cost guardrails.

Apply these rules in CI templates, service README files, and platform reviews so every Go deployment gets predictable latency, bills, and incident response.

How to Use This List

Adopt Tier A items before production traffic on any new platform.
Wire Tier B observability into dashboards and alerts during the first deploy week.
Revisit cost rules monthly - serverless bills surprise teams when concurrency or memory defaults drift.

A - Cold start and packaging

Cross-compile with CGO_ENABLED=0 for Lambda and minimal containers. Avoids dynamic linker surprises on provided runtimes.
Strip binaries with -ldflags="-s -w" unless you need Delve symbols in dev. Smaller zips load faster on cold start.
Name Lambda artifacts bootstrap at zip root for provided.al2023. Wrong names fail at runtime with opaque errors.
Lazy-init AWS/GCP clients with sync.Once, not in init(). Keeps unused code paths off the cold path.
Benchmark arm64 vs amd64 for your dependency tree. Graviton often wins for Go but verify before fleet-wide cutover.
Keep container images under a few hundred MB. Large layers slow Cloud Run and ACA cold starts.

B - Handlers and correctness

Design for at-least-once delivery on queues and webhooks. Idempotent keys and dedupe stores are mandatory.
Pass context.Context from the platform into all SDK calls. Honors API Gateway, Cloud Run, and client disconnect deadlines.
Bound work with context.WithTimeout slightly below platform limits. Return structured errors before hard kills.
Avoid mutable local disk except /tmp on Lambda. Treat instance storage as ephemeral.
Reuse DB pools across warm invocations; size pools to concurrency. Prevents connection storms after scale-out events.
Return distinct errors for retryable vs terminal failures. Drives DLQ behavior and alert noise quality.

C - Security and identity

Never commit long-lived cloud access keys to git or images. Use execution roles, workload identity, managed identity.
Scope IAM to resource ARN prefixes per service. Reject * admin policies on runtime identities.
Fetch secrets from Secrets Manager / Secret Manager at runtime. Cache in memory with rotation awareness.
Enable encryption at rest on buckets holding user data. Default bucket policies or SSE-KMS on sensitive prefixes.
Require authentication on public HTTP services unless explicitly a health demo. Lock down --allow-unauthenticated defaults after spikes.
Run govulncheck on modules touching cloud SDKs. Supply-chain issues land in credential and HTTP paths first.

D - Observability

Emit JSON logs with log/slog to stdout/stderr. CloudWatch, Cloud Logging, and App Insights parse structured fields.
Include request ID, trace ID, and cold-start flag in log attributes. Separates warm vs cold latency incidents.
Export RED metrics: rate, errors, duration per route and per dependency. Prometheus or vendor agents on containers; Lambda extensions where needed.
Alert on error rate and p99 duration, not CPU alone. Serverless autoscaler hides CPU saturation until timeouts spike.
Trace outbound SDK calls when latency SLOs tighten. OpenTelemetry wrappers exist for aws-sdk-go-v2 and gRPC.
Log platform metadata (function version, revision) on startup once. Speeds rollback decisions during incidents.

E - Networking and resilience

Set ReadHeaderTimeout on every http.Server in containers. Blocks slowloris before handlers run.
Handle SIGTERM with bounded http.Server.Shutdown. Cloud Run and ECS send terminate signals before SIGKILL.
Separate liveness and readiness health endpoints. Stop traffic without restarting tasks when dependencies fail.
Document VPC connector / security group requirements in README. On-call should not grep Terraform during outages.
Retry idempotent SDK operations only; use jittered backoff for app-level retries. Prevents amplify loops on throttling.
Configure DLQs for async triggers with alarm on depth. Poison messages must surface within minutes.

F - Cost guardrails

Right-size Lambda memory after load tests - memory scales CPU. Over-provisioning doubles bills without latency wins.
Set Cloud Run max instances and concurrency caps in prod. Protects databases from unbounded scale events.
Use scale-to-zero only when cold starts fit SLOs; otherwise set min instances. Economic vs latency trade-off made explicit.
Delete unused preview environments and stale function versions. Orphaned resources accumulate quietly.
Monitor egress to object storage and cross-AZ traffic. Data transfer often exceeds compute on export workloads.
Schedule non-urgent batch as jobs, not always-on pods. Fargate hours add up for nightly-only work.

FAQs

What is the first metric to watch after launch?

p99 end-to-end latency split by cold vs warm path.

If cold dominates, fix packaging and init before raising memory spend.

How strict should IAM be on dev accounts?

Dev can be looser for velocity but must not share prod roles or secrets.

Use separate accounts or namespaces with parallel policies.

Should I use Lambda layers for shared Go dependencies?

Layers help polyglot teams share agents; pure Go binaries rarely need layers unless distributing observability extensions.

Prefer smaller single binaries first.

How do I test idempotency?

Chaos-test duplicate SQS deliveries and HTTP retries in integration tests; assert one visible side effect.

When do I leave serverless for Kubernetes?

When you need operators, custom CRDs, service mesh features, or uniform multi-team cluster standards that managed functions cannot host.

Does Go's GC affect Lambda memory sizing?

Yes - leave headroom above heap live size; OOM kills are harder to debug than slightly higher memory settings.

Go in the Cloud: Lambda, Cloud Run, and Beyond - conceptual foundation
Cloud Deploy Basics - hands-on deploy examples
AWS Lambda Custom Runtime & provided.al2023 - packaging details
Secrets, IAM & Workload Identity - credential posture
Serverless Go Architecture Decision Guide - platform choice
Incident Response for Go Services - when alerts fire

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Cloud & Serverless Best Practices

Search Documentation

Cloud & Serverless Best Practices