Kubernetes Operators Best Practices

Q: How do I enforce this list in CI?

Run make test, make manifests diff check, bundle validate, and golangci-lint on controller packages. Block merges when RBAC markers change without regenerated YAML.

Idempotent reconcilers, level-driven design, and production operator checklist.

These rules turn recurring operator sharp edges into review habits, CI gates, and packaging choices that keep controllers safe at scale.

How to Use This List

Walk sections A-E during design review before adding new API fields or child resource types.
Run envtest and operator-sdk bundle validate in CI on every release tag.
Enable leader election and scrape reconcile metrics before raising Deployment replicas above one.
Match OLM install modes and RBAC scope to the smallest tenancy model your product supports.

A - Reconcile design

Treat every reconcile as a full convergence pass. Compare desired spec to observed state; never assume a single event represents the full delta.
Keep reconcile idempotent. Running twice with the same cluster state must not create duplicate external resources or double patch children.
Return client.IgnoreNotFound on deleted primary objects. Avoid error metrics for races between delete and final reconcile.
Use Status().Update for conditions and readiness. Never patch status fields through generic Update on the full object.
Prefer RequeueAfter for expected waiting. Reserve returned errors for genuine failures worth alerting.
Set owner references on every child object you create. Enable garbage collection and Owns watches without label-only glue.

B - API, webhooks, and CRDs

Validate simple bounds in CRD markers; use webhooks for cross-field rules. Keep OpenAPI and admission layers complementary, not duplicated inconsistently.
Default missing spec in mutating webhooks. Reconcile should still tolerate legacy objects missing new fields.
Use failurePolicy: Fail for security-critical validation. Pair with HA webhook pods and cert monitoring.
Bump API version instead of breaking JSON field names. Add conversion when storage version changes shape.
Regenerate manifests after marker edits. Never hand-edit config/crd/bases without syncing Go types.
Expose meaningful kubectl get columns. Support engineers should see phase and readiness without YAML dumps.

C - Lifecycle, finalizers, and deletion

Add a finalizer before creating external dependencies. Block CR removal until cloud databases, DNS, or IAM teardown completes.
Make teardown idempotent. External delete APIs may be called more than once during retry storms.
Remove finalizers only after successful cleanup. Objects stuck in Terminating signal missing teardown logic.
Use predicates to reduce status noise on primary objects. Let child Owns traffic drive readiness updates.
Document orphan behavior for optional child objects. Clarify whether deleting the CR should delete shared cluster resources.

D - Testing, observability, and operations

Mirror production Scheme registration in tests. Missing types in envtest hide production decode failures.
Cover finalizer add, active, and delete paths in envtest or fake client tables. Deletion bugs are production incidents.
Enable leader election when replicas can exceed one. Standby pods should not reconcile concurrently.
Expose Prometheus metrics and alert on retry rates. workqueue_retries_total spikes precede incident pages.
Wire liveness and readiness probes to dedicated health ports. Do not tie probes to metrics scrape endpoints.
Start Manager with ctrl.SetupSignalHandler. Release leases promptly on rolling upgrades.

E - Packaging, RBAC, and multi-tenancy

Align CSV permissions with generated RBAC. OLM installs fail silently partial when rules drift from reconciler needs.
Disable install modes you do not implement. Claiming AllNamespaces without cluster RBAC blocks security review.
Scope Roles to tenant namespaces when possible. Prefer OwnNamespace over cluster-wide Secret list for SaaS platforms.
Pass --leader-elect and resource limits in CSV Deployment. HA without election duplicates external side effects.
Validate bundles in CI before catalog publish. operator-sdk bundle validate catches missing owned CRD descriptors.
Plan upgrade skips for bad intermediate releases. Document replaces and skips in CSV metadata.

FAQs

Which rules matter most for a first production operator?

Idempotent reconcile, owner references on children, finalizers for external resources, leader election with 2 replicas, and envtest coverage for create/delete paths.

Should every PR run the full checklist?

Use section A and D on every controller change.

Reserve OLM and multi-tenant sections for release and packaging PRs.

How does level-driven design differ from event handlers?

Level-driven reconcile reads current spec and cluster state each pass.

Event handlers that only append side effects miss drift and periodic resync.

When are webhooks mandatory?

When validation spans multiple spec fields or depends on admission-only context.

Simple min/max bounds belong in CRD markers first.

What is the most common production outage pattern?

Multiple active reconcilers without leader election mutating the same external system, or finalizers never cleared after failed teardown.

How do I enforce this list in CI?

Run make test, make manifests diff check, bundle validate, and golangci-lint on controller packages.

Block merges when RBAC markers change without regenerated YAML.

Should I use AllNamespaces install mode for convenience?

Only when customers explicitly need cluster-wide CR instances and accept cluster-scoped RBAC.

Otherwise OwnNamespace or SingleNamespace reduces blast radius.

How many metrics are enough?

Built-in controller-runtime workqueue metrics plus one or two domain counters (reconcile result, external API latency) per controller.

What belongs in status conditions?

Ready, progressing, degraded, and terminating states with Kubernetes meta condition conventions and clear reason codes.

When should I split multiple controllers?

When unrelated CRDs have independent failure domains or different RBAC needs.

A single Manager is fine for one product domain with shared caches.

Writing a Reconciler: ctrl.Request & Reconcile Loop - idempotent reconcile patterns
Watches, Predicates, Owner References & Finalizers - lifecycle safety
Operator Testing with envtest & fake client - CI testing harness
Leader Election, Metrics & Operator Observability - production operations
OLM Bundles, Upgrades & Multi-Tenant RBAC - packaging and tenancy

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Kubernetes Operators Best Practices

Idempotent reconcilers, level-driven design, and production operator checklist.

These rules turn recurring operator sharp edges into review habits, CI gates, and packaging choices that keep controllers safe at scale.

How to Use This List

Walk sections A-E during design review before adding new API fields or child resource types.
Run envtest and operator-sdk bundle validate in CI on every release tag.
Enable leader election and scrape reconcile metrics before raising Deployment replicas above one.
Match OLM install modes and RBAC scope to the smallest tenancy model your product supports.

A - Reconcile design

Treat every reconcile as a full convergence pass. Compare desired spec to observed state; never assume a single event represents the full delta.
Keep reconcile idempotent. Running twice with the same cluster state must not create duplicate external resources or double patch children.
Return client.IgnoreNotFound on deleted primary objects. Avoid error metrics for races between delete and final reconcile.
Use Status().Update for conditions and readiness. Never patch status fields through generic Update on the full object.
Prefer RequeueAfter for expected waiting. Reserve returned errors for genuine failures worth alerting.
Set owner references on every child object you create. Enable garbage collection and Owns watches without label-only glue.

B - API, webhooks, and CRDs

Validate simple bounds in CRD markers; use webhooks for cross-field rules. Keep OpenAPI and admission layers complementary, not duplicated inconsistently.
Default missing spec in mutating webhooks. Reconcile should still tolerate legacy objects missing new fields.
Use failurePolicy: Fail for security-critical validation. Pair with HA webhook pods and cert monitoring.
Bump API version instead of breaking JSON field names. Add conversion when storage version changes shape.
Regenerate manifests after marker edits. Never hand-edit config/crd/bases without syncing Go types.
Expose meaningful kubectl get columns. Support engineers should see phase and readiness without YAML dumps.

C - Lifecycle, finalizers, and deletion

Add a finalizer before creating external dependencies. Block CR removal until cloud databases, DNS, or IAM teardown completes.
Make teardown idempotent. External delete APIs may be called more than once during retry storms.
Remove finalizers only after successful cleanup. Objects stuck in Terminating signal missing teardown logic.
Use predicates to reduce status noise on primary objects. Let child Owns traffic drive readiness updates.
Document orphan behavior for optional child objects. Clarify whether deleting the CR should delete shared cluster resources.

D - Testing, observability, and operations

Mirror production Scheme registration in tests. Missing types in envtest hide production decode failures.
Cover finalizer add, active, and delete paths in envtest or fake client tables. Deletion bugs are production incidents.
Enable leader election when replicas can exceed one. Standby pods should not reconcile concurrently.
Expose Prometheus metrics and alert on retry rates. workqueue_retries_total spikes precede incident pages.
Wire liveness and readiness probes to dedicated health ports. Do not tie probes to metrics scrape endpoints.
Start Manager with ctrl.SetupSignalHandler. Release leases promptly on rolling upgrades.

E - Packaging, RBAC, and multi-tenancy

Align CSV permissions with generated RBAC. OLM installs fail silently partial when rules drift from reconciler needs.
Disable install modes you do not implement. Claiming AllNamespaces without cluster RBAC blocks security review.
Scope Roles to tenant namespaces when possible. Prefer OwnNamespace over cluster-wide Secret list for SaaS platforms.
Pass --leader-elect and resource limits in CSV Deployment. HA without election duplicates external side effects.
Validate bundles in CI before catalog publish. operator-sdk bundle validate catches missing owned CRD descriptors.
Plan upgrade skips for bad intermediate releases. Document replaces and skips in CSV metadata.

FAQs

Which rules matter most for a first production operator?

Idempotent reconcile, owner references on children, finalizers for external resources, leader election with 2 replicas, and envtest coverage for create/delete paths.

Should every PR run the full checklist?

Use section A and D on every controller change.

Reserve OLM and multi-tenant sections for release and packaging PRs.

How does level-driven design differ from event handlers?

Level-driven reconcile reads current spec and cluster state each pass.

Event handlers that only append side effects miss drift and periodic resync.

When are webhooks mandatory?

When validation spans multiple spec fields or depends on admission-only context.

Simple min/max bounds belong in CRD markers first.

What is the most common production outage pattern?

Multiple active reconcilers without leader election mutating the same external system, or finalizers never cleared after failed teardown.

How do I enforce this list in CI?

Run make test, make manifests diff check, bundle validate, and golangci-lint on controller packages.

Block merges when RBAC markers change without regenerated YAML.

Should I use AllNamespaces install mode for convenience?

Only when customers explicitly need cluster-wide CR instances and accept cluster-scoped RBAC.

Otherwise OwnNamespace or SingleNamespace reduces blast radius.

How many metrics are enough?

Built-in controller-runtime workqueue metrics plus one or two domain counters (reconcile result, external API latency) per controller.

What belongs in status conditions?

Ready, progressing, degraded, and terminating states with Kubernetes meta condition conventions and clear reason codes.

When should I split multiple controllers?

When unrelated CRDs have independent failure domains or different RBAC needs.

A single Manager is fine for one product domain with shared caches.

Writing a Reconciler: ctrl.Request & Reconcile Loop - idempotent reconcile patterns
Watches, Predicates, Owner References & Finalizers - lifecycle safety
Operator Testing with envtest & fake client - CI testing harness
Leader Election, Metrics & Operator Observability - production operations
OLM Bundles, Upgrades & Multi-Tenant RBAC - packaging and tenancy

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Kubernetes Operators Best Practices

Search Documentation

Kubernetes Operators Best Practices