Observability Basics

9 examples to get you started with Observability - 7 basic and 2 intermediate.

Prerequisites

Install Go 1.26.x or later from go.dev/dl.
Create a module: mkdir oblab && cd oblab && go mod init example.com/oblab.
For metrics and tracing examples later in the section: go get github.com/prometheus/client_golang/prometheus/promhttp and OpenTelemetry modules as needed.

Basic Examples

1. Default slog Logger

Go 1.21+ sets a default structured logger you can call without setup.

package main
 
import (
    "log/slog"
)
 
func main() {
    slog.Info("server starting", "port", 8080)
}

slog.Info writes key-value pairs after the message.
Default output is text format to stderr.
Replace the default with JSON in production services.

Related: Structured Logging with slog - handlers and levels

2. JSON slog Handler

JSON logs parse cleanly in Loki, Elasticsearch, and CloudWatch.

package main
 
import (
    "log/slog"
    "os"
)
 
func main() {
    logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
    slog.SetDefault(logger)
    slog.Info("ready", "service", "api")
}

NewJSONHandler emits one JSON object per line.
SetDefault makes package-level slog calls use your handler.
Set Level to slog.LevelDebug only in development.

Related: Production Go: Logs, Metrics, Traces, and Signals - observability overview

3. Request ID Middleware

Attach a correlation ID to context for logs and downstream calls.

package main
 
import (
    "context"
    "crypto/rand"
    "encoding/hex"
    "net/http"
)
 
type ctxKey string
 
const requestIDKey ctxKey = "request_id"
 
func withRequestID(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        id := r.Header.Get("X-Request-ID")
        if id == "" {
            var b [16]byte
            rand.Read(b[:])
            id = hex.EncodeToString(b[:])
        }
        ctx := context.WithValue(r.Context(), requestIDKey, id)
        w.Header().Set("X-Request-ID", id)
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

Prefer incoming X-Request-ID when present for trace continuity.
Store IDs in context with unexported key types to avoid collisions.
Echo the ID on the response for client-side support tickets.

Related: Middleware Chains & Request Context - wrapping handlers

4. Logging Middleware with slog

Log method, path, and status for every HTTP request.

package main
 
import (
    "log/slog"
    "net/http"
    "time"
)
 
type statusWriter struct {
    http.ResponseWriter
    code int
}
 
func (w *statusWriter) WriteHeader(code int) {
    w.code = code
    w.ResponseWriter.WriteHeader(code)
}
 
func loggingMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        sw := &statusWriter{ResponseWriter: w, code: http.StatusOK}
        next.ServeHTTP(sw, r)
        slog.Info("request",
            "method", r.Method,
            "path", r.URL.Path,
            "status", sw.code,
            "duration_ms", time.Since(start).Milliseconds(),
        )
    })
}

Wrap ResponseWriter to capture the final status code.
Log duration in milliseconds for quick grep in log UIs.
Keep health check paths at DEBUG level in production to reduce noise.

Related: Structured Logging with slog - middleware patterns

5. Liveness Handler `/healthz`

Liveness should be cheap and never call external systems.

package main
 
import (
    "net/http"
)
 
func healthz(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("ok"))
}
 
func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("GET /healthz", healthz)
    http.ListenAndServe(":8080", mux)
}

Return 200 when the process and HTTP server are running.
Do not check databases on liveness - that causes restart loops.
Kubernetes livenessProbe hits this path on a short interval.

Related: Health, Readiness & Liveness Probes - probe design

6. Readiness Handler `/readyz`

Readiness gates traffic when dependencies are unhealthy.

package main
 
import (
    "net/http"
    "sync/atomic"
)
 
var ready atomic.Bool
 
func readyz(w http.ResponseWriter, r *http.Request) {
    if !ready.Load() {
        http.Error(w, "not ready", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("ready"))
}
 
func main() {
    ready.Store(true)
    mux := http.NewServeMux()
    mux.HandleFunc("GET /readyz", readyz)
    http.ListenAndServe(":8080", mux)
}

Return 503 when the service cannot accept traffic.
Flip ready to false during startup until migrations finish.
Load balancers and Kubernetes use readiness to stop sending requests.

Related: Health, Readiness & Liveness Probes - dependency checks

7. Environment Variable Config

12-factor apps read config from the environment at startup.

package main
 
import (
    "log/slog"
    "os"
)
 
func main() {
    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }
    slog.Info("config loaded", "port", port, "env", os.Getenv("APP_ENV"))
}

Fail fast in main when required variables are missing.
Log non-secret config at startup for deploy verification.
Document every env var in the service README.

Related: Configuration Management: Env, Flags & Viper - full config patterns

Intermediate Examples

8. Prometheus Counter on `/metrics`

Expose a scrape endpoint for request totals.

package main
 
import (
    "net/http"
 
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)
 
var requests = prometheus.NewCounterVec(
    prometheus.CounterOpts{Name: "http_requests_total", Help: "HTTP requests"},
    []string{"method", "code"},
)
 
func main() {
    prometheus.MustRegister(requests)
    mux := http.NewServeMux()
    mux.Handle("GET /metrics", promhttp.Handler())
    mux.HandleFunc("GET /hello", func(w http.ResponseWriter, r *http.Request) {
        requests.WithLabelValues(r.Method, "200").Inc()
        w.Write([]byte("hello"))
    })
    http.ListenAndServe(":8080", mux)
}

Register metrics once in main, not per request handler.
Use bounded label values (method, status class) to control cardinality.
Restrict /metrics to internal networks or auth middleware.

Related: Prometheus Metrics & RED/USE Dashboards - RED dashboards

9. Graceful Shutdown on SIGTERM

Drain in-flight HTTP requests before process exit.

package main
 
import (
    "context"
    "log/slog"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)
 
func main() {
    srv := &http.Server{Addr: ":8080", Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("ok"))
    })}
    go func() {
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            slog.Error("listen failed", "err", err)
            os.Exit(1)
        }
    }()
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    <-quit
    ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
    defer cancel()
    if err := srv.Shutdown(ctx); err != nil {
        slog.Error("shutdown failed", "err", err)
    }
    slog.Info("server stopped")
}

Shutdown stops accepting new connections and waits for active requests.
Bound shutdown context to Kubernetes terminationGracePeriodSeconds.
Log shutdown start and completion for deploy correlation.

Related: Graceful Shutdown & Signal Handling - errgroup patterns

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Prerequisites

Install Go 1.26.x or later from go.dev/dl.
Create a module: mkdir oblab && cd oblab && go mod init example.com/oblab.
For metrics and tracing examples later in the section: go get github.com/prometheus/client_golang/prometheus/promhttp and OpenTelemetry modules as needed.

Basic Examples

1. Default slog Logger

Go 1.21+ sets a default structured logger you can call without setup.

package main
 
import (
    "log/slog"
)
 
func main() {
    slog.Info("server starting", "port", 8080)
}

slog.Info writes key-value pairs after the message.
Default output is text format to stderr.
Replace the default with JSON in production services.

Related: Structured Logging with slog - handlers and levels

2. JSON slog Handler

JSON logs parse cleanly in Loki, Elasticsearch, and CloudWatch.

package main
 
import (
    "log/slog"
    "os"
)
 
func main() {
    logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
    slog.SetDefault(logger)
    slog.Info("ready", "service", "api")
}

NewJSONHandler emits one JSON object per line.
SetDefault makes package-level slog calls use your handler.
Set Level to slog.LevelDebug only in development.

Related: Production Go: Logs, Metrics, Traces, and Signals - observability overview

3. Request ID Middleware

Attach a correlation ID to context for logs and downstream calls.

package main
 
import (
    "context"
    "crypto/rand"
    "encoding/hex"
    "net/http"
)
 
type ctxKey string
 
const requestIDKey ctxKey = "request_id"
 
func withRequestID(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        id := r.Header.Get("X-Request-ID")
        if id == "" {
            var b [16]byte
            rand.Read(b[:])
            id = hex.EncodeToString(b[:])
        }
        ctx := context.WithValue(r.Context(), requestIDKey, id)
        w.Header().Set("X-Request-ID", id)
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

Prefer incoming X-Request-ID when present for trace continuity.
Store IDs in context with unexported key types to avoid collisions.
Echo the ID on the response for client-side support tickets.

Related: Middleware Chains & Request Context - wrapping handlers

4. Logging Middleware with slog

Log method, path, and status for every HTTP request.

package main
 
import (
    "log/slog"
    "net/http"
    "time"
)
 
type statusWriter struct {
    http.ResponseWriter
    code int
}
 
func (w *statusWriter) WriteHeader(code int) {
    w.code = code
    w.ResponseWriter.WriteHeader(code)
}
 
func loggingMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        sw := &statusWriter{ResponseWriter: w, code: http.StatusOK}
        next.ServeHTTP(sw, r)
        slog.Info("request",
            "method", r.Method,
            "path", r.URL.Path,
            "status", sw.code,
            "duration_ms", time.Since(start).Milliseconds(),
        )
    })
}

Wrap ResponseWriter to capture the final status code.
Log duration in milliseconds for quick grep in log UIs.
Keep health check paths at DEBUG level in production to reduce noise.

Related: Structured Logging with slog - middleware patterns

5. Liveness Handler `/healthz`

Liveness should be cheap and never call external systems.

package main
 
import (
    "net/http"
)
 
func healthz(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("ok"))
}
 
func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("GET /healthz", healthz)
    http.ListenAndServe(":8080", mux)
}

Return 200 when the process and HTTP server are running.
Do not check databases on liveness - that causes restart loops.
Kubernetes livenessProbe hits this path on a short interval.

Related: Health, Readiness & Liveness Probes - probe design

6. Readiness Handler `/readyz`

Readiness gates traffic when dependencies are unhealthy.

package main
 
import (
    "net/http"
    "sync/atomic"
)
 
var ready atomic.Bool
 
func readyz(w http.ResponseWriter, r *http.Request) {
    if !ready.Load() {
        http.Error(w, "not ready", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("ready"))
}
 
func main() {
    ready.Store(true)
    mux := http.NewServeMux()
    mux.HandleFunc("GET /readyz", readyz)
    http.ListenAndServe(":8080", mux)
}

Return 503 when the service cannot accept traffic.
Flip ready to false during startup until migrations finish.
Load balancers and Kubernetes use readiness to stop sending requests.

Related: Health, Readiness & Liveness Probes - dependency checks

7. Environment Variable Config

12-factor apps read config from the environment at startup.

package main
 
import (
    "log/slog"
    "os"
)
 
func main() {
    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }
    slog.Info("config loaded", "port", port, "env", os.Getenv("APP_ENV"))
}

Fail fast in main when required variables are missing.
Log non-secret config at startup for deploy verification.
Document every env var in the service README.

Related: Configuration Management: Env, Flags & Viper - full config patterns

Intermediate Examples

8. Prometheus Counter on `/metrics`

Expose a scrape endpoint for request totals.

package main
 
import (
    "net/http"
 
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)
 
var requests = prometheus.NewCounterVec(
    prometheus.CounterOpts{Name: "http_requests_total", Help: "HTTP requests"},
    []string{"method", "code"},
)
 
func main() {
    prometheus.MustRegister(requests)
    mux := http.NewServeMux()
    mux.Handle("GET /metrics", promhttp.Handler())
    mux.HandleFunc("GET /hello", func(w http.ResponseWriter, r *http.Request) {
        requests.WithLabelValues(r.Method, "200").Inc()
        w.Write([]byte("hello"))
    })
    http.ListenAndServe(":8080", mux)
}

Register metrics once in main, not per request handler.
Use bounded label values (method, status class) to control cardinality.
Restrict /metrics to internal networks or auth middleware.

Related: Prometheus Metrics & RED/USE Dashboards - RED dashboards

9. Graceful Shutdown on SIGTERM

Drain in-flight HTTP requests before process exit.

package main
 
import (
    "context"
    "log/slog"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)
 
func main() {
    srv := &http.Server{Addr: ":8080", Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("ok"))
    })}
    go func() {
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            slog.Error("listen failed", "err", err)
            os.Exit(1)
        }
    }()
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    <-quit
    ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
    defer cancel()
    if err := srv.Shutdown(ctx); err != nil {
        slog.Error("shutdown failed", "err", err)
    }
    slog.Info("server stopped")
}

Shutdown stops accepting new connections and waits for active requests.
Bound shutdown context to Kubernetes terminationGracePeriodSeconds.
Log shutdown start and completion for deploy correlation.

Related: Graceful Shutdown & Signal Handling - errgroup patterns

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Observability Basics

Search Documentation

Observability Basics