Prometheus & OpenTelemetry Client Libraries

Production Go services expose Prometheus metrics for dashboards and OpenTelemetry traces for request latency debugging.

The client libraries sit at the HTTP/gRPC edge and wrap outbound calls without replacing your logging stack.

Recipe

Quick-reference recipe card - copy-paste ready.

import (
    "net/http"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)
 
http.Handle("/metrics", promhttp.Handler())

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)
 
handler := otelhttp.NewHandler(mux, "api")

When to reach for this:

• SREs need RED metrics (rate, errors, duration) per route and dependency
• On-call engineers trace slow requests across services
• Kubernetes scrapes /metrics or an OTel collector receives OTLP
• You standardize observability libraries across chi, Gin, and gRPC servers

Working Example

package main
 
import (
    "context"
    "log"
    "net/http"
    "time"
 
    "github.com/go-chi/chi/v5"
    "github.com/go-chi/chi/v5/middleware"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/otel"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/trace"
)
 
var (
    httpDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "HTTP request latency",
        Buckets: prometheus.DefBuckets,
    }, []string{"method", "route", "status"})
)
 
func initTracer() trace.TracerProvider {
    tp := sdktrace.NewTracerProvider()
    otel.SetTracerProvider(tp)
    return tp
}
 
func metricsMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        ww := middleware.NewWrapResponseWriter(w, r.ProtoMajor)
        next.ServeHTTP(ww, r)
        route := chi.RouteContext(r.Context()).RoutePattern()
        httpDuration.WithLabelValues(r.Method, route, http.StatusText(ww.Status())).Observe(time.Since(start).Seconds())
    })
}
 
func main() {
    tp := initTracer()
    defer func() { _ = tp.Shutdown(context.Background()) }()
 
    r := chi.NewRouter()
    r.Use(metricsMiddleware)
    r.Get("/health", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(200) })
    r.Handle("/metrics", promhttp.Handler())
 
    root := otelhttp.NewHandler(r, "api")
    log.Fatal(http.ListenAndServe(":8080", root))
}

What this demonstrates:

• Prometheus histogram with low-cardinality labels (method, route, status)
• chi route patterns for label stability (avoid raw URLs with IDs)
• promhttp.Handler on a dedicated path for scrapers
• OTel HTTP middleware wrapping the router for trace propagation

Deep Dive

How It Works

• Prometheus client registers collectors in a registry; promhttp exposes text exposition format.
• OpenTelemetry SDK configures TracerProvider, exporters (OTLP gRPC/HTTP), and resource attributes (service.name).
• otelhttp creates spans around handlers and propagates traceparent headers.
• Metrics answer "how much and how fast"; traces answer "which hop was slow".

Metric Types (Prometheus)

OTel Bootstrap Sketch

exporter, _ := otlptracegrpc.New(ctx)
tp := sdktrace.NewTracerProvider(
    sdktrace.WithBatcher(exporter),
    sdktrace.WithResource(resource.NewWithAttributes(
        semconv.SchemaURL,
        semconv.ServiceName("orders-api"),
    )),
)
otel.SetTracerProvider(tp)

• Run collector sidecar or cluster-wide OTLP endpoint in Kubernetes.
• Align service.name with deployment labels for trace search.

Go Notes

• Use promauto only when a single registry is fine; libraries should accept prometheus.Registerer parameters.
• gRPC uses go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc interceptors similarly.
• Never put user IDs or unbounded path segments in metric labels.

Gotchas

• High-cardinality labels - /users/123 per request explodes series. Fix: use chi.RoutePattern() or templated route names.
• Double instrumentation - framework metrics plus custom middleware duplicate counts. Fix: one owner per metric name in ADR.
• Global Prometheus registry in libraries - imported packages register collectors unexpectedly. Fix: pass prometheus.Registerer into constructors.
• Missing tracer shutdown - batch exporters drop spans on exit. Fix: Shutdown on SIGTERM with graceful drain timeout.
• Scraping main port from internet - /metrics leaks operational detail. Fix: separate listener or network policy on admin port.
• Trace sampling at 100% always - cost blows up at scale. Fix: parent-based or probabilistic sampling tuned per environment.

NzA2MjAyfDI5M29pZ2N8b2kuc2VkaXVnZWRvYw==

Alternatives

FAQs

Related

• Prometheus Metrics & RED USE Dashboards - RED methodology
• OpenTelemetry Tracing in Go Services - trace pipeline
• Production Go: Logs, Metrics, Traces, and Signals - three pillars
• Observability Basics - health and slog starters
• Logging: zap vs zerolog vs slog - correlate logs with traces

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).