Production Troubleshooting Basics

10 examples to get you started with Production Troubleshooting - 7 basic and 3 intermediate.

Prerequisites

Install Go 1.26.x or later from go.dev/dl.
Create a module: mkdir triagelab && cd triagelab && go mod init example.com/triagelab.
For local experiments: go get github.com/prometheus/client_golang/prometheus/promhttp when exporting metrics.

Basic Examples

1. Log Build and Runtime Version at Startup

Support bundles need the exact Go version and build metadata during incidents.

package main
 
import (
    "log/slog"
    "runtime"
    "runtime/debug"
)
 
func main() {
    info, _ := debug.ReadBuildInfo()
    slog.Info("starting",
        "go", runtime.Version(),
        "module", info.Main.Path,
        "version", info.Main.Version,
    )
}

runtime.Version() reports the toolchain that compiled the binary.
debug.ReadBuildInfo() embeds module version from -ldflags or tagged releases.
Log once at startup so incident logs always include build context.

Related: Incident Response for Go Services - triage mental model

2. Correlate Errors with Request IDs

Structured logs tie scattered timeout messages to one customer request.

package main
 
import (
    "log/slog"
    "net/http"
)
 
func withRequestID(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        id := r.Header.Get("X-Request-ID")
        if id == "" {
            id = "generated-" + r.URL.Path
        }
        logger := slog.With("request_id", id)
        ctx := r.Context()
        _ = logger
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

Propagate X-Request-ID from ingress or generate one per request.
Attach IDs with slog.With so every log line in the handler chain shares it.
Search logs by request_id during incident triage.

Related: Structured Logging with slog - production handlers

3. Register pprof on a Local Admin Port

Runtime profiles are the fastest path from "slow" to "which function."

package main
 
import (
    "log"
    "net/http"
    _ "net/http/pprof"
)
 
func main() {
    go func() {
        admin := http.NewServeMux()
        admin.Handle("/debug/", http.DefaultServeMux)
        log.Println(http.ListenAndServe("127.0.0.1:6060", admin))
    }()
    select {}
}

_ "net/http/pprof" registers standard profile handlers on DefaultServeMux.
Bind admin to 127.0.0.1 or an internal network only.
Use go tool pprof http://127.0.0.1:6060/debug/pprof/profile?seconds=20 under load.

Related: Live CPU & Heap Profiling Under Incident Load - incident-safe collection

4. Export RED Metrics with Prometheus

Rate, errors, and duration histograms answer "is this a Go problem or traffic?"

package main
 
import (
    "net/http"
    "time"
 
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)
 
var reqDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
    Name:    "http_request_duration_seconds",
    Buckets: prometheus.DefBuckets,
}, []string{"route", "code"})
 
func main() {
    http.Handle("/metrics", promhttp.Handler())
    http.HandleFunc("/api", func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        w.WriteHeader(http.StatusOK)
        reqDuration.WithLabelValues("/api", "200").Observe(time.Since(start).Seconds())
    })
    http.ListenAndServe(":8080", nil)
}

Histogram http_request_duration_seconds powers p99 dashboards.
Label cardinality must stay bounded - use route templates, not raw paths.
Compare error ratio and latency before opening profiles.

Related: Prometheus Metrics & RED-USE Dashboards - dashboard design

5. Print Goroutine Count from /debug/vars

A climbing goroutine count often precedes memory and latency incidents.

package main
 
import (
    "expvar"
    "runtime"
)
 
func init() {
    expvar.Publish("goroutines", expvar.Func(func() any {
        return runtime.NumGoroutine()
    }))
}

expvar exposes values at /debug/vars when mounted on your admin mux.
Plot goroutines alongside heap and request rate during incidents.
Sudden plateaus far above baseline warrant a goroutine dump.

Related: Goroutine Dump & Leak Profile Analysis - reading stacks

6. Check sql.DB Pool Stats in Logs

Pool exhaustion looks like app timeouts while the database is healthy.

package main
 
import (
    "database/sql"
    "log"
    "time"
 
    _ "github.com/jackc/pgx/v5/stdlib"
)
 
func logDBStats(db *sql.DB) {
    ticker := time.NewTicker(30 * time.Second)
    for range ticker.C {
        s := db.Stats()
        log.Printf("db open=%d inUse=%d idle=%d waitCount=%d waitDur=%s",
            s.OpenConnections, s.InUse, s.Idle, s.WaitCount, s.WaitDuration)
    }
}

WaitCount and WaitDuration rising mean handlers block on db.Conn.
Compare with DB server connection count before blaming queries.
Export the same fields as Prometheus gauges in production.

Related: Database Connection Pool Exhaustion - saturation playbook

7. Record Recent Deploy SHA in /healthz

Deploy correlation is the fastest incident hypothesis.

package main
 
import (
    "encoding/json"
    "net/http"
    "os"
)
 
var buildSHA = os.Getenv("BUILD_SHA")
 
func health(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(map[string]string{
        "status": "ok",
        "build":  buildSHA,
    })
}

Inject BUILD_SHA from CI into the container environment.
Compare health responses across replicas during rolling deploys.
Pair with deploy event markers on Grafana dashboards.

Related: Health, Readiness & Liveness Probes - probe semantics

Intermediate Examples

8. CPU Profile Hook for On-Call

Trigger a short profile from a secured endpoint during an active incident.

package main
 
import (
    "net/http"
    _ "net/http/pprof"
    "os"
    "time"
)
 
func profileHandler(w http.ResponseWriter, r *http.Request) {
    if r.Header.Get("X-On-Call-Token") != os.Getenv("ONCALL_TOKEN") {
        http.Error(w, "forbidden", http.StatusForbidden)
        return
    }
    http.Redirect(w, r, "/debug/pprof/profile?seconds=20", http.StatusTemporaryRedirect)
}

Gate profile endpoints with a shared secret or mTLS.
Keep sample windows short (20-30s) on production pods.
Save resulting profiles to ticket storage for post-mortems.

Related: CPU & Heap Profiling with pprof - reading flame graphs

9. Graceful Drain Before Restart

Restarting mid-request causes false 5xx spikes that look like code regressions.

package main
 
import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)
 
func main() {
    srv := &http.Server{Addr: ":8080"}
    go srv.ListenAndServe()
 
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    <-quit
 
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    if err := srv.Shutdown(ctx); err != nil {
        log.Fatal(err)
    }
}

Shutdown stops accepting new connections and waits for in-flight requests.
Set timeout below Kubernetes terminationGracePeriodSeconds.
Incidents during bad deploys improve when rollouts drain cleanly.

Related: Graceful Shutdown & Signal Handling - production patterns

10. Snapshot Incident Evidence to a Bundle Directory

Post-mortems need files, not only screenshots.

mkdir -p /tmp/incident-$(date +%Y%m%d%H%M)
curl -s "http://127.0.0.1:6060/debug/pprof/profile?seconds=20" > /tmp/incident-*/cpu.prof
curl -s "http://127.0.0.1:6060/debug/pprof/heap" > /tmp/incident-*/heap.prof
curl -s "http://127.0.0.1:6060/debug/pprof/goroutine?debug=2" > /tmp/incident-*/goroutine.txt
go tool pprof -top /tmp/incident-*/cpu.prof > /tmp/incident-*/cpu-top.txt

Collect CPU, heap, and goroutine artifacts before pods recycle.
Add dashboard PDF links and deploy SHA to the same folder.
Attach the bundle to the post-mortem doc.

Related: Post-Mortem Template for Go Incidents - RCA structure

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Production Troubleshooting Basics

10 examples to get you started with Production Troubleshooting - 7 basic and 3 intermediate.

Prerequisites

Install Go 1.26.x or later from go.dev/dl.
Create a module: mkdir triagelab && cd triagelab && go mod init example.com/triagelab.
For local experiments: go get github.com/prometheus/client_golang/prometheus/promhttp when exporting metrics.

Basic Examples

1. Log Build and Runtime Version at Startup

Support bundles need the exact Go version and build metadata during incidents.

package main
 
import (
    "log/slog"
    "runtime"
    "runtime/debug"
)
 
func main() {
    info, _ := debug.ReadBuildInfo()
    slog.Info("starting",
        "go", runtime.Version(),
        "module", info.Main.Path,
        "version", info.Main.Version,
    )
}

runtime.Version() reports the toolchain that compiled the binary.
debug.ReadBuildInfo() embeds module version from -ldflags or tagged releases.
Log once at startup so incident logs always include build context.

Related: Incident Response for Go Services - triage mental model

2. Correlate Errors with Request IDs

Structured logs tie scattered timeout messages to one customer request.

package main
 
import (
    "log/slog"
    "net/http"
)
 
func withRequestID(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        id := r.Header.Get("X-Request-ID")
        if id == "" {
            id = "generated-" + r.URL.Path
        }
        logger := slog.With("request_id", id)
        ctx := r.Context()
        _ = logger
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

Propagate X-Request-ID from ingress or generate one per request.
Attach IDs with slog.With so every log line in the handler chain shares it.
Search logs by request_id during incident triage.

Related: Structured Logging with slog - production handlers

3. Register pprof on a Local Admin Port

Runtime profiles are the fastest path from "slow" to "which function."

package main
 
import (
    "log"
    "net/http"
    _ "net/http/pprof"
)
 
func main() {
    go func() {
        admin := http.NewServeMux()
        admin.Handle("/debug/", http.DefaultServeMux)
        log.Println(http.ListenAndServe("127.0.0.1:6060", admin))
    }()
    select {}
}

_ "net/http/pprof" registers standard profile handlers on DefaultServeMux.
Bind admin to 127.0.0.1 or an internal network only.
Use go tool pprof http://127.0.0.1:6060/debug/pprof/profile?seconds=20 under load.

Related: Live CPU & Heap Profiling Under Incident Load - incident-safe collection

4. Export RED Metrics with Prometheus

Rate, errors, and duration histograms answer "is this a Go problem or traffic?"

package main
 
import (
    "net/http"
    "time"
 
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)
 
var reqDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
    Name:    "http_request_duration_seconds",
    Buckets: prometheus.DefBuckets,
}, []string{"route", "code"})
 
func main() {
    http.Handle("/metrics", promhttp.Handler())
    http.HandleFunc("/api", func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        w.WriteHeader(http.StatusOK)
        reqDuration.WithLabelValues("/api", "200").Observe(time.Since(start).Seconds())
    })
    http.ListenAndServe(":8080", nil)
}

Histogram http_request_duration_seconds powers p99 dashboards.
Label cardinality must stay bounded - use route templates, not raw paths.
Compare error ratio and latency before opening profiles.

Related: Prometheus Metrics & RED-USE Dashboards - dashboard design

5. Print Goroutine Count from /debug/vars

A climbing goroutine count often precedes memory and latency incidents.

package main
 
import (
    "expvar"
    "runtime"
)
 
func init() {
    expvar.Publish("goroutines", expvar.Func(func() any {
        return runtime.NumGoroutine()
    }))
}

expvar exposes values at /debug/vars when mounted on your admin mux.
Plot goroutines alongside heap and request rate during incidents.
Sudden plateaus far above baseline warrant a goroutine dump.

Related: Goroutine Dump & Leak Profile Analysis - reading stacks

6. Check sql.DB Pool Stats in Logs

Pool exhaustion looks like app timeouts while the database is healthy.

package main
 
import (
    "database/sql"
    "log"
    "time"
 
    _ "github.com/jackc/pgx/v5/stdlib"
)
 
func logDBStats(db *sql.DB) {
    ticker := time.NewTicker(30 * time.Second)
    for range ticker.C {
        s := db.Stats()
        log.Printf("db open=%d inUse=%d idle=%d waitCount=%d waitDur=%s",
            s.OpenConnections, s.InUse, s.Idle, s.WaitCount, s.WaitDuration)
    }
}

WaitCount and WaitDuration rising mean handlers block on db.Conn.
Compare with DB server connection count before blaming queries.
Export the same fields as Prometheus gauges in production.

Related: Database Connection Pool Exhaustion - saturation playbook

7. Record Recent Deploy SHA in /healthz

Deploy correlation is the fastest incident hypothesis.

package main
 
import (
    "encoding/json"
    "net/http"
    "os"
)
 
var buildSHA = os.Getenv("BUILD_SHA")
 
func health(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(map[string]string{
        "status": "ok",
        "build":  buildSHA,
    })
}

Inject BUILD_SHA from CI into the container environment.
Compare health responses across replicas during rolling deploys.
Pair with deploy event markers on Grafana dashboards.

Related: Health, Readiness & Liveness Probes - probe semantics

Intermediate Examples

8. CPU Profile Hook for On-Call

Trigger a short profile from a secured endpoint during an active incident.

package main
 
import (
    "net/http"
    _ "net/http/pprof"
    "os"
    "time"
)
 
func profileHandler(w http.ResponseWriter, r *http.Request) {
    if r.Header.Get("X-On-Call-Token") != os.Getenv("ONCALL_TOKEN") {
        http.Error(w, "forbidden", http.StatusForbidden)
        return
    }
    http.Redirect(w, r, "/debug/pprof/profile?seconds=20", http.StatusTemporaryRedirect)
}

Gate profile endpoints with a shared secret or mTLS.
Keep sample windows short (20-30s) on production pods.
Save resulting profiles to ticket storage for post-mortems.

Related: CPU & Heap Profiling with pprof - reading flame graphs

9. Graceful Drain Before Restart

Restarting mid-request causes false 5xx spikes that look like code regressions.

package main
 
import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)
 
func main() {
    srv := &http.Server{Addr: ":8080"}
    go srv.ListenAndServe()
 
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    <-quit
 
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    if err := srv.Shutdown(ctx); err != nil {
        log.Fatal(err)
    }
}

Shutdown stops accepting new connections and waits for in-flight requests.
Set timeout below Kubernetes terminationGracePeriodSeconds.
Incidents during bad deploys improve when rollouts drain cleanly.

Related: Graceful Shutdown & Signal Handling - production patterns

10. Snapshot Incident Evidence to a Bundle Directory

Post-mortems need files, not only screenshots.

mkdir -p /tmp/incident-$(date +%Y%m%d%H%M)
curl -s "http://127.0.0.1:6060/debug/pprof/profile?seconds=20" > /tmp/incident-*/cpu.prof
curl -s "http://127.0.0.1:6060/debug/pprof/heap" > /tmp/incident-*/heap.prof
curl -s "http://127.0.0.1:6060/debug/pprof/goroutine?debug=2" > /tmp/incident-*/goroutine.txt
go tool pprof -top /tmp/incident-*/cpu.prof > /tmp/incident-*/cpu-top.txt

Collect CPU, heap, and goroutine artifacts before pods recycle.
Add dashboard PDF links and deploy SHA to the same folder.
Attach the bundle to the post-mortem doc.

Related: Post-Mortem Template for Go Incidents - RCA structure

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

Production Troubleshooting Basics

Search Documentation

Production Troubleshooting Basics