S3/GCS SDK Patterns with aws-sdk-go-v2

Object storage is the backbone of serverless Go: user uploads, export files, event payloads, and static assets.

aws-sdk-go-v2 and cloud.google.com/go/storage both reward the same habits: shared clients, context deadlines, idempotent keys, and explicit retry understanding.

Summary

Build S3 clients once with config.LoadDefaultConfig and reuse them across invocations in Lambda or goroutines in Cloud Run.

Thread r.Context() from HTTP handlers into PutObject and GetObject so platform timeouts cancel in-flight uploads.

Use content-addressed or deterministic object keys so retries do not create duplicate user-visible files.

Recipe

Quick-reference recipe card - copy-paste ready.

cfg, err := config.LoadDefaultConfig(ctx)
client := s3.NewFromConfig(cfg, func(o *s3.Options) {
    o.RetryMaxAttempts = 5
})
_, err = client.PutObject(ctx, &s3.PutObjectInput{
    Bucket: aws.String(bucket),
    Key:    aws.String(key),
    Body:   body,
    ContentType: aws.String("application/json"),
})

When to reach for this:

• Upload/download paths in Lambda behind API Gateway or S3 event notifications.
• Cloud Run export jobs writing large objects with cancellable contexts.
• Dual-cloud modules abstracting Put/Get behind a small interface.

Working Example

package storage
 
import (
    "bytes"
    "context"
    "crypto/sha256"
    "encoding/hex"
    "fmt"
    "io"
 
    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)
 
type S3Uploader struct {
    client *s3.Client
    bucket string
}
 
func NewS3Uploader(ctx context.Context, bucket, region string) (*S3Uploader, error) {
    cfg, err := config.LoadDefaultConfig(ctx, config.WithRegion(region))
    if err != nil {
        return nil, fmt.Errorf("load aws config: %w", err)
    }
    return &S3Uploader{
        client: s3.NewFromConfig(cfg),
        bucket: bucket,
    }, nil
}
 
func (u *S3Uploader) PutJSONIdempotent(ctx context.Context, prefix string, payload []byte) (string, error) {
    sum := sha256.Sum256(payload)
    key := fmt.Sprintf("%s/%s.json", prefix, hex.EncodeToString(sum[:]))
    _, err := u.client.PutObject(ctx, &s3.PutObjectInput{
        Bucket:      aws.String(u.bucket),
        Key:         aws.String(key),
        Body:        bytes.NewReader(payload),
        ContentType: aws.String("application/json"),
        // IfNoneMatch prevents overwrite when retry races occur (S3 conditional headers)
    })
    if err != nil {
        return "", fmt.Errorf("put object %s: %w", key, err)
    }
    return key, nil
}
 
func (u *S3Uploader) Get(ctx context.Context, key string) ([]byte, error) {
    out, err := u.client.GetObject(ctx, &s3.GetObjectInput{
        Bucket: aws.String(u.bucket),
        Key:    aws.String(key),
    })
    if err != nil {
        return nil, fmt.Errorf("get object: %w", err)
    }
    defer out.Body.Close()
    return io.ReadAll(out.Body)
}

What this demonstrates:

• Client constructed once in NewS3Uploader.
• Content-addressed key for idempotent retries.
• ctx on both PutObject and GetObject.
• Explicit error wrapping for observability.

Deep Dive

How It Works

• aws-sdk-go-v2 uses middleware stacks for signing, retries, and deserialization.

Retries apply to throttling and transient 5xx responses with backoff - not a substitute for idempotent application logic.

• GCS client uses gRPC/HTTP under ADC:

gcsClient, err := storage.NewClient(ctx)
w := gcsClient.Bucket(bucket).Object(key).NewWriter(ctx)
if _, err := io.Copy(w, bytes.NewReader(payload)); err != nil { /* ... */ }
if err := w.Close(); err != nil { /* ... */ }

Always Close() GCS writers to commit objects.

Context Deadline Layers

If the handler ctx expires mid-upload, the SDK aborts and S3 may or may not have committed - design keys so a retry is safe.

Multipart and Large Files

• Use manager.Uploader from feature/s3/manager for files larger than a few MB.
• Set PartSize and Concurrency conservatively in Lambda memory-constrained environments.
• For downloads, manager.Downloader streams to io.Writer without loading entire objects into RAM.

Retry vs Idempotency

Gotchas

• New client per request - Slow and can exhaust sockets on Cloud Run. Fix: Package-level sync.Once client or inject on startup.
• Ignoring ctx cancellation - Uploads continue after client disconnect. Fix: Pass handler context into SDK calls.
• Loading entire object into memory - OOM on large exports. Fix: Stream with io.Copy and GetObject body.
• Missing GCS Writer.Close - Objects never finalize. Fix: Always check Close() error.
• Assuming exactly-once S3 events - S3 notifications are at-least-once. Fix: Idempotent handlers keyed by object etag.
• Wildcard IAM s3:* on * - Overbroad roles. Fix: Prefix ARNs per bucket and PutObject/GetObject only.

Alternatives

FAQs

Related

• Secrets, IAM & Workload Identity - credentials for SDK clients
• AWS Lambda Custom Runtime & provided.al2023 - client init on cold start
• GCP Cloud Run & AWS Fargate Patterns - long-running upload handlers
• os, io, bufio & path/filepath - streaming I/O primitives
• Cloud & Serverless Best Practices - cost and timeout guardrails
• Cloud Deploy Basics - module layout for cloud adapters

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

49454e4f4d5f434e4f5904434556494d434519121356181a181c1a1d