os, io, bufio & path/filepath

File and stream I/O in Go layers small interfaces into composable pipelines.

os talks to the filesystem, io defines movement of bytes, bufio reduces syscall overhead, and path/filepath builds portable paths.

Summary

Most Go programs read config, serve uploads, or pipe data between processes.

The stdlib models all of that as io.Reader and io.Writer implementations.

os.Open returns an *os.File you can pass to io.Copy, bufio.NewScanner, or json.NewDecoder.

Cross-platform tools must never hardcode / or \; filepath.Join picks the correct separator.

Recipe

Quick-reference recipe card - copy-paste ready.

path := filepath.Join("data", "input.txt")
f, err := os.Open(path)
if err != nil {
    return err
}
defer f.Close()
 
var out bytes.Buffer
if _, err := io.Copy(&out, f); err != nil {
    return err
}

When to reach for this:

• Reading or writing files and directories on disk.
• Streaming data between HTTP bodies, compressors, and hashers without loading everything into RAM.
• Scanning logs or CSV-style text line by line with bufio.Scanner.
• Building file paths that work on Linux, macOS, and Windows.

Working Example

package main
 
import (
    "bufio"
    "fmt"
    "io"
    "os"
    "path/filepath"
    "strings"
)
 
func copyFiltered(srcPath, dstPath, prefix string) (int, error) {
    srcPath = filepath.Clean(srcPath)
    dstPath = filepath.Clean(dstPath)
 
    src, err := os.Open(srcPath)
    if err != nil {
        return 0, fmt.Errorf("open src: %w", err)
    }
    defer src.Close()
 
    if err := os.MkdirAll(filepath.Dir(dstPath), 0o755); err != nil {
        return 0, fmt.Errorf("mkdir: %w", err)
    }
 
    dst, err := os.Create(dstPath)
    if err != nil {
        return 0, fmt.Errorf("create dst: %w", err)
    }
    defer func() {
        if cerr := dst.Close(); cerr != nil && err == nil {
            err = cerr
        }
    }()
 
    scanner := bufio.NewScanner(src)
    writer := bufio.NewWriter(dst)
    defer writer.Flush()
 
    lines := 0
    for scanner.Scan() {
        line := scanner.Text()
        if strings.HasPrefix(line, prefix) {
            if _, err := writer.WriteString(line + "\n"); err != nil {
                return lines, err
            }
            lines++
        }
    }
    if err := scanner.Err(); err != nil {
        return lines, err
    }
    return lines, nil
}
 
func main() {
    n, err := copyFiltered("input.log", filepath.Join("out", "filtered.log"), "ERR")
    if err != nil {
        panic(err)
    }
    fmt.Println("wrote lines:", n)
}

What this demonstrates:

• filepath.Clean and filepath.Join for safe path assembly.
• os.MkdirAll before creating nested output files.
• bufio.Scanner for line iteration and bufio.Writer for batched writes.
• Deferred close with error capture on the destination file.

Deep Dive

How It Works

• io.Copy reads from any Reader and writes to any Writer using an internal buffer pool.
• *os.File reads and writes map to OS file descriptors; offsets are tracked per handle.
• bufio.Reader and bufio.Writer wrap streams to batch syscalls.
• filepath operates on paths as strings; it does not access the filesystem (use os for that).

Package Roles

Go Notes

// Prefer io.Copy over manual Read loops for piping.
// Limit uploads to prevent memory exhaustion:
limited := io.LimitReader(r.Body, 1<<20) // 1 MiB cap
 
// Walk directory trees portably:
filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
    if err != nil {
        return err
    }
    if d.IsDir() {
        return nil
    }
    // process file
    return nil
})

Gotchas

• Forgetting to close files - Descriptor leaks exhaust process limits under load. Fix: defer f.Close() and propagate close errors on writers.
• Using path instead of path/filepath - path is for URL-like forward-slash paths, not Windows filesystem paths. Fix: import path/filepath for OS paths.
• Ignoring Scanner.Err() - Loop ends on I/O errors silently. Fix: check sc.Err() after for sc.Scan().
• Huge lines with default Scanner buffer - Token too long errors on very long lines. Fix: sc.Buffer(make([]byte, 0, 64*1024), maxToken).
• Assuming Read fills the buffer - Read may return n < len(buf) with nil error. Fix: loop until io.EOF or use io.ReadFull.
• TOCTOU with os.Stat then os.Open - File may change between checks. Fix: open first, then validate content or use file locks when required.

Alternatives

FAQs

Related

• Standard Library Basics - Intro examples for I/O and JSON
• The Go Standard Library: Batteries Included - Composition philosophy
• log, regexp & text/template - Text processing after reading files
• crypto/*, compress & math/rand/v2 - Hashing and compression streams
• net/http: Go's Batteries-Included HTTP Stack - HTTP bodies as io.Reader

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).