os, io, bufio & path/filepath

Q: What is the difference between io.Reader and os.File?

io.Reader is an interface with one Read method. *os.File implements io.Reader (and io.Writer) for filesystem-backed streams. Functions accepting io.Reader work with files, network bodies, strings, and buffers alike.

Q: When should I use bufio versus raw os.File reads?

Use bufio when you perform many small reads or writes and want fewer syscalls. Use direct reads when you already have large buffers or memory-mapped data.

Q: Why filepath.Join instead of string concatenation?

Join inserts the correct separator for the OS and cleans redundant elements. Manual concatenation breaks on Windows and invites double-slash bugs.

Q: Does filepath.Clean make paths safe from traversal attacks?

Clean simplifies . and .. segments but does not validate trust. Validate user input roots separately before opening files.

Q: How do I read an entire file into a string?

data, err := os.ReadFile(path) then string(data). For large files, stream with bufio.Scanner or io.Copy instead.

Q: What permissions should I use with os.WriteFile?

Use the minimum needed; 0o600 for private config, 0o644 for world-readable artifacts. Never default to 0777 in application code.

Q: Can Scanner split on commas or custom delimiters?

Yes. Call sc.Split with a custom SplitFunc or use bufio.NewReader with ReadString.

Q: How do I duplicate a Reader for two consumers?

Read once into a buffer, or use io.TeeReader to copy bytes to a secondary writer while reading. There is no built-in fan-out Reader.

Q: What does defer f.Close() miss?

Close errors are ignored unless you assign them in a deferred closure. Writers especially need close error checks to flush buffers.

Q: When is io.LimitReader required?

When accepting untrusted input streams (HTTP uploads, RPC bodies) to cap memory and CPU usage. Pair with max size checks before io.ReadAll.

File and stream I/O in Go layers small interfaces into composable pipelines.

os talks to the filesystem, io defines movement of bytes, bufio reduces syscall overhead, and path/filepath builds portable paths.

Summary

Most Go programs read config, serve uploads, or pipe data between processes.

The stdlib models all of that as io.Reader and io.Writer implementations.

os.Open returns an *os.File you can pass to io.Copy, bufio.NewScanner, or json.NewDecoder.

Cross-platform tools must never hardcode / or \; filepath.Join picks the correct separator.

Recipe

Quick-reference recipe card - copy-paste ready.

path := filepath.Join("data", "input.txt")
f, err := os.Open(path)
if err != nil {
    return err
}
defer f.Close()
 
var out bytes.Buffer
if _, err := io.Copy(&out, f); err != nil {
    return err
}

When to reach for this:

Reading or writing files and directories on disk.
Streaming data between HTTP bodies, compressors, and hashers without loading everything into RAM.
Scanning logs or CSV-style text line by line with bufio.Scanner.
Building file paths that work on Linux, macOS, and Windows.

Working Example

package main
 
import (
    "bufio"
    "fmt"
    "io"
    "os"
    "path/filepath"
    "strings"
)
 
func copyFiltered(srcPath, dstPath, prefix string) (int, error) {
    srcPath = filepath.Clean(srcPath)
    dstPath = filepath.Clean(dstPath)
 
    src, err := os.Open(srcPath)
    if err != nil {
        return 0, fmt.Errorf("open src: %w", err)
    }
    defer src.Close()
 
    if err := os.MkdirAll(filepath.Dir(dstPath), 0o755); err != nil {
        return 0, fmt.Errorf("mkdir: %w", err)
    }
 
    dst, err := os.Create(dstPath)
    if err != nil {
        return 0, fmt.Errorf("create dst: %w", err)
    }
    defer func() {
        if cerr := dst.Close(); cerr != nil && err == nil {
            err = cerr
        }
    }()
 
    scanner := bufio.NewScanner(src)
    writer := bufio.NewWriter(dst)
    defer writer.Flush()
 
    lines := 0
    for scanner.Scan() {
        line := scanner.Text()
        if strings.HasPrefix(line, prefix) {
            if _, err := writer.WriteString(line + "\n"); err != nil {
                return lines, err
            }
            lines++
        }
    }
    if err := scanner.Err(); err != nil {
        return lines, err
    }
    return lines, nil
}
 
func main() {
    n, err := copyFiltered("input.log", filepath.Join("out", "filtered.log"), "ERR")
    if err != nil {
        panic(err)
    }
    fmt.Println("wrote lines:", n)
}

What this demonstrates:

filepath.Clean and filepath.Join for safe path assembly.
os.MkdirAll before creating nested output files.
bufio.Scanner for line iteration and bufio.Writer for batched writes.
Deferred close with error capture on the destination file.

Deep Dive

How It Works

io.Copy reads from any Reader and writes to any Writer using an internal buffer pool.
*os.File reads and writes map to OS file descriptors; offsets are tracked per handle.
bufio.Reader and bufio.Writer wrap streams to batch syscalls.
filepath operates on paths as strings; it does not access the filesystem (use os for that).

Package Roles

Package	Responsibility	Typical entry points
`os`	Files, env, process, args	`Open`, `ReadFile`, `WriteFile`, `Getenv`
`io`	Stream contracts and helpers	`Copy`, `ReadAll`, `LimitReader`, `Pipe`
`bufio`	Buffering and scanning	`NewScanner`, `NewReader`, `NewWriter`
`path/filepath`	OS-specific path ops	`Join`, `Clean`, `Walk`, `Rel`

Go Notes

// Prefer io.Copy over manual Read loops for piping.
// Limit uploads to prevent memory exhaustion:
limited := io.LimitReader(r.Body, 1<<20) // 1 MiB cap
 
// Walk directory trees portably:
filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
    if err != nil {
        return err
    }
    if d.IsDir() {
        return nil
    }
    // process file
    return nil
})

Gotchas

Forgetting to close files - Descriptor leaks exhaust process limits under load. Fix: defer f.Close() and propagate close errors on writers.
Using path instead of path/filepath - path is for URL-like forward-slash paths, not Windows filesystem paths. Fix: import path/filepath for OS paths.
Ignoring Scanner.Err() - Loop ends on I/O errors silently. Fix: check sc.Err() after for sc.Scan().
Huge lines with default Scanner buffer - Token too long errors on very long lines. Fix: sc.Buffer(make([]byte, 0, 64*1024), maxToken).
Assuming Read fills the buffer - Read may return n < len(buf) with nil error. Fix: loop until io.EOF or use io.ReadFull.
TOCTOU with os.Stat then os.Open - File may change between checks. Fix: open first, then validate content or use file locks when required.

Alternatives

Alternative	Use When	Don't Use When
`os.ReadFile` / `os.WriteFile`	Small, bounded files	Streaming large uploads or logs
`mmap` via syscall or third-party	Zero-copy read of huge files	Portability and simplicity matter more
`embed.FS`	Ship static assets in binary	User-provided paths on disk
`afero` or `io/fs` abstractions	Test filesystem without disk	Simple CLI tools with real files only

FAQs

What is the difference between io.Reader and os.File?

io.Reader is an interface with one Read method.

*os.File implements io.Reader (and io.Writer) for filesystem-backed streams.

Functions accepting io.Reader work with files, network bodies, strings, and buffers alike.

When should I use bufio versus raw os.File reads?

Use bufio when you perform many small reads or writes and want fewer syscalls.

Use direct reads when you already have large buffers or memory-mapped data.

Why filepath.Join instead of string concatenation?

Join inserts the correct separator for the OS and cleans redundant elements.

Manual concatenation breaks on Windows and invites double-slash bugs.

Does filepath.Clean make paths safe from traversal attacks?

Clean simplifies . and .. segments but does not validate trust.

Validate user input roots separately before opening files.

How do I read an entire file into a string?

data, err := os.ReadFile(path) then string(data).

For large files, stream with bufio.Scanner or io.Copy instead.

What permissions should I use with os.WriteFile?

Use the minimum needed; 0o600 for private config, 0o644 for world-readable artifacts.

Never default to 0777 in application code.

Can Scanner split on commas or custom delimiters?

Yes.

Call sc.Split with a custom SplitFunc or use bufio.NewReader with ReadString.

How do I duplicate a Reader for two consumers?

Read once into a buffer, or use io.TeeReader to copy bytes to a secondary writer while reading.

There is no built-in fan-out Reader.

What does defer f.Close() miss?

Close errors are ignored unless you assign them in a deferred closure.

Writers especially need close error checks to flush buffers.

When is io.LimitReader required?

When accepting untrusted input streams (HTTP uploads, RPC bodies) to cap memory and CPU usage.

Pair with max size checks before io.ReadAll.

Should I use os.RemoveAll for temp directories?

Yes for scratch dirs you created.

Never point RemoveAll at user-supplied paths without validating they stay under an expected root.

How does WalkDir differ from Walk?

filepath.WalkDir passes os.DirEntry without extra Stat calls, reducing syscalls on large trees.

Prefer WalkDir in new code.

Standard Library Basics - Intro examples for I/O and JSON
The Go Standard Library: Batteries Included - Composition philosophy
log, regexp & text/template - Text processing after reading files
crypto/*, compress & math/rand/v2 - Hashing and compression streams
net/http: Go's Batteries-Included HTTP Stack - HTTP bodies as io.Reader

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

os talks to the filesystem, io defines movement of bytes, bufio reduces syscall overhead, and path/filepath builds portable paths.

Summary

Most Go programs read config, serve uploads, or pipe data between processes.

The stdlib models all of that as io.Reader and io.Writer implementations.

os.Open returns an *os.File you can pass to io.Copy, bufio.NewScanner, or json.NewDecoder.

Cross-platform tools must never hardcode / or \; filepath.Join picks the correct separator.

Recipe

Quick-reference recipe card - copy-paste ready.

path := filepath.Join("data", "input.txt")
f, err := os.Open(path)
if err != nil {
    return err
}
defer f.Close()
 
var out bytes.Buffer
if _, err := io.Copy(&out, f); err != nil {
    return err
}

When to reach for this:

Reading or writing files and directories on disk.
Streaming data between HTTP bodies, compressors, and hashers without loading everything into RAM.
Scanning logs or CSV-style text line by line with bufio.Scanner.
Building file paths that work on Linux, macOS, and Windows.

Working Example

package main
 
import (
    "bufio"
    "fmt"
    "io"
    "os"
    "path/filepath"
    "strings"
)
 
func copyFiltered(srcPath, dstPath, prefix string) (int, error) {
    srcPath = filepath.Clean(srcPath)
    dstPath = filepath.Clean(dstPath)
 
    src, err := os.Open(srcPath)
    if err != nil {
        return 0, fmt.Errorf("open src: %w", err)
    }
    defer src.Close()
 
    if err := os.MkdirAll(filepath.Dir(dstPath), 0o755); err != nil {
        return 0, fmt.Errorf("mkdir: %w", err)
    }
 
    dst, err := os.Create(dstPath)
    if err != nil {
        return 0, fmt.Errorf("create dst: %w", err)
    }
    defer func() {
        if cerr := dst.Close(); cerr != nil && err == nil {
            err = cerr
        }
    }()
 
    scanner := bufio.NewScanner(src)
    writer := bufio.NewWriter(dst)
    defer writer.Flush()
 
    lines := 0
    for scanner.Scan() {
        line := scanner.Text()
        if strings.HasPrefix(line, prefix) {
            if _, err := writer.WriteString(line + "\n"); err != nil {
                return lines, err
            }
            lines++
        }
    }
    if err := scanner.Err(); err != nil {
        return lines, err
    }
    return lines, nil
}
 
func main() {
    n, err := copyFiltered("input.log", filepath.Join("out", "filtered.log"), "ERR")
    if err != nil {
        panic(err)
    }
    fmt.Println("wrote lines:", n)
}

What this demonstrates:

filepath.Clean and filepath.Join for safe path assembly.
os.MkdirAll before creating nested output files.
bufio.Scanner for line iteration and bufio.Writer for batched writes.
Deferred close with error capture on the destination file.

Deep Dive

How It Works

io.Copy reads from any Reader and writes to any Writer using an internal buffer pool.
*os.File reads and writes map to OS file descriptors; offsets are tracked per handle.
bufio.Reader and bufio.Writer wrap streams to batch syscalls.
filepath operates on paths as strings; it does not access the filesystem (use os for that).

Package Roles

Package	Responsibility	Typical entry points
`os`	Files, env, process, args	`Open`, `ReadFile`, `WriteFile`, `Getenv`
`io`	Stream contracts and helpers	`Copy`, `ReadAll`, `LimitReader`, `Pipe`
`bufio`	Buffering and scanning	`NewScanner`, `NewReader`, `NewWriter`
`path/filepath`	OS-specific path ops	`Join`, `Clean`, `Walk`, `Rel`

Go Notes

// Prefer io.Copy over manual Read loops for piping.
// Limit uploads to prevent memory exhaustion:
limited := io.LimitReader(r.Body, 1<<20) // 1 MiB cap
 
// Walk directory trees portably:
filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
    if err != nil {
        return err
    }
    if d.IsDir() {
        return nil
    }
    // process file
    return nil
})

Gotchas

Forgetting to close files - Descriptor leaks exhaust process limits under load. Fix: defer f.Close() and propagate close errors on writers.
Using path instead of path/filepath - path is for URL-like forward-slash paths, not Windows filesystem paths. Fix: import path/filepath for OS paths.
Ignoring Scanner.Err() - Loop ends on I/O errors silently. Fix: check sc.Err() after for sc.Scan().
Huge lines with default Scanner buffer - Token too long errors on very long lines. Fix: sc.Buffer(make([]byte, 0, 64*1024), maxToken).
Assuming Read fills the buffer - Read may return n < len(buf) with nil error. Fix: loop until io.EOF or use io.ReadFull.
TOCTOU with os.Stat then os.Open - File may change between checks. Fix: open first, then validate content or use file locks when required.

Alternatives

Alternative	Use When	Don't Use When
`os.ReadFile` / `os.WriteFile`	Small, bounded files	Streaming large uploads or logs
`mmap` via syscall or third-party	Zero-copy read of huge files	Portability and simplicity matter more
`embed.FS`	Ship static assets in binary	User-provided paths on disk
`afero` or `io/fs` abstractions	Test filesystem without disk	Simple CLI tools with real files only

FAQs

What is the difference between io.Reader and os.File?

io.Reader is an interface with one Read method.

*os.File implements io.Reader (and io.Writer) for filesystem-backed streams.

Functions accepting io.Reader work with files, network bodies, strings, and buffers alike.

When should I use bufio versus raw os.File reads?

Use bufio when you perform many small reads or writes and want fewer syscalls.

Use direct reads when you already have large buffers or memory-mapped data.

Why filepath.Join instead of string concatenation?

Join inserts the correct separator for the OS and cleans redundant elements.

Manual concatenation breaks on Windows and invites double-slash bugs.

Does filepath.Clean make paths safe from traversal attacks?

Clean simplifies . and .. segments but does not validate trust.

Validate user input roots separately before opening files.

How do I read an entire file into a string?

data, err := os.ReadFile(path) then string(data).

For large files, stream with bufio.Scanner or io.Copy instead.

What permissions should I use with os.WriteFile?

Use the minimum needed; 0o600 for private config, 0o644 for world-readable artifacts.

Never default to 0777 in application code.

Can Scanner split on commas or custom delimiters?

Yes.

Call sc.Split with a custom SplitFunc or use bufio.NewReader with ReadString.

How do I duplicate a Reader for two consumers?

Read once into a buffer, or use io.TeeReader to copy bytes to a secondary writer while reading.

There is no built-in fan-out Reader.

What does defer f.Close() miss?

Close errors are ignored unless you assign them in a deferred closure.

Writers especially need close error checks to flush buffers.

When is io.LimitReader required?

When accepting untrusted input streams (HTTP uploads, RPC bodies) to cap memory and CPU usage.

Pair with max size checks before io.ReadAll.

Should I use os.RemoveAll for temp directories?

Yes for scratch dirs you created.

Never point RemoveAll at user-supplied paths without validating they stay under an expected root.

How does WalkDir differ from Walk?

filepath.WalkDir passes os.DirEntry without extra Stat calls, reducing syscalls on large trees.

Prefer WalkDir in new code.

Standard Library Basics - Intro examples for I/O and JSON
The Go Standard Library: Batteries Included - Composition philosophy
log, regexp & text/template - Text processing after reading files
crypto/*, compress & math/rand/v2 - Hashing and compression streams
net/http: Go's Batteries-Included HTTP Stack - HTTP bodies as io.Reader

Stack versions: This page was written for Go 1.26.x (Green Tea GC default, go fix modernizers - verify patch at build), chi (latest - verify at build), gin (latest - verify at build), echo (latest - verify at build), google.golang.org/grpc (latest - verify at build), sigs.k8s.io/controller-runtime (latest - verify at build), kubebuilder (latest - verify at build), tinygo (latest - verify board targets at build), wazero (latest - verify at build), and golangci-lint (latest - verify linter set at build).

os, io, bufio & path/filepath

Search Documentation

os, io, bufio & path/filepath