ZFS tuning

This guide covers ZFS pool and dataset configuration optimized for PutFS workloads – primarily small file random reads, writes, and prefix listing.

See Data integrity and Scaling for what ZFS brings to PutFS.

Pool topology

Special vdev (the biggest win)

A special vdev stores ZFS metadata and small files (anything under special_small_blocks) on a dedicated device – typically NVMe. This is the single most impactful optimization for PutFS.

# Create pool with mirrored data vdevs and mirrored NVMe special vdev
zpool create tank \
    mirror /dev/sda /dev/sdb \
    mirror /dev/sdc /dev/sdd \
    special mirror /dev/nvme0n1 /dev/nvme1n1

Warning

Always mirror the special vdev. If it's a single device and it fails, you lose the pool.

The special vdev always stores metadata (dnodes, indirect blocks, directory entries). To also store small files on it, set special_small_blocks:

# Files under 128K go to special vdev (NVMe)
zfs set special_small_blocks=128K tank/putfs

With special_small_blocks=0 (the default), only metadata lives on the special vdev – no file data. This still accelerates directory listings and stat calls, but reads still hit the HDD pool. Setting it to 128K moves small files to NVMe too, which is where PutFS gets the biggest read speedup.

With this, every small file read is an NVMe read. Directory lookups, inodes, and metadata are also on NVMe. The spinning disks only handle large files.

SLOG

A SLOG (Separate intent LOG) absorbs synchronous write latency. PutFS uploads are synchronous writes – without a SLOG, every PUT waits for the data to hit spinning rust.

# Add NVMe SLOG
zpool add tank log /dev/nvme2n1

A small, fast NVMe device (even 16–32 GB) is enough. The SLOG only holds in-flight transactions, not persistent data.

L2ARC

L2ARC extends the ARC (RAM cache) onto SSD. Files that fall out of ARC but are still warm get served from SSD instead of spinning disk.

# Add SSD L2ARC
zpool add tank cache /dev/sde

L2ARC is most useful when your working set exceeds available RAM but fits on SSD. If your entire dataset fits in ARC, L2ARC adds nothing.

Recommended topology summary

NVMe 1+2 (mirror)  → special vdev  (metadata + small files)
NVMe 3             → SLOG          (write intent log)
SSD 1              → L2ARC         (read cache overflow)
HDD 1+2 (mirror)   → data vdevs   (large files)
HDD 3+4 (mirror)   → data vdevs   (large files)

Synchronous vs asynchronous writes

ZFS has a per-dataset sync property that controls when a write is acknowledged to the client:

PutFS issues no fsync on the write path, so ZFS treats its PUTs as asynchronous unless you force otherwise:

Value	Behavior with PutFS	A just-acknowledged PUT is...
`sync=standard` (default)	PutFS makes no synchronous write request, so its PUTs are asynchronous: batched into the open transaction group and flushed at the next commit (up to `zfs_txg_timeout`, default 5s).	atomically visible, but not yet on stable storage – a power loss within ~5s can drop it (never torn).
`sync=always`	Forces every write to the ZIL (SLOG) before it returns, regardless of the application.	on the SLOG (or main pool) before the `201` – the setting that makes an acknowledged PUT power-loss-safe for PutFS.
`sync=disabled`	All writes asynchronous; any `fsync` is ignored. Identical to `standard` for PutFS, which never calls `fsync`.	in the same ~5s window as `standard`; no faster than `standard` for PutFS.

Trade-offs

sync=standard (default): Because PutFS issues no fsync, PUTs return as soon as the body is in RAM and the rename is done – sub-millisecond, with no SLOG wait. Fast, but a just-acknowledged PUT is not yet durable: a power loss can drop up to zfs_txg_timeout (default 5s) of acknowledged writes. Copy-on-write still guarantees you never see a torn object, only a missing one.

sync=always (durable on ack): Forces every write to the ZIL before it returns, so this is the only setting under which a 201 means "survives power loss" for PutFS. With a fast SLOG the cost is small (~0.1ms per write on NVMe); without one, each write hits the main pool (~5-15ms per PUT on HDDs, and every chunk of a large upload pays it). Use it for primary storage where an acknowledged upload must not be lost.

sync=disabled: No effect on PutFS relative to standard – both are already asynchronous, since PutFS never requests a sync. It only changes behavior for other tools that do fsync on the same dataset (e.g. zfs receive), removing their durability. No reason to set it for PutFS's own writes.

# Default: safe, uses SLOG if available
zfs set sync=standard tank/putfs

# Fast but unsafe: scratch data, regenerable datasets
zfs set sync=disabled tank/putfs/acme-corp/cache

# Per-dataset: safe for important data, fast for temp
zfs set sync=standard tank/putfs/acme-corp/documents
zfs set sync=disabled tank/putfs/acme-corp/processing-tmp

Durability is opt-in for PutFS

Since PutFS never calls fsync, only sync=always makes an acknowledged PUT power-loss-safe; standard and disabled both leave a ~5s window. Set sync=always on datasets holding primary data that must survive a crash. sync=disabled remains a footgun for other tools sharing the dataset – it silently ignores their fsync – and children inherit the parent's sync property unless explicitly overridden.

Dataset properties

Per-dataset datasets

Create a dataset per dataset for independent tuning:

zfs create tank/putfs
zfs create tank/putfs/acme-corp
zfs create tank/putfs/acme-corp/invoices

recordsize

The default recordsize is 128K – optimal for large sequential files but wastes space for small files. For datasets that primarily store small files:

# Small file dataset – 4K records
zfs set recordsize=4K tank/putfs/acme-corp/thumbnails

# Mixed dataset – keep default 128K
# ZFS only allocates what's needed, 128K is the *maximum*

Note

recordsize is the maximum block size, not the minimum. A 2KB file in a 128K recordsize dataset uses a single 4K block (ZFS minimum), not 128K. Lowering recordsize primarily helps when you have many files between 4K and 128K, as it reduces metadata overhead per block.

Compression

Always enable compression. With zstd, the effective ARC and L2ARC size increases because cached data is compressed:

zfs set compression=zstd tank/putfs

For datasets with already-compressed content (JPEG, video, ZIP):

zfs set compression=off tank/putfs/acme-corp/media

Extended attributes

Pack extended attributes into the dnode itself to avoid extra I/O:

zfs set xattr=sa tank/putfs
zfs set dnodesize=auto tank/putfs

atime

Disable access time updates – every GET would otherwise trigger a metadata write:

zfs set atime=off tank/putfs

Quotas

Per-dataset storage limits:

zfs set quota=100G tank/putfs/acme-corp/invoices
zfs set quota=1T tank/putfs/acme-corp/documents

PutFS returns 507 Insufficient Storage when the filesystem is full.

ARC tuning

Size

By default, ZFS uses up to 50% of system RAM for ARC. For a dedicated PutFS server, give it more:

# /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=34359738368  # 32 GB

Monitoring

Check ARC hit rate – anything above 90% means most reads are served from RAM:

arc_summary | grep -A5 "ARC size"
arcstat 1  # live monitoring

Key metrics:

ARC hit rate – above 90% is good, above 98% is excellent
Demand metadata hits – directory lookups and stat calls. If this drops, small file performance degrades.
L2ARC hit rate – if low, your working set exceeds SSD + RAM

Complete example

A PutFS server with 64 GB RAM, 2x NVMe, 1x SSD, 4x HDD:

# Create pool
zpool create tank \
    mirror /dev/sda /dev/sdb \
    mirror /dev/sdc /dev/sdd \
    special mirror /dev/nvme0n1 /dev/nvme1n1 \
    cache /dev/sde

# Root dataset
zfs create tank/putfs
zfs set compression=zstd tank/putfs
zfs set atime=off tank/putfs
zfs set xattr=sa tank/putfs
zfs set dnodesize=auto tank/putfs
zfs set special_small_blocks=128K tank/putfs

# ARC – give it 48 GB of the 64 GB
echo 'options zfs zfs_arc_max=51539607552' > /etc/modprobe.d/zfs.conf

# Per-dataset datasets
zfs create tank/putfs/acme-corp
zfs create -o recordsize=4K tank/putfs/acme-corp/thumbnails
zfs create -o quota=500G tank/putfs/acme-corp/documents

With this setup:

Small files (< 128K) are stored and read from NVMe via the special vdev
Hot files are served from RAM (ARC)
Warm files fall back to SSD (L2ARC)
Large files live on mirrored HDDs
Compression makes everything effectively larger in cache
No access time writes on reads