ZFS tuning

Pool and dataset configuration for PutFS workloads: small-file random reads, writes and prefix listings. The feature mapping lists what ZFS brings to PutFS beyond speed.

Pool topology

Special vdev (the biggest win)

A special vdev stores ZFS metadata and small files (under special_small_blocks) on a dedicated device, typically NVMe.

# Mirrored data vdevs and a mirrored NVMe special vdev
zpool create tank \
    mirror /dev/sda /dev/sdb \
    mirror /dev/sdc /dev/sdd \
    special mirror /dev/nvme0n1 /dev/nvme1n1

Warning

Always mirror the special vdev. If a single one fails, the pool is lost.

Metadata (dnodes, indirect blocks, directory entries) always lives on the special vdev, which alone speeds up listings and stat calls. Small file data follows only with:

# Files under 128K go to the special vdev
zfs set special_small_blocks=128K tank/putfs

With that, every small-file read and every directory lookup is an NVMe read; the spinning disks only handle large files.

SLOG

A SLOG (separate intent log) absorbs synchronous write latency. A small, fast NVMe device (16–32 GB) is enough; it holds in-flight transactions only.

zpool add tank log /dev/nvme2n1

L2ARC

L2ARC extends the ARC (RAM cache) onto SSD for a working set that exceeds RAM but fits on SSD. If the dataset fits in ARC, it adds nothing.

zpool add tank cache /dev/sde

Recommended topology

NVMe 1+2 (mirror)  → special vdev  (metadata + small files)
NVMe 3             → SLOG          (write intent log)
SSD 1              → L2ARC         (read cache overflow)
HDD 1+2 (mirror)   → data vdevs   (large files)
HDD 3+4 (mirror)   → data vdevs   (large files)

Synchronous vs asynchronous writes

PutFS issues no fsync, so the dataset's sync property decides when a PUT is on stable storage:

Value	Behavior with PutFS	A just-acknowledged PUT is...
`sync=standard` (default)	PUTs are asynchronous: batched into the open transaction group and flushed at the next commit (`zfs_txg_timeout`, default 5s).	atomically visible, not yet on stable storage; a power loss within ~5s drops it whole, never torn.
`sync=always`	Every write goes to the ZIL (SLOG) before it returns.	on the SLOG before the `201`: the setting that makes an acknowledged PUT power-loss-safe.
`sync=disabled`	All writes asynchronous, every `fsync` ignored. Identical to `standard` for PutFS.	in the same ~5s window; no faster than `standard` for PutFS.

sync=standard returns a PUT as soon as the body is in RAM and the rename is done, sub-millisecond, with no SLOG wait. sync=always costs ~0.1ms per write with an NVMe SLOG, 5–15ms per write on HDDs without one, every chunk of a large upload included; use it for primary data that must survive a crash. sync=disabled only removes durability for other tools that do fsync on the dataset (zfs receive); there is no reason to set it for PutFS.

# Default: asynchronous for PutFS
zfs set sync=standard tank/putfs

# Durable on ack, for primary data
zfs set sync=always tank/putfs/acme-corp/documents

Durability is opt-in for PutFS

Only sync=always makes an acknowledged PUT power-loss-safe; standard and disabled both leave a ~5s window. Children inherit the parent's sync unless overridden.

Dataset properties

One dataset per bucket

zfs create tank/putfs
zfs create tank/putfs/acme-corp
zfs create tank/putfs/acme-corp/invoices

recordsize

recordsize is the maximum block size, not the minimum: a 2KB file in a 128K dataset uses one 4K block. Lower it only for datasets of many files between 4K and 128K, where it cuts per-block metadata:

zfs set recordsize=4K tank/putfs/acme-corp/thumbnails

Compression

Always on; with zstd the ARC and L2ARC effectively grow, since cached data stays compressed. Turn it off only for already-compressed content (JPEG, video, ZIP):

zfs set compression=zstd tank/putfs
zfs set compression=off tank/putfs/acme-corp/media

Extended attributes

Pack xattrs into the dnode to save an I/O:

zfs set xattr=sa tank/putfs
zfs set dnodesize=auto tank/putfs

atime

Every GET would otherwise write metadata:

zfs set atime=off tank/putfs

Quotas

zfs set quota=100G tank/putfs/acme-corp/invoices

PutFS returns 507 Insufficient Storage when the filesystem is full.

ARC tuning

Size

ZFS uses up to 50% of RAM for the ARC by default. On a dedicated server, give it more:

# /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=34359738368  # 32 GB

Monitoring

arcstat 1                    # live hit rate, size, evictions
arc_summary                  # detailed ARC breakdown
zpool iostat -v tank 5       # per-vdev I/O
zpool iostat -l tank 5       # latency histograms

ARC hit rate: above 90% is good, above 98% means most reads come from RAM.
Demand metadata hits: directory lookups and stat calls; a drop here degrades listings.
L2ARC hit rate: if low, the working set exceeds SSD + RAM.

For Prometheus, zfs_exporter exposes pool, dataset and ARC metrics, and nginx-prometheus-exporter scrapes nginx's stub_status. Alert on pool capacity above 85% (PutFS answers 507 when full), an ARC demand hit rate below 0.9, and replication lag.

Complete example

A PutFS server with 64 GB RAM, 2x NVMe, 1x SSD, 4x HDD:

# Create pool
zpool create tank \
    mirror /dev/sda /dev/sdb \
    mirror /dev/sdc /dev/sdd \
    special mirror /dev/nvme0n1 /dev/nvme1n1 \
    cache /dev/sde

# Root dataset
zfs create tank/putfs
zfs set compression=zstd tank/putfs
zfs set atime=off tank/putfs
zfs set xattr=sa tank/putfs
zfs set dnodesize=auto tank/putfs
zfs set special_small_blocks=128K tank/putfs

# ARC – give it 48 GB of the 64 GB
echo 'options zfs zfs_arc_max=51539607552' > /etc/modprobe.d/zfs.conf

# Per-bucket datasets
zfs create tank/putfs/acme-corp
zfs create -o recordsize=4K tank/putfs/acme-corp/thumbnails
zfs create -o quota=500G tank/putfs/acme-corp/documents

Small files (< 128K) live on NVMe, hot files in RAM, warm files on SSD, large files on mirrored HDDs; everything is compressed in cache and no read writes an atime.