Design choices
Separation of concerns
PutFS implements the write path only: PUT, DELETE and the QUERY listing. nginx serves reads with sendfile, so object bytes never pass through the application, and nginx holds authentication and authorization (two-header API keys, presigned URLs). Redundancy, checksums, snapshots, encryption, quotas and durability belong to the filesystem.
What remains is the one operation neither nginx nor the filesystem can make safe alone: turning a streamed HTTP body into a complete, atomically published object. Anything that could live in the reverse proxy or the filesystem is rejected by design, which keeps the server small enough to audit line by line.
The filesystem is the only truth
No metadata database, no index, no sidecar files: the bytes on disk are the state PutFS serves, so ls, mv, rsync, zfs send and snapshot rollbacks all work without knowing PutFS exists (see Data model).
The one bounded exception is the in-flight temp file. While a PUT streams, its .putfs-tmp.<random> file is on the filesystem but not part of the exposed truth: filtered from listings, its prefix refused as a key, holding an incomplete object. It becomes truth at the atomic publish, never before.
Streaming upload, then atomic publish
Every PUT streams into a private temp file in the target's directory, verifies the checksum if the key is content-addressed, and publishes with one atomic operation: rename(2) for plain keys (last-writer-wins) or a create-only link(2) for WORM keys (first-committer-wins, EEXIST otherwise). The temp is unlinked in a finally, so a failed, aborted or checksum-rejected upload leaves only its own temp.
One invariant follows by construction: a key exists if and only if it is a complete, verified object. No reader or racing PUT can observe a partial write, because a partial write only exists under the private temp name.
The temp shares a directory with the target because rename(2) and link(2) cannot cross a filesystem boundary (EXDEV), and per-prefix ZFS datasets are routine. Its name is unguessable, so a client cannot address an in-flight upload through nginx, and the .putfs-tmp. prefix is reserved: a key using it is a 400.
What it replaced
Earlier versions wrote the body straight into the canonical file under an advisory lock (flock, later OFD fcntl) plus a lock probe to tell a finished object from one still being written. That question is inherently racy, since the canonical file is visible before its bytes are complete, and it carried a family of races: a false 204/409 on a re-PUT, a create-then-lock gap, crash partials fooling the probe, torn reads on overwrite. Temp+rename removed all of it, and the 423 Locked status with it: plain keys are last-writer-wins, WORM keys resolve by whoever commits first.
The standard idiom
Write-to-temp-then-publish is the canonical pattern for one-file-per-object storage on POSIX:
- MinIO streams every object into a private
.minio.sys/tmp/<uuid>/(openedO_EXCL) and commits with arename(2); it never writes in place. - OpenStack Swift, Git (loose objects), Nix, Perkeep, IPFS and Garage write a temp and
rename/linkit into place, several of them to a content-addressed name. - POSIX and the Linux man page guarantee that
rename()replaces the destination atomically: a concurrent reader sees the old file or the new one, never a mix.
Content addressing makes it safer still: a half-written temp can never occupy a valid hash path, and two writers of identical content are idempotent, which is why WORM keys publish with a create-only link rather than a replace.
Durability
The atomic publish guarantees visibility, not persistence. PutFS issues no fsync, so a 201 means "atomically visible", not "flushed to disk"; see Write atomicity and durability for the ZFS sync setting that makes it power-loss-safe.
A hard crash (SIGKILL, power loss) can orphan a temp before it is published. Reclaim orphans out of band, e.g. a cron find /srv/putfs -name '.putfs-tmp.*' -mmin +120 -delete with the age comfortably above your longest upload.