Skip to content

Design choices

Separation of concerns

PutFS implements exactly one thing: the write path – PUT, DELETE, and listing. Reads (GET/HEAD) are served directly by nginx with sendfile, so object bytes never pass through the application at all. Authentication and authorization live in nginx too (a two-header API-key model plus presigned URLs), never in the storage layer. Redundancy, checksums, snapshots, encryption, quotas, and durability belong to the filesystem – ZFS on a well-run deployment.

What is left for PutFS is the single operation neither nginx nor the filesystem can make safe on its own: turning a streamed HTTP body into a complete, atomically-published object. Anything that could be pushed down to the reverse proxy or the filesystem is rejected by design, which is what keeps the server small enough to audit line by line.

Filesystem is only truth

PutFS keeps no metadata database, no index, and no sidecar files: the bytes on disk are the state it serves. A PUT is visible to ls the next instant, an mv or rm on disk shows up in the next listing with no reindex step, and rsync, zfs send, or a snapshot rollback all just work – none of them need to know PutFS exists.

There is exactly one caveat, and it is the in-flight temp file. While a PUT streams, its .putfs-tmp.<random> file sits on the filesystem but is deliberately not part of the truth PutFS exposes: it is filtered from listings, its reserved prefix is refused as a key, and it holds a still-incomplete object. It becomes truth only at the instant of the atomic publish described below, never before – the one bounded exception, and it exists precisely so that everything PutFS does expose is always a complete object.

Streaming upload, then atomic publish

Every PUT streams the request body into a private temporary file (.putfs-tmp.<random>) in the same directory as the target key, verifies the checksum if the path is content-addressed, and only then publishes it onto the canonical key with a single atomic operation: rename(2) for plain keys (last-writer-wins) or a create-only link(2) for WORM keys (first-committer-wins, EEXIST otherwise). The temp is unlinked in a finally, so a failed, aborted, or checksum-rejected upload leaves only its own temp and never a half-written key.

This makes one invariant true by construction: the canonical key exists if and only if it is a complete, verified object. There is no window in which a reader, or a racing PUT, can observe a partial write, because a partial write only ever exists under the private temp name.

The temp must share a directory with the target because rename(2) and link(2) cannot cross a filesystem boundary (EXDEV), and operators routinely put per-prefix ZFS datasets on the tree. Its name is unguessable so a client cannot address an in-flight upload through nginx, it is filtered from listings, and the .putfs-tmp. prefix is reserved – any key using it is rejected with 400.

What it replaced: in-place write + a lock and a probe

Earlier versions wrote the body directly into the canonical file and guarded it with an advisory write lock (originally flock, later OFD fcntl locks, F_OFD_SETLK) plus a F_OFD_GETLK probe to answer the question "is this key a finished object, or is someone still writing it?".

That question is inherently racy, because the canonical file becomes visible (an O_EXCL EEXIST to the next writer) before its bytes are complete. Keeping it safe required a growing pile of subtle code, and it still carried a family of races: a re-PUT could observe a not-yet-complete file and return a false 204/409; there was a create-then-lock gap in which a legitimate winner could be bounced; a crash could leave a partial that fooled the probe; an overwrite exposed a torn read to concurrent GETs; and cleanup could only safely unlink while holding the lock.

Temp+rename removes all of it. With completeness guaranteed by the atomic publish, there is no lock to take, no probe to run, and no torn state to reason about – the most delicate, most-audited code in the project collapsed into a textbook idiom, and an entire family of race conditions went to zero. The 423 Locked status disappeared with the per-key lock: plain keys are clean last-writer-wins, WORM keys resolve by whoever commits first.

Why this is the standard idiom, not a local invention

Write-to-a-temp-then-atomically-publish is the canonical pattern for one-file-per-object storage on a POSIX filesystem, and it is what comparable systems do:

  • MinIO streams every object into a private .minio.sys/tmp/<uuid>/ (opened O_EXCL), then commits with RenameData() down to a POSIX rename(2). It never writes in place: each write uses a fresh data directory and the old one is removed only after the rename. This is PutFS's design at a larger scale.
  • OpenStack Swift, Git (loose objects), Nix, Perkeep, IPFS, and Garage all write to a temp file and then rename/link it into place – several of them, like PutFS's checksum paths, to a content-addressed name.
  • The guarantee underneath is stated verbatim in both the Linux man page and POSIX: rename() replaces the destination atomically, so a concurrent reader sees the old file or the new file, never a mix.

Content-addressing (checksum paths) makes the idiom even safer: a half-written temp can never occupy a valid hash path, and two writers of identical content are idempotent. IPFS's flat store states it in the same terms PutFS uses – "first-successful-writer-wins … safe for content-addressed data because identical keys guarantee identical values" – which is exactly why WORM keys publish with a create-only link rather than a replace.

Durability

The atomic publish guarantees visibility (no torn or partial object), not persistence. PutFS issues no fsync, so a 201 means "atomically visible", not "flushed to disk"; durability is delegated to the filesystem. See Write atomicity and durability for what a 201 guarantees and the ZFS sync setting that makes it power-loss-safe.

A hard crash (SIGKILL, power loss) can orphan a temp before it is published. Reclaim orphans out of band, e.g. a cron find /srv/putfs -name '.putfs-tmp.*' -mmin +120 -delete with the age set comfortably above your longest expected upload.