Skip to content

Replication

PutFS is single-node by design and has no replication protocol of its own. The data is plain files, so any tool that copies files or filesystems replicates it: zrepl, rsync, rclone or DRBD. The primary accepts writes; secondaries hold read-only copies and can be promoted. Replication is asynchronous: writes between the last snapshot and a failure are not yet replicated, but can be recovered from the failed node once it is back (see disaster recovery). Read downtime is zero.

zrepl (ZFS)

Continuous incremental replication via ZFS send/receive:

# /etc/zrepl/zrepl.yml
jobs:
  - name: putfs-push
    type: push
    connect:
      type: tcp
      address: backup-host:8888
    filesystems:
      "tank/putfs<": true
    snapshotting:
      type: periodic
      interval: 15m
      prefix: zrepl_

rsync

Any filesystem:

*/15 * * * * rsync -a --delete /srv/putfs/ backup-host:/srv/putfs/

rclone

Between PutFS instances over WebDAV:

rclone sync :webdav,url=https://primary/dav/ :webdav,url=https://secondary/dav/

Load balancer failover

Writes (PUT/DELETE) go to the primary; reads (GET, HEAD and the QUERY listing) round-robin over all nodes. A 404 from a lagging secondary is retried on the primary. Lag has no such signal for listings: a QUERY on a lagging secondary answers 200 without the newest keys, so route QUERY to the primary too if that matters.

upstream primary {
    server primary-host:8000;
}

upstream readers {
    server primary-host:8000;
    server secondary-host:8000;
}

server {
    listen 80;

    location / {
        # Explicit method match, not limit_except GET HEAD: that would
        # send QUERY listings to the primary (and QUERY cannot be named
        # in limit_except – nginx rejects unknown methods there).
        if ($request_method ~ ^(PUT|DELETE)$) {
            proxy_pass http://primary;
        }
        proxy_pass http://readers;
        proxy_next_upstream error timeout http_404;
    }
}

global
    maxconn 4096

defaults
    mode http
    timeout connect 5s
    timeout client 30s
    timeout server 30s

frontend putfs
    bind *:80
    # QUERY (listing) is a read: it stays on the readers backend.
    acl is_write method PUT DELETE
    use_backend writers if is_write
    default_backend readers

backend writers
    server primary primary-host:8000 check

backend readers
    balance roundrobin
    retry-on 404
    retries 1
    server primary primary-host:8000 check
    server secondary secondary-host:8000 check

Dataset routing

Horizontal scaling works as long as one dataset fits one node: route whole datasets to different servers at the load balancer, each running its own PutFS + nginx + pool. New datasets go to the server with capacity; existing data stays put, no rebalancing.

map $uri $backend {
    ~^/bucket-a/  server1:8000;
    ~^/bucket-b/  server2:8000;
    default       server1:8000;
}

server {
    location / {
        proxy_pass http://$backend;
    }
}

Disaster recovery (zrepl)

For a two-node zrepl setup (primary pushes snapshots to the secondary), the recovery path depends on what failed. Usually the pool is intact: the data disks are mirrored and survive an OS disk failure, a crash or a network outage. Then recover the primary; do not fail over. Reads continue from the secondary meanwhile, and applications should retry write errors anyway, so a temporary 5xx is expected behaviour, not a reason to fail over.

Primary fails
    │
    ├─ Pool intact (OS disk failure, software crash)
    │    → Recover primary, do NOT failover
    │    → Secondary serves reads during recovery
    │    → zrepl resumes automatically once primary is back
    │
    └─ Unrecoverable (hardware dead, days to replace)
         → Promote secondary to primary
         → When old primary recovers:
              1. Transfer gap files (written after last snapshot, before failure)
              2. Rollback old primary to shared ancestor snapshot
              3. Reverse zrepl direction (new primary pushes to recovered node)
              4. Optional: flip back to original topology once synced

Failover: promote the secondary

  1. Stop the zrepl sink on the secondary
  2. Start the PutFS write path on the secondary
  3. Point DNS / the load balancer at the secondary
  4. Record the last shared snapshot name, the common ancestor for recovery

Recovery: the old primary comes back

Its filesystem may hold gap files, written after the last replicated snapshot and never snapshotted.

  1. Find them: find -newer .zfs/snapshot/<shared> on the old primary
  2. rsync them to the new primary (WORM data is purely additive, no conflicts)
  3. zfs rollback -r the old primary to the shared ancestor
  4. Reverse zrepl: the new primary pushes to the recovered node
  5. zrepl finds the shared ancestor and sends only the delta

Monitoring replication lag

Larger lag means more gap files on failure:

zrepl status | grep -A5 "push_to_replica"

With Prometheus (zrepl exposes metrics):

- alert: ZreplReplicationLag
  expr: time() - zrepl_replication_last_successful_step_timestamp > 900
  for: 5m

Further reading