CLI

The PutFS command-line client works like awscli or mc. It moves data between a local filesystem and a PutFS server, either way round, or between two PutFS servers; sync is multi-threaded.

pip install putfs[client]

Authentication

Via flags or environment variables:

# Environment (recommended)
export PUTFS_API_KEY=my-key
export PUTFS_API_SECRET=my-secret
export PUTFS_ENDPOINT_URL=putfs://localhost:8000
export PUTFS_HTTPS=0

# Or per-command
putfs --key my-key --secret my-secret <command>

Commands

sync

Sync a local directory and a PutFS store, either way round, or two PutFS stores, with threaded streaming transfers.

# Local directory → PutFS
putfs sync /data/exports putfs://localhost:8000/acme/exports

# PutFS → local directory
putfs sync putfs://localhost:8000/acme/exports /backup/exports

# Between two PutFS instances
putfs sync putfs://primary:8000/acme putfs://secondary:8000/acme

# Force re-upload everything (skip diff checks)
putfs sync --overwrite /data/exports putfs://localhost:8000/acme/exports

# Delete extra files on target
putfs sync --delete /data/exports putfs://localhost:8000/acme/exports

# Control parallelism (default: min(CPU count, 4))
putfs sync -w 16 /data/exports putfs://localhost:8000/acme/exports

# Retry budget per key on transient faults (default: 5)
putfs sync --retries 10 /data/exports putfs://localhost:8000/acme/exports

# Exclude a prefix from scanning and syncing entirely, in both directions (repeatable)
putfs sync --exclude tmp/ --exclude logs/ /data/exports putfs://localhost:8000/acme/exports

How a run decides:

• Both sides are listed directory by directory with each entry's size and mtime (the server's stat listing), several directories at once and alongside the transfers, so memory stays bounded and the first transfer starts before the whole tree is known. No key is probed: a key the target lacks, or whose source mtime is newer (size as the fallback), is transferred, the rest skipped. A PutFS source is read as one stream, without a HEAD.
• --skip-existing keeps every key the target holds. --overwrite transfers every source file without comparing and, without --delete, never lists the target.
• --delete removes the target's extra keys once the transfers are done. If any listing failed, nothing is deleted, since the extra keys are unknown. An unreadable local directory is a failed listing, never a silent skip.
• --exclude prunes a key prefix (tmp, tmp/ and /tmp are the same) on both sides before it is listed, so excluded target content is never taken for an orphan under --delete.
• A name that is a file on one side and a directory on the other counts as a failed key and is left alone on both sides; remove one to resolve it.
• A local source must be an existing directory. A missing PutFS source prefix lists as empty, since the server cannot tell it from an empty one.
• The target's mtime is the server's clock at upload. A server clock behind the source's by more than an upload takes makes that file look newer on every run.

Failures and retries:

• Transient faults (read timeouts, dropped connections, 429, transient 5xx) are retried per key with exponential backoff and jitter. The budget is --retries (default PUTFS_SYNC_RETRIES, 5), the base delay PUTFS_SYNC_RETRY_BACKOFF (default 1s, capped at 60s).
• Permanent answers are not retried: 403, 404, 501 and 507 fail the key; 409 (WORM re-PUT) and 422 (checksum mismatch) are skips.
• A key that exhausts its budget is logged and counted, and the run goes on. A directory listing is retried the same way; one that still fails, on either side, counts once and its subtree is left out. That includes a target that cannot be listed: a key whose scope lacks QUERY, a directory over PUTFS_MAX_LIST_KEYS, a server too old for stat. --overwrite without --delete never lists the target, so it still gets files into one.
• A key whose name anystore refuses (a whitespace-only one) fails on its own, neither transferred nor deleted.
• The final line reports keys synced and deleted. If anything failed, the count goes to stderr and the exit code is 1.

ls

# List all keys
putfs ls putfs://localhost:8000/acme/

# Exclude prefix
putfs ls --exclude-prefix temp/ putfs://localhost:8000/acme/

cp

Copy a single file between stores.

# Upload
putfs cp /local/file.pdf putfs://localhost:8000/acme/file.pdf

# Download
putfs cp putfs://localhost:8000/acme/file.pdf /local/file.pdf

# Between two PutFS instances
putfs cp putfs://primary:8000/acme/file.pdf putfs://secondary:8000/acme/file.pdf

cp copies the way sync does: a PutFS source is read as one stream, transient faults retry the whole copy (--retries, default PUTFS_SYNC_RETRIES), and 409/422 are skips. A pipe (-) gets a single attempt. The progress bar on stderr counts bytes against the source size (stat for a local file, HEAD for a PutFS key).

rm

# Delete a single file
putfs rm putfs://localhost:8000/acme/old-file.pdf

# Delete all files under a prefix
putfs rm --recursive putfs://localhost:8000/acme/temp/

URI schemes

Other storage backends are out of scope; a sync between them belongs on anystore's Store interface, which the Python client builds on.