Skip to content

S3 (not implemented)

PutFS ships no S3 endpoint. We prototyped a minimal S3-compatible layer and removed it. This page records why, and what to watch out for if you reimplement it on top of PutFS.

Why we removed it

The prototype verified SigV4 with awssig, the only readily available pure-Python verifier on PyPI, via AWSSigV4S3Verifier(..., body=b""):

  1. No body integrity. awssig echoes the client-supplied X-Amz-Content-Sha256 into the canonical string and never compares it to the bytes received; AWS expects the server to verify it independently.
  2. Unmaintained. Last release 0.5.0 (2019-12-14), still depends on six. Signature comparison uses == (not constant-time), and its S3 URI normalisation preserves .. and //, so the signed canonical path can differ from what the OS resolves.

Design notes for re-implementers

SigV4 verification

  • Don't use awssig, or treat it as unaudited: re-implement the canonical request (the spec is ~50 lines of code) and compare with hmac.compare_digest.
  • Re-hash the body. Stream it through hashlib.sha256() and compare to the verified X-Amz-Content-Sha256; refuse a mismatch with 400 XAmzContentSHA256Mismatch.
  • Decide a policy for UNSIGNED-PAYLOAD. boto3 defaults to it for streaming uploads, which is spec-compliant but leaves body integrity to TLS. Refuse it for writes unless the deployer opts in.
  • STREAMING-AWS4-HMAC-SHA256-PAYLOAD signs each chunk and needs chunk-trailer parsing.
  • Pin a strict timestamp window (15 min, as AWS). Without nonce tracking a captured signed request replays within the window over plain HTTP, so require HTTPS.

X-Accel-Redirect

Serve GET via nginx sendfile, building the redirect from the resolved relative path:

rel = path.relative_to(root).as_posix()
headers["x-accel-redirect"] = f"/_internal/{rel}"

location /_internal/ {
    internal;
    alias /data/;
}

In the proxy block, set Host explicitly so the SigV4 signature matches the host the client signed against:

proxy_set_header Host $http_host;

What the prototype implemented

GetObject, PutObject, HeadObject, DeleteObject, ListObjectsV2 (prefix, delimiter, continuation-token pagination), ListBuckets, CreateBucket, HeadBucket, DeleteBucket. No multipart upload. XML responses used xml.sax.saxutils.escape, fine for static fields but not for binary in object keys.