Nginx tuning

The shipped contrib/putfs.nginx.conf and its snippets are tuned for blob storage. This page explains the directives and the extras for high concurrency.

Static file serving

sendfile on;
tcp_nopush on;
tcp_nodelay on;

• sendfile on: files go from the file descriptor to the socket in the kernel, bypassing userspace. The single most important directive.
• tcp_nopush on: batches the headers and the first chunk into one packet (TCP_CORK).
• tcp_nodelay on: sends the final chunk immediately instead of waiting to fill a segment. With tcp_nopush that is batched headers and no trailing delay.

Connection handling

keepalive_timeout 65;
keepalive_requests 1000;
reset_timedout_connection on;

• keepalive_timeout 65: clients that list and then download reuse the connection.
• keepalive_requests 1000: requests per connection before a reconnect (the default; explicit is defensive).
• reset_timedout_connection on: RST instead of FIN for timed-out connections, freeing resources without a TIME_WAIT.

Upload handling

client_max_body_size 0;
proxy_request_buffering off;

• client_max_body_size 0: no upload limit. Enforce quotas at the filesystem (zfs set quota) instead.
• proxy_request_buffering off: streams the body to the API instead of spooling it to a temp file first, which doubles disk I/O. Auth runs before the body is read, so unauthenticated uploads are rejected at the header stage.

Logging

access_log off;
log_not_found off;

• access_log off: a log line per request is blocking I/O that dominates latency. In our benchmarks access_log /dev/stdout cut throughput from 104K req/s to 3.8K req/s. If you need logs, buffer them: access_log /var/log/nginx/access.log buffer=64k flush=5s;.
• log_not_found off: with try_files $uri =404, missing files are normal operation and would flood the error log.

High-concurrency settings

For thousands of concurrent connections, at the top level:

worker_processes auto;
worker_rlimit_nofile 65535;

events {
    worker_connections 4096;
    multi_accept on;
}

• worker_rlimit_nofile 65535: every connection and every open file costs a descriptor; the default ~1024 is too low. Needs a matching host ulimit -n.
• worker_connections 4096: per worker, so cores × 4096 in total. Default 1024.
• multi_accept on: accepts all pending connections per event loop iteration.

Async I/O for large files

location / {
    sendfile on;
    aio threads;
    directio 8m;
    # ...
}

• aio threads: large reads go to a thread pool instead of blocking the worker and every other connection on it. Needs nginx built --with-threads (standard on distros).
• directio 8m: files of 8MB and more bypass the page cache and use aio threads; smaller files keep the sendfile fast path. directio disables sendfile for affected requests itself; there is no conflict.

location / {
    sendfile on;
    sendfile_max_chunk 2m;   # fairer scheduling between large-file streams
    aio threads;             # offloads blocking reads to a thread pool
    # directio  ...          # OMIT on ZFS
}

File descriptor cache

open_file_cache max=10000 inactive=60s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;

Caches nginx's own stat() and open() results (descriptors, sizes, mtimes), not content. For a write-once, read-a-few-times workload the kernel's dentry/inode cache and the ZFS ARC already cover this; enable it only if strace -c or perf top show high stat()/open() counts under load.

Gzip compression

gzip on;
gzip_types text/plain application/json text/css application/javascript;
gzip_min_length 1024;
gzip_proxied any;

Blobs don't compress further, but listings (plain-text keys) do, especially for large buckets. gzip_types skips binary content, gzip_min_length skips tiny payloads, gzip_proxied any includes the proxied API responses.

Upstream keepalive

upstream api {
    server unix:/run/putfs/putfs.sock;
    keepalive 32;
}

Without a pool nginx opens a TCP connection per proxied request. Size keepalive to your API concurrency. Proxied requests then need HTTP/1.1:

proxy_http_version 1.1;
proxy_set_header Connection "";

Auth

Authentication and authorization are two map lookups folded into one $deny gate, resolved from nginx memory with no I/O: if ($deny) { return 403; } at server level. auth_request (a subrequest per request) is the alternative for dynamic auth such as OAuth or token validation. See Auth.

Timeouts

client_body_timeout 300s;
proxy_read_timeout 300s;

The 60s defaults time out multi-GB uploads on slow links and long listings.

Unix socket

A Unix socket skips TCP loopback: no port allocation, no handshake, no TIME_WAIT.

granian --interface asgi --uds /run/putfs/putfs.sock \
    --workers 4 --runtime-mode mt --loop uvloop \
    --backpressure 16 --http 1 --no-ws \
    putfs.api:app

upstream api {
    server unix:/run/putfs/putfs.sock;
    keepalive 32;
}

In docker-compose the socket lives on a volume shared between the api and nginx containers, as the shipped docker-compose.yml does (putfs_run).

Further reading

• ngx_http_core_module: sendfile, tcp_nopush, tcp_nodelay, aio, directio, open_file_cache, keepalive_timeout, client_max_body_size
• ngx_http_proxy_module: proxy_request_buffering, proxy_http_version
• ngx_http_upstream_module: keepalive
• nginx performance tuning gist