MinIO migration
PutFS was originally researched and developed to find a replacement for self-hosted MinIO deployments. Almost the entire feature set can be replicated on the Linux/ZFS stack with UNIX tooling.
Assumptions
With OpenAleph and our managed deployments as the target in mind, for our use case there are a few assumptions that make the features below and therefore the MinIO replacement possible:
- Small number of Buckets, not added every hour (each new Bucket is a
zfs create ...) - Only "operators" can manage Buckets and access
- Simple IAM Policies, not frequent changes
- Nature of data / access patterns are mostly (but not necessary) WORM
In other words, tenant / bucket privisioning and access control is part of the deployment. Otherwise, the lack of an admin UI is a bit annoying.
Feature mapping
| MinIO feature | PutFS | Reference |
|---|---|---|
| Object storage | Plain filesystem | Data model |
| Bucket management | Directories, or ZFS datasets | Data model |
| IAM policies | nginx map / auth_request |
Auth |
| API key management | keys.conf + nginx -s reload |
Auth |
| S3 SigV4 auth | awssig verification |
Auth |
| Presigned URLs | nginx secure_link |
Presigned URLs |
| Site replication | zrepl, rsync, rclone, DRBD | Replication |
| Object versioning | ZFS snapshots, or btrfs/LVM snapshots, rsync --link-dest |
Versioning |
| Point-in-time restore | zfs rollback, or btrfs/LVM restore |
Versioning |
| Erasure coding | ZFS mirror / RAIDZ, or mdadm RAID 5/6, btrfs | Data integrity |
| Self-healing / bit-rot | ZFS checksums + scrub, or dm-integrity + mdadm, btrfs scrub | Data integrity |
| Encryption at rest | ZFS native encryption, or LUKS / dm-crypt | Encryption |
| Storage tiering | ZFS ARC / L2ARC / special vdev, or OS page cache + bcache | Storage tiering |
| S3 API | Minimal S3 subset (GET/PUT/DELETE/HEAD/List) | Sync & mount |
| WebDAV / SFTP | nginx dav_methods, ssh |
Sync & mount |
| Desktop mount | WebDAV via Finder / Cyberduck (also sftp/ftps) | Sync & mount |
| Quota management | zfs set quota, or setquota / disk limits |
Lifecycle |
| Object lifecycle / expiry | cron + find |
Lifecycle |
| Object WORM | chattr +i (any filesystem) |
WORM |
| Bucket read-only | zfs set readonly=on, or mount -o remount,ro / chattr -R +i |
WORM |
| Legal hold | ZFS snapshot holds, or chattr +i on backup |
WORM |
| Read cache | ZFS ARC, or OS page cache | ZFS tuning |
| SSD cache tier | ZFS L2ARC, or bcache / dm-cache | ZFS tuning |
| Write acceleration | ZFS SLOG (NVMe) | ZFS tuning |
Honest gaps
Features with no clean PutFS equivalent. Out of scope by design.
| MinIO feature | Why not |
|---|---|
| Active-active multi-site | Requires distributed consensus (Raft/Paxos). Use Garage or Ceph. |
| Full S3 API (multipart, ACLs, etc.) | Partial – core subset works. Multipart upload on roadmap if needed. |
| S3 Select (SQL over objects) | Niche. Use your application layer. |
| Lambda / trigger integration | AWS-ecosystem specific. Use nginx log pipelines. |
| Admin UI | Use your filesystem, ZFS CLI, or build your own. |
| Cross-account IAM federation | Use your identity provider directly via auth_request. |
Alternatives
If the gaps above are dealbreakers:
| System | Best for | Trade-off |
|---|---|---|
| SeaweedFS | High-volume distributed with filer API | 3 components, own metadata layer |
| Ceph | Large-scale, erasure-coded, multi-tenant | Complex to operate, needs dedicated infra |
| Garage | Geo-distributed, multi-site S3 | Designed for 3+ nodes |
| RustFS | MinIO clone in Rust | Alpha stage and we find it very sketchy |
| GlusterFS | Distributed POSIX filesystem | Heavy, not designed for object storage |
Migration steps
One of the main features is the flat filesystem. So the migration is straight forward: ssh into the PutFS server and directly mc mirror on the ZFS / filesystem.
For the S3 API path, most clients just need endpoint_url changed – boto3, aws s3, and rclone work with the PutFS S3 subset.