Self-Hosted S3-Compatible Object Storage — Garage vs SeaweedFS vs Ceph RadosGW

Object storage has become the de facto standard for storing unstructured data — backups, media files, logs, and machine learning datasets. While Amazon S3 dominates the cloud, several open-source projects provide S3-compatible object storage you can run on your own hardware. This guide compares Garage, SeaweedFS, and Ceph RadosGW, three distinct approaches to self-hosted object storage.

Project Overview

Architecture Comparison

Garage — Lightweight Geo-Distributed Storage

Garage is designed specifically for small, self-hosted, geo-distributed deployments. Key design principles:

• Single binary: No complex dependencies or separate services
• Geo-distribution: Works across sites with high-latency links
• Erasure coding: Built-in support for storage efficiency
• S3-first: Designed from day one for S3 API compatibility
• Simplicity: Aims to be the simplest object store to deploy and operate

Garage uses a ring-based data distribution algorithm with configurable replication factors. Each node stores a subset of data, and the system automatically replicates across nodes based on a topology configuration. This makes it ideal for deployments spanning multiple data centers or edge locations.

SeaweedFS — High-Performance Storage with Filer

SeaweedFS implements a two-layer architecture:

• Volume servers: Store small files efficiently (handles billions of files)
• Filer service: Provides directory structure, metadata, and S3 gateway
• Master servers: Coordinate volume allocation and replication

SeaweedFS is inspired by Facebook’s Haystack paper and excels at storing large numbers of small files. The S3 gateway runs as part of the filer service, providing a drop-in S3-compatible API. Unlike Garage, SeaweedFS also provides a POSIX-like filesystem interface through its filer.

Ceph RadosGW — Enterprise-Scale Object Storage

Ceph RadosGW (RADOS Gateway) is the S3/Swift-compatible interface layer on top of the Ceph storage cluster:

• RADOS core: Distributed object store with CRUSH placement algorithm
• RGW layer: HTTP gateway implementing S3 and Swift APIs
• Monitors: Cluster state management and consensus
• OSDs: Storage daemons managing individual disks

Ceph is the most feature-complete and battle-tested option, but also the most complex. It provides unified block (RBD), object (RGW), and file (CephFS) storage on the same cluster. For organizations already running Ceph, RadosGW adds S3 compatibility with no additional infrastructure.

Deployment & Setup

Garage Docker Compose

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

version: '3.8'
services:
  garage:
    image: ghcr.io/deuxfleurs-org/garage:latest
    command: ["garage", "server", "-c", "/etc/garage/garage.toml"]
    ports:
      - "3900:3900"  # S3 API
      - "3901:3901"  # RPC (internal)
      - "3902:3902"  # HTTP admin
    volumes:
      - ./garage.toml:/etc/garage/garage.toml:ro
      - garage_data:/var/lib/garage
    environment:
      - RUST_LOG=info
    deploy:
      resources:
        limits:
          memory: 2G

volumes:
  garage_data:

Configuration file (garage.toml):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
rpc_bind_addr = "0.0.0.0:3901"
s3_api_bind_addr = "0.0.0.0:3900"
s3_root_domain = "s3.garage.example.com"
db_engine = "sqlite"

[replication]
mode = "zone"

[replication.layout]
zones = ["dc1"]
replication_factor = 1

For multi-node deployments, add more nodes to the cluster and configure the topology with zone assignments for geo-distribution.

SeaweedFS Docker Compose

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

version: '3.8'
services:
  weed-master:
    image: chrislusf/seaweedfs:latest
    command: ["master", "-ip=weed-master", "-ip.bind=0.0.0.0", "-defaultReplication=001"]
    ports:
      - "9333:9333"
      - "19333:19333"
    volumes:
      - master_data:/data

  weed-volume:
    image: chrislusf/seaweedfs:latest
    command: ["volume", "-mserver=weed-master:9333", "-ip.bind=0.0.0.0", "-dataCenter=dc1", "-rack=rack1"]
    ports:
      - "8080:8080"
      - "18080:18080"
    volumes:
      - volume_data:/data
    depends_on:
      - weed-master

  weed-filer:
    image: chrislusf/seaweedfs:latest
    command: ["filer", "-master=weed-master:9333", "-ip.bind=0.0.0.0"]
    ports:
      - "8888:8888"
      - "8889:8889"
    volumes:
      - filer_data:/data
    depends_on:
      - weed-master
      - weed-volume

  weed-s3:
    image: chrislusf/seaweedfs:latest
    command: ["s3", "-filer=weed-filer:8888", "-ip.bind=0.0.0.0"]
    ports:
      - "8333:8333"
    depends_on:
      - weed-filer

volumes:
  master_data:
  volume_data:
  filer_data:

Ceph RadosGW — Minimal Deployment

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

version: '3.8'
services:
  ceph-mon:
    image: docker.io/ceph/ceph:v18
    command: ["mon", "node1", "--public-addr", "0.0.0.0:6789"]
    ports:
      - "6789:6789"
    volumes:
      - /etc/ceph:/etc/ceph
      - mon_data:/var/lib/ceph/mon
    environment:
      - CEPH_DEPLOY_CEPH_RELEASE=reef
      - CEPH_PUBLIC_NETWORK=0.0.0.0/0

  ceph-osd:
    image: docker.io/ceph/ceph:v18
    command: ["osd", "ceph", "--osd-data", "/var/lib/ceph/osd/ceph-0"]
    volumes:
      - /etc/ceph:/etc/ceph
      - osd_data:/var/lib/ceph/osd
    privileged: true

  ceph-rgw:
    image: docker.io/ceph/ceph:v18
    command: ["rgw", "node1", "--rgw-frontends", "civetweb port=8080"]
    ports:
      - "8080:8080"
    volumes:
      - /etc/ceph:/etc/ceph
    depends_on:
      - ceph-mon
      - ceph-osd

volumes:
  mon_data:
  osd_data:

Note: Ceph in production requires careful tuning of CRUSH maps, OSD placement, and monitor quorum. This minimal config is for testing only.

Performance Comparison

Why Self-Host Your Object Storage?

Cost Savings at Scale: Cloud S3 storage pricing adds up quickly when storing terabytes of data. Self-hosted object storage on commodity hardware costs a fraction of cloud pricing, especially for cold data that is rarely accessed. Garage running on three $100 VPS instances can replace an S3 bucket costing $50+/month.

Bandwidth Control: When your storage is in the same network as your compute, data transfer is free and fast. Cloud object storage charges egress fees that can exceed storage costs for data-intensive workloads like machine learning training or video processing.

Data Sovereignty: Regulations like GDPR require data to remain within specific geographic boundaries. Self-hosted object storage gives you complete control over where your data physically resides.

No API Rate Limits: Cloud object storage providers throttle API requests. Self-hosted solutions have no rate limits, which is critical for high-throughput applications like log aggregation or CI/CD artifact storage.

Simplified Backup Strategy: Self-hosted object storage can be replicated between your own data centers using the same tool, eliminating the need for third-party backup services and cross-cloud data transfer costs.

For related reading, see our self-hosted cloud storage aggregators comparison and our NAS solutions guide. If you are comparing cold storage options, our complete cold storage guide covers the broader landscape.

FAQ

Can Garage run as a single node for development?

Yes. Garage is specifically designed to work well in single-node configurations. Simply start one Garage instance with a local data directory and you have a fully functional S3-compatible endpoint. This makes it ideal for local development and testing before deploying to a multi-node production cluster.

Does SeaweedFS support erasure coding?

No, SeaweedFS uses replication for data durability, not erasure coding. The replication factor is configurable (default is 3 copies). For erasure coding support, consider Garage or Ceph. However, SeaweedFS’s volume-based architecture is highly efficient for storing billions of small files, which partially compensates for the lack of erasure coding.

How does Ceph RadosGW compare to MinIO for S3 compatibility?

Ceph RadosGW and MinIO both provide excellent S3 API compatibility. The key difference is that Ceph is a unified storage system (object + block + file), while MinIO is purpose-built for object storage only. If you need block or file storage in addition to S3, Ceph is the better choice. If you only need object storage, MinIO is simpler to deploy and manage. Both are production-proven at massive scale.

What is the minimum hardware requirement for each?

Garage: 512 MB RAM, 1 CPU core, and any amount of disk. SeaweedFS: 1 GB RAM, 1 CPU core, and disk for volume data. Ceph RadosGW: 4 GB RAM per OSD, 2 CPU cores, and dedicated disks. Ceph has the highest resource requirements due to its distributed architecture and consensus mechanisms.

Can I migrate from one object storage to another?

Yes. All three support the S3 API, so you can use tools like rclone to migrate data between them. For example, rclone sync garage-remote:bucket s3-remote:bucket will copy all objects from a Garage instance to any S3-compatible target. The S3 API compatibility makes migrations straightforward.

Does Garage support lifecycle policies and versioning?

As of the latest release, Garage supports basic S3 operations (GET, PUT, DELETE, LIST, multipart upload) but does not yet implement S3 lifecycle policies or object versioning. These features are on the roadmap. If lifecycle management is critical, SeaweedFS or Ceph RadosGW provide more complete S3 API coverage.