Self-Hosted Read-Only Filesystem for Containers: EROFS vs SquashFS vs zstd Compression

When deploying self-hosted container workloads at scale, the choice of read-only filesystem and image compression strategy directly impacts pull times, disk usage, and runtime performance. Three approaches dominate this space: EROFS (Enhanced Read-Only Filesystem), SquashFS, and zstd-compressed container images. Each offers different tradeoffs in compression ratio, random-access performance, and ecosystem maturity.

This guide compares all three approaches with practical Docker Compose configurations, benchmark results, and deployment recommendations for homelab and production environments.

What Are Read-Only Filesystems for Containers?

Container images are inherently read-only layers that get stacked to form the final filesystem view. The format used to store and distribute these layers affects:

• Image pull speed — how fast layers download and decompress
• Disk footprint — storage efficiency on registry and host
• Runtime performance — random read latency when containers access files
• Startup time — time from docker pull to container ready

Traditional OCI images use tar+gzip layers. Modern alternatives leverage dedicated read-only filesystem formats or advanced compression algorithms to optimize each of these dimensions.

EROFS (Enhanced Read-Only Filesystem)

EROFS is a Linux kernel filesystem designed specifically for read-only scenarios. Originally developed by Huawei for Android, it has been mainlined since kernel 5.4 and is now gaining adoption in container runtimes.

Key Features

• In-place decompression — no need to decompress entire images before mounting
• Fixed-sized output compression — enables random access without full decompression
• Page cache sharing — multiple containers share the same cached pages
• Metadata compression — reduces inode and directory entry overhead

Docker Compose with EROFS

While EROFS is primarily a kernel-level feature, container runtimes like containerd and nerdctl support EROFS-based image storage via stargz-snapshotter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# docker-compose.yml for stargz-snapshotter with EROFS support
version: "3.8"
services:
  snapshotter:
    image: ghcr.io/containerd/stargz-snapshotter:latest
    container_name: stargz-snapshotter
    restart: always
    volumes:
      - /var/lib/containerd-stargz-grpc:/var/lib/containerd-stargz-grpc
      - /etc/containerd-stargz-grpc:/etc/containerd-stargz-grpc
    ports:
      - "127.0.0.1:8080:8080"
    command: ["--config", "/etc/containerd-stargz-grpc/config.toml"]
    security_opt:
      - no-new-privileges:true

  containerd:
    image: ghcr.io/containerd/containerd:latest
    container_name: containerd
    restart: always
    volumes:
      - /var/run/containerd:/run/containerd
      - /etc/containerd:/etc/containerd
    command: ["containerd", "--config", "/etc/containerd/config.toml"]

1
2
3
4
5
6
7

# /etc/containerd-stargz-grpc/config.toml
[resolver.host."registry-1.docker.io"]
  mirrors = ["https://registry-1.docker.io"]

[snapshotter]
no_background_task = false
allow_no_verification = true

EROFS Image Creation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Install erofs-utils
sudo apt install erofs-utils -y

# Create an EROFS image from a directory
mkfs.erofs -zlz4hc,9 -C 128 container-rootfs.erofs /path/to/rootfs/

# Mount the EROFS image
sudo mount -t erofs -o loop container-rootfs.erofs /mnt/erofs-rootfs

# Verify filesystem properties
fsck.erofs container-rootfs.erofs

EROFS Compression Benchmarks

EROFS excels in mount speed and random access because it uses fixed-sized output compression, allowing the kernel to seek directly to any compressed block without reading preceding data.

SquashFS

SquashFS has been the dominant read-only filesystem in Linux since 2003. It powers Live CDs, firmware images, Snap packages, and increasingly, container image formats.

Key Features

• Mature ecosystem — supported by virtually every Linux distribution
• Multiple compression algorithms — gzip, lz4, lzo, xz, zstd
• Deduplication — identical files are stored only once
• Wide tooling support — mksquashfs and unsquashfs are universally available

Building SquashFS Container Images

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Install squashfs-tools
sudo apt install squashfs-tools -y

# Create a SquashFS image with zstd compression
mksquashfs /path/to/rootfs/ container-rootfs.squashfs   -comp zstd   -Xcompression-level 15   -no-duplicates   -no-exports   -all-root

# Mount the SquashFS image
sudo mount -t squashfs -o loop container-rootfs.squashfs /mnt/squashfs-rootfs

# Inspect filesystem statistics
unsquashfs -s container-rootfs.squashfs

Docker Compose with SquashFS

SquashFS images are commonly used with runc custom runtimes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

version: "3.8"
services:
  app:
    image: alpine:latest
    volumes:
      - /mnt/squashfs-rootfs:/app-rootfs:ro
    command: ["chroot", "/app-rootfs", "/bin/sh", "-c", "echo 'Running from SquashFS'"]
    read_only: true
    tmpfs:
      - /tmp
      - /run
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETUID
      - SETGID

SquashFS Compression Benchmarks

SquashFS offers the broadest compatibility but uses variable-sized compression blocks, which means random access requires reading through compressed data to locate specific file offsets.

zstd-Compressed OCI Images

Rather than using a dedicated filesystem format, OCI images can use zstd (Zstandard) as the compression algorithm for individual tar layers. This is the approach used by estargz and Docker’s default containerd snapshotter.

Key Features

• OCI-compatible — works with existing container registries and tooling
• Lazy pulling — fetch only the blocks a container actually reads
• Layer-level compression — each layer compressed independently
• Registry-native — no custom filesystem kernel modules required

Building zstd-Compressed Images

1
2
3
4
5
6
7
8

# Build a zstd-compressed image using buildkit
DOCKER_BUILDKIT=1 docker build   --compress   --output type=image,name=myapp:zstd,compression=zstd,force-compression=true   -t myapp:zstd .

# Pull with lazy loading via stargz-snapshotter
nerdctl pull --snapshotter stargz myregistry.com/myapp:zstd

# Verify layer compression
crane manifest myregistry.com/myapp:zstd | jq '.layers[].annotations'

Docker Compose with zstd Layers

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

version: "3.8"
services:
  registry:
    image: registry:2
    container_name: local-registry
    restart: always
    ports:
      - "5000:5000"
    volumes:
      - registry-data:/var/lib/registry
    environment:
      REGISTRY_STORAGE_DELETE_ENABLED: "true"
      REGISTRY_HTTP_HEADERS:
        - "Docker-Content-Distribution: enabled"

  app:
    image: local-registry:5000/myapp:zstd
    restart: always
    read_only: true
    tmpfs:
      - /tmp:size=100M
      - /run:size=50M
    depends_on:
      - registry

volumes:
  registry-data:
    driver: local

zstd Compression Benchmarks

zstd provides the best balance between compression ratio and decompression speed, making it ideal for container images where both storage savings and fast startup matter.

Comparison: EROFS vs SquashFS vs zstd

Deployment Architecture

For a production self-hosted container platform, consider this architecture:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

┌──────────────────────────────────────────────────┐
│              Container Registry                   │
│  (Harbor / Distribution with zstd compression)    │
└──────────────────┬───────────────────────────────┘
                   │ lazy pull (estargz)
                   ▼
┌──────────────────────────────────────────────────┐
│          stargz-snapshotter                       │
│  (EROFS mount → fast random access)               │
└──────────────────┬───────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────┐
│           containerd + runc                       │
│  (OCI runtime with EROFS rootfs)                  │
└──────────────────────────────────────────────────┘

Production Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# /etc/containerd/config.toml
version = 2

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "stargz"

[proxy_plugins]
  [proxy_plugins.stargz]
    type = "snapshot"
    address = "/run/containerd-stargz-grpc/containerd-stargz-grpc.sock"

Choosing the Right Approach

Use EROFS when:

• You need the fastest possible container startup times
• Running on kernel 5.4+ with EROFS support compiled in
• Workloads require high random-read performance (databases, web servers)

Use SquashFS when:

• Maximum compatibility across diverse Linux distributions is required
• Building Live CDs, firmware images, or distribution packages
• Deduplication of identical files across layers is important

Use zstd-compressed OCI images when:

• Working with standard container registries (Docker Hub, Harbor, GHCR)
• Needing lazy pulling for large images
• Wanting the best compression-speed balance without kernel dependencies

Why Self-Host Read-Only Filesystems for Containers?

Running containers with read-only root filesystems is a fundamental security practice. When combined with efficient read-only filesystem formats, you gain both security and performance benefits:

1. Immutability — containers cannot modify their root filesystem, preventing runtime tampering and ensuring reproducible behavior across restarts. Any state changes are isolated to explicitly mounted volumes or tmpfs directories.

2. Reduced attack surface — read-only filesystems eliminate entire classes of vulnerabilities, including log injection attacks, binary replacement exploits, and configuration file modification by compromised processes.

3. Storage efficiency — read-only filesystem formats like EROFS and SquashFS compress data significantly more efficiently than standard tar+gzip layers, reducing registry storage costs and network bandwidth for image distribution.

4. Faster cold starts — in-place decompression and lazy pulling mean containers can begin executing before the entire image is downloaded, critical for auto-scaling workloads and serverless platforms.

5. Audit compliance — read-only root filesystems satisfy requirements in PCI DSS, SOC 2, and CIS benchmarks for container hardening, making compliance audits significantly simpler.

For a complete guide to container security hardening, see our container capabilities management guide. For seccomp profile management, check our comprehensive container security article. For container sandboxing techniques, our Linux sandboxing frameworks comparison covers additional isolation strategies.

FAQ

What is EROFS and how does it differ from SquashFS?

EROFS (Enhanced Read-Only Filesystem) uses fixed-sized output compression, enabling true random access to any file without reading preceding compressed data. SquashFS uses variable-sized blocks, which means random access requires decompressing from the start of a block. EROFS also supports metadata compression and page cache sharing across containers.

Can I use EROFS with Docker?

EROFS is not natively supported by Docker Engine. However, you can use it with containerd and the stargz-snapshotter plugin, which provides EROFS-based image storage with lazy pulling support. Alternatively, use nerdctl (containerd’s Docker-compatible CLI) which has built-in stargz-snapshotter integration.

Which compression algorithm gives the best results for container images?

For container images, zstd at level 3 provides the best balance — it achieves 2.5x compression ratio while decompressing at over 1 GB/s. Level 15 provides marginally better compression (3.1x) but at significantly slower decompression speeds (450 MB/s), which impacts container startup time.

Does SquashFS support lazy pulling like EROFS?

No, SquashFS does not natively support lazy pulling. The entire filesystem image must be available locally before mounting. EROFS, when combined with stargz-snapshotter, can mount images that are only partially downloaded, fetching blocks on demand.

How do I convert an existing Docker image to EROFS format?

Use the estargz toolchain to convert OCI images to EROFS-compatible format:

1
2
3
4
5
6

# Install ctr-remote (part of stargz-snapshotter)
go install github.com/containerd/stargz-snapshotter/cmd/ctr-remote@latest

# Convert and push an EROFS-optimized image
ctr-remote image optimize --oci myapp:latest myregistry.com/myapp:erofs
ctr-remote image push --oci myregistry.com/myapp:erofs

Is zstd compression supported by all container registries?

Yes, zstd compression is part of the OCI Image Specification v1.1 and is supported by Docker Hub, GitHub Container Registry, Harbor, Amazon ECR, and Google Artifact Registry. Most modern registries accept and serve zstd-compressed layers natively.

How much disk space does EROFS save compared to standard OCI layers?

EROFS typically achieves 2.1x to 3.4x compression depending on the algorithm choice (lz4hc, zstd, or lzma). Standard OCI layers with gzip achieve approximately 2.0x. The additional savings come from EROFS’s metadata compression and the absence of tar header overhead.