zfs/readme.md

ZFS Snapshot Manager

A distributed ZFS snapshot management system with S3-compatible storage support. This project provides client, server, and restore tools for managing ZFS snapshots across multiple machines.

Features

• S3 Storage Support: Store snapshots in any S3-compatible storage (AWS S3, MinIO, Backblaze B2, Wasabi, DigitalOcean Spaces)
• Local ZFS Storage: Option to use local ZFS datasets for maximum performance
• Multi-client Architecture: Support for multiple clients with isolated storage and per-client quotas
• Automatic Compression: Gzip compression for reduced storage costs
• Snapshot Rotation: Automatic cleanup of old snapshots based on quota
• Server-Managed Rotation Policies: Centralized control of client rotation policies - clients must use server-configured retention settings
• API Key Authentication: Secure client-server communication

Project Structure

zfs/
├── cmd/
│   ├── zfs-server/     # Server executable
│   ├── zfs-client/     # Client executable
│   └── zfs-restore/    # Restore tool executable
├── internal/
│   ├── server/         # Server package (config, storage, HTTP handlers)
│   ├── client/         # Client package (snapshot creation, upload)
│   └── restore/        # Restore package (download, restore operations)
├── go.mod
├── go.sum
├── .env                # Configuration file
└── readme.md

Installation

Using Go Install

# Install server
go install git.ma-al.com/goc_marek/zfs/cmd/zfs-server@latest

# Install client
go install git.ma-al.com/goc_marek/zfs/cmd/zfs-client@latest

# Install restore tool
go install git.ma-al.com/goc_marek/zfs/cmd/zfs-restore@latest

Build from Source

# Clone the repository
git clone https://git.ma-al.com/goc_marek/zfs.git
cd zfs

# Build all binaries
go build -o bin/zfs-server ./cmd/zfs-server
go build -o bin/zfs-client ./cmd/zfs-client
go build -o bin/zfs-restore ./cmd/zfs-restore

Configuration

Server Configuration

Create a .env file in the working directory:

# S3 Configuration
S3_ENABLED=true
S3_ENDPOINT=s3.amazonaws.com
S3_ACCESS_KEY=YOUR_ACCESS_KEY
S3_SECRET_KEY=YOUR_SECRET_KEY
S3_BUCKET=zfs-snapshots
S3_USE_SSL=true

# Local ZFS fallback
ZFS_BASE_DATASET=backup

# Server settings
CONFIG_FILE=clients.json
METADATA_FILE=metadata.json
PORT=8080

Client Configuration

CLIENT_ID=client1
API_KEY=secret123
SERVER_URL=http://backup-server:8080
LOCAL_DATASET=tank/data
COMPRESS=true
STORAGE_TYPE=s3

Restore Tool Configuration

CLIENT_ID=client1
API_KEY=secret123
SERVER_URL=http://backup-server:8080

Usage

Server

# Start the backup server
zfs-server

# The server listens on port 8080 by default
# Endpoints:
#   POST /upload           - Request upload authorization
#   POST /upload-stream/   - Stream snapshot data
#   GET  /status           - Check client status
#   POST /rotate           - Rotate old snapshots
#   GET  /download         - Download a snapshot
#   GET  /rotation-policy  - Get client rotation policy
#   GET  /health           - Health check

Client Commands

The zfs-client tool provides the following commands for managing ZFS snapshots:

backup

Creates a snapshot and sends it to the server. Automatically uses incremental backup if a bookmark exists.

zfs-client backup

backup-full

Forces a full backup (no incremental). Use for the initial backup or when you want to resend the complete dataset.

zfs-client backup-full

backup-incremental

Creates an incremental backup from the last bookmark. Requires an existing bookmark from a previous full backup.

zfs-client backup-incremental

snapshot <type>

Creates a typed snapshot (hourly, daily, weekly, monthly) with automatic rotation. The rotation policy is fetched from the server if configured.

zfs-client snapshot hourly
zfs-client snapshot daily
zfs-client snapshot weekly
zfs-client snapshot monthly

rotate

Rotates local snapshots based on the retention policy. If the server has a rotation policy configured, it will be used; otherwise, default values apply.

zfs-client rotate

rotate-remote

Requests the server to rotate (delete old) remote snapshots to free up storage quota.

zfs-client rotate-remote

status

Displays the current backup status including storage usage, quota, and snapshot count from the server.

zfs-client status

bookmarks

Lists ZFS bookmarks on the local system. Bookmarks are used as reference points for incremental backups.

zfs-client bookmarks

help

Shows the help message with all available commands and options.

zfs-client help

Restore Tool Commands

The zfs-restore tool provides commands for listing and restoring snapshots from the backup server:

list

Lists all available snapshots for the configured client from the server.

zfs-restore list

Output example:

#  Snapshot ID                    Timestamp           Size
1  client1/tank_data_2024-02-13   2024-02-13 14:30    1.2 GB
2  client1/tank_data_2024-02-12   2024-02-12 14:30    1.1 GB

restore <number> <dataset>

Restores a snapshot by its list number to a specified ZFS dataset.

zfs-restore restore 1 tank/restored

Options:

• --force or -f - Overwrite existing dataset if it exists

zfs-restore restore 1 tank/restored --force

latest <dataset>

Restores the most recent snapshot to a specified dataset.

zfs-restore latest tank/restored

save <number> <filename>

Downloads a snapshot and saves it to a local file without restoring.

zfs-restore save 1 backup.zfs.gz

mount <dataset> <mountpoint>

Mounts a restored ZFS dataset to a specified directory for file access.

zfs-restore mount tank/restored /mnt/restore

S3 Provider Configuration

AWS S3

S3_ENDPOINT=s3.amazonaws.com
S3_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE
S3_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
S3_BUCKET=my-zfs-backups
S3_USE_SSL=true

MinIO (Self-Hosted)

S3_ENDPOINT=minio.example.com:9000
S3_ACCESS_KEY=minioadmin
S3_SECRET_KEY=minioadmin
S3_BUCKET=zfs-snapshots
S3_USE_SSL=false

Backblaze B2

S3_ENDPOINT=s3.us-west-000.backblazeb2.com
S3_ACCESS_KEY=your_key_id
S3_SECRET_KEY=your_application_key
S3_BUCKET=zfs-backups
S3_USE_SSL=true

Wasabi

S3_ENDPOINT=s3.wasabisys.com
S3_ACCESS_KEY=your_access_key
S3_SECRET_KEY=your_secret_key
S3_BUCKET=zfs-backups
S3_USE_SSL=true

Client Configuration File

The server maintains a clients.json file with client configurations:

[
  {
    "client_id": "client1",
    "api_key": "hashed_key",
    "max_size_bytes": 107374182400,
    "dataset": "backup/client1",
    "enabled": true,
    "storage_type": "s3",
    "rotation_policy": {
      "keep_hourly": 24,
      "keep_daily": 7,
      "keep_weekly": 4,
      "keep_monthly": 12
    }
  }
]

Server-Managed Rotation Policy

When rotation_policy is configured for a client in clients.json, the client must use this policy and cannot override it. This enables centralized control of snapshot retention policies:

• Server-Managed: If rotation_policy is set, the client fetches the policy from the server and applies it
• Client-Autonomous: If no rotation_policy is set, the client uses its default policy

The rotation policy fields are:

• keep_hourly: Number of hourly snapshots to keep (default: 24)
• keep_daily: Number of daily snapshots to keep (default: 7)
• keep_weekly: Number of weekly snapshots to keep (default: 4)
• keep_monthly: Number of monthly snapshots to keep (default: 12)

API Endpoint

The server exposes a /rotation-policy endpoint for clients to fetch their configured policy:

GET /rotation-policy?client_id=client1&api_key=secret123

Response:

{
  "success": true,
  "message": "Server-managed rotation policy",
  "rotation_policy": {
    "keep_hourly": 24,
    "keep_daily": 7,
    "keep_weekly": 4,
    "keep_monthly": 12
  },
  "server_managed": true
}

Architecture

┌─────────────┐    ZFS send      ┌──────────────────┐
│  Client 1   │───────┬─────────▶│  Backup Server   │
│ (S3 mode)   │       │          │                  │
└─────────────┘       │          │  ┌────────────┐  │
                      │          │  │ S3 Backend │  │
┌─────────────┐       │  HTTP    │  └─────┬──────┘  │
│  Client 2   │───────┤  Stream  │        │         │
│ (S3 mode)   │       │          │        ▼         │
└─────────────┘       │          │  ┌────────────┐  │
                      │          │  │   MinIO    │  │
┌─────────────┐       │          │  │     or     │  │
│  Client 3   │───────┘          │  │   AWS S3   │  │
│ (Local ZFS) │─────────────────▶│  └────────────┘  │
└─────────────┘   ZFS recv       │                  │
                                 │  ┌────────────┐  │
                                 │  │  Local ZFS │  │
                                 │  │   Backend  │  │
                                 │  └────────────┘  │
                                 └──────────────────┘

Storage Format

Snapshots are stored in S3 with the following naming convention:

s3://bucket/client1/tank_data_2024-02-13_14:30:00.zfs.gz
                ^         ^           ^
              client   dataset    timestamp

Security

• API keys are hashed using SHA-256
• S3 bucket policies can restrict access to backup server only
• Server-side encryption available in S3
• Client-side encryption possible via custom compression pipeline

Monitoring

Health Check

curl http://localhost:8080/health

Server Logs

# SystemD
journalctl -u zfs-server -f

# Docker
docker logs -f zfs-server

Development

Project Layout

• cmd/ - Main applications (entry points)
• internal/ - Private application code
  • server/ - Server logic, HTTP handlers, storage backends
  • client/ - Client logic for creating and uploading snapshots
  • restore/ - Restore logic for downloading and restoring snapshots

Building

# Build all
go build ./...

# Run tests
go test ./...

# Lint
go vet ./...

License

MIT License