zfs/readme.md

ZFS Snapshot Manager

A distributed ZFS snapshot management system with S3-compatible storage support. This project provides client, server, and restore tools for managing ZFS snapshots across multiple machines.

Features

• S3 Storage Support: Store snapshots in any S3-compatible storage (AWS S3, MinIO, Backblaze B2, Wasabi, DigitalOcean Spaces)
• Local ZFS Storage: Option to use local ZFS datasets for maximum performance
• Multi-client Architecture: Support for multiple clients with isolated storage and per-client quotas
• Automatic Compression: Gzip compression for reduced storage costs
• Snapshot Rotation: Automatic cleanup of old snapshots based on quota
• Server-Managed Rotation Policies: Centralized control of client rotation policies - clients must use server-configured retention settings
• API Key Authentication: Secure client-server communication

Project Structure

zfs/
├── cmd/
│   ├── zfs-server/     # Server executable
│   ├── zfs-client/     # Client executable
│   └── zfs-restore/    # Restore tool executable
├── internal/
│   ├── server/         # Server package (config, storage, HTTP handlers)
│   ├── client/         # Client package (snapshot creation, upload)
│   └── restore/        # Restore package (download, restore operations)
├── go.mod
├── go.sum
├── .env                # Configuration file
└── readme.md

Installation

Using Go Install

# Install server
go install git.ma-al.com/goc_marek/zfs/cmd/zfs-server@latest

# Install client
go install git.ma-al.com/goc_marek/zfs/cmd/zfs-client@latest

# Install restore tool
go install git.ma-al.com/goc_marek/zfs/cmd/zfs-restore@latest

Build from Source

# Clone the repository
git clone https://git.ma-al.com/goc_marek/zfs.git
cd zfs

# Build all binaries
go build -o bin/zfs-server ./cmd/zfs-server
go build -o bin/zfs-client ./cmd/zfs-client
go build -o bin/zfs-restore ./cmd/zfs-restore

Configuration

Server Configuration

Create a .env file in the working directory:

# S3 Configuration
S3_ENABLED=true
S3_ENDPOINT=s3.amazonaws.com
S3_ACCESS_KEY=YOUR_ACCESS_KEY
S3_SECRET_KEY=YOUR_SECRET_KEY
S3_BUCKET=zfs-snapshots
S3_USE_SSL=true

# Local ZFS fallback
ZFS_BASE_DATASET=backup

# Database Configuration (SQLite)
DATABASE_PATH=zfs-backup.db

# Server settings
PORT=8080

Note

: All client configuration and snapshot metadata is stored in a SQLite database (zfs-backup.db by default). The server automatically creates a default client (client1 with API key secret123) if no clients exist.

Client Configuration

CLIENT_ID=client1
API_KEY=secret123
SERVER_URL=http://backup-server:8080
LOCAL_DATASET=tank/data
COMPRESS=true

Important

:

• The API_KEY in the client .env file must be the raw (unhashed) key. The server stores the SHA-256 hash in the database.
• Storage type is determined by the server, not the client. The server decides whether to use S3 or local ZFS storage based on its configuration.

Restore Tool Configuration

CLIENT_ID=client1
API_KEY=secret123
SERVER_URL=http://backup-server:8080

Usage

Server

# Start the backup server
zfs-server

# The server listens on port 8080 by default
# Endpoints:
#   POST /upload           - Request upload authorization
#   POST /upload-stream/   - Stream snapshot data
#   GET  /status           - Check client status
#   POST /rotate           - Rotate old snapshots
#   GET  /download         - Download a snapshot
#   GET  /rotation-policy  - Get client rotation policy
#   GET  /health           - Health check

Client Commands

The zfs-client tool provides the following commands for managing ZFS snapshots:

backup

Creates a snapshot and sends it to the server. Automatically uses incremental backup if a bookmark exists.

zfs-client backup

backup-full

Forces a full backup (no incremental). Use for the initial backup or when you want to resend the complete dataset.

zfs-client backup-full

backup-incremental

Creates an incremental backup from the last bookmark. Requires an existing bookmark from a previous full backup.

zfs-client backup-incremental

snapshot <type>

Creates a typed snapshot (hourly, daily, weekly, monthly) with automatic rotation. The rotation policy is fetched from the server if configured.

zfs-client snapshot hourly
zfs-client snapshot daily
zfs-client snapshot weekly
zfs-client snapshot monthly

rotate

Rotates local snapshots based on the retention policy. If the server has a rotation policy configured, it will be used; otherwise, default values apply.

zfs-client rotate

rotate-remote

Requests the server to rotate (delete old) remote snapshots to free up storage quota.

zfs-client rotate-remote

status

Displays the current backup status including storage usage, quota, and snapshot count from the server.

zfs-client status

bookmarks

Lists ZFS bookmarks on the local system. Bookmarks are used as reference points for incremental backups.

zfs-client bookmarks

help

Shows the help message with all available commands and options.

zfs-client help

Restore Tool Commands

The zfs-restore tool provides commands for listing and restoring snapshots from the backup server:

list

Lists all available snapshots for the configured client from the server.

zfs-restore list

Output example:

#  Snapshot ID                    Timestamp           Size
1  client1/tank_data_2024-02-13   2024-02-13 14:30    1.2 GB
2  client1/tank_data_2024-02-12   2024-02-12 14:30    1.1 GB

restore <number> <dataset>

Restores a snapshot by its list number to a specified ZFS dataset.

zfs-restore restore 1 tank/restored

Options:

• --force or -f - Overwrite existing dataset if it exists

zfs-restore restore 1 tank/restored --force

latest <dataset>

Restores the most recent snapshot to a specified dataset.

zfs-restore latest tank/restored

save <number> <filename>

Downloads a snapshot and saves it to a local file without restoring.

zfs-restore save 1 backup.zfs.gz

mount <dataset> <mountpoint>

Mounts a restored ZFS dataset to a specified directory for file access.

zfs-restore mount tank/restored /mnt/restore

S3 Provider Configuration

AWS S3

S3_ENDPOINT=s3.amazonaws.com
S3_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE
S3_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
S3_BUCKET=my-zfs-backups
S3_USE_SSL=true

MinIO (Self-Hosted)

S3_ENDPOINT=minio.example.com:9000
S3_ACCESS_KEY=minioadmin
S3_SECRET_KEY=minioadmin
S3_BUCKET=zfs-snapshots
S3_USE_SSL=false

Setting Up MinIO Locally

Option A: Using Docker (Recommended)

# Create a directory for MinIO data
mkdir -p ~/minio-data

# Run MinIO container
docker run -d \
  --name minio \
  -p 9000:9000 \
  -p 9001:9001 \
  -v ~/minio-data:/data \
  -e MINIO_ROOT_USER=minioadmin \
  -e MINIO_ROOT_PASSWORD=minioadmin \
  minio/minio server /data --console-address ":9001"

Option B: Using Binary

# Download MinIO
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mv minio /usr/local/bin/

# Create data directory
mkdir -p ~/minio-data

# Start MinIO
MINIO_ROOT_USER=minioadmin MINIO_ROOT_PASSWORD=minioadmin \
  minio server ~/minio-data --console-address ":9001"

Create the Bucket

After starting MinIO, create the bucket using the MinIO Client (mc) or web console:

# Install MinIO Client (mc)
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/local/bin/

# Configure alias to local MinIO
mc alias set local http://localhost:9000 minioadmin minioadmin

# Create bucket
mc mb local/zfs

# Verify bucket was created
mc ls local

Alternatively, access the MinIO Web Console at http://localhost:9001 and create the bucket through the UI (login: minioadmin / minioadmin).

Backblaze B2

S3_ENDPOINT=s3.us-west-000.backblazeb2.com
S3_ACCESS_KEY=your_key_id
S3_SECRET_KEY=your_application_key
S3_BUCKET=zfs-backups
S3_USE_SSL=true

Wasabi

S3_ENDPOINT=s3.wasabisys.com
S3_ACCESS_KEY=your_access_key
S3_SECRET_KEY=your_secret_key
S3_BUCKET=zfs-backups
S3_USE_SSL=true

Database Storage

The server uses SQLite to store all configuration and metadata in a single database file (zfs-backup.db by default). This includes:

• Admin users: Authentication credentials for the admin panel
• Client configurations: Authentication, quotas, storage type, rotation policies
• Snapshot metadata: Timestamps, sizes, storage keys, incremental relationships

Database Schema

The database contains four main tables:

admins table:

• id - Unique identifier
• username - Admin username (unique)
• password_hash - SHA-256 hashed password
• role - Admin role (default: "admin")
• created_at, updated_at - Timestamps

admin_sessions table:

• id - Unique identifier
• admin_id - Foreign key to admins table
• token - Session token
• expires_at - Session expiration time

clients table:

• client_id - Unique identifier
• api_key - SHA-256 hashed API key
• max_size_bytes - Storage quota
• dataset - Target dataset for local ZFS storage
• enabled - Client status
• storage_type - "s3" or "local"
• keep_hourly, keep_daily, keep_weekly, keep_monthly - Rotation policy

snapshots table:

• client_id - Owner of the snapshot
• snapshot_id - Unique identifier
• timestamp - When the snapshot was taken
• size_bytes - Snapshot size
• storage_key - Location in storage
• storage_type - Where it's stored
• compressed, incremental, base_snapshot - Snapshot properties

Server-Managed Rotation Policy

When a rotation policy is configured for a client in the database, the client must use this policy and cannot override it. This enables centralized control of snapshot retention policies:

• Server-Managed: If rotation policy is set, the client fetches the policy from the server and applies it
• Client-Autonomous: If no rotation policy is set, the client uses its default policy

The rotation policy fields are:

• keep_hourly: Number of hourly snapshots to keep (default: 24)
• keep_daily: Number of daily snapshots to keep (default: 7)
• keep_weekly: Number of weekly snapshots to keep (default: 4)
• keep_monthly: Number of monthly snapshots to keep (default: 12)

API Endpoint

The server exposes a /rotation-policy endpoint for clients to fetch their configured policy:

GET /rotation-policy?client_id=client1&api_key=secret123

Response:

{
  "success": true,
  "message": "Server-managed rotation policy",
  "rotation_policy": {
    "keep_hourly": 24,
    "keep_daily": 7,
    "keep_weekly": 4,
    "keep_monthly": 12
  },
  "server_managed": true
}

Admin Panel

The server includes a web-based admin panel for managing clients, snapshots, and admin users. Access it at http://localhost:8080/admin/.

Default Admin Credentials

When the server starts for the first time, it creates a default admin user:

• Username: admin
• Password: admin123

Important

: Change the default password immediately after first login!

Admin Panel Features

• Dashboard: View statistics (client count, total snapshots, storage usage)
• Client Management:
  • Create, view, and delete clients
  • Configure storage type (S3 or local ZFS)
  • Set quotas and rotation policies
  • Enable/disable clients
• Snapshot Management:
  • View all snapshots across all clients
  • Filter by client
  • Delete individual snapshots
• Admin User Management:
  • Create additional admin users
  • Delete admin accounts

Admin API Endpoints

All admin endpoints require authentication via session cookie.

Endpoint	Method	Description
/admin/login	POST	Login with username/password
/admin/logout	POST	Logout current session
/admin/check	GET	Check authentication status
/admin/clients	GET	List all clients with usage stats
/admin/client	GET	Get specific client details
/admin/client/create	POST	Create new client
/admin/client/update	PUT	Update client configuration
/admin/client/delete	POST	Delete client and all snapshots
/admin/snapshots	GET	List all snapshots
/admin/snapshot/delete	POST	Delete specific snapshot
/admin/stats	GET	Get server statistics
/admin/admins	GET	List all admin users
/admin/admin/create	POST	Create new admin user
/admin/admin/delete	POST	Delete admin user

Creating a Client via API

# Login first (saves session cookie)
curl -c cookies.txt -X POST http://localhost:8080/admin/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"admin123"}'

# Create a new client
curl -b cookies.txt -X POST http://localhost:8080/admin/client/create \
  -H "Content-Type: application/json" \
  -d '{
    "client_id": "myclient",
    "api_key": "secretkey123",
    "storage_type": "s3",
    "dataset": "backup/myclient",
    "max_size_bytes": 107374182400,
    "enabled": true,
    "rotation_policy": {
      "keep_hourly": 24,
      "keep_daily": 7,
      "keep_weekly": 4,
      "keep_monthly": 12
    }
  }'

Architecture

┌─────────────┐    ZFS send      ┌──────────────────┐
│  Client 1   │───────┬─────────▶│  Backup Server   │
│ (S3 mode)   │       │          │                  │
└─────────────┘       │          │  ┌────────────┐  │
                      │          │  │ S3 Backend │  │
┌─────────────┐       │  HTTP    │  └─────┬──────┘  │
│  Client 2   │───────┤  Stream  │        │         │
│ (S3 mode)   │       │          │        ▼         │
└─────────────┘       │          │  ┌────────────┐  │
                      │          │  │   MinIO    │  │
┌─────────────┐       │          │  │     or     │  │
│  Client 3   │───────┘          │  │   AWS S3   │  │
│ (Local ZFS) │─────────────────▶│  └────────────┘  │
└─────────────┘   ZFS recv       │                  │
                                 │  ┌────────────┐  │
                                 │  │  Local ZFS │  │
                                 │  │   Backend  │  │
                                 │  └────────────┘  │
                                 └──────────────────┘

Storage Format

Snapshots are stored in S3 with the following naming convention:

s3://bucket/client1/tank_data_2024-02-13_14:30:00.zfs.gz
                ^         ^           ^
              client   dataset    timestamp

Security

• API keys are hashed using SHA-256
• S3 bucket policies can restrict access to backup server only
• Server-side encryption available in S3
• Client-side encryption possible via custom compression pipeline

Monitoring

Health Check

curl http://localhost:8080/health

Server Logs

# SystemD
journalctl -u zfs-server -f

# Docker
docker logs -f zfs-server

Development

Project Layout

• cmd/ - Main applications (entry points)
• internal/ - Private application code
  • server/ - Server logic, HTTP handlers, storage backends
  • client/ - Client logic for creating and uploading snapshots
  • restore/ - Restore logic for downloading and restoring snapshots

Building

# Build all
go build ./...

# Run tests
go test ./...

# Lint
go vet ./...

License

MIT License