goc_marek/zfs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fb9bb2fc82eed7c12e1daa51b1de686f0a9ebd14
zfs/readme.md
Marek Goc fb9bb2fc82 fix
2026-02-13 18:38:55 +01:00

491 lines
12 KiB
Markdown
Raw Blame History

# ZFS Snapshot Manager
A distributed ZFS snapshot management system with S3-compatible storage support. This project provides client, server, and restore tools for managing ZFS snapshots across multiple machines.
## Features
- **S3 Storage Support**: Store snapshots in any S3-compatible storage (AWS S3, MinIO, Backblaze B2, Wasabi, DigitalOcean Spaces)
- **Local ZFS Storage**: Option to use local ZFS datasets for maximum performance
- **Multi-client Architecture**: Support for multiple clients with isolated storage and per-client quotas
- **Automatic Compression**: Gzip compression for reduced storage costs
- **Snapshot Rotation**: Automatic cleanup of old snapshots based on quota
- **Server-Managed Rotation Policies**: Centralized control of client rotation policies - clients must use server-configured retention settings
- **API Key Authentication**: Secure client-server communication
## Project Structure
```
zfs/
├── cmd/
│ ├── zfs-server/ # Server executable
│ ├── zfs-client/ # Client executable
│ └── zfs-restore/ # Restore tool executable
├── internal/
│ ├── server/ # Server package (config, storage, HTTP handlers)
│ ├── client/ # Client package (snapshot creation, upload)
│ └── restore/ # Restore package (download, restore operations)
├── go.mod
├── go.sum
├── .env # Configuration file
└── readme.md
```
## Installation
### Using Go Install
```bash
# Install server
go install git.ma-al.com/goc_marek/zfs/cmd/zfs-server@latest
# Install client
go install git.ma-al.com/goc_marek/zfs/cmd/zfs-client@latest
# Install restore tool
go install git.ma-al.com/goc_marek/zfs/cmd/zfs-restore@latest
```
### Build from Source
```bash
# Clone the repository
git clone https://git.ma-al.com/goc_marek/zfs.git
cd zfs
# Build all binaries
go build -o bin/zfs-server ./cmd/zfs-server
go build -o bin/zfs-client ./cmd/zfs-client
go build -o bin/zfs-restore ./cmd/zfs-restore
```
## Configuration
### Server Configuration
Create a `.env` file in the working directory:
```env
# S3 Configuration
S3_ENABLED=true
S3_ENDPOINT=s3.amazonaws.com
S3_ACCESS_KEY=YOUR_ACCESS_KEY
S3_SECRET_KEY=YOUR_SECRET_KEY
S3_BUCKET=zfs-snapshots
S3_USE_SSL=true
# Local ZFS fallback
ZFS_BASE_DATASET=backup
# Server settings
CONFIG_FILE=clients.json
METADATA_FILE=metadata.json
PORT=8080
```
### Client Configuration
```env
CLIENT_ID=client1
API_KEY=secret123
SERVER_URL=http://backup-server:8080
LOCAL_DATASET=tank/data
COMPRESS=true
STORAGE_TYPE=s3
```
> **Important**: The `API_KEY` in the client `.env` file must be the **raw (unhashed)** key. The server stores the SHA-256 hash in `clients.json`, and the client sends the raw key which the server then hashes for comparison. For example, if `clients.json` has `api_key: "fcf730b6d95236ecd3c9fc2d92d7b6b2bb061514961aec041d6c7a7192f592e4"` (hash of "secret123"), the client `.env` should have `API_KEY=secret123`.
### Restore Tool Configuration
```env
CLIENT_ID=client1
API_KEY=secret123
SERVER_URL=http://backup-server:8080
```
## Usage
### Server
```bash
# Start the backup server
zfs-server
# The server listens on port 8080 by default
# Endpoints:
# POST /upload - Request upload authorization
# POST /upload-stream/ - Stream snapshot data
# GET /status - Check client status
# POST /rotate - Rotate old snapshots
# GET /download - Download a snapshot
# GET /rotation-policy - Get client rotation policy
# GET /health - Health check
```
### Client Commands
The `zfs-client` tool provides the following commands for managing ZFS snapshots:
#### `backup`
Creates a snapshot and sends it to the server. Automatically uses incremental backup if a bookmark exists.
```bash
zfs-client backup
```
#### `backup-full`
Forces a full backup (no incremental). Use for the initial backup or when you want to resend the complete dataset.
```bash
zfs-client backup-full
```
#### `backup-incremental`
Creates an incremental backup from the last bookmark. Requires an existing bookmark from a previous full backup.
```bash
zfs-client backup-incremental
```
#### `snapshot <type>`
Creates a typed snapshot (hourly, daily, weekly, monthly) with automatic rotation. The rotation policy is fetched from the server if configured.
```bash
zfs-client snapshot hourly
zfs-client snapshot daily
zfs-client snapshot weekly
zfs-client snapshot monthly
```
#### `rotate`
Rotates local snapshots based on the retention policy. If the server has a rotation policy configured, it will be used; otherwise, default values apply.
```bash
zfs-client rotate
```
#### `rotate-remote`
Requests the server to rotate (delete old) remote snapshots to free up storage quota.
```bash
zfs-client rotate-remote
```
#### `status`
Displays the current backup status including storage usage, quota, and snapshot count from the server.
```bash
zfs-client status
```
#### `bookmarks`
Lists ZFS bookmarks on the local system. Bookmarks are used as reference points for incremental backups.
```bash
zfs-client bookmarks
```
#### `help`
Shows the help message with all available commands and options.
```bash
zfs-client help
```
### Restore Tool Commands
The `zfs-restore` tool provides commands for listing and restoring snapshots from the backup server:
#### `list`
Lists all available snapshots for the configured client from the server.
```bash
zfs-restore list
```
Output example:
```
# Snapshot ID Timestamp Size
1 client1/tank_data_2024-02-13 2024-02-13 14:30 1.2 GB
2 client1/tank_data_2024-02-12 2024-02-12 14:30 1.1 GB
```
#### `restore <number> <dataset>`
Restores a snapshot by its list number to a specified ZFS dataset.
```bash
zfs-restore restore 1 tank/restored
```
Options:
- `--force` or `-f` - Overwrite existing dataset if it exists
```bash
zfs-restore restore 1 tank/restored --force
```
#### `latest <dataset>`
Restores the most recent snapshot to a specified dataset.
```bash
zfs-restore latest tank/restored
```
#### `save <number> <filename>`
Downloads a snapshot and saves it to a local file without restoring.
```bash
zfs-restore save 1 backup.zfs.gz
```
#### `mount <dataset> <mountpoint>`
Mounts a restored ZFS dataset to a specified directory for file access.
```bash
zfs-restore mount tank/restored /mnt/restore
```
## S3 Provider Configuration
### AWS S3
```env
S3_ENDPOINT=s3.amazonaws.com
S3_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE
S3_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
S3_BUCKET=my-zfs-backups
S3_USE_SSL=true
```
### MinIO (Self-Hosted)
```env
S3_ENDPOINT=minio.example.com:9000
S3_ACCESS_KEY=minioadmin
S3_SECRET_KEY=minioadmin
S3_BUCKET=zfs-snapshots
S3_USE_SSL=false
```
#### Setting Up MinIO Locally
**Option A: Using Docker (Recommended)**
```bash
# Create a directory for MinIO data
mkdir -p ~/minio-data
# Run MinIO container
docker run -d \
--name minio \
-p 9000:9000 \
-p 9001:9001 \
-v ~/minio-data:/data \
-e MINIO_ROOT_USER=minioadmin \
-e MINIO_ROOT_PASSWORD=minioadmin \
minio/minio server /data --console-address ":9001"
```
**Option B: Using Binary**
```bash
# Download MinIO
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mv minio /usr/local/bin/
# Create data directory
mkdir -p ~/minio-data
# Start MinIO
MINIO_ROOT_USER=minioadmin MINIO_ROOT_PASSWORD=minioadmin \
minio server ~/minio-data --console-address ":9001"
```
**Create the Bucket**
After starting MinIO, create the bucket using the MinIO Client (mc) or web console:
```bash
# Install MinIO Client (mc)
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/local/bin/
# Configure alias to local MinIO
mc alias set local http://localhost:9000 minioadmin minioadmin
# Create bucket
mc mb local/zfs
# Verify bucket was created
mc ls local
```
Alternatively, access the MinIO Web Console at http://localhost:9001 and create the bucket through the UI (login: `minioadmin` / `minioadmin`).
### Backblaze B2
```env
S3_ENDPOINT=s3.us-west-000.backblazeb2.com
S3_ACCESS_KEY=your_key_id
S3_SECRET_KEY=your_application_key
S3_BUCKET=zfs-backups
S3_USE_SSL=true
```
### Wasabi
```env
S3_ENDPOINT=s3.wasabisys.com
S3_ACCESS_KEY=your_access_key
S3_SECRET_KEY=your_secret_key
S3_BUCKET=zfs-backups
S3_USE_SSL=true
```
## Client Configuration File
The server maintains a `clients.json` file with client configurations:
```json
[
{
"client_id": "client1",
"api_key": "hashed_key",
"max_size_bytes": 107374182400,
"dataset": "backup/client1",
"enabled": true,
"storage_type": "s3",
"rotation_policy": {
"keep_hourly": 24,
"keep_daily": 7,
"keep_weekly": 4,
"keep_monthly": 12
}
}
]
```
### Server-Managed Rotation Policy
When `rotation_policy` is configured for a client in `clients.json`, the client **must** use this policy and cannot override it. This enables centralized control of snapshot retention policies:
- **Server-Managed**: If `rotation_policy` is set, the client fetches the policy from the server and applies it
- **Client-Autonomous**: If no `rotation_policy` is set, the client uses its default policy
The rotation policy fields are:
- `keep_hourly`: Number of hourly snapshots to keep (default: 24)
- `keep_daily`: Number of daily snapshots to keep (default: 7)
- `keep_weekly`: Number of weekly snapshots to keep (default: 4)
- `keep_monthly`: Number of monthly snapshots to keep (default: 12)
#### API Endpoint
The server exposes a `/rotation-policy` endpoint for clients to fetch their configured policy:
```bash
GET /rotation-policy?client_id=client1&api_key=secret123
```
Response:
```json
{
"success": true,
"message": "Server-managed rotation policy",
"rotation_policy": {
"keep_hourly": 24,
"keep_daily": 7,
"keep_weekly": 4,
"keep_monthly": 12
},
"server_managed": true
}
```
## Architecture
```
┌─────────────┐ ZFS send ┌──────────────────┐
│ Client 1 │───────┬─────────▶│ Backup Server │
│ (S3 mode) │ │ │ │
└─────────────┘ │ │ ┌────────────┐ │
│ │ │ S3 Backend │ │
┌─────────────┐ │ HTTP │ └─────┬──────┘ │
│ Client 2 │───────┤ Stream │ │ │
│ (S3 mode) │ │ │ ▼ │
└─────────────┘ │ │ ┌────────────┐ │
│ │ │ MinIO │ │
┌─────────────┐ │ │ │ or │ │
│ Client 3 │───────┘ │ │ AWS S3 │ │
│ (Local ZFS) │─────────────────▶│ └────────────┘ │
└─────────────┘ ZFS recv │ │
│ ┌────────────┐ │
│ │ Local ZFS │ │
│ │ Backend │ │
│ └────────────┘ │
└──────────────────┘
```
## Storage Format
Snapshots are stored in S3 with the following naming convention:
```
s3://bucket/client1/tank_data_2024-02-13_14:30:00.zfs.gz
^ ^ ^
client dataset timestamp
```
## Security
- API keys are hashed using SHA-256
- S3 bucket policies can restrict access to backup server only
- Server-side encryption available in S3
- Client-side encryption possible via custom compression pipeline
## Monitoring
### Health Check
```bash
curl http://localhost:8080/health
```
### Server Logs
```bash
# SystemD
journalctl -u zfs-server -f
# Docker
docker logs -f zfs-server
```
## Development
### Project Layout
- `cmd/` - Main applications (entry points)
- `internal/` - Private application code
- `server/` - Server logic, HTTP handlers, storage backends
- `client/` - Client logic for creating and uploading snapshots
- `restore/` - Restore logic for downloading and restoring snapshots
### Building
```bash
# Build all
go build ./...
# Run tests
go test ./...
# Lint
go vet ./...
```
## License
MIT License
Reference in New Issue View Git Blame Copy Permalink