fix: env

2026-02-12 15:18:55 +01:00
parent ba32991201
commit 002e05eb89
15 changed files with 373 additions and 106 deletions
--- a/README.md
+++ b/README.md
@@ -7,7 +7,7 @@ A robust MySQL/MariaDB binlog streaming replication service with automatic initi
 ### Quick Install (Go)

 ```bash
-go install git.ma-al.com/goc_marek/replica/cmd/replica@latest
+go install git.ma-al.com/goc_marek/replica@latest
 ```

 ### Build from Source
@@ -18,18 +18,15 @@ git clone https://git.ma-al.com/goc_marek/replica.git
 cd replica

 # Build the service
-go build -o replica ./cmd/replica
+go build -o bin/replica ./main.go

-# Or install globally
-go install ./cmd/replica
+# Run the service
+./bin/replica
 ```

-### Docker
+### Docker Compose

 ```bash
-# Build the image
-docker build -t replica .
-
 # Run with docker-compose
 docker-compose up -d
 ```
@@ -76,14 +73,14 @@ nano .env

 | File | Purpose |
 |------|---------|
-| [`cmd/replica/main.go`](cmd/replica/main.go) | Application entry point and configuration |
-| [`pkg/replica/service.go`](pkg/replica/service.go) | BinlogSyncService - core replication orchestration |
-| [`pkg/replica/handlers.go`](pkg/replica/handlers.go) | EventHandlers - binlog event processing with resilience |
-| [`pkg/replica/initial_transfer.go`](pkg/replica/initial_transfer.go) | InitialTransfer - bulk data transfer management |
-| [`pkg/replica/position.go`](pkg/replica/position.go) | PositionManager - binlog position persistence |
-| [`pkg/replica/sqlbuilder.go`](pkg/replica/sqlbuilder.go) | SQLBuilder - SQL statement generation |
-| [`pkg/replica/config.go`](pkg/replica/config.go) | Configuration types |
-| [`pkg/replica/logging.go`](pkg/replica/logging.go) | Structured logging with Graylog support |
+| [`main.go`](main.go) | Application entry point and configuration |
+| [`replica/service.go`](replica/service.go) | BinlogSyncService - core replication orchestration |
+| [`replica/handlers.go`](replica/handlers.go) | EventHandlers - binlog event processing with resilience |
+| [`replica/initial_transfer.go`](replica/initial_transfer.go) | InitialTransfer - bulk data transfer management |
+| [`replica/position.go`](replica/position.go) | PositionManager - binlog position persistence |
+| [`replica/sqlbuilder.go`](replica/sqlbuilder.go) | SQLBuilder - SQL statement generation |
+| [`replica/config.go`](replica/config.go) | Configuration types |
+| [`replica/logging.go`](replica/logging.go) | Structured logging with Graylog support |

 ### Data Flow

@@ -116,10 +113,10 @@ BinlogSyncService.processEvent()

 ```bash
 # Build the service
-go build -o replica
+go build -o bin/replica ./main.go

 # Run the service
-./replica
+./bin/replica
 ```

 ### Configuration
@@ -213,7 +210,8 @@ GRAYLOG_SOURCE=binlog-sync-prod

 | Variable | Description | Default |
 |----------|-------------|---------|
-| `TRANSFER_BATCH_SIZE` | Rows per transfer chunk | `1000` |
+| `TRANSFER_BATCH_SIZE` | Rows per transfer chunk | `10000` |
+| `TRANSFER_WORKER_COUNT` | Number of parallel transfer workers | `4` |
 | `LOCAL_PROJECT_NAME` | Project name for logging | `naluconcept` |

 ## Resilience Features
@@ -359,7 +357,7 @@ When resync is needed (empty replica or no saved position), the service performs
   - Get table schema (column definitions)
   - Check row count
   - Transfer in chunks using primary key or LIMIT/OFFSET
-5. **Progress Checkpointing**: Save progress to JSON file every 1000 rows
+5. **Progress Checkpointing**: Save progress to JSON file every 10000 rows
 6. **Position Reset**: Clear saved binlog position after successful transfer
 7. **Binlog Streaming**: Start streaming from current position

@@ -375,7 +373,7 @@ SELECT * FROM table WHERE pk >= 1000 AND pk < 2000 ORDER BY pk
 SELECT * FROM table LIMIT 1000 OFFSET 1000
 ```

-**Batch Size:** Configurable (default: 1000 rows per chunk)
+**Batch Size:** Configurable (default: 10000 rows per chunk)

 ### Progress Checkpointing

@@ -400,7 +398,7 @@ If the transfer is interrupted, it resumes from the last checkpoint.
 Transfers can be paused and resumed programmatically:

 ```go
-transfer := NewInitialTransfer(dsn, dsn, 1000, 1)
+transfer := NewInitialTransfer(dsn, dsn, 10000, 4)

 // Pause during transfer
 transfer.Pause()
@@ -445,8 +443,8 @@ type EventHandlers struct {
 type InitialTransfer struct {
    primaryDB      *sql.DB
    secondaryDB    *sql.DB
-    batchSize      int               // Default: 1000
-    workerCount    int               // Default: 1
+    batchSize      int               // Default: 10000
+    workerCount    int               // Default: 4
    excludedDBs    map[string]bool
    checkpointFile string
    progress       TransferProgress
@@ -575,15 +573,15 @@ SELECT COUNT(*) FROM your_table;

 ## Performance Considerations

-1. **Batch Size**: Start with 1000, adjust based on table size and memory
+1. **Batch Size**: Start with 10000, adjust based on table size and memory
 2. **Connection Pooling**: `SetMaxOpenConns(25)` for moderate load
-3. **Worker Count**: Currently single-threaded (multi-worker planned)
+3. **Worker Count**: Default: 4 workers for parallel processing
 4. **Schema Caching**: Table schemas cached in memory (auto-updated on drift)
 5. **Index Usage**: Chunked transfers require indexed primary key

 ## Limitations

- **Single-threaded**: One worker processes events sequentially
+- **Multi-threaded**: Multiple workers process events in parallel (configurable via TRANSFER_WORKER_COUNT)
 - **Position-based**: No GTID support yet (position-based only)
 - **Integer PKs**: Chunking requires integer primary key for efficiency
 - **No Conflict Resolution**: Concurrent writes not handled
@@ -618,18 +616,20 @@ SELECT COUNT(*) FROM your_table;

 ```
 replica/
-├── cmd/
-│   └── replica/
-│       └── main.go                    # Entry point
-├── pkg/
-│   └── replica/
-│       ├── service.go                 # Replication orchestration
-│       ├── handlers.go                # Event processing
-│       ├── initial_transfer.go        # Bulk data transfer
-│       ├── position.go                # Position persistence
-│       ├── sqlbuilder.go              # SQL generation
-│       ├── config.go                  # Configuration types
-│       └── logging.go                 # Structured logging
+├── main.go                            # Entry point
+├── replica/
+│   ├── service.go                     # Replication orchestration
+│   ├── handlers.go                    # Event processing
+│   ├── initial_transfer.go            # Bulk data transfer
+│   ├── position.go                    # Position persistence
+│   ├── sqlbuilder.go                  # SQL generation
+│   ├── config.go                      # Configuration types
+│   └── logging.go                     # Structured logging
+├── examples/
+│   └── binlog-listener/
+│       └── main.go                    # Example binlog listener
+├── bin/
+│   └── replica                        # Compiled binary
 ├── example.env                        # Environment template
 ├── .env                               # Environment (gitignored)
 ├── docker-compose.yml                 # Local development