Capacity Planning

One daemon captures, indexes and rotates. The index MySQL holds the state: size that.

Disk

Per-change cost grows with row width and with binlog_row_image=FULL. Filtering schemas cuts it.

How retention works

Archive compression

zstd by default, snappy and gzip supported. S3 keeps archives per your bucket lifecycle rules. Settings: Retention.

Lag

Change on the source to row in the index: sub-second for typical OLTP, hundreds to low thousands of writes per second. A checkpoint is saved every 10 seconds (--checkpoint); restart resumes from it.

What affects lag

• Bulk writes: large INSERTs, batch UPDATEs
• Wide rows: big TEXT or BLOB columns
• Network distance or an SSH tunnel to the source
• Index disk speed: SSD; very write-heavy sources may need provisioned IOPS

Restart policy: systemd or Docker (restart: unless-stopped in the bundled stack). Binlogs purged past the checkpoint: take a fresh snapshot.

A busy server, measured

One source at 5,200 row changes a second, on 16 vCPU and 128 GB.

Rules of thumb

• Index disk: 1.1 KB per row change, about three times the binlog.
• Index MySQL: give it memory first. With the default 128 MB buffer pool it captured 1,400 changes a second. With 48 GB, 16,000. How to set it: below.
• First copy: 100 million rows in 5 to 7 minutes.
• Parquet copy: 7 to 18 times smaller than the tables in MySQL.
• Each update: 15 to 30 seconds while the busiest table changes under a million rows per update.

Measured with DBTrail 0.98.0 under a sysbench-tpcc load, on a 16,000 IOPS SSD.

Give the index MySQL memory

MySQL's default is a 128 MB buffer pool on any machine. On a busy source that caps capture at about 1,400 row changes a second, and the lag grows without end while the machine sits mostly idle. Nothing stops; the copy just falls further behind.

The bundled stack sizes it through INDEX_BUFFER_POOL in the .env next to docker-compose.yml. A fresh install.sh writes it for you: a quarter of the memory Docker has, whole gigabytes from 1G up, at most 32G. A quarter, not more, because the SQL engine in the web interface takes up to 4 GB and your source MySQL may run on the same machine. Raise it when the machine runs nothing else.

To set or change it on a stack you already have (the installer never rewrites a value you set, and a re-run only prints the line to add):

echo 'INDEX_BUFFER_POOL=8G' >> .env   # whole GB, or e.g. 512M
docker compose up -d                  # restarts the index MySQL; capture resumes from its checkpoint

A docker-compose.yml from before v0.100.0 does not pass the value on. Take the current file first: Upgrading the stack.

bintrail doctor, the startup checklist and the server checks in the web interface warn Index buffer pool when an index on this machine still runs with 128 MB and the machine has about 4 GB or more. DBTrail only reads the setting; it never changes your index. With your own MySQL as the index (INDEX_DSN), set innodb_buffer_pool_size yourself, 50 to 70% of the RAM it has to itself.

Sizing

Past that, one busy server:

Next

• Status
• Retention settings
• Snapshot strategy