Fleet docs v0.2.0-rc.448

Backup and restore

A timer on the server box makes a backup each day. A restore puts the backup on a new box. The nodes then find the new server.

Each run writes a new dated copy, for example copies/2026-11-08T002808Z-nightly/, and never writes over another one (#298). <kind> is nightly from fleet-backup-nightly.service, the unit the timer starts, and manual from systemctl start fleet-backup or a run by hand. SHA256SUMS is written last, after the copy on the target matched the local files, so a copy without it is incomplete. Pruning comes after that check and never touches blobs. It keeps:

  • the newest copy of each of the newest 14 days that have one; a nightly is preferred, so an on-demand run never takes the place of that day's nightly (BACKUP_KEEP_DAILY);
  • of those day copies, the newest in each of the newest 8 ISO weeks (BACKUP_KEEP_WEEKLY);
  • every manual copy of the last 7 days (BACKUP_KEEP_MANUAL_DAYS), every copy younger than 24 hours, and the new copy;
  • incomplete copies from today. Older incomplete copies are removed.

A copy is about 1.5 MB (2026-11-08: fleet.db 1.45 MB, keys.tar.age 10 KB), so about 30 copies take under 50 MB, next to 48 MB of blobs. fleet-backup list names the copies on the target.

Restore on a new box

In the backup Not in the backup
fleet.db: the registry and the ledger table ledger.jsonl: a copy of the ledger table, not the source
server.key, owner.token, age.key, creds.age.key, creds/, secrets/, encrypted with age turns/: the files of the running turns
blobs/: briefs and conversations /etc/fleet/serve.yaml: keep it in your deploy repo
  • The scripts are in fleet/deploy/backup/. install.sh installs them as fleet-backup and fleet-restore, with fleet-backup.service (a backup now), fleet-backup-nightly.service and the timer that starts it. There is no fleet backup command.
  • The timer is OnCalendar=daily, RandomizedDelaySec=30min, Persistent=true. If the box was off at the time, the backup runs at the next start.
  • fleet ledger replay <file> --db <new.db> can make a new registry from a ledger file. It is not part of the restore steps. The restore copies the database. --until <seq> stops after that event: an agent that a later fleet prune removed is in that registry.
  • A replay of a ledger goes past the rows that the ledger does not carry: a message's body, and a profile from before profile events (sqlite.ReplayRow). The README lists every known difference between a replayed registry and the database.
  • Restore an older copy: BACKUP_COPY=2026-11-07T002640Z-nightly fleet-restore. The blobs are the full set (blobs are never removed), so a restored database finds every blob it names.
  • Check a backup without touching the server: restore it on another machine, start a check server on a loopback port, and compare with the live ledger replayed --until the backup's last seq (Check a backup). First done on 2026-11-08 with the 00:28 UTC nightly (#295): counts, fleet costs and both replays matched.
  • The cleanvm layout restore does a full backup, erases the box, and restores (fleet/test/cleanvm/layouts/restore.sh).
  • Steps: Back up and restore.