Fleet docs v0.2.0-rc.448

A release goes out and is reverted

A release is a git tag vX.Y.Z. CI builds the binaries from the tag. An operator installs the release on the server first, then on each node. A revert installs the earlier tag again.

A revert

Step Fact Code
The gate fleet.yml runs three jobs at once on each branch push that touches fleet/ or docs/: lint (gofmt, vet, golangci-lint, docs and neutral checks), test (go test -race as concurrent processes, fleet/scripts/ci-test.sh) and build. The gate passes only when all three pass. .forgejo/workflows/fleet.yml
The merge train Repos on merge.mode: train (swarm-lab until #576). A green branch that main fast-forwards to goes on main at once. A branch behind main is rebased and gated again on train/<sha> (a conflict where both sides only added lines is resolved, main's lines first; with merger.resolver on in serve.yaml, a same-line conflict goes to one cheap model call, resolve.go; any other conflict is sent back), unless the rebase changed no file (its tree is the tree of a green head): then it goes on main with no train, and main's own push runs the gate. A Forgejo status webhook makes the train read its gate at once; with none it reads every 15 s. fleet/internal/merger/merger.go
The trunk guard swarm-lab is trunk-based (#571, #576): workers land on main with fleet land after the local gate; the guard in fleet serve pushes check/<sha12> for main's newest head, and fleet.yml runs only on check/**, so main itself runs no gate. A red is rerun once on check/<sha12>-r, then bisected by landing and the culprit reverted. A tag push runs no gate. fleet/internal/merger/trunk.go, docs/fleet/trunk.md
Where the time went Forgejo's API gives no job logs, so the lint, test and release jobs post a commit status time/<job> (always success; the merge train reads only the fleet / <job> contexts). Examples: setup 18s, make -j 26s; setup 10s, run 70s; ci-test: 6 processes, 50s (building the split packages 15s before), slowest: internal/cli#3 50s. Read it with GET /api/v1/repos/<owner>/<repo>/commits/<sha>/status. .forgejo/scripts/time-note.sh
The build A tag vX.Y.Z or vX.Y.Z-* starts the release. It publishes the binary that the build job made for that commit, with the tag written into its version slot (from the runner's /var/cache/fleet-ci/<sha>, else the actions cache fleet-bin-<sha>), and builds only when it finds neither. A tag with a suffix is a pre-release. .forgejo/workflows/fleet-release.yml, .forgejo/scripts/stamp-version.sh
The test shards The suite runs as three jobs, test, test-2 and test-3 (CI_TEST_SHARD=1/3, 2/3 and 3/3 of fleet/scripts/ci-test.sh: every process goes, heaviest first, to the lighter shard). The three run on self-hosted on every branch: the full suite on train/** and main, tests by impact on t/**. The x86-64-v3 runners are off (2026-11-12). All three are gate jobs. A heavy package's tests are found in its source and its test binary is built once per shard (go test -race -c), then run in groups; the group weights are fleet/scripts/ci-test-times.tsv. .forgejo/workflows/fleet.yml, .fleet/ship.yaml
The upload route The release talks to Forgejo at the repo or org variable DIRECT_FORGEJO_URL (Forgejo refuses variable names that start with FORGEJO_; set to http://127.0.0.1:3000, a tunnel to Forgejo on every runner), else the runner's environment FORGEJO_DIRECT_URL, when that address answers /api/v1/version within 5 s, else at the server URL, which goes through Cloudflare (34-42 s for the binary, #354). The runners' own setup provides the direct route. time/release names the route it used. .forgejo/workflows/fleet-release.yml
The image The deploy builds the fleet-agent image for kubejob pods fleet/deploy/agent-image/build.sh
The zstd copy The release also attaches fleet-linux-amd64.zst (zstd -3, about 20 MB for the 69 MB binary) and lists it in SHA256SUMS, when the runner has zstd. The deploy can download it and send it to its hosts as it is (#354). install.sh uses the plain binary. .forgejo/workflows/fleet-release.yml
The install install.sh compares the sha256 with SHA256SUMS. If they are not the same, it installs nothing. fleet/deploy/install.sh
The restart A turn on the server box runs under fleet turn-shim (KillMode=process), so a restart does not stop it fleet/internal/backend/process/detach.go
The stop On SIGTERM fleet serve stops its listeners at once and ends every open request's context 1 s later: the page's event streams, a node's 25 s live pull and MCP streams never end by themselves, and each one held systemctl restart fleet for the whole 10 s shutdown timeout on every deploy (#354: 10-11 s on rc.151-159). fleet/internal/cli/httpstop.go
The node's join A deploy restarts the server and the nodes at the same time. While the server does not answer at all (connection refused, no route), fleet node joins again every second for up to 2 minutes, then exits as before. Before #354 it exited at once and systemd started it again 5 s later, more than once. fleet/internal/node/keep.go

Where push to live goes (#354, measured)

Each step, with its source: Forgejo commit statuses (fleet / <job>, time/<job>), the live: line that the deploy writes on the release, and the deploy's stamped log.

Step Time What sets it
Push → the gate's jobs start 0-90 s free runners
The gate (lint, test, test-2, test-3, build at once) 2m-2m50s on warm runners, up to 6m40s when they are busy the slowest shard
Ready → main 10-15 s on a fast-forward; a whole second gate when it goes by a train the merge train
Main → tag 8-16 s (44 s once) the deploy's 10 s timer and its gate check
Tag → release done 34-54 s (0-35 s of it waiting for a runner) checkout 5-9 s, build 7-10 s, uploads 16-23 s
Release → live, before #354 round 6 67-73 s upload to the server over ssh 40-47 s, server stop 10 s
Release → live, rc.161 and rc.162 10-12 s the server fetches the release itself (about 1 s); stop 10 s
  • Not built: a version check between the server and a node. The rule "the same release on every box" is a procedure for people. See the fleet-release skill (.claude/skills/fleet-release/SKILL.md).
  • Not built: fleet upgrade or a rollback command. A revert is install.sh --version with the earlier tag, or a revert commit and a new tag.
  • Guard on the tag (#670): the release workflow's first step (.forgejo/scripts/require-green.sh) refuses a tag whose commit has no success status for each of lint, test, test-2, test-3 and build (fleet / <job> (push)). govulncheck is not in the gate: vuln.yml runs it daily and opens an incident issue.
  • Warning: the database migrations go forward only, and there are no down migrations (fleet/internal/registry/sqlite/sqlite.go). An earlier binary starts on a newer schema with no check. Before you revert across a release that adds a migration, restore a backup from before the upgrade. See backup and restore.
  • Before an upgrade on Kubernetes, read the release notes for new permissions. Since v0.1.5, the node needs create on pods/exec.