5. More machines: nodes
A node is a machine that runs turns for the server. It runs
fleet node. The node asks the server for work. The server never opens a connection to a node.
-
Make a join token on the server with
fleet node token. It is good for one join and 1 hour. -
max_runninginnode.yamllimits the turns that run at the same time. The default is 1. -
admitinnode.yamlsets when the box is too busy to take a new turn (#336). A busy node claims no new turn; the work stays queued and says why ("node-a busy: load 1.9 per core"). The load is the whole box's, not only fleet's. A live chat of an existing conversation is never refused, so a busy box does not block your chat. Autoscale counts refused work as waiting, so overflow goes to Spot first and then the cloud.admit: max_load_per_core: 0.8 # 1-minute load average divided by cores (default 0.8) max_mem_pressure: 10 # /proc/pressure/memory "some" avg10, percent (default 10) min_mem_available: 4GiB # MemAvailable (default 4GiB; bytes, KiB, MiB, GiB) min_disk_free: 2GiB # free disk under the node's state and projects dirs (default 2GiB) max_disk_use: 85 # percent of that disk in use, as /machines shows it (default 85, #459)A box that is busy takes work again only when it is clearly under the limits: load and pressure under 80% of the limit, available memory and free disk over 125% of the floor, disk use 5 points under
max_disk_use(80%). So it does not flap. The /machines page shows a "busy, not taking new work" badge, and the Work page card shows the reason. InGET /v1/machinesit is the card'ssample.busy. The node's journal has a line when it turns busy ("node: busy, takes no new work") and when it takes work again (#512). -
housekeepinginnode.yamlkeeps the disk clear (#459). Every 10 minutes, and every 2 minutes while the disk is overhigh_disk_use, the node:- removes go-build cache entries not used for
gocache_age(Go itself keeps them 5 days). Go moves an entry's time on use only once an hour, sogocache_ageis at least 2h (#487); under disk pressure it trims in steps (#512): while the disk is still at 70% or more it also removes entries unused for 1 h, at 80% those unused for 15 min, at 85% those unused for 5 min. "Used" is the later of the entry's atime and mtime, and an entry that a process has open stays. A removed entry costs only a rebuild. The report names the last step ("go-build 752 files 151 MB, to 15m unused at 86%"); - with docker: removes dangling images and unnamed volumes that no container uses, both older
than
gocache_age(a named volume is never removed); - removes the own clones (
<dir>/work/<agent id>) of agents that are done, stopped or failed, after it pushes their uncommitted changes and unpushed commits towip/<agent id>, and each other local branch and stash entry with commits origin does not have towip/<agent id>-<branch>andwip/<agent id>-stash-<n>(#483). If a push fails, the clone stays; - over
high_disk_use, also removes the worktrees of ended agents whose branch is not merged (the 30-minute sweep keeps those), after it pushes their unpushed work towip/<name>.
A turn that runs on the node, or a process in a directory, keeps that directory. The machine card on /machines shows the last report ("housekeeping: disk 92% to 61%: go-build ..."). A disk that stays at 90% or more for 10 minutes anyway opens an incident issue (
type:incident, one per machine, in the fleet alert repo) that closes by itself once the disk is under 90% again. CI runners and build helpers never count.housekeeping: every: 10m # or off (default 10m) high_disk_use: 70 # percent (default 70) gocache: /home/fleet/.cache/go-build # default <home>/.cache/go-build; off: no trim gocache_age: 3h # default 3h; at least 2h docker: /usr/bin/docker # default docker on PATH, if any; off: no prune - removes go-build cache entries not used for
-
kindsinnode.yamllimits the agent kinds that the node takes. -
claudeinnode.yamlpins the Claude Code version that the node runs, for exampleclaude: {version: "2.1.321"}. The server'sclaude.versionis the fleet's one version. See Claude Code version. -
The working directory is on the node. The registry, the ledger and the blobs stay on the server.
-
A node that is silent for more than 30 s is lost. Its agents are
lost. After 15 minutes, the server gives them to another node, which continues the conversation. See a machine joins and dies. A rented node whose server the provider deleted or took back does not wait 15 minutes: its agents go to another node once it is lost. -
With nodes, the server box can stop running turns:
--no-local-workers. -
Per-turn caps (#349). Admission only stops new claims. Each turn also runs in its own systemd scope with a CPU and memory cap, so one worker cannot crush the box. Worker turns run at
nice10; a live chat process gets less and is not niced. The process view shows the cap of the turn.turn_cpu_percent: 400 # CPUQuota; 400 = 4 cores (default 400) turn_memory_high: 8G # MemoryHigh: throttles, does not kill (default 8G) turn_nice: 10 # worker turns only (default 10) turn_live_cpu_percent: 200 # a live chat process (default 200) turn_live_memory_high: 4G # (default 4G)A key left out takes its default;
0turns that cap off. The caps need the user's systemd manager (a user bus): on a node run by a service,loginctl enable-linger <user>. The node logs a warning at start when it has no bus and the turns run uncapped. -
Workers on the live process (#376). On a node whose probe of Claude Code passed (a node with a
claude:pin), a worker turn runs as one live process, the same process a chat uses.fleet sendand board posts reach the worker while it works: the message goes into the running process asMessage from <sender>, sent while you work: ..., and the ledger marks it delivered in that turn. The process ends as soon as its work is done and no message waits. The turn deadline, the budget, the worker caps andnicestay as they are. A node with no probe, a failed probe or a driver with no live mode runs one process per turn, as before. The node config sets it per kind:live: worker: false # one process per turn for this kind elastic: true # live even on a node with no probeAn elastic (rented) node cannot probe, because it holds no Claude login and takes no chats. Its node.yaml, written by
fleet autoscale, sets each of its worker kindslive: true, so workers there takefleet sendmid-turn. A Spot node is also pinned (claude: {version, path: /usr/local/bin/claude}) to the Claude Code version that its install put on the box. A node that booted before this keeps its old node.yaml and runs one process per turn.
The This box panel shows one section for each node that reports.
Next: 6. A team: people and projects. More than one person uses the fleet.
The Machines page
/machines shows every box: the control server, each node, each rented (elastic) server and
each CI runner and build helper (#321, #327).
-
A rented server is on the page from the moment
fleet autoscalerents it: Renting a server, Starting up, Joining the fleet, then its live numbers once its node joins, with working or idle, deleted in N m. After the controller deletes it the card says Gone and leaves the page 2 minutes later. An idle server is deleted by how its provider bills: a Spot server 5 minutes after its last turn, a cloud server at minute 55 of its current billing hour (the hour starts when it is created). A server with a running or queued turn for it is never deleted;fleet autoscale run --idle Dsets one flat idle time for every provider (#339). Its node runs the controller's own fleet binary (the deployed release), copied in before the node starts, so an older snapshot still reports. -
fleet autoscalerents a server only when work waits that the servers it holds (idle, booting, or still being created) cannot take, and never in a tick when a provider failed to list its servers. It keeps one row per machine it rents or tries to rent (#424), and writes the row before it asks the cloud, so a restart counts what it was renting. The rows are therented_machinestable of the server's database:fleet autoscale runruns only inside fleet serve and fails without them (#659). A row keeps the idle clock, the "every machine full" clock, why the machine was rented and why it went, its rate and bid, and a provider's cooldown after a failed create, across restarts; a gone row stays as history for 400 days. A failed create is cleaned up: the hub peer at once, the server when a listing shows it; the next tick tries the next provider. A failed tick is logged, never an exit (#350). -
Placement prefers a joined elastic server with a free slot, since it is paid for already. A persistent node leaves an unpinned turn that such a server can take for up to 15 seconds, then takes it itself.
fleet autoscalenever rents while one of its servers is idle or booting: it rents the next one only when every server it holds is busy (ruling 2026-11-11, #350). -
Rented machines are packed like rented mules (#371). A rented node takes turns until its 1-minute load reaches 2 per vCPU, so load 16 on 8 vCPU and 32 on 16 vCPU (
--rent-load-per-cpu; itsnode.yamlgetsadmit: {max_load_per_core: 2}, not the 0.8 per core of a persistent box;--rent-load Nsets one whole-box limit instead,admit: {max_load: N}), withmax_running: 64(--per-node) and no per-turn CPU or memory cap; memory pressure and available memory still stop it. A waiting turn goes to the most loaded rented machine that still takes work: a less loaded one leaves it for up to 15 seconds. A second machine is rented only when work waits and every rented machine has been full (its 5-minute load at its limit, its node refusing work for memory or disk, or every slot taken; a 1-minute spike does not count, #559) for--rent-after(default 10 minutes, 5 live), never while one is under the load, idle, booting or joining. Idle machines go by their billing rule, so the busiest stays. The warm spare (--spare) is the last choice for a turn, since it is the least loaded. A spare that took a turn is replaced only under the same rule: when every rented machine has been full for 10 minutes (ruling 2026-11-11). A rented machine counts as taking work for 20 seconds after each claim, because a far machine claims about every 11.5 seconds. For its first 20 seconds after a start, the control server gives a rented node no unpinned turn while another node that reports has not yet claimed from it, because that node may be fuller. The card of a rented machine says packing: load5 12.3 of 32 or full since 7 min: load5 33.1 of 32, and the line under the title says why the next machine is or is not rented: next machine: not renting: elastic-1 full for 7 min of 10 min. The controller sends its--per-nodeto the control server with each report, and the server gives it to every rented node as its capacity at the next heartbeat. A machine rented under an older--per-nodetakes the new one with no reprovision or node restart (#371 a); its card shows the newmax_running. -
The persistent nodes have a placement order too (#351). In
serve.yaml:placement: order: [node-b, node-a] # hostnames or node ids, first preferredThe order is: a joined elastic server with a free slot, then the listed nodes in turn, then the nodes not listed. A node leaves an unpinned turn for a polling node earlier in the order that takes the turn's kind and has a free slot; it takes the turn itself at once when that node is full or busy, and after 15 seconds at most in any case. A pinned turn ignores the order. Without
placement, the first node to claim takes the turn. -
One scheduler (#351):
fleet servecan run the autoscale controller itself, so the process that places the turns also rents and deletes the machines. Inserve.yaml:autoscale: run: true token_file: /etc/fleet/autoscale.token # the token it calls this server with url: https://fleet.example.com # "": fleet_url args: [--providers, "spot,cloud", --hub, root@hub.example.com, --max-spend, "2"]argsare the flags offleet autoscale run; the cloud tokens (FLEET_CLOUD_TOKEN,SPOT_REFRESH_TOKEN,GITHUB_TOKENorFORGEJO_TOKEN) come from fleet serve's environment. The module is the only controller (#659): there is no standalone unit and no state file. An old--stateinargsis accepted and ignored. -
Spend ceiling (#351):
fleet autoscalerents no machine that would take the rented machines together past $2 an hour (--max-spend; a negative value turns it off). EUR prices count at 1.2 USD (--eur-usd). An offer with no price is not rented under a ceiling. The log line names the ceiling when it stops a rent. A Spot machine counts at its bid, the most it can bill, not at the market price (#434). -
Spot bid cap (#434): a Spot bid is never over $0.50 an hour (
--spot-max-bid) nor over 10 times the class's market price or min bid, whichever is higher (--spot-max-bid-x), on Spot's bid grid; 0 turns a cap off. A class whose bid (its 80th percentile, or its listed bid) would pass a cap is skipped, and the scheduler log says why: autoscale: spot skips ch.vs1.2xlarge-syd in aus-syd-1: its bid 1.950 USD/h (p80 1.921, …) is over the cap 0.350.--spot-bid-floorraises a bid up to the cap, never past it. -
A CI runner or build helper runs
fleet machine-report --role ci(or--role builder) withFLEET_URLandFLEET_TOKEN_FILE. Its token is the machine token: on the control server,fleet machine-report token --dir /var/lib/fleetprints it. That token can send machine reports and nothing else; a new server key revokes it. A runner that stops reporting leaves the page 2 minutes later. -
The line under the title counts them: N machines, M elastic, K CI.
-
Each card shows who provides the box and what it costs an hour: a provider badge, then €0.04/h, or own hardware. The line under the title adds what the boxes with a rate cost now: rented now: €0.10/h + $0.19/h (one sum for each currency).
-
A fixed box sets it in its config,
node.yamlorserve.yaml:machine: provider: home # a label: home, or the name of a cloud own_hardware: true # or: rate_per_hour: 0.0164 and rate_currency: EUR -
A CI runner or build helper:
fleet machine-report --provider P --rate R --currency C, orFLEET_MACHINE_PROVIDER,FLEET_MACHINE_RATEandFLEET_MACHINE_CURRENCY. -
A rented server:
fleet autoscalereports the provider and the price it pays; it wins over what the box says.
-