Fleet docs v0.2.0-rc.448

5. More machines: nodes

A node is a machine that runs turns for the server. It runs fleet node. The node asks the server for work. The server never opens a connection to a node.

  • Make a join token on the server with fleet node token. It is good for one join and 1 hour.

  • max_running in node.yaml limits the turns that run at the same time. The default is 1.

  • admit in node.yaml sets when the box is too busy to take a new turn (#336). A busy node claims no new turn; the work stays queued and says why ("node-a busy: load 1.9 per core"). The load is the whole box's, not only fleet's. A live chat of an existing conversation is never refused, so a busy box does not block your chat. Autoscale counts refused work as waiting, so overflow goes to Spot first and then the cloud.

    admit:
      max_load_per_core: 0.8   # 1-minute load average divided by cores (default 0.8)
      max_mem_pressure: 10     # /proc/pressure/memory "some" avg10, percent (default 10)
      min_mem_available: 4GiB  # MemAvailable (default 4GiB; bytes, KiB, MiB, GiB)
      min_disk_free: 2GiB      # free disk under the node's state and projects dirs (default 2GiB)
      max_disk_use: 85         # percent of that disk in use, as /machines shows it (default 85, #459)
    

    A box that is busy takes work again only when it is clearly under the limits: load and pressure under 80% of the limit, available memory and free disk over 125% of the floor, disk use 5 points under max_disk_use (80%). So it does not flap. The /machines page shows a "busy, not taking new work" badge, and the Work page card shows the reason. In GET /v1/machines it is the card's sample.busy. The node's journal has a line when it turns busy ("node: busy, takes no new work") and when it takes work again (#512).

  • housekeeping in node.yaml keeps the disk clear (#459). Every 10 minutes, and every 2 minutes while the disk is over high_disk_use, the node:

    • removes go-build cache entries not used for gocache_age (Go itself keeps them 5 days). Go moves an entry's time on use only once an hour, so gocache_age is at least 2h (#487); under disk pressure it trims in steps (#512): while the disk is still at 70% or more it also removes entries unused for 1 h, at 80% those unused for 15 min, at 85% those unused for 5 min. "Used" is the later of the entry's atime and mtime, and an entry that a process has open stays. A removed entry costs only a rebuild. The report names the last step ("go-build 752 files 151 MB, to 15m unused at 86%");
    • with docker: removes dangling images and unnamed volumes that no container uses, both older than gocache_age (a named volume is never removed);
    • removes the own clones (<dir>/work/<agent id>) of agents that are done, stopped or failed, after it pushes their uncommitted changes and unpushed commits to wip/<agent id>, and each other local branch and stash entry with commits origin does not have to wip/<agent id>-<branch> and wip/<agent id>-stash-<n> (#483). If a push fails, the clone stays;
    • over high_disk_use, also removes the worktrees of ended agents whose branch is not merged (the 30-minute sweep keeps those), after it pushes their unpushed work to wip/<name>.

    A turn that runs on the node, or a process in a directory, keeps that directory. The machine card on /machines shows the last report ("housekeeping: disk 92% to 61%: go-build ..."). A disk that stays at 90% or more for 10 minutes anyway opens an incident issue (type:incident, one per machine, in the fleet alert repo) that closes by itself once the disk is under 90% again. CI runners and build helpers never count.

    housekeeping:
      every: 10m               # or off (default 10m)
      high_disk_use: 70        # percent (default 70)
      gocache: /home/fleet/.cache/go-build  # default <home>/.cache/go-build; off: no trim
      gocache_age: 3h          # default 3h; at least 2h
      docker: /usr/bin/docker  # default docker on PATH, if any; off: no prune
    
  • kinds in node.yaml limits the agent kinds that the node takes.

  • claude in node.yaml pins the Claude Code version that the node runs, for example claude: {version: "2.1.321"}. The server's claude.version is the fleet's one version. See Claude Code version.

  • The working directory is on the node. The registry, the ledger and the blobs stay on the server.

  • A node that is silent for more than 30 s is lost. Its agents are lost. After 15 minutes, the server gives them to another node, which continues the conversation. See a machine joins and dies. A rented node whose server the provider deleted or took back does not wait 15 minutes: its agents go to another node once it is lost.

  • With nodes, the server box can stop running turns: --no-local-workers.

  • Per-turn caps (#349). Admission only stops new claims. Each turn also runs in its own systemd scope with a CPU and memory cap, so one worker cannot crush the box. Worker turns run at nice 10; a live chat process gets less and is not niced. The process view shows the cap of the turn.

    turn_cpu_percent: 400        # CPUQuota; 400 = 4 cores (default 400)
    turn_memory_high: 8G         # MemoryHigh: throttles, does not kill (default 8G)
    turn_nice: 10                # worker turns only (default 10)
    turn_live_cpu_percent: 200   # a live chat process (default 200)
    turn_live_memory_high: 4G    # (default 4G)
    

    A key left out takes its default; 0 turns that cap off. The caps need the user's systemd manager (a user bus): on a node run by a service, loginctl enable-linger <user>. The node logs a warning at start when it has no bus and the turns run uncapped.

  • Workers on the live process (#376). On a node whose probe of Claude Code passed (a node with a claude: pin), a worker turn runs as one live process, the same process a chat uses. fleet send and board posts reach the worker while it works: the message goes into the running process as Message from <sender>, sent while you work: ..., and the ledger marks it delivered in that turn. The process ends as soon as its work is done and no message waits. The turn deadline, the budget, the worker caps and nice stay as they are. A node with no probe, a failed probe or a driver with no live mode runs one process per turn, as before. The node config sets it per kind:

    live:
      worker: false   # one process per turn for this kind
      elastic: true   # live even on a node with no probe
    

    An elastic (rented) node cannot probe, because it holds no Claude login and takes no chats. Its node.yaml, written by fleet autoscale, sets each of its worker kinds live: true, so workers there take fleet send mid-turn. A Spot node is also pinned (claude: {version, path: /usr/local/bin/claude}) to the Claude Code version that its install put on the box. A node that booted before this keeps its old node.yaml and runs one process per turn.

The This box panel shows one section for each node that reports.

Next: 6. A team: people and projects. More than one person uses the fleet.

The Machines page

/machines shows every box: the control server, each node, each rented (elastic) server and each CI runner and build helper (#321, #327).

  • A rented server is on the page from the moment fleet autoscale rents it: Renting a server, Starting up, Joining the fleet, then its live numbers once its node joins, with working or idle, deleted in N m. After the controller deletes it the card says Gone and leaves the page 2 minutes later. An idle server is deleted by how its provider bills: a Spot server 5 minutes after its last turn, a cloud server at minute 55 of its current billing hour (the hour starts when it is created). A server with a running or queued turn for it is never deleted; fleet autoscale run --idle D sets one flat idle time for every provider (#339). Its node runs the controller's own fleet binary (the deployed release), copied in before the node starts, so an older snapshot still reports.

  • fleet autoscale rents a server only when work waits that the servers it holds (idle, booting, or still being created) cannot take, and never in a tick when a provider failed to list its servers. It keeps one row per machine it rents or tries to rent (#424), and writes the row before it asks the cloud, so a restart counts what it was renting. The rows are the rented_machines table of the server's database: fleet autoscale run runs only inside fleet serve and fails without them (#659). A row keeps the idle clock, the "every machine full" clock, why the machine was rented and why it went, its rate and bid, and a provider's cooldown after a failed create, across restarts; a gone row stays as history for 400 days. A failed create is cleaned up: the hub peer at once, the server when a listing shows it; the next tick tries the next provider. A failed tick is logged, never an exit (#350).

  • Placement prefers a joined elastic server with a free slot, since it is paid for already. A persistent node leaves an unpinned turn that such a server can take for up to 15 seconds, then takes it itself. fleet autoscale never rents while one of its servers is idle or booting: it rents the next one only when every server it holds is busy (ruling 2026-11-11, #350).

  • Rented machines are packed like rented mules (#371). A rented node takes turns until its 1-minute load reaches 2 per vCPU, so load 16 on 8 vCPU and 32 on 16 vCPU (--rent-load-per-cpu; its node.yaml gets admit: {max_load_per_core: 2}, not the 0.8 per core of a persistent box; --rent-load N sets one whole-box limit instead, admit: {max_load: N}), with max_running: 64 (--per-node) and no per-turn CPU or memory cap; memory pressure and available memory still stop it. A waiting turn goes to the most loaded rented machine that still takes work: a less loaded one leaves it for up to 15 seconds. A second machine is rented only when work waits and every rented machine has been full (its 5-minute load at its limit, its node refusing work for memory or disk, or every slot taken; a 1-minute spike does not count, #559) for --rent-after (default 10 minutes, 5 live), never while one is under the load, idle, booting or joining. Idle machines go by their billing rule, so the busiest stays. The warm spare (--spare) is the last choice for a turn, since it is the least loaded. A spare that took a turn is replaced only under the same rule: when every rented machine has been full for 10 minutes (ruling 2026-11-11). A rented machine counts as taking work for 20 seconds after each claim, because a far machine claims about every 11.5 seconds. For its first 20 seconds after a start, the control server gives a rented node no unpinned turn while another node that reports has not yet claimed from it, because that node may be fuller. The card of a rented machine says packing: load5 12.3 of 32 or full since 7 min: load5 33.1 of 32, and the line under the title says why the next machine is or is not rented: next machine: not renting: elastic-1 full for 7 min of 10 min. The controller sends its --per-node to the control server with each report, and the server gives it to every rented node as its capacity at the next heartbeat. A machine rented under an older --per-node takes the new one with no reprovision or node restart (#371 a); its card shows the new max_running.

  • The persistent nodes have a placement order too (#351). In serve.yaml:

    placement:
      order: [node-b, node-a]   # hostnames or node ids, first preferred
    

    The order is: a joined elastic server with a free slot, then the listed nodes in turn, then the nodes not listed. A node leaves an unpinned turn for a polling node earlier in the order that takes the turn's kind and has a free slot; it takes the turn itself at once when that node is full or busy, and after 15 seconds at most in any case. A pinned turn ignores the order. Without placement, the first node to claim takes the turn.

  • One scheduler (#351): fleet serve can run the autoscale controller itself, so the process that places the turns also rents and deletes the machines. In serve.yaml:

    autoscale:
      run: true
      token_file: /etc/fleet/autoscale.token   # the token it calls this server with
      url: https://fleet.example.com           # "": fleet_url
      args: [--providers, "spot,cloud", --hub, root@hub.example.com, --max-spend, "2"]
    

    args are the flags of fleet autoscale run; the cloud tokens (FLEET_CLOUD_TOKEN, SPOT_REFRESH_TOKEN, GITHUB_TOKEN or FORGEJO_TOKEN) come from fleet serve's environment. The module is the only controller (#659): there is no standalone unit and no state file. An old --state in args is accepted and ignored.

  • Spend ceiling (#351): fleet autoscale rents no machine that would take the rented machines together past $2 an hour (--max-spend; a negative value turns it off). EUR prices count at 1.2 USD (--eur-usd). An offer with no price is not rented under a ceiling. The log line names the ceiling when it stops a rent. A Spot machine counts at its bid, the most it can bill, not at the market price (#434).

  • Spot bid cap (#434): a Spot bid is never over $0.50 an hour (--spot-max-bid) nor over 10 times the class's market price or min bid, whichever is higher (--spot-max-bid-x), on Spot's bid grid; 0 turns a cap off. A class whose bid (its 80th percentile, or its listed bid) would pass a cap is skipped, and the scheduler log says why: autoscale: spot skips ch.vs1.2xlarge-syd in aus-syd-1: its bid 1.950 USD/h (p80 1.921, …) is over the cap 0.350. --spot-bid-floor raises a bid up to the cap, never past it.

  • A CI runner or build helper runs fleet machine-report --role ci (or --role builder) with FLEET_URL and FLEET_TOKEN_FILE. Its token is the machine token: on the control server, fleet machine-report token --dir /var/lib/fleet prints it. That token can send machine reports and nothing else; a new server key revokes it. A runner that stops reporting leaves the page 2 minutes later.

  • The line under the title counts them: N machines, M elastic, K CI.

  • Each card shows who provides the box and what it costs an hour: a provider badge, then €0.04/h, or own hardware. The line under the title adds what the boxes with a rate cost now: rented now: €0.10/h + $0.19/h (one sum for each currency).

    • A fixed box sets it in its config, node.yaml or serve.yaml:

      machine:
        provider: home          # a label: home, or the name of a cloud
        own_hardware: true      # or: rate_per_hour: 0.0164 and rate_currency: EUR
      
    • A CI runner or build helper: fleet machine-report --provider P --rate R --currency C, or FLEET_MACHINE_PROVIDER, FLEET_MACHINE_RATE and FLEET_MACHINE_CURRENCY.

    • A rented server: fleet autoscale reports the provider and the price it pays; it wins over what the box says.