Fleet docs v0.2.0-rc.448

Capacity: how many agents run at once

Fleet runs as many dispatched workers as the machines can carry (#559). The limit is machine load over time, not a fixed count of workers.

The dispatcher

Each pass (every minute), for each node that reports:

  • Room is a 15-minute load per vCPU under the threshold, more than 20% of memory available, a free turn slot (max_running), and a node that takes work (its card shows no "busy").
  • The threshold is dispatch.max_load_per_cpu for a fixed box (default 0.8) and dispatch.rent_load_per_cpu for a rented one (default: autoscale's --rent-load-per-cpu, 2).
  • Each node with room takes one new worker a pass. The 15-minute load moves slowly, so the pass after sees only part of the load the last worker added.
  • A dispatched worker that is still queued (no node took it yet) uses that room first.
  • When no node has room and autoscale may rent another machine under its --max-spend, the dispatcher starts one worker. That worker waits in the queue, and autoscale rents for it.
  • An inference profile must also have room. When every profile the workers can use is limited (on any window) or at its max_concurrent, pace or usage threshold, the tickets wait.
  • dispatch.max_workers (default 40) is a hard safety ceiling only.

A ticket that waits shows why on its Work card. The server logs a line for each pass that has a ticket waiting:

dispatch: load room=1 rent=false running=7 queued=0 ceiling=40 waiting=3
  nodes="elastic-1: load15 0.18/vCPU of 2, mem 81% free, 5/64 turns: room; node-b: load15 0.49/vCPU of 0.8, mem 40% free, 1/1 turns: full; ..."

dispatch.limit_by: max_workers turns the load rule off: then only max_workers limits.

Autoscale

Autoscale rents another machine when every rented machine has been full for --rent-after. A machine is full when its 5-minute load reaches --rent-load-per-cpu times its vCPUs (or --rent-load), every slot is taken, or its node refuses work for memory or disk. A node that refuses work only for its 1-minute load is not full: a short spike does not start the clock. The live value of --rent-after is 5m. Idle machines are deleted as before. The spend ceiling (--max-spend) is not changed by any of this.

History

The Machines page shows each machine's 15-minute load per vCPU for the last 24 hours, and a fleet-wide panel of agents running against fleet load. GET /v1/machines/{id}/history?since= serves the points: one a minute, kept in memory, so a server restart starts it empty.