Fleet docs v0.2.0-rc.448

A machine joins, and a machine dies

A node joins with a one-use token. Then it asks the server for work every 10 s. If it is silent for more than 30 s, the server marks it lost. After 15 minutes, the server gives its agents to another node.

A machine joins

  • The node id is tag:fleet-node/<hostname>.
  • A personal join token makes a node that runs only its owner's agents. A shared join token (--shared) makes a node for every member. It needs uid or pod isolation, or the server refuses the join (fleet/internal/node/tokens.go).
  • kinds in node.yaml limits the agent kinds: worker, front-door, chief, kubejob. An empty list takes any kind.
  • A claim takes a lease of 90 s. The node also claims at once when a turn ends.
  • If the server forgets the node (HTTP 401 or 404), the node joins again with its node token.

A machine dies, and its work moves

Fact Value Code
Heartbeat and claim Every 10 s fleet/internal/node/coordinator.go (HeartbeatInterval)
Lost No heartbeat for more than 3 × 10 s. In practice 30 to 40 s. coordinator.go (Sweep)
Made again elsewhere After 15 min coordinator.go (RecreateAfter)
The new turn's prompt Tells the agent its branch and to read RESUME.md coordinator.go (MovedPrompt)
  • The working directory stays on the dead machine. The new node does not get work that the agent did not push. Tell workers to push after each commit (to wip/<n>, Q14) and to write RESUME.md.
  • At each turn's end the node stores the agent's RESUME.md, the files that .fleet/keep names (one path, glob or directory per line, relative to the worktree) and the uncommitted tracked changes as a patch, in the blob <agent>/keep.tar.zst. The next turn's node, any node, restores them into the worktree before the agent starts. A file already there is not overwritten. The patch is applied only to a worktree with no uncommitted change; one that does not apply is left in the git dir as fleet-wip.patch (#442). A lost node's turn ends with no store, so the next turn gets what the turn before it kept.
  • The node sends the conversation to the server's blobs at the end of each turn. The new turn continues from the last finished turn. The part of the turn that was in progress is lost. If there is no blob, the new turn starts from the brief.
  • The node token never expires and has no revoke. Only a new server key ends it.
  • The This box panel of a silent node shows a stale badge.