Fleet docs v0.2.0-rc.448

A worker crashes, or the server restarts

A crash is a turn that ends with no result. The supervisor tries the turn again three times. A server restart does not stop a turn on the server's box. A node restart stops the turns on that node, and the server starts them again.

A crash

Fact Value Code
The waits 30 s, 2 min, 8 min fleet/internal/supervisor/supervisor.go (DefaultBackoff)
failed At the 4th crash in a row supervisor.go
The count starts again After a turn that completes or needs input supervisor.go
Straight to failed Errors of kind model, workdir, permanent, and the budget supervisor.go
Login error The profile pauses and the agent spends no retry supervisor.go
Resume Only from failed, blocked or held. Other states get HTTP 409. fleet/internal/api/lifecycle.go
  • A person sees retrying 2/3 on the page and in fleet ls. The supervisor log lists the agent. No Telegram message goes out for a crash.
  • Not built: a fleet resume command. The held reason tells you to run it, but the command is not in the CLI. Use the resume tool of the control MCP, or the API.

The server restarts

  • fleet turn-shim ignores SIGTERM, SIGINT and SIGHUP. It writes the exit code to a file (fleet/internal/backend/process/detach.go).
  • The shim is not used with uid isolation. On a shared node, the node starts the turns, not the server.
  • Retry timers are in memory. After a restart, an agent whose retry is due starts at once (fleet/internal/sched/sched.go).
  • The server does not touch the turns of other nodes when it restarts.

A node restarts

  • A node's turns run as child processes of the node. A node stop ends them.
  • A clean stop counts as "stuck", so the agent can move up the model ladder (see a stuck worker). This is a known quirk of the code (fleet/internal/supervisor/supervisor.go).
  • If the node does not return, see a machine dies.