A worker crashes, or the server restarts
A crash is a turn that ends with no result. The supervisor tries the turn again three times. A server restart does not stop a turn on the server's box. A node restart stops the turns on that node, and the server starts them again.
A crash
| Fact | Value | Code |
|---|---|---|
| The waits | 30 s, 2 min, 8 min | fleet/internal/supervisor/supervisor.go (DefaultBackoff) |
failed |
At the 4th crash in a row | supervisor.go |
| The count starts again | After a turn that completes or needs input | supervisor.go |
Straight to failed |
Errors of kind model, workdir, permanent, and the budget | supervisor.go |
| Login error | The profile pauses and the agent spends no retry | supervisor.go |
| Resume | Only from failed, blocked or held. Other states get HTTP 409. |
fleet/internal/api/lifecycle.go |
- A person sees
retrying 2/3on the page and infleet ls. The supervisor log lists the agent. No Telegram message goes out for a crash. - Not built: a
fleet resumecommand. The held reason tells you to run it, but the command is not in the CLI. Use theresumetool of the control MCP, or the API.
The server restarts
fleet turn-shimignores SIGTERM, SIGINT and SIGHUP. It writes the exit code to a file (fleet/internal/backend/process/detach.go).- The shim is not used with
uidisolation. On a shared node, the node starts the turns, not the server. - Retry timers are in memory. After a restart, an agent whose retry is due starts at once
(
fleet/internal/sched/sched.go). - The server does not touch the turns of other nodes when it restarts.
A node restarts
- A node's turns run as child processes of the node. A node stop ends them.
- A clean stop counts as "stuck", so the agent can move up the model ladder (see
a stuck worker). This is a known quirk of the code
(
fleet/internal/supervisor/supervisor.go). - If the node does not return, see a machine dies.