Deploy re-queue
A deploy job can die with its CI runner: a Spot machine is taken back, a runner is deleted
while it is busy, or a runner pod is evicted. No step failed, but the run ends failure or
cancelled, and the change does not reach production. Fleet runs such a deploy again, once
(#513).
What fleet does
- Each push to a repo's default branch opens a watch, when the repo's
.fleet/ship.yamlhasdeploy.trigger: pushand adeploy.workflow. Fleet learns of the push from the Forgejo webhook (#320). - The watch follows the push run of the deploy workflow for that sha, every 30 s, until it
ends. It stops when no run shows in 10 minutes (a push that
paths-ignoreskips), or after 3 hours. A poll that fails (a 502, a timeout) does not count as "no run", and once the run has shown, the 10 minutes no longer apply (#532). - When the run ends
failureorcancelled, fleet reads the commit statuses of the sha. A failed job of the run with noend-note/<job>status for that run never reached its last step, so its runner died. - Fleet then dispatches the workflow (
workflow_dispatch) on the default branch. When the workflow has ashainput, fleet passes the sha, so the same commit is deployed. - Fleet adds a comment to each issue that the commit subject names (
(#12),Refs #12), and tells the first admin (Telegram) and the chief (incidents.chief). When the new run ends, fleet says how it ended in the same places. - When the dispatch fails (a 5xx, a timeout), fleet tries it again 30 s later, up to 3 times. Before each retry it looks for a newer run: a dispatched run means the dispatch that seemed to fail went through, and a push run means a later push took the dead run's place. When every try fails, the same places hear "deploy NOT re-queued" with the error, so a person re-runs it (#533).
Fleet does not re-queue:
- a run where every failed job posted its end-note: a step failed, and the incident watcher
(
operators/incidents.md) handles red CI; - a run that a newer run of the same workflow follows: a later push, or a re-run by a person, takes its place;
- a dispatched run. The watch follows push runs only, so a re-queued run is never re-queued again;
- a deploy workflow that has no end-note step, or no
workflow_dispatch. Without the end-note, fleet cannot tell a dead runner from a failed step.
The end-note step
Forgejo's API shows the jobs of a run, but not their steps or logs. So the deploy job says
itself that it reached its end. Add this step as the last step of each job of the deploy
workflow. Replace <job> with the job's name (its name:, else its key).
- name: end-note (fleet re-queues a deploy whose runner died, swarm-lab#513)
if: always()
env:
TOKEN: ${{ github.token }}
SHA: ${{ inputs.sha || github.sha }}
run: |
curl -fsS --max-time 30 -o /dev/null -X POST \
-H "Authorization: token $TOKEN" -H "Content-Type: application/json" \
-d "{\"state\":\"success\",\"context\":\"end-note/<job>\",\"description\":\"run $GITHUB_RUN_NUMBER: ${{ job.status }}\"}" \
"${GITHUB_SERVER_URL%/}/api/v1/repos/${GITHUB_REPOSITORY}/statuses/${SHA}" || echo "end-note: not posted"
The description must start with run <n>, the run number. A note from an earlier run of the
same sha does not count for a later run.
The workflow also needs on: workflow_dispatch, best with a sha input that it deploys.
Limits
- Watches are in memory. A server restart drops them, so a deploy that dies while the server restarts is not re-queued.
- Only
deploy.trigger: pushis watched. A tag release (trigger: tag) is not.