Incidents
Fleet opens an incident issue when CI goes red or Alertmanager sends an alert (#368). Each
incident is a Forgejo issue with the labels type:incident and prio:now. Fleet closes it
with a comment when the cause is gone.
Red CI
Fleet reads the result of every push that the Forgejo webhook reports (see #320). It reads results in every repo and on every branch.
- When a check on the default branch fails, fleet opens an incident in that repo.
- When 3 checks in a row in a repo fail at the same step, on at least 2 branches, fleet also opens an incident. If one branch fails again and again, that is a problem for its worker, so fleet opens no incident.
- The issue names the failing step, the first error line and the links to the tasks.
- A repo has one open incident for each cause. The cause is the failing step. When a check fails again, fleet adds a comment to the open issue and opens no new one.
- When the default branch is green again, fleet closes the repo's CI incidents with a comment.
The Forgejo API shows the jobs of a run, but not their steps or logs. So each gate job in
fleet.yml writes its log to a file with tee. When the job fails, its last step runs
.forgejo/scripts/fail-note.sh. The script posts a commit status with the context
fail-note/<job> and the description <step>: <first error line>. The step is the make
target that failed. If a repo has no such status, fleet uses the job's name as the step and
the issue has no error line.
Alertmanager
POST /v1/hooks/alertmanager takes Alertmanager's webhook. Alertmanager must send
Authorization: Bearer <secret>. The secret is FLEET_ALERTMANAGER_TOKEN. Fleet reads it
from the environment, or from the secrets file. If the server has no secret, the path answers
404.
- When an alert fires, fleet opens an incident with its labels, annotations and source link. The alert's fingerprint is the cause, so an alert that Alertmanager sends again opens no second issue.
- When the alert resolves, fleet adds a comment and closes the issue.
- The labels of the alert pick the repo. Fleet looks at the label
repo, thenservice, thennamespace, thenjob, and uses the first value thatincidents.alert_reposnames. If the labelrepois in the formowner/name, fleet uses it as it is. All other alerts go toincidents.infra_repo. - When Forgejo refuses a write, fleet answers 502. Alertmanager then sends the alert again.
# serve.yaml
forgejo_url: https://git.example.com
incidents:
infra_repo: owner/infra
alert_repos:
fleet: owner/swarm-lab
chief: chief # the agent, by name or id, that gets each incident line
Incidents need forgejo_url and a FORGEJO_TOKEN that can write issues and labels.
Version drift
Each machine on the machine list reports the fleet release it runs. When a machine runs another release than the control server for more than 30 minutes, fleet opens an incident (#369).
- There is one incident per cause, not per machine (#403). Machines of one class share a cause:
every CI runner (
ci-runner), every build helper (builder), every rented node (elastic-node). A fixed box (a home box, the server's own node) is its own class. - The first machine of a class past the 30 minutes opens the incident. Each further machine of the class adds one comment to it.
- A machine that comes and goes within the 30 minutes (a short-lived CI runner) files nothing.
- Fleet looks at the versions every minute.
- The incident goes to the repo that
incidents.alert_reposnames forfleet, else toincidents.infra_repo. - Fleet closes the incident when every machine of the class runs the server's release, goes offline or gone, or leaves the machine list.
- An incident from before #403 (one machine each) still covers its machine and closes when that machine stops drifting.
- The time each drift began is kept in
drift.jsonin the server's state directory, so a restart offleet servedoes not start the 30 minutes again. - A machine that moves to another release, even one still behind the server's, starts the 30 minutes again (#418). A machine that follows the releases is not an incident because the server shipped again before the machine caught up.
Signals
When fleet opens an incident, it:
- sends a Telegram line to the first admin, through the bot of
fleet notify, - sends a message from
fleet-incidentsto the agent thatincidents.chiefnames, by name or id, when that agent is not stopped or failed. With noincidents.chief, the message goes to the newest agent of kind chief. Ops desks and a night chief are of kind chief too, so a fleet that has them must setincidents.chief. The journal hasincident: the chief toldfor each message andincident: no live chiefwhen no agent matches, - shows the ticket in Needs attention on the Work page until a worker is active on it,
- tells the dispatcher (#367) to start a worker on it.
When fleet closes an incident, it sends a Telegram line too. Each Telegram line that is sent
logs incident: telegram sent in the journal, and a failure logs incident: telegram.