Fleet docs v0.2.0-rc.448

Incidents

Fleet opens an incident issue when CI goes red or Alertmanager sends an alert (#368). Each incident is a Forgejo issue with the labels type:incident and prio:now. Fleet closes it with a comment when the cause is gone.

Red CI

Fleet reads the result of every push that the Forgejo webhook reports (see #320). It reads results in every repo and on every branch.

  • When a check on the default branch fails, fleet opens an incident in that repo.
  • When 3 checks in a row in a repo fail at the same step, on at least 2 branches, fleet also opens an incident. If one branch fails again and again, that is a problem for its worker, so fleet opens no incident.
  • The issue names the failing step, the first error line and the links to the tasks.
  • A repo has one open incident for each cause. The cause is the failing step. When a check fails again, fleet adds a comment to the open issue and opens no new one.
  • When the default branch is green again, fleet closes the repo's CI incidents with a comment.

The Forgejo API shows the jobs of a run, but not their steps or logs. So each gate job in fleet.yml writes its log to a file with tee. When the job fails, its last step runs .forgejo/scripts/fail-note.sh. The script posts a commit status with the context fail-note/<job> and the description <step>: <first error line>. The step is the make target that failed. If a repo has no such status, fleet uses the job's name as the step and the issue has no error line.

Alertmanager

POST /v1/hooks/alertmanager takes Alertmanager's webhook. Alertmanager must send Authorization: Bearer <secret>. The secret is FLEET_ALERTMANAGER_TOKEN. Fleet reads it from the environment, or from the secrets file. If the server has no secret, the path answers 404.

  • When an alert fires, fleet opens an incident with its labels, annotations and source link. The alert's fingerprint is the cause, so an alert that Alertmanager sends again opens no second issue.
  • When the alert resolves, fleet adds a comment and closes the issue.
  • The labels of the alert pick the repo. Fleet looks at the label repo, then service, then namespace, then job, and uses the first value that incidents.alert_repos names. If the label repo is in the form owner/name, fleet uses it as it is. All other alerts go to incidents.infra_repo.
  • When Forgejo refuses a write, fleet answers 502. Alertmanager then sends the alert again.
# serve.yaml
forgejo_url: https://git.example.com
incidents:
  infra_repo: owner/infra
  alert_repos:
    fleet: owner/swarm-lab
  chief: chief   # the agent, by name or id, that gets each incident line

Incidents need forgejo_url and a FORGEJO_TOKEN that can write issues and labels.

Version drift

Each machine on the machine list reports the fleet release it runs. When a machine runs another release than the control server for more than 30 minutes, fleet opens an incident (#369).

  • There is one incident per cause, not per machine (#403). Machines of one class share a cause: every CI runner (ci-runner), every build helper (builder), every rented node (elastic-node). A fixed box (a home box, the server's own node) is its own class.
  • The first machine of a class past the 30 minutes opens the incident. Each further machine of the class adds one comment to it.
  • A machine that comes and goes within the 30 minutes (a short-lived CI runner) files nothing.
  • Fleet looks at the versions every minute.
  • The incident goes to the repo that incidents.alert_repos names for fleet, else to incidents.infra_repo.
  • Fleet closes the incident when every machine of the class runs the server's release, goes offline or gone, or leaves the machine list.
  • An incident from before #403 (one machine each) still covers its machine and closes when that machine stops drifting.
  • The time each drift began is kept in drift.json in the server's state directory, so a restart of fleet serve does not start the 30 minutes again.
  • A machine that moves to another release, even one still behind the server's, starts the 30 minutes again (#418). A machine that follows the releases is not an incident because the server shipped again before the machine caught up.

Signals

When fleet opens an incident, it:

  • sends a Telegram line to the first admin, through the bot of fleet notify,
  • sends a message from fleet-incidents to the agent that incidents.chief names, by name or id, when that agent is not stopped or failed. With no incidents.chief, the message goes to the newest agent of kind chief. Ops desks and a night chief are of kind chief too, so a fleet that has them must set incidents.chief. The journal has incident: the chief told for each message and incident: no live chief when no agent matches,
  • shows the ticket in Needs attention on the Work page until a worker is active on it,
  • tells the dispatcher (#367) to start a worker on it.

When fleet closes an incident, it sends a Telegram line too. Each Telegram line that is sent logs incident: telegram sent in the journal, and a failure logs incident: telegram.