8. Kubernetes and some clusters
A node can run each turn as a Kubernetes Job instead of a process. This node has the
backend kubejob. It takes only the agents of the
kind kubejob.
- Make the agent with
fleet spawn <name> --kind kubejob. The server never runs akubejobagent itself. - The node needs one Role in one namespace: Jobs, pods,
pods/logandpods/exec. - The pod's entrypoint waits for the copy, runs the turn, then waits again until the node has
taken the results out (
fleet/internal/backend/kubejob/copy.go). - There is no shared disk. The node copies the directories into the pod and back out. Secret
values go in the same way, not in the Job spec (
fleet/internal/backend/kubejob/copy.go). - The pod gets no service account token. The pod must reach the server.
Some clusters
- Each cluster has its own
kubejobnode. A node finds its cluster from the in-cluster config, or else fromKUBECONFIG(fleet/internal/cli/node.go). - Not built: a way to send one agent to one cluster. Any
kubejobnode can take anykubejobturn. - Not built: a
kubeconfigkey innode.yaml. To reach two clusters from one machine, run two node processes, each with its own--dirandKUBECONFIG.
Details: Kubernetes Jobs, some clusters. Install: Kubernetes Jobs.
Next: 9. Separate fleets. Some teams must not share one fleet.