The scheduler in ephemerd is event-driven. GitHub sends a workflow_job webhook when a job is queued, when it starts and when it completes. ephemerd creates and destroys runners to match. For a while that worked well enough that I stopped thinking about it. Then a docker-build job failed with “The self-hosted runner lost communication with the server”. I pulled the node’s journal. ephemerd had destroyed that runner itself, on purpose, while it was running the build.

go-proxmox had its first commit in June 2021 and v0.8.2 went out on September 30th. In between there were 366 commits and 43 people besides me with a merged PR. More to the point of this post, there are downstream projects that I don’t control pinning it in their go.mod. IONOS’s Cluster API provider and Sidero’s Omni Proxmox provider are both on v0.8.1 today. A library with users like that gets judged less on its API. It gets judged more on what happens to their build when they…

Two short ones from the Windows node of ephemerd from the last few weeks. Neither is complicated once you know what’s going on. Both took longer than they should have to find, because the failure showed up a long way from the cause.

Every shipmates crew member runs shell commands. Before any of them runs, it goes through a permission gate. The gate uses the same rule syntax as Claude Code settings: Bash(git status), Bash(rm *), Write(**/.claude/settings.json). The rules are merged from several layers:

Every turn a shipmates crew member takes is a subprocess. It’s claude or the codex app-server. That in turn spawns shells, go test, npm or whatever the task calls for. Two things have to be true about that tree. When a turn ends or gets killed, the whole tree goes, including the grandchildren. And if somebody sets a memory or CPU budget, the budget applies to the whole tree, not just the process I happened to start.

On August 16th a server.test.exe from shipmates climbed to roughly 1 TB of commit charge on my Windows workstation. Windows Terminal was hosting several agent sessions in tabs. It hard-faulted with E_OUTOFMEMORY, and every one of those sessions died at the same moment. It wasn’t the first time either. An earlier one hit about 236 GB through an infinite filepath.Dir loop.

shipmates spends most of its life spawning claude (and codex) processes and reading newline-delimited JSON off their stdout. The last frame of a turn is the important one. It’s a result frame with the cost, the model and the final text. Stderr matters too, because that’s where Claude Code tells you Invalid API key · Please run /login. For a while I was throwing both away under load. The cause is documented right there in the os/exec godoc. I fixed it in one package and then found the…

A few shipmates bugs don’t need a whole post each. They were too good to leave in commit messages, though.

A lot of my day is spent in k9s. Sometimes I need something from the Linode side of the house. Maybe it’s a node that won’t boot, a volume nobody cleaned up or the kubeconfig for a cluster I just made. Then I’m back in a browser tab or typing linode-cli incantations. I wanted the k9s workflow for the Linode API itself:

shipmates ships connect to Fleet Command over outbound websocket tunnels. That way nothing on the ship side has to be exposed to the internet. I didn’t write that tunnel layer. I used rancher/remotedialer, the same library Rancher uses to reach downstream cluster agents. I spent years around that code at Rancher, so it was the obvious pick. What I didn’t expect was for my CI’s race job to find two data races in it in the first week.