ephemerd gives every CI job a brand new runner. A new container on Linux, a new VM on macOS and Windows. The job runs, the runner is thrown away, and the next job starts from a clean image. That is the whole point. Nothing a job does can leak into the next one.

ephemerd gives every CI job a brand new environment and throws it away after. On Linux and Windows that environment is a container. On macOS it can’t be. Apple’s license wants macOS running on Apple hardware, so the Mac in my rack is the real thing. The isolation comes from a VM instead.

The scheduler in ephemerd is event-driven. GitHub sends a workflow_job webhook when a job is queued, when it starts and when it completes. ephemerd creates and destroys runners to match. For a while that worked well enough that I stopped thinking about it. Then a docker-build job failed with “The self-hosted runner lost communication with the server”. I pulled the node’s journal. ephemerd had destroyed that runner itself, on purpose, while it was running the build.

go-proxmox had its first commit in June 2021 and v0.8.2 went out on September 30th. In between there were 366 commits and 43 people besides me with a merged PR. More to the point of this post, there are downstream projects that I don’t control pinning it in their go.mod. IONOS’s Cluster API provider and Sidero’s Omni Proxmox provider are both on v0.8.1 today. A library with users like that gets judged less on its API. It gets judged more on what happens to their build when they…

Two short ones from the Windows node of ephemerd from the last few weeks. Neither is complicated once you know what’s going on. Both took longer than they should have to find, because the failure showed up a long way from the cause.

Every shipmates crew member runs shell commands. Before any of them runs, it goes through a permission gate. The gate uses the same rule syntax as Claude Code settings: Bash(git status), Bash(rm *), Write(**/.claude/settings.json). The rules are merged from several layers:

Every turn a shipmates crew member takes is a subprocess. It’s claude or the codex app-server. That in turn spawns shells, go test, npm or whatever the task calls for. Two things have to be true about that tree. When a turn ends or gets killed, the whole tree goes, including the grandchildren. And if somebody sets a memory or CPU budget, the budget applies to the whole tree, not just the process I happened to start.

On August 16th a server.test.exe from shipmates climbed to roughly 1 TB of commit charge on my Windows workstation. Windows Terminal was hosting several agent sessions in tabs. It hard-faulted with E_OUTOFMEMORY, and every one of those sessions died at the same moment. It wasn’t the first time either. An earlier one hit about 236 GB through an infinite filepath.Dir loop.

shipmates spends most of its life spawning claude (and codex) processes and reading newline-delimited JSON off their stdout. The last frame of a turn is the important one. It’s a result frame with the cost, the model and the final text. Stderr matters too, because that’s where Claude Code tells you Invalid API key · Please run /login. For a while I was throwing both away under load. The cause is documented right there in the os/exec godoc. I fixed it in one package and then found the…

A few shipmates bugs don’t need a whole post each. They were too good to leave in commit messages, though.