A lot of my day is spent in k9s. Sometimes I need something from the Linode side of the house. Maybe it’s a node that won’t boot, a volume nobody cleaned up or the kubeconfig for a cluster I just made. Then I’m back in a browser tab or typing linode-cli incantations. I wanted the k9s workflow for the Linode API itself:

shipmates ships connect to Fleet Command over outbound websocket tunnels. That way nothing on the ship side has to be exposed to the internet. I didn’t write that tunnel layer. I used rancher/remotedialer, the same library Rancher uses to reach downstream cluster agents. I spent years around that code at Rancher, so it was the obvious pick. What I didn’t expect was for my CI’s race job to find two data races in it in the first week.

Ask most people what their PHP server does when it’s overloaded and you’ll hear “it returns 503s.” Ask them how they know and it usually comes down to a config setting they read about. I was in that camp for ePHPm. We had a [server.limits] max_connections knob, and the docs said it returned 503 at the cap. Every benchmark we had ever run reported zero errors under heavy load. Then I wrote a load generator that could produce overload. The honest answer was that an…

Since June most of my evenings have gone into shipmates, a Go CLI and daemon for running crews of AI coding agents. I tagged v0.5.0 on August 17th. That was two months and about 240 commits after the first architecture doc. So it’s a good time to write down what it is and why it exists.

I have CI that needs Linux amd64, Linux arm64, Windows and macOS. GitHub-hosted runners for all four get expensive fast, especially the arm64 and Mac minutes. Self-hosted runners fix the bill and create a different problem. A stock actions/runner on a box you own runs whatever the workflow says. It keeps whatever the last job left behind. And it’s one fork PR away from somebody running code on your LAN.

shipmates writes a lot of small state files that other processes read. Voyage plans and state, tracker files. Each of them has to be replaced atomically, so a reader sees the old file or the new one and never half of each. Each also has to be replaced durably. When the function returns success, a power cut doesn’t take it back. Every Unix programmer knows the recipe for that. It doesn’t work on Windows, and the reasons are more interesting than “Windows is different”.

Issue #144 on go-proxmox was opened in April 2024 with the title “Weird sshkeys SSH public key validation error”. It is one of my less proud threads. It’s thirteen comments of people, including me, trying to get a public key into a cloud-init VM through the API. The working answer took more than a year to show up. The fix that finally landed in May is one line. Here’s why it took so long.

Here are two changes from the v0.5 to v0.7 stretch of go-proxmox. They’re too small for a post each. They’re worth knowing about if you write long-running tooling against Proxmox. One is a bug in code I wrote about in 2022. The other is a feature that replaces a retry loop everybody was writing for themselves.

On May 9th I merged #264. It added an AGENTS.md to go-proxmox and a stub CLAUDE.md that just points at it. The next six weeks ran through the v0.8.0 migration guide on June 15th. In that time main took 77 commits and +31,794 / -1,144 lines. The repo went from 84 to 186 .go files. Endpoint coverage went from 30% to 100% of the Proxmox API. Most of that code was typed by coding agents, a lot of it by several at once. This is what that looked like from the maintainer’s chair. It’s…

In 2023 I wrote about why go-proxmox has StringOrInt and friends. Proxmox sends the same number quoted in one place and bare in another, so the client has types that accept both. That post ended on a pretty confident note. Since then three more shapes of “the JSON is valid and Go still won’t take it” have come in. One of them is a bug I’d already fixed once and then reintroduced myself.