Luther Monson, Staff Software Engineer at SUSE

Luther Monson

Go by way of PHP · Containers · VMs · Windows · Kubernetes

shipmates: Persona-Based AI Coding Crews That Remember

shipmates is a Go CLI and daemon that runs crews of AI coding agents across machines. It adds per-project memory, a shared task graph, voice control and a real permission model.

7-Minute Read

shipmates hierarchy: Admiral, Fleet Command, ships, captains and mates

Since June most of my evenings have gone into shipmates, a Go CLI and daemon for running crews of AI coding agents. I tagged v0.5.0 on August 17th. That was two months and about 240 commits after the first architecture doc. So it’s a good time to write down what it is and why it exists.

ephemerd: One Daemon, a Fresh Runner Per Job, Every OS

A single Go binary that runs self-hosted GitHub Actions runners on Linux, Windows and macOS, with a new container or VM for every job.

6-Minute Read

ephemerd architecture across Linux, Windows and macOS hosts

I have CI that needs Linux amd64, Linux arm64, Windows and macOS. GitHub-hosted runners for all four get expensive fast, especially the arm64 and Mac minutes. Self-hosted runners fix the bill and create a different problem. A stock actions/runner on a box you own runs whatever the workflow says. It keeps whatever the last job left behind. And it’s one fork PR away from somebody running code on your LAN.

Atomic File Replacement on NTFS Is Not rename + fsync

The Unix recipe for durable atomic file replacement doesn't translate to Windows. This is what shipmates uses instead, plus the delete-pending window that made an 8-way test fail only when the whole suite ran.

7-Minute Read

How long a losing concurrent replacer waits on NTFS, idle disk vs a disk under fsync load

shipmates writes a lot of small state files that other processes read. Voyage plans and state, tracker files. Each of them has to be replaced atomically, so a reader sees the old file or the new one and never half of each. Each also has to be replaced durably. When the function returns success, a power cut doesn’t take it back. Every Unix programmer knows the recipe for that. It doesn’t work on Windows, and the reasons are more interesting than “Windows is different”.

Five Ways to URL-Encode an SSH Key (and Why Proxmox Rejects Them)

Proxmox's sshkeys parameter wants Python-style percent encoding that no single Go function produces. This covers a two-year-old issue, its one-line fix, and a related double-encoding bug in guest file writes.

6-Minute Read

Table of seven ways to encode an SSH public key and whether each passes Proxmox's urlencoded validator

Issue #144 on go-proxmox was opened in April 2024 with the title “Weird sshkeys SSH public key validation error”. It is one of my less proud threads. It’s thirteen comments of people, including me, trying to get a public key into a cloud-init VM through the API. The working answer took more than a year to show up. The fix that finally landed in May is one line. Here’s why it took so long.

Two Small go-proxmox Fixes: A Watch Channel That Never Closed, and WithRetry

go-proxmox had a task log channel that only closed on the happy path. It also gained a composable RoundTripper that retries the 502s and 503s Proxmox throws during cluster transitions.

6-Minute Read

Proxmox VE Logo

Here are two changes from the v0.5 to v0.7 stretch of go-proxmox. They’re too small for a post each. They’re worth knowing about if you write long-running tooling against Proxmox. One is a bug in code I wrote about in 2022. The other is a feature that replaces a retry loop everybody was writing for themselves.

Maintaining an Open-Source Library With AI Agents: AGENTS.md and the Rule They Kept Breaking

Six weeks of go-proxmox with coding agents doing most of the typing taught me what belongs in AGENTS.md. Five parallel agents got the same API shape wrong, and a coverage metric bent the code around it.

7-Minute Read

Decision diagram: endpoints without an id in the path become methods on the parent, endpoints with an id get a handle type

On May 9th I merged #264. It added an AGENTS.md to go-proxmox and a stub CLAUDE.md that just points at it. The next six weeks ran through the v0.8.0 migration guide on June 15th. In that time main took 77 commits and +31,794 / -1,144 lines. The repo went from 84 to 186 .go files. Endpoint coverage went from 30% to 100% of the Proxmox API. Most of that code was typed by coding agents, a lot of it by several at once. This is what that looked like from the maintainer’s chair. It’s…

Proxmox JSON, Part 2: null PIDs, 1.12589990684262e+15 and net10

The Proxmox VE API found three more ways for its JSON not to fit Go's encoding/json. There's a null I fixed and then broke, petabyte storage in scientific notation, and devices past index 9 that were silently dropped.

7-Minute Read

Proxmox VE Logo

In 2023 I wrote about why go-proxmox has StringOrInt and friends. Proxmox sends the same number quoted in one place and bare in another, so the client has types that accept both. That post ended on a pretty confident note. Since then three more shapes of “the JSON is valid and Go still won’t take it” have come in. One of them is a bug I’d already fixed once and then reintroduced myself.

Escaping WSL: Booting My Own Linux VM Through Hyper-V's HCS API

WSL2 refuses to run under LocalSystem, so ephemerd boots an Alpine kernel directly through vmcompute.dll instead.

9-Minute Read

Diagram of ephemerd booting a Linux VM through HCS and reading its IP from the serial console

I’ve been building ephemerd, a daemon that runs self-hosted GitHub Actions runners with a fresh environment per job. On a Windows host, Windows jobs run in Hyper-V isolated containers. Linux jobs still need a Linux kernel from somewhere. The obvious answer on Windows in 2026 is WSL2, and that’s what I used first. It worked great from my terminal and fell over the moment ephemerd became a real Windows service.

Your Zero Value Just Turned Off My TLS Verification: omitempty, Go Defaults and Proxmox

A plain Go field can't tell unset from false. In go-proxmox the fix for one omitempty bug created the opposite bug, until an audit against the Proxmox schema found 75 fields and v0.7.0 moved them to pointers.

7-Minute Read

Table showing how verify-certificates was sent by go-proxmox before #210, between v0.2.3 and v0.6.x, and in v0.7.0

Every Go developer knows omitempty drops zero values. What takes longer to sink in is that a plain bool only has two states. A caller filling in an options struct means one of three:

Recent Posts

Categories

About

Luther is a 20+ year software veteran specializing in backend APIs and architecting products. Employed at SUSE as the Team Lead for the Containers on Windows Team working on Rancher + Kubernetes