Go on Luther Monson

Webhooks Arrive Out of Order: How My Scheduler Killed Live Builds

Wed, 07 Oct 2026 21:10:00 -0700

The scheduler in ephemerd is event-driven. GitHub sends a workflow_job webhook when a job is queued, when it starts and when it completes. ephemerd creates and destroys runners to match. For a while that worked well enough that I stopped thinking about it. Then a docker-build job failed with “The self-hosted runner lost communication with the server”. I pulled the node’s journal. ephemerd had destroyed that runner itself, on purpose, while it was running the build.

Breaking Changes Without Breaking Trust: Five Years to go-proxmox v0.8

Sun, 04 Oct 2026 19:40:00 -0700

go-proxmox had its first commit in June 2021 and v0.8.2 went out on September 30th. In between there were 366 commits and 43 people besides me with a merged PR. More to the point of this post, there are downstream projects that I don’t control pinning it in their go.mod. IONOS’s Cluster API provider and Sidero’s Omni Proxmox provider are both on v0.8.1 today. A library with users like that gets judged less on its API. It gets judged more on what happens to their build when they bump the version. I didn’t always get that right, so here is the history, measured.

Windows Grab-Bag: SIGTERM That Doesn't Exist and Sparse Files That Break Docker

Sat, 03 Oct 2026 14:30:00 -0700

Two short ones from the Windows node of ephemerd from the last few weeks. Neither is complicated once you know what’s going on. Both took longer than they should have to find, because the failure showed up a long way from the cause.

Part 1: there is no SIGTERM on Windows

ephemerd can run a Cloudflare tunnel so GitHub webhooks reach a node behind NAT. It does that by running cloudflared as a managed subprocess. On Linux the child is tied to ephemerd’s lifetime by the kernel with Pdeathsig:

Your Agent's Shell Allowlist Is Bypassable

Tue, 29 Sep 2026 20:30:00 -0700

Every shipmates crew member runs shell commands. Before any of them runs, it goes through a permission gate. The gate uses the same rule syntax as Claude Code settings: Bash(git status), Bash(rm *), Write(**/.claude/settings.json). The rules are merged from several layers:

a fleet-wide deny list pushed from Fleet Command
per-persona policy files
project settings
the Brig, which compiles shipmates’ fifteen “Ship’s Articles” into deny and ask rules

Deny beats ask, ask beats allow.

Cross-Platform Process Containment: Job Objects vs Process Groups vs cgroups

Sat, 26 Sep 2026 16:20:00 -0700

Every turn a shipmates crew member takes is a subprocess. It’s claude or the codex app-server. That in turn spawns shells, go test, npm or whatever the task calls for. Two things have to be true about that tree. When a turn ends or gets killed, the whole tree goes, including the grandchildren. And if somebody sets a memory or CPU budget, the budget applies to the whole tree, not just the process I happened to start.

A 1 TB Test: Building a Memory Watchdog for go test

Tue, 22 Sep 2026 20:48:00 -0700

On August 16th a server.test.exe from shipmates climbed to roughly 1 TB of commit charge on my Windows workstation. Windows Terminal was hosting several agent sessions in tabs. It hard-faulted with E_OUTOFMEMORY, and every one of those sessions died at the same moment. It wasn’t the first time either. An earlier one hit about 236 GB through an infinite filepath.Dir loop.

The machine has 128 GB of RAM. Commit charge doesn’t care. It’s backed by the page file, and Windows will happily let a process reserve-and-commit until the commit limit is gone. Then everything else on the desktop starts failing allocations too.

os/exec's StdoutPipe Footgun: How cmd.Wait() Ate My Last Frame (Twice)

Thu, 17 Sep 2026 20:12:00 -0700

shipmates spends most of its life spawning claude (and codex) processes and reading newline-delimited JSON off their stdout. The last frame of a turn is the important one. It’s a result frame with the cost, the model and the final text. Stderr matters too, because that’s where Claude Code tells you Invalid API key · Please run /login. For a while I was throwing both away under load. The cause is documented right there in the os/exec godoc. I fixed it in one package and then found the exact same bug again a month later in another.

shipmates Short Ones: 8 Minutes Extracting Go, and Three Voice-Loop Bugs

Sat, 12 Sep 2026 14:35:00 -0700

A few shipmates bugs don’t need a whole post each. They were too good to leave in commit messages, though.

setup-go spends 8 minutes extracting Go on Windows CI

shipmates CI runs on my own ephemerd runners. On Windows those jobs run inside a Hyper-V utility VM. The windows-x64 leg kept timing out in actions/setup-go. Here’s the log from PR #48:

20:58:46  Setup go version spec 1.26.6
20:58:46  Attempting to download 1.26.6...     <- tool-cache MISS
20:58:48  Acquiring 1.26.6 from ...zip
20:58:53  Extracting Go...
21:06:46  ERROR: The action 'Run actions/setup-go@v5' has timed out after 8 minutes

Seven seconds from the start of the step to Extracting Go.... Then almost eight minutes extracting until the step timeout killed it.

Building k9s for Linode With Bubble Tea

Wed, 09 Sep 2026 21:22:00 -0700

A lot of my day is spent in k9s. Sometimes I need something from the Linode side of the house. Maybe it’s a node that won’t boot, a volume nobody cleaned up or the kubeconfig for a cluster I just made. Then I’m back in a browser tab or typing linode-cli incantations. I wanted the k9s workflow for the Linode API itself:

: to jump to a resource type
/ to filter
single-key actions on the selected row
Enter to go deeper

That turned into linode-tui. It’s a Go TUI built on Bubble Tea, linodego and urfave/cli v3.

Finding a Data Race in rancher/remotedialer, Then Pinning Only Two-Thirds of the Fix

Tue, 01 Sep 2026 21:05:00 -0700

shipmates ships connect to Fleet Command over outbound websocket tunnels. That way nothing on the ship side has to be exposed to the internet. I didn’t write that tunnel layer. I used rancher/remotedialer, the same library Rancher uses to reach downstream cluster agents. I spent years around that code at Rancher, so it was the obvious pick. What I didn’t expect was for my CI’s race job to find two data races in it in the first week.

shipmates: Persona-Based AI Coding Crews That Remember

Fri, 21 Aug 2026 21:18:00 -0700

Since June most of my evenings have gone into shipmates, a Go CLI and daemon for running crews of AI coding agents. I tagged v0.5.0 on August 17th. That was two months and about 240 commits after the first architecture doc. So it’s a good time to write down what it is and why it exists.

The problem

I run a lot of agent sessions. On a normal night this workstation has four or five projects going at once. Each has a few Claude Code sessions doing different jobs: one reviewing, one writing tests, one chasing a bug. Two things kept annoying me.

ephemerd: One Daemon, a Fresh Runner Per Job, Every OS

Tue, 18 Aug 2026 21:05:00 -0700

I have CI that needs Linux amd64, Linux arm64, Windows and macOS. GitHub-hosted runners for all four get expensive fast, especially the arm64 and Mac minutes. Self-hosted runners fix the bill and create a different problem. A stock actions/runner on a box you own runs whatever the workflow says. It keeps whatever the last job left behind. And it’s one fork PR away from somebody running code on your LAN.

Atomic File Replacement on NTFS Is Not rename + fsync

Thu, 06 Aug 2026 21:27:00 -0700

shipmates writes a lot of small state files that other processes read. Voyage plans and state, tracker files. Each of them has to be replaced atomically, so a reader sees the old file or the new one and never half of each. Each also has to be replaced durably. When the function returns success, a power cut doesn’t take it back. Every Unix programmer knows the recipe for that. It doesn’t work on Windows, and the reasons are more interesting than “Windows is different”.

Five Ways to URL-Encode an SSH Key (and Why Proxmox Rejects Them)

Tue, 28 Jul 2026 20:12:00 -0700

Issue #144 on go-proxmox was opened in April 2024 with the title “Weird sshkeys SSH public key validation error”. It is one of my less proud threads. It’s thirteen comments of people, including me, trying to get a public key into a cloud-init VM through the API. The working answer took more than a year to show up. The fix that finally landed in May is one line. Here’s why it took so long.

Two Small go-proxmox Fixes: A Watch Channel That Never Closed, and WithRetry

Sat, 11 Jul 2026 17:26:00 -0700

Here are two changes from the v0.5 to v0.7 stretch of go-proxmox. They’re too small for a post each. They’re worth knowing about if you write long-running tooling against Proxmox. One is a bug in code I wrote about in 2022. The other is a feature that replaces a retry loop everybody was writing for themselves.

The channel that only closed when things went well

In 2022 I wrote up how tasks and Watch work. Watch returns a channel of log lines. A goroutine tails the task log until the task stops. Then it closes the channel so you can range over it. Here is the tail loop from that post:

Maintaining an Open-Source Library With AI Agents: AGENTS.md and the Rule They Kept Breaking

Sat, 27 Jun 2026 18:47:00 -0700

On May 9th I merged #264. It added an AGENTS.md to go-proxmox and a stub CLAUDE.md that just points at it. The next six weeks ran through the v0.8.0 migration guide on June 15th. In that time main took 77 commits and +31,794 / -1,144 lines. The repo went from 84 to 186 .go files. Endpoint coverage went from 30% to 100% of the Proxmox API. Most of that code was typed by coding agents, a lot of it by several at once. This is what that looked like from the maintainer’s chair. It’s mostly the parts that went wrong, because that’s where AGENTS.md got its content.

Proxmox JSON, Part 2: null PIDs, 1.12589990684262e+15 and net10

Tue, 23 Jun 2026 20:31:00 -0700

In 2023 I wrote about why go-proxmox has StringOrInt and friends. Proxmox sends the same number quoted in one place and bare in another, so the client has types that accept both. That post ended on a pretty confident note. Since then three more shapes of “the JSON is valid and Go still won’t take it” have come in. One of them is a bug I’d already fixed once and then reintroduced myself.

Stop Ctrl-C-ing Your tail -f | grep

Sat, 20 Jun 2026 20:31:00 -0700

Here’s a loop I end up in all the time:

tail -f app.log
something scrolls by that looks wrong, Ctrl-C
tail -f app.log | grep ERROR and wait for it to happen again
decide I also want to drop the health checks, Ctrl-C
| grep -v healthz and wait again

Every filter tweak throws away the stream and every line that already went past. I got tired of it and wrote tail-gunner. It’s a tail replacement that you can promote into a filtering TUI with one keystroke. You can drop back out of it without the stream ever stopping.

Escaping WSL: Booting My Own Linux VM Through Hyper-V's HCS API

Sun, 14 Jun 2026 20:12:00 -0700

I’ve been building ephemerd, a daemon that runs self-hosted GitHub Actions runners with a fresh environment per job. On a Windows host, Windows jobs run in Hyper-V isolated containers. Linux jobs still need a Linux kernel from somewhere. The obvious answer on Windows in 2026 is WSL2, and that’s what I used first. It worked great from my terminal and fell over the moment ephemerd became a real Windows service.

WSL2 doesn’t do LocalSystem

A Windows service that wants to create containers and Hyper-V VMs runs as LocalSystem. WSL2 doesn’t support that context at all. You get WSL_E_LOCAL_SYSTEM_NOT_SUPPORTED and that’s the end of the conversation. There’s no flag for it. WSL is a per-user feature and a service account isn’t a user.

Your Zero Value Just Turned Off My TLS Verification: omitempty, Go Defaults and Proxmox

Sat, 13 Jun 2026 19:52:00 -0700

Every Go developer knows omitempty drops zero values. What takes longer to sink in is that a plain bool only has two states. A caller filling in an options struct means one of three:

“set it true”
“set it false”
“I didn’t touch this, do whatever the server does by default”

With omitempty you lose “false”. Without it you lose “didn’t touch it”. In go-proxmox we managed to ship both bugs, in that order. The second one quietly turned off TLS certificate verification for anyone downloading ISOs through the API.

From 52% to 100%: Measuring a Go Client Against the Proxmox API Schema

Sun, 07 Jun 2026 21:36:00 -0700

For five years the honest answer to “how much of the Proxmox API does go-proxmox cover?” was “a lot of it? the parts people asked for?”. Issues would come in for endpoints that didn’t exist in the client. I’d have no idea how many more were missing. At the end of May I finally built the thing that answers the question. Three days after the first run the answer was 100%. The more interesting part is how many times the measuring stick was wrong on the way there.

Cloud-Init ISOs in Pure Go with go-proxmox

Sat, 06 Jun 2026 15:20:00 -0700

Proxmox’s built-in cloud-init support is fine for the basics: a user, an SSH key, an IP config. It falls apart the second you want your own user-data, vendor-data or a real network-config. The GUI and the ci* options only expose a handful of fields. The escape hatch is cicustom pointing at snippet files, and that means getting files onto the Proxmox host’s filesystem first. The usual alternative is building a nocloud ISO yourself with genisoimage or mkisofs. That means a system dependency on whatever machine runs your automation. You get a different package name on every OS and a subprocess to babysit.

Staying Logged In to Proxmox: Ticket Renewal, Expiry and Where API Tokens Stop

Sun, 31 May 2026 20:24:00 -0700

Back in 2023 I wrote up how go-proxmox logs in and ended it with a shrug. Once a session existed the client never logged in again. A ticket-based client died at the two hour mark. My advice was “use an API token for anything long running”. It took until this month to fix that properly. It took three PRs from two of us to get there. Here’s the whole arc, plus the one place where an API token isn’t an option at all.

go-diskfs: Getting Rock Ridge Working for Joliet on Windows

Sun, 17 May 2026 20:40:00 -0700

go-proxmox has had a CloudInit() helper since 2023. It builds a small nocloud ISO in Go, uploads it to Proxmox and mounts it on the VM as a CD-ROM. It’s always worked fine for Linux guests. It has never worked for Windows guests running cloudbase-init. The reason is a filesystem extension from 1995 called Joliet.

Getting it fixed took three PRs to go-diskfs over about seven weeks. It also took a bug report from a user who ran it on Windows. That turned out to be the most important part.

Maintaining a Go Tool That Edits /etc/hosts on Every OS

Sat, 03 Feb 2024 19:22:00 -0700

Editing the hosts file sounds like the most boring problem in software. Read a text file, add a line, write it back. Then you find out your binary is being shelled out to from a Ruby Vagrant plugin on a Windows laptop with UAC prompts. Your library is being called in a loop by ddev. And every hosts file you parse was hand-edited by somebody at some point. I’ve been maintaining goodhosts for a few years now. I just spent a chunk of the fall and the holidays on it, so here’s what that looks like.

Proxmox Doesn't Know What Type a Number Is

Sat, 25 Nov 2023 20:05:00 -0700

Every couple of months somebody opens an issue on go-proxmox with a title that starts with json: cannot unmarshal string into Go struct field. The most recent one was fixed last week. It’s the same root cause every time. The Proxmox VE API is backed by Perl, and Perl doesn’t really care whether a scalar is a number or a string. The JSON encoder on the other end writes whatever the scalar happens to be at the time. So a field that is documented as an integer comes back as 2048 from one endpoint and "2048" from another. Sometimes it changes on the same endpoint depending on how the value got into the config.

Logging In to Proxmox From Go: Tickets, CSRF Tokens and API Tokens

Sun, 27 Aug 2023 19:18:00 -0700

I just spent a week of evenings on the access side of go-proxmox. Realms, groups, users and roles all landed. Before any of that, I reworked how a client gets its credentials in the first place. Proxmox gives you two different ways to authenticate. They behave differently enough that it’s worth writing down how each one works on the wire and what the client does with it.

Waiting on Proxmox: UPIDs, Tasks and WaitFor in Go

Sun, 20 Mar 2022 16:42:00 -0700

If you start automating Proxmox VE through the API, one thing trips you up first. Almost nothing you ask it to do is finished when the HTTP call returns. Start a VM, clone a template, download an ISO to storage, delete a disk… the response is a single string that looks like a typo. The actual work is happening in a worker process on the node. If your next call depends on that work being done, you have to go find out when it’s done.

Recreating Proxmox Ve Xtermjs in Go

Sun, 06 Feb 2022 21:54:53 -0700

Part of a hobby of mine is running a home lab and using on Proxmox VE to do the virtualization. Since I work in the space I’m slowly trying to build tooling around Proxmox that will let me use the home lab full time for work. Usually my work involves building Kubernetes management tooling. You can imagine waiting for public cloud APIs all day can take away hours of your day. That’s at cost to your employer for both your time and the compute resources.

Practical Cue Example, Produce YAML Config from Go Types

Mon, 31 Jan 2022 09:57:26 -0700

I recently had to spend some time to learn Cue and felt the examples/tutorials were a bit lacking. The paradigm shift to the entire language sort of messed with my head, and some things Cue does were a bit foreign to me. To get started, note that I spent all of my time getting this information from two places: their main docs and a community Cuetorials site. This will be a tutorial on how you can use Cue to generate two different YAML documents, all starting with Go types to generate Cue files.