Luther Monson, Staff Software Engineer at SUSE

Luther Monson

Go by way of PHP · Containers · VMs · Windows · Kubernetes

Breaking Changes Without Breaking Trust: Five Years to go-proxmox v0.8

I measured five years of go-proxmox releases and found a broken tag that can never be deleted, a release that didn't compile and 285 breaking changes in v0.7.0. This is that history and the rule I follow now.

9-Minute Read

Horizontal bar chart of apidiff incompatible and compatible changes for every go-proxmox release from v0.1.0 to v0.8.2

go-proxmox had its first commit in June 2021 and v0.8.2 went out on September 30th. In between there were 366 commits and 43 people besides me with a merged PR. More to the point of this post, there are downstream projects that I don’t control pinning it in their go.mod. IONOS’s Cluster API provider and Sidero’s Omni Proxmox provider are both on v0.8.1 today. A library with users like that gets judged less on its API. It gets judged more on what happens to their build when they…

Rescuing a Windows Dynamic Disk Over USB with WSL2 and USB/IP

Windows refused to bring an old dynamic disk online over USB. Handing the raw USB device to WSL2's Linux kernel with usbipd-win got every file back.

10-Minute Read

GPT layout of the 1 TB drive with LDM metadata and a 930 GiB LDM data partition, and the two paths: the Windows storage stack refusing it and usbipd passing it to WSL2

I pulled a 1 TB NVMe out of an old Windows box, dropped it in a USB enclosure and plugged it in. Windows showed nothing. No drive letter, no “initialize this disk” prompt, nothing in Explorer. Disk Management listed it as Invalid. The enclosure’s booklet said to “bring the disk online”, which silently did nothing. The data was completely intact. Windows was refusing on policy grounds, not because anything was broken. Getting it back meant going around the Windows…

Five Ways to URL-Encode an SSH Key (and Why Proxmox Rejects Them)

Proxmox's sshkeys parameter wants Python-style percent encoding that no single Go function produces. This covers a two-year-old issue, its one-line fix, and a related double-encoding bug in guest file writes.

6-Minute Read

Table of seven ways to encode an SSH public key and whether each passes Proxmox's urlencoded validator

Issue #144 on go-proxmox was opened in April 2024 with the title “Weird sshkeys SSH public key validation error”. It is one of my less proud threads. It’s thirteen comments of people, including me, trying to get a public key into a cloud-init VM through the API. The working answer took more than a year to show up. The fix that finally landed in May is one line. Here’s why it took so long.

Two Small go-proxmox Fixes: A Watch Channel That Never Closed, and WithRetry

go-proxmox had a task log channel that only closed on the happy path. It also gained a composable RoundTripper that retries the 502s and 503s Proxmox throws during cluster transitions.

6-Minute Read

Proxmox VE Logo

Here are two changes from the v0.5 to v0.7 stretch of go-proxmox. They’re too small for a post each. They’re worth knowing about if you write long-running tooling against Proxmox. One is a bug in code I wrote about in 2022. The other is a feature that replaces a retry loop everybody was writing for themselves.

Maintaining an Open-Source Library With AI Agents: AGENTS.md and the Rule They Kept Breaking

Six weeks of go-proxmox with coding agents doing most of the typing taught me what belongs in AGENTS.md. Five parallel agents got the same API shape wrong, and a coverage metric bent the code around it.

7-Minute Read

Decision diagram: endpoints without an id in the path become methods on the parent, endpoints with an id get a handle type

On May 9th I merged #264. It added an AGENTS.md to go-proxmox and a stub CLAUDE.md that just points at it. The next six weeks ran through the v0.8.0 migration guide on June 15th. In that time main took 77 commits and +31,794 / -1,144 lines. The repo went from 84 to 186 .go files. Endpoint coverage went from 30% to 100% of the Proxmox API. Most of that code was typed by coding agents, a lot of it by several at once. This is what that looked like from the maintainer’s chair. It’s…

Proxmox JSON, Part 2: null PIDs, 1.12589990684262e+15 and net10

The Proxmox VE API found three more ways for its JSON not to fit Go's encoding/json. There's a null I fixed and then broke, petabyte storage in scientific notation, and devices past index 9 that were silently dropped.

7-Minute Read

Proxmox VE Logo

In 2023 I wrote about why go-proxmox has StringOrInt and friends. Proxmox sends the same number quoted in one place and bare in another, so the client has types that accept both. That post ended on a pretty confident note. Since then three more shapes of “the JSON is valid and Go still won’t take it” have come in. One of them is a bug I’d already fixed once and then reintroduced myself.

Your Zero Value Just Turned Off My TLS Verification: omitempty, Go Defaults and Proxmox

A plain Go field can't tell unset from false. In go-proxmox the fix for one omitempty bug created the opposite bug, until an audit against the Proxmox schema found 75 fields and v0.7.0 moved them to pointers.

7-Minute Read

Table showing how verify-certificates was sent by go-proxmox before #210, between v0.2.3 and v0.6.x, and in v0.7.0

Every Go developer knows omitempty drops zero values. What takes longer to sink in is that a plain bool only has two states. A caller filling in an options struct means one of three:

From 52% to 100%: Measuring a Go Client Against the Proxmox API Schema

I built a stdlib-only mage target that pulls Proxmox's own API schema and regex-scans go-proxmox for HTTP call sites to report coverage. The tool was wrong more often than the code was.

8-Minute Read

Line chart of go-proxmox endpoint coverage from 4% in 2021 to 100% in May 2026

For five years the honest answer to “how much of the Proxmox API does go-proxmox cover?” was “a lot of it? the parts people asked for?”. Issues would come in for endpoints that didn’t exist in the client. I’d have no idea how many more were missing. At the end of May I finally built the thing that answers the question. Three days after the first run the answer was 100%. The more interesting part is how many times the measuring stick was wrong on the way there.

Cloud-Init ISOs in Pure Go with go-proxmox

Building nocloud cloud-init ISOs in memory with go-diskfs, uploading them to Proxmox and mounting them on a VM in one call, with no genisoimage.

8-Minute Read

Flow: makeCloudInitISO, upload to Proxmox storage, mount as CD-ROM, VM boots and cloud-init reads the cidata volume, cleanup

Proxmox’s built-in cloud-init support is fine for the basics: a user, an SSH key, an IP config. It falls apart the second you want your own user-data, vendor-data or a real network-config. The GUI and the ci* options only expose a handful of fields. The escape hatch is cicustom pointing at snippet files, and that means getting files onto the Proxmox host’s filesystem first. The usual alternative is building a nocloud ISO yourself with genisoimage or mkisofs. That means a system…

Staying Logged In to Proxmox: Ticket Renewal, Expiry and Where API Tokens Stop

Proxmox tickets die after two hours. How go-proxmox went from never re-authenticating to renewing tickets in place, plus the one thing API tokens can't do.

7-Minute Read

Timeline of a Proxmox ticket: valid for two hours server side, considered stale by the client after one

Back in 2023 I wrote up how go-proxmox logs in and ended it with a shrug. Once a session existed the client never logged in again. A ticket-based client died at the two hour mark. My advice was “use an API token for anything long running”. It took until this month to fix that properly. It took three PRs from two of us to get there. Here’s the whole arc, plus the one place where an API token isn’t an option at all.

Recent Posts

Categories

About

Luther is a 20+ year software veteran specializing in backend APIs and architecting products. Employed at SUSE as the Team Lead for the Containers on Windows Team working on Rancher + Kubernetes