Luther Monson, Staff Software Engineer at SUSE

Luther Monson

Go by way of PHP · Containers · VMs · Windows · Kubernetes

Breaking Changes Without Breaking Trust: Five Years to go-proxmox v0.8

I measured five years of go-proxmox releases and found a broken tag that can never be deleted, a release that didn't compile and 285 breaking changes in v0.7.0. This is that history and the rule I follow now.

9-Minute Read

Horizontal bar chart of apidiff incompatible and compatible changes for every go-proxmox release from v0.1.0 to v0.8.2

go-proxmox had its first commit in June 2021 and v0.8.2 went out on September 30th. In between there were 366 commits and 43 people besides me with a merged PR. More to the point of this post, there are downstream projects that I don’t control pinning it in their go.mod. IONOS’s Cluster API provider and Sidero’s Omni Proxmox provider are both on v0.8.1 today. A library with users like that gets judged less on its API. It gets judged more on what happens to their build when they…

Five Ways to URL-Encode an SSH Key (and Why Proxmox Rejects Them)

Proxmox's sshkeys parameter wants Python-style percent encoding that no single Go function produces. This covers a two-year-old issue, its one-line fix, and a related double-encoding bug in guest file writes.

6-Minute Read

Table of seven ways to encode an SSH public key and whether each passes Proxmox's urlencoded validator

Issue #144 on go-proxmox was opened in April 2024 with the title “Weird sshkeys SSH public key validation error”. It is one of my less proud threads. It’s thirteen comments of people, including me, trying to get a public key into a cloud-init VM through the API. The working answer took more than a year to show up. The fix that finally landed in May is one line. Here’s why it took so long.

Two Small go-proxmox Fixes: A Watch Channel That Never Closed, and WithRetry

go-proxmox had a task log channel that only closed on the happy path. It also gained a composable RoundTripper that retries the 502s and 503s Proxmox throws during cluster transitions.

6-Minute Read

Proxmox VE Logo

Here are two changes from the v0.5 to v0.7 stretch of go-proxmox. They’re too small for a post each. They’re worth knowing about if you write long-running tooling against Proxmox. One is a bug in code I wrote about in 2022. The other is a feature that replaces a retry loop everybody was writing for themselves.

Maintaining an Open-Source Library With AI Agents: AGENTS.md and the Rule They Kept Breaking

Six weeks of go-proxmox with coding agents doing most of the typing taught me what belongs in AGENTS.md. Five parallel agents got the same API shape wrong, and a coverage metric bent the code around it.

7-Minute Read

Decision diagram: endpoints without an id in the path become methods on the parent, endpoints with an id get a handle type

On May 9th I merged #264. It added an AGENTS.md to go-proxmox and a stub CLAUDE.md that just points at it. The next six weeks ran through the v0.8.0 migration guide on June 15th. In that time main took 77 commits and +31,794 / -1,144 lines. The repo went from 84 to 186 .go files. Endpoint coverage went from 30% to 100% of the Proxmox API. Most of that code was typed by coding agents, a lot of it by several at once. This is what that looked like from the maintainer’s chair. It’s…

Proxmox JSON, Part 2: null PIDs, 1.12589990684262e+15 and net10

The Proxmox VE API found three more ways for its JSON not to fit Go's encoding/json. There's a null I fixed and then broke, petabyte storage in scientific notation, and devices past index 9 that were silently dropped.

7-Minute Read

Proxmox VE Logo

In 2023 I wrote about why go-proxmox has StringOrInt and friends. Proxmox sends the same number quoted in one place and bare in another, so the client has types that accept both. That post ended on a pretty confident note. Since then three more shapes of “the JSON is valid and Go still won’t take it” have come in. One of them is a bug I’d already fixed once and then reintroduced myself.

Your Zero Value Just Turned Off My TLS Verification: omitempty, Go Defaults and Proxmox

A plain Go field can't tell unset from false. In go-proxmox the fix for one omitempty bug created the opposite bug, until an audit against the Proxmox schema found 75 fields and v0.7.0 moved them to pointers.

7-Minute Read

Table showing how verify-certificates was sent by go-proxmox before #210, between v0.2.3 and v0.6.x, and in v0.7.0

Every Go developer knows omitempty drops zero values. What takes longer to sink in is that a plain bool only has two states. A caller filling in an options struct means one of three:

From 52% to 100%: Measuring a Go Client Against the Proxmox API Schema

I built a stdlib-only mage target that pulls Proxmox's own API schema and regex-scans go-proxmox for HTTP call sites to report coverage. The tool was wrong more often than the code was.

8-Minute Read

Line chart of go-proxmox endpoint coverage from 4% in 2021 to 100% in May 2026

For five years the honest answer to “how much of the Proxmox API does go-proxmox cover?” was “a lot of it? the parts people asked for?”. Issues would come in for endpoints that didn’t exist in the client. I’d have no idea how many more were missing. At the end of May I finally built the thing that answers the question. Three days after the first run the answer was 100%. The more interesting part is how many times the measuring stick was wrong on the way there.

Staying Logged In to Proxmox: Ticket Renewal, Expiry and Where API Tokens Stop

Proxmox tickets die after two hours. How go-proxmox went from never re-authenticating to renewing tickets in place, plus the one thing API tokens can't do.

7-Minute Read

Timeline of a Proxmox ticket: valid for two hours server side, considered stale by the client after one

Back in 2023 I wrote up how go-proxmox logs in and ended it with a shrug. Once a session existed the client never logged in again. A ticket-based client died at the two hour mark. My advice was “use an API token for anything long running”. It took until this month to fix that properly. It took three PRs from two of us to get there. Here’s the whole arc, plus the one place where an API token isn’t an option at all.

Proxmox Doesn't Know What Type a Number Is

The Proxmox VE API returns the same field as 1234 in one place and "1234" in another. This is why go-proxmox has StringOrInt, StringOrUint64, StringOrFloat64 and IntOrBool instead of plain Go types.

5-Minute Read

Proxmox VE Logo

Every couple of months somebody opens an issue on go-proxmox with a title that starts with json: cannot unmarshal string into Go struct field. The most recent one was fixed last week. It’s the same root cause every time. The Proxmox VE API is backed by Perl, and Perl doesn’t really care whether a scalar is a number or a string. The JSON encoder on the other end writes whatever the scalar happens to be at the time. So a field that is documented as an integer comes back as 2048 from…

Logging In to Proxmox From Go: Tickets, CSRF Tokens and API Tokens

Proxmox VE has two ways to authenticate against its API. Here's how go-proxmox handles each one and why the client is now configured entirely with functional options.

5-Minute Read

Diagram of the ticket login flow and the API token header for the Proxmox VE API

I just spent a week of evenings on the access side of go-proxmox. Realms, groups, users and roles all landed. Before any of that, I reworked how a client gets its credentials in the first place. Proxmox gives you two different ways to authenticate. They behave differently enough that it’s worth writing down how each one works on the wire and what the client does with it.

Recent Posts

Categories

About

Luther is a 20+ year software veteran specializing in backend APIs and architecting products. Employed at SUSE as the Team Lead for the Containers on Windows Team working on Rancher + Kubernetes