Luther Monson, Founding Engineer at Gas City

Luther Monson

Open Source · AI · Containers · VMs · Kubernetes · Windows · Go by way of PHP

A Lock Is Only as Good as Its Release

SETNX gets all the attention, but most of the bugs in ePHPm's KV locks were on the release side. This is how blind deletes, a missing compare-and-delete and a cluster where two nodes could both win got fixed.

9-Minute Read

Timeline: request A's lock expires, B acquires it, A's blind DEL frees B's lock and C gets in

Every “build a lock on Redis” tutorial spends its time on acquisition: SET key value NX PX 30000, one winner, done. The release gets one line, DEL key. That line is where most of the lock bugs I’ve had in ePHPm lived. Over about three months the same mistake showed up in three places. First the native session handler, then two of the Composer wrapper packages, and finally the cluster. Each time it wore a slightly different costume.

Step Debugging a Worker That Never Ends

A persistent PHP worker never finishes its request, so Xdebug never sees your trigger. Here's how ePHPm routes the debugged request to its own fresh worker instead of trying to reset anything.

12-Minute Read

Diagram of the ePHPm router sending untriggered requests to the warm worker pool and Xdebug-triggered requests to a one-request debug lane

Persistent workers are the big performance win in modern PHP. You boot Laravel or Symfony once, keep the container in memory, and serve thousands of requests out of the same process. ePHPm has had a worker mode for a while. Like most worker runtimes it had a quiet hole in it. You couldn’t step debug a request with Xdebug. I wanted that fixed without patching Xdebug and without giving up the warm pool. The answer ended up being to stop fighting the constraint and route around it. The debug…

WebSockets in ePHPm, From Zero to a Working Chat

How to run WebSockets in ePHPm with a single PHP file. It covers turning them on, sharing the login session, pushing messages from normal HTTP requests, serving over TLS and the limits to know about.

14-Minute Read

Diagram: browsers hold sockets to ePHPm's Rust session tasks and registry; each connect, message and disconnect runs websocket.php once on the PHP thread pool, and ordinary HTTP requests push through ephpm_ws_broadcast

I covered the design of ePHPm’s native WebSockets in a few paragraphs of the odds and ends post. Rust owns the socket and PHP runs once per event. An idle connection never costs you a PHP thread. That was the why. This post is the how. We’ll build a small chat app with login, per-user push notifications from ordinary HTTP requests and a browser client. Then we’ll serve it over TLS. Last come the limits, the sharp edges and how to debug it when the browser only tells you 1006.

Webhooks Arrive Out of Order: How My Scheduler Killed Live Builds

ephemerd's orphan sweep trusted webhook history over reality and destroyed a runner mid-build. Now the rules only nominate, and a live busy check gets a veto.

9-Minute Read

Timeline of three same-label jobs where a runner executing a build is destroyed 2.3 seconds before its in_progress webhook arrives

The scheduler in ephemerd is event-driven. GitHub sends a workflow_job webhook when a job is queued, when it starts and when it completes. ephemerd creates and destroys runners to match. For a while that worked well enough that I stopped thinking about it. Then a docker-build job failed with “The self-hosted runner lost communication with the server”. I pulled the node’s journal. ephemerd had destroyed that runner itself, on purpose, while it was running the build.

Breaking Changes Without Breaking Trust: Five Years to go-proxmox v0.8

I measured five years of go-proxmox releases and found a broken tag that can never be deleted, a release that didn't compile and 285 breaking changes in v0.7.0. This is that history and the rule I follow now.

9-Minute Read

Horizontal bar chart of apidiff incompatible and compatible changes for every go-proxmox release from v0.1.0 to v0.8.2

go-proxmox had its first commit in June 2021 and v0.8.2 went out on September 30th. In between there were 366 commits and 43 people besides me with a merged PR. More to the point of this post, there are downstream projects that I don’t control pinning it in their go.mod. IONOS’s Cluster API provider and Sidero’s Omni Proxmox provider are both on v0.8.1 today. A library with users like that gets judged less on its API. It gets judged more on what happens to their build when they…

Windows Grab-Bag: SIGTERM That Doesn't Exist and Sparse Files That Break Docker

Go can't SIGTERM a child on Windows, so ephemerd's cloudflared processes piled up. Separately, Delivery Optimization's sparse temp files broke its Windows image layer commits.

7-Minute Read

Bar chart: 96 Cloudflare edge connections held by orphaned cloudflared processes versus 4 for the live daemon

Two short ones from the Windows node of ephemerd from the last few weeks. Neither is complicated once you know what’s going on. Both took longer than they should have to find, because the failure showed up a long way from the cause.

An Upstream PHP JIT Use-After-Free, and Why Our Default Flipped Twice

ePHPm turned PHP's tracing JIT on by default, then off three days later when stock Laravel died after three requests. The bug was upstream, and this is the bisect, the root cause and where the fix landed.

10-Minute Read

Bar chart of requests served before a crash per PHP version: 8.4.14, 8.5.0 and 8.5.4 never crash; 8.4.24 dies at 7, 8.5.5 at 1, 8.5.7 and 8.5.9 at 2

In August, ePHPm turned PHP’s tracing JIT on by default in single-site serve mode. Three days later it went back off on every platform, and it’s still off. The reason turned out to be a use-after-free in PHP’s tracing JIT that has nothing to do with ePHPm. Stock php -S with a stock Laravel skeleton dies on it in a couple of requests. The fix is now released upstream. So this seemed like the right time to write up how we got there, and what still has to happen before the default…

Your Agent's Shell Allowlist Is Bypassable

A permission gate for AI agents' shell commands judged spellings instead of commands. I found several classes of bypass in shipmates' own gate, and one idea fixes most of them.

9-Minute Read

A shell command decomposed into the nested command lines the permission gate has to judge

Every shipmates crew member runs shell commands. Before any of them runs, it goes through a permission gate. The gate uses the same rule syntax as Claude Code settings: Bash(git status), Bash(rm *), Write(**/.claude/settings.json). The rules are merged from several layers:

Cross-Platform Process Containment: Job Objects vs Process Groups vs cgroups

How shipmates bounds and tears down agent process trees on Windows, Linux and macOS. It also covers why I deleted 1,206 lines of cgroup code and the grandchild that escaped the memory cap everywhere except my dev box.

8-Minute Read

Agent process tree with the scope of kill and measurement on Windows, Linux and macOS

Every turn a shipmates crew member takes is a subprocess. It’s claude or the codex app-server. That in turn spawns shells, go test, npm or whatever the task calls for. Two things have to be true about that tree. When a turn ends or gets killed, the whole tree goes, including the grandchildren. And if somebody sets a memory or CPU budget, the budget applies to the whole tree, not just the process I happened to start.

Bind Mounts Across Mount Namespaces: Four PRs and a Container Escape

Making GitHub Actions container: jobs work through ephemerd's Docker API shim took four tries, then a TOCTOU race turned the working version into an escape.

8-Minute Read

Sequence diagram of the bind mount TOCTOU race and the openat2 plus staging fix

GitHub Actions has a container: key that runs a job’s steps inside an image of your choosing. It’s popular because it gives you a reproducible toolchain without building a custom runner image. On a self-hosted runner it works by having the runner drive Docker. On ephemerd, “Docker” is a shim that speaks the Docker API and creates containers in containerd. Getting bind mounts right through that shim took four PRs in a week. A couple of months later the version that worked…

Recent Posts

Categories

About

Luther is a 20+ year software veteran specializing in backend APIs and architecting products. Founding engineer at Gas City building open-source AI software factories on Beads, the durable work graph for coding agents.