Luther Monson, Founding Engineer at Gas City

Luther Monson

Open Source · AI · Containers · VMs · Kubernetes · Windows · Go by way of PHP

Building k9s for Linode With Bubble Tea

A k9s-style terminal app for managing Linode. Search for any resource, browse it, and jump straight into k9s, a database shell or SSH.

8-Minute Read

linode-tui showing the Linodes view with the command bar open and tab completion suggesting lke

A lot of my day is spent in k9s. Sometimes I need something from the Linode side of the house. Maybe it’s a node that won’t boot, a volume nobody cleaned up or the kubeconfig for a cluster I just made. Then I’m back in a browser tab or typing linode-cli incantations. I wanted the k9s workflow for the Linode API itself:

P99 From 280 ms to 58 ms by Making the Queue Fair

ePHPm had the best throughput in a Laravel runtime comparison and the worst tail of the fast group. The cause was a bounded channel that let new requests cut in line.

9-Minute Read

Bar chart of P50, P75, P90 and P99 latency for FIFO versus barging admission in ePHPm worker mode

A third-party Laravel runtime comparison put ePHPm’s worker mode at the top on throughput. It was about 19% ahead of FrankenPHP on /api/static. It was also at the bottom of the fast group on tail latency, with a P99 of 157 ms against FrankenPHP’s 57 ms. Best throughput plus the fattest tail is a shape worth chasing, because fast execution can’t produce it. Making requests faster lowers the mean and the tail. Only waiting can pull them apart. Specifically, unfair waiting.

We Disabled PHP's Stack Guard Ourselves: Containing Segfaults in a Single-Process PHP Server

How ePHPm went from dying silently on a segfault to containing stack overflows per thread. It also covers the linker flag that had turned off PHP's own guard.

12-Minute Read

Diagram of a PHP thread stack, the guard window below it, and which faults get contained versus which kill the process

php-fpm has a property nobody thinks about until they leave it. When a worker segfaults, you lose one request. The master forks a new child and life goes on. ePHPm runs every tenant’s PHP as threads inside one process. That’s where the memory and latency wins come from. It also means one tenant’s segfault takes down every other tenant on the box.

Stop Running PHP on tokio's spawn_blocking - Own Your Thread Pool

Why ePHPm moved PHP execution off tokio's generic blocking pool onto threads it owns. How the switch rolled out behind a knob and what it did to memory, throughput and overload.

10-Minute Read

Bar chart of server RSS at 32, 128 and 512 concurrent clients: spawn_blocking grows with concurrency, the owned pool stays flat

For most of its life, ePHPm ran every PHP request on tokio’s spawn_blocking pool. It works. It passes tests and it serves benchmarks fine. It’s the first thing anybody reaches for when they need to call blocking code from async Rust. It’s also quietly wrong for a language runtime. The problems only show up under overload or after a crash. Over the last three weeks it got replaced with a pool of OS threads that ePHPm owns. That pool shipped first behind a config flag, then as…

Kata Containers for CI: Boot Cost Isn't the Problem, File I/O Is

Kata containers are safer for CI, but slower. Starting one takes 12x longer, which is fine. Reading small files is about 40x slower for the entire job, which isn't.

7-Minute Read

Bar chart of Kata slowdown versus runc for each measurement

Of the three platforms ephemerd runs jobs on, Linux has the weakest isolation. Windows jobs get a Hyper-V isolated container with its own kernel. macOS jobs get a whole VM. A Linux job is an ordinary runc container sharing the host kernel. A kernel escape from a Linux job is a host compromise. Kata Containers fixes that by putting each container in a lightweight VM with its own kernel. It plugs into containerd as just another runtime handler. So I added it in #155, measured it, and left it off…

Clustered ACME in Production: Three Bugs That Took TLS Down

Moving wildcard TLS onto a live 3-node ePHPm cluster found a leader election that wasn't, a certificate only two nodes could see, and an HTTP port speaking TLS.

10-Minute Read

Diagram comparing a sharded KV write that leaves one of three nodes without the certificate to a broadcast write that reaches all three

ePHPm gets its own certificates from Let’s Encrypt. In a cluster only one node should be talking to the CA. The nodes elect an ACME leader over the shared KV store. The leader orders the cert, and everybody else installs it from KV. That all had unit tests and had worked on single nodes for months. Then I tried to move wildcard TLS termination for our preview environment off a separate proxy and onto the 3-node cluster itself. In about two days it found three separate bugs that each took…

Finding a Data Race in rancher/remotedialer, Then Pinning Only Two-Thirds of the Fix

Two data races in remotedialer v0.6.1 that upstream's tests never ran with -race, and how my replace directive ended up pinned to the middle commit of my own fix.

7-Minute Read

Three commits of remotedialer PR 265 and which one the replace directive pointed at

shipmates ships connect to Fleet Command over outbound websocket tunnels. That way nothing on the ship side has to be exposed to the internet. I didn’t write that tunnel layer. I used rancher/remotedialer, the same library Rancher uses to reach downstream cluster agents. I spent years around that code at Rancher, so it was the obvious pick. What I didn’t expect was for my CI’s race job to find two data races in it in the first week.

Middleware in PHP: Compiling It With elephc, Then Not Needing To

Fitting PHP into ePHPm's native middleware ABI with an ahead-of-time compiler, and what that experiment said about where PHP middleware should actually run.

9-Minute Read

Timeline of an ePHPm request showing the native middleware lane before the body read and php: mounts inside the PHP request

ePHPm middleware runs before PHP. It’s a small versioned C ABI, loaded with dlopen, that gets to look at a request and reject it before PHP boots or the request body is read. That’s the right shape for a gateway. It’s the wrong shape for most PHP shops, who want to write their middleware in PHP. This is the story of compiling PHP into a native .so to fit that ABI and finding exactly where it broke. Then I built the lane that should have existed in the first place.

OPcache on Windows: Two Silent Failures

An ASLR fatal on Windows looked like a memory-size lottery. Separately, Windows PHP 8.3 builds never had an opcode cache, and the fix sat unpicked for three weeks.

8-Minute Read

Two ePHPm processes computing the same OPcache shared-memory name; the second reattaches and dies on the ASLR check

Windows is the platform where ePHPm gets the least traffic and the most surprises. Last week closed out two OPcache bugs that only exist there. Both were quiet in their own way. One killed the server at startup with nothing in our own logs. The other didn’t kill anything. It just ran PHP 8.3 with no opcode cache at all. For three weeks after a fixed SDK was published, every check stayed green.

Recent Posts

Categories

About

Luther is a 20+ year software veteran specializing in backend APIs and architecting products. Founding engineer at Gas City building open-source AI software factories on Beads, the durable work graph for coding agents.