Luther Monson, Founding Engineer at Gas City

Luther Monson

Open Source · AI · Containers · VMs · Kubernetes · Windows · Go by way of PHP

Per-Tenant Network Policy in One Process: ePHPm's eBPF Thread Tags

Every site on an ePHPm server runs inside the same process, so a normal firewall can't tell them apart. A small eBPF program tags each thread with its site. It keeps sites from reaching each other's local ports for under a microsecond per connection.

17-Minute Read

Flow from a PHP pool thread writing its vhost tag into a BPF map, through the bind and connect cgroup hooks, to allow, rewrite or EPERM

In multi-tenant mode ePHPm serves every site in sites_dir from one process. Every tenant’s PHP runs as the same OS user on the same pool of ZTS threads. A thread that just served blog.example will serve shop.example a few milliseconds later. That’s the density win. It also means the kernel has no idea which tenant is behind a given connect(). As far as Linux is concerned it’s all ephpm-web.

Security and Performance Hardening of a Rust Database Proxy

What a week of hardening litewire turned up. A dependency had a memory bug, one site could open its neighbor's database, a security rule lived only in a comment and every query paid for a needless 1 MB clear.

11-Minute Read

Bar chart of point SELECT throughput over the litewire MySQL frontend at three revisions: before the command filter, with the filter, and with the poll_read fix

litewire is the piece of ePHPm that lets PHP apps talk MySQL to a SQLite file. I wrote about how it works and about the MySQL quirks WordPress depends on. This one is the less visible half of August:

My iptables Rule Matched 0 Packets in 9.8 GB

Container-to-host traffic goes through INPUT, not FORWARD, so ephemerd's carefully written allow-list never matched a single packet.

8-Minute Read

Netfilter flow showing container traffic to the host taking INPUT instead of FORWARD

After a week of Windows firewall rules that listed fine and enforced nothing, I went back to the Linux side of ephemerd feeling pretty good about it. iptables is the boring, well-understood one. Then I looked at the packet counters on a live node. I found a rule that had never matched a packet in its life. That node had pushed 9.8 GB through the same chain.

What Does One More WordPress Site Cost?

I measured what each extra WordPress tenant on one ePHPm instance costs in RAM, file descriptors, CPU and disk. The sweep ran from 10 to 1000 sites.

9-Minute Read

Line chart of ePHPm steady RSS against tenant site count: linear at about 2.2 MB per site with every site open, flat past the default LRU cap of 256

ePHPm v0.7.0 shipped per-site multi-tenancy. Point [server] sites_dir at a directory and every Host maps to its own docroot and its own Turso database file. One process serves all of them. The first question anybody asks about that is “how many sites fit on a box?” The honest way to answer it is to turn the question around. What does one more site cost? If you know the marginal RAM, file descriptors, CPU and disk for a tenant, the box size falls out of arithmetic.

What an Overloaded PHP Server Actually Returns

Closed-loop benchmarks can't show you overload. An open-loop load generator with an error taxonomy showed ePHPm returned nothing at all, and then verified the fix.

11-Minute Read

Bar chart of what 48,001 arrivals at 800 req/s got back: before the fixes almost everything was a client timeout, after them every arrival got a 200 or a fast 503

Ask most people what their PHP server does when it’s overloaded and you’ll hear “it returns 503s.” Ask them how they know and it usually comes down to a config setting they read about. I was in that camp for ePHPm. We had a [server.limits] max_connections knob, and the docs said it returned 503 at the cap. Every benchmark we had ever run reported zero errors under heavy load. Then I wrote a load generator that could produce overload. The honest answer was that an…

Five Ways to Not Firewall a Windows Container (and the One That Works)

Five different Windows firewall approaches failed to stop a container from reaching my home network. The sixth, a different network mode for the container, finally worked.

10-Minute Read

Packet path of a Windows container on an HNS NAT network versus an L2Bridge network

ephemerd runs CI jobs from pull requests. One of the rules is that a job can reach the internet but not my LAN. On Linux that’s a few iptables rules in the FORWARD path, and I didn’t think about it much. On Windows I spent about a week finding out how many ways there are to write a firewall rule that Windows accepts, lists and never applies.

shipmates: Persona-Based AI Coding Crews That Remember

shipmates is a Go CLI and daemon that runs crews of AI coding agents across machines. It adds per-project memory, a shared task graph, voice control and a real permission model.

7-Minute Read

shipmates hierarchy: Admiral, Fleet Command, ships, captains and mates

Since June most of my evenings have gone into shipmates, a Go CLI and daemon for running crews of AI coding agents. I tagged v0.5.0 on August 17th. That was two months and about 240 commits after the first architecture doc. So it’s a good time to write down what it is and why it exists.

The Fast PHP Interpreter Windows Never Got

Windows PHP being slow is mostly a missing-implementation story, not an OS fact. Here is why ePHPm ships a separate clang-cl build with the TAILCALL interpreter, and what it proves.

9-Minute Read

Bar chart of upstream bench.php run time for three Zend dispatch kinds on one machine: CALL (MSVC, the slow fallback) at 1.783s, HYBRID (GCC, default since 7.2) at 1.006s, and TAILCALL (Clang, new in PHP 8.5) at 1.017s, showing TAILCALL recovers almost all of HYBRID's advantage over CALL

“PHP is slow on Windows” is one of those things everyone knows and nobody questions. I believed it too, right up until I went looking for where the slowness actually lives. Part of it is real and unfixable. But a big chunk of the interpreter half turns out to be something much more boring: nobody ever wrote the fast dispatch loop for the Windows compiler. That is not the same claim as “the OS is slower.” It is a missing implementation.

My Benchmark Was 4x Wrong Because of WSL

Launching the server with its working directory on /mnt/c cost ePHPm about 4x on a light workload. The report built a wrong explanation on top of it, and the strace shows why.

8-Minute Read

Bar chart of ePHPm requests per second by working directory: native ext4 far ahead of DrvFs, with DrvFs throughput falling as the cwd path gets deeper

I do most of my benchmarking on a Windows box in WSL2. It’s the machine on my desk with 32 threads and 62 GB of RAM. Everybody who does that learns the first rule early: don’t put your data on /mnt/c. DrvFs is the 9p-backed mount that exposes the Windows drives inside WSL, and it’s slow for file-heavy work. So the databases, the docroot and the build output all live on the WSL ext4 disk. My scaling harness for ePHPm even says so in capital letters in its README.

Clustered SQLite in ePHPm, Part 2: Adding Turso

How I decided a beta database engine was ready by testing it instead of reading its version number. ePHPm v0.7.0 replaces both rusqlite and sqld with Turso.

9-Minute Read

Bar chart of four SQLite lanes at 16 concurrent connections: rusqlite and Turso single node, sqld and Turso CDC clustered

Part 1 ended with ePHPm running two SQLite engines. rusqlite ran in-process for a single node, and an embedded sqld child process ran for a cluster. The sidecar fell off a cliff past four concurrent writers. The best fix I could ship was a semaphore set to 1. This post is about replacing both engines with one, and about how I decided that engine was ready.

Recent Posts

Categories

About

Luther is a 20+ year software veteran specializing in backend APIs and architecting products. Founding engineer at Gas City building open-source AI software factories on Beads, the durable work graph for coding agents.