Luther Monson, Staff Software Engineer at SUSE

Luther Monson

Go by way of PHP · Containers · VMs · Windows · Kubernetes

Making ephpm php Byte-for-Byte Indistinguishable From php-cli

A differential test harness against upstream php, and the bugs it found in an embedded PHP CLI on day one.

9-Minute Read

Bar chart of conformance corpus results: 65 pass and 10 xfail when the harness landed, 79 pass and 1 xfail after the fixes

ePHPm embeds PHP into a single Rust binary. The obvious follow-up question from anybody deploying it is “do I still need a php on the box?” Composer, WP-CLI, artisan, PHPUnit and every build script in the world call php. If ephpm php can stand in for it then the answer is no. The single binary really means a single binary. “Mostly works” isn’t good enough for that though. These tools parse PHP’s output, check its exit codes and re-invoke it. So the goal was…

P99 From 280 ms to 58 ms by Making the Queue Fair

ePHPm had the best throughput in a Laravel runtime comparison and the worst tail of the fast group. The cause was a bounded channel that let new requests cut in line.

9-Minute Read

Bar chart of P50, P75, P90 and P99 latency for FIFO versus barging admission in ePHPm worker mode

A third-party Laravel runtime comparison put ePHPm’s worker mode at the top on throughput. It was about 19% ahead of FrankenPHP on /api/static. It was also at the bottom of the fast group on tail latency, with a P99 of 157 ms against FrankenPHP’s 57 ms. Best throughput plus the fattest tail is a shape worth chasing, because fast execution can’t produce it. Making requests faster lowers the mean and the tail. Only waiting can pull them apart. Specifically, unfair waiting.

We Disabled PHP's Stack Guard Ourselves: Containing Segfaults in a Single-Process PHP Server

How ePHPm went from dying silently on a segfault to containing stack overflows per thread. It also covers the linker flag that had turned off PHP's own guard.

12-Minute Read

Diagram of a PHP thread stack, the guard window below it, and which faults get contained versus which kill the process

php-fpm has a property nobody thinks about until they leave it. When a worker segfaults, you lose one request. The master forks a new child and life goes on. ePHPm runs every tenant’s PHP as threads inside one process. That’s where the memory and latency wins come from. It also means one tenant’s segfault takes down every other tenant on the box.

Stop Running PHP on tokio's spawn_blocking - Own Your Thread Pool

Why ePHPm moved PHP execution off tokio's generic blocking pool onto threads it owns. How the switch rolled out behind a knob and what it did to memory, throughput and overload.

10-Minute Read

Bar chart of server RSS at 32, 128 and 512 concurrent clients: spawn_blocking grows with concurrency, the owned pool stays flat

For most of its life, ePHPm ran every PHP request on tokio’s spawn_blocking pool. It works. It passes tests and it serves benchmarks fine. It’s the first thing anybody reaches for when they need to call blocking code from async Rust. It’s also quietly wrong for a language runtime. The problems only show up under overload or after a crash. Over the last three weeks it got replaced with a pool of OS threads that ePHPm owns. That pool shipped first behind a config flag, then as…

Clustered ACME in Production: Three Bugs That Took TLS Down

Moving wildcard TLS onto a live 3-node ePHPm cluster found a leader election that wasn't, a certificate only two nodes could see, and an HTTP port speaking TLS.

10-Minute Read

Diagram comparing a sharded KV write that leaves one of three nodes without the certificate to a broadcast write that reaches all three

ePHPm gets its own certificates from Let’s Encrypt. In a cluster only one node should be talking to the CA. The nodes elect an ACME leader over the shared KV store. The leader orders the cert, and everybody else installs it from KV. That all had unit tests and had worked on single nodes for months. Then I tried to move wildcard TLS termination for our preview environment off a separate proxy and onto the 3-node cluster itself. In about two days it found three separate bugs that each took…

Middleware in PHP: Compiling It With elephc, Then Not Needing To

Fitting PHP into ePHPm's native middleware ABI with an ahead-of-time compiler, and what that experiment said about where PHP middleware should actually run.

9-Minute Read

Timeline of an ePHPm request showing the native middleware lane before the body read and php: mounts inside the PHP request

ePHPm middleware runs before PHP. It’s a small versioned C ABI, loaded with dlopen, that gets to look at a request and reject it before PHP boots or the request body is read. That’s the right shape for a gateway. It’s the wrong shape for most PHP shops, who want to write their middleware in PHP. This is the story of compiling PHP into a native .so to fit that ABI and finding exactly where it broke. Then I built the lane that should have existed in the first place.

Per-Tenant Network Policy in One Process: ePHPm's eBPF Thread Tags

Every site on an ePHPm server runs inside the same process, so a normal firewall can't tell them apart. A small eBPF program tags each thread with its site. It keeps sites from reaching each other's local ports for under a microsecond per connection.

17-Minute Read

Flow from a PHP pool thread writing its vhost tag into a BPF map, through the bind and connect cgroup hooks, to allow, rewrite or EPERM

In multi-tenant mode ePHPm serves every site in sites_dir from one process. Every tenant’s PHP runs as the same OS user on the same pool of ZTS threads. A thread that just served blog.example will serve shop.example a few milliseconds later. That’s the density win. It also means the kernel has no idea which tenant is behind a given connect(). As far as Linux is concerned it’s all ephpm-web.

Security and Performance Hardening of a Rust Database Proxy

What a week of hardening litewire turned up. A dependency had a memory bug, one site could open its neighbor's database, a security rule lived only in a comment and every query paid for a needless 1 MB clear.

11-Minute Read

Bar chart of point SELECT throughput over the litewire MySQL frontend at three revisions: before the command filter, with the filter, and with the poll_read fix

litewire is the piece of ePHPm that lets PHP apps talk MySQL to a SQLite file. I wrote about how it works and about the MySQL quirks WordPress depends on. This one is the less visible half of August:

What an Overloaded PHP Server Actually Returns

Closed-loop benchmarks can't show you overload. An open-loop load generator with an error taxonomy showed ePHPm returned nothing at all, and then verified the fix.

11-Minute Read

Bar chart of what 48,001 arrivals at 800 req/s got back: before the fixes almost everything was a client timeout, after them every arrival got a 200 or a fast 503

Ask most people what their PHP server does when it’s overloaded and you’ll hear “it returns 503s.” Ask them how they know and it usually comes down to a config setting they read about. I was in that camp for ePHPm. We had a [server.limits] max_connections knob, and the docs said it returned 503 at the cap. Every benchmark we had ever run reported zero errors under heavy load. Then I wrote a load generator that could produce overload. The honest answer was that an…

Clustered SQLite in ePHPm, Part 2: Adding Turso

How I decided a beta database engine was ready by testing it instead of reading its version number. ePHPm v0.7.0 replaces both rusqlite and sqld with Turso.

9-Minute Read

Bar chart of four SQLite lanes at 16 concurrent connections: rusqlite and Turso single node, sqld and Turso CDC clustered

Part 1 ended with ePHPm running two SQLite engines. rusqlite ran in-process for a single node, and an embedded sqld child process ran for a cluster. The sidecar fell off a cliff past four concurrent writers. The best fix I could ship was a semaphore set to 1. This post is about replacing both engines with one, and about how I decided that engine was ready.

Recent Posts

Categories

About

Luther is a 20+ year software veteran specializing in backend APIs and architecting products. Employed at SUSE as the Team Lead for the Containers on Windows Team working on Rancher + Kubernetes