Luther Monson, Staff Software Engineer at SUSE

Luther Monson

Go by way of PHP · Containers · VMs · Windows · Kubernetes

Per-Tenant Network Policy in One Process: ePHPm's eBPF Thread Tags

Every site on an ePHPm server runs inside the same process, so a normal firewall can't tell them apart. A small eBPF program tags each thread with its site. It keeps sites from reaching each other's local ports for under a microsecond per connection.

17-Minute Read

Flow from a PHP pool thread writing its vhost tag into a BPF map, through the bind and connect cgroup hooks, to allow, rewrite or EPERM

In multi-tenant mode ePHPm serves every site in sites_dir from one process. Every tenant’s PHP runs as the same OS user on the same pool of ZTS threads. A thread that just served blog.example will serve shop.example a few milliseconds later. That’s the density win. It also means the kernel has no idea which tenant is behind a given connect(). As far as Linux is concerned it’s all ephpm-web.

What Does One More WordPress Site Cost?

I measured what each extra WordPress tenant on one ePHPm instance costs in RAM, file descriptors, CPU and disk. The sweep ran from 10 to 1000 sites.

9-Minute Read

Line chart of ePHPm steady RSS against tenant site count: linear at about 2.2 MB per site with every site open, flat past the default LRU cap of 256

ePHPm v0.7.0 shipped per-site multi-tenancy. Point [server] sites_dir at a directory and every Host maps to its own docroot and its own Turso database file. One process serves all of them. The first question anybody asks about that is “how many sites fit on a box?” The honest way to answer it is to turn the question around. What does one more site cost? If you know the marginal RAM, file descriptors, CPU and disk for a tenant, the box size falls out of arithmetic.

What an Overloaded PHP Server Actually Returns

Closed-loop benchmarks can't show you overload. An open-loop load generator with an error taxonomy showed ePHPm returned nothing at all, and then verified the fix.

11-Minute Read

Bar chart of what 48,001 arrivals at 800 req/s got back: before the fixes almost everything was a client timeout, after them every arrival got a 200 or a fast 503

Ask most people what their PHP server does when it’s overloaded and you’ll hear “it returns 503s.” Ask them how they know and it usually comes down to a config setting they read about. I was in that camp for ePHPm. We had a [server.limits] max_connections knob, and the docs said it returned 503 at the cap. Every benchmark we had ever run reported zero errors under heavy load. Then I wrote a load generator that could produce overload. The honest answer was that an…

My Benchmark Was 4x Wrong Because of WSL

Launching the server with its working directory on /mnt/c cost ePHPm about 4x on a light workload. The report built a wrong explanation on top of it, and the strace shows why.

8-Minute Read

Bar chart of ePHPm requests per second by working directory: native ext4 far ahead of DrvFs, with DrvFs throughput falling as the cwd path gets deeper

I do most of my benchmarking on a Windows box in WSL2. It’s the machine on my desk with 32 threads and 62 GB of RAM. Everybody who does that learns the first rule early: don’t put your data on /mnt/c. DrvFs is the 9p-backed mount that exposes the Windows drives inside WSL, and it’s slow for file-heavy work. So the databases, the docroot and the build output all live on the WSL ext4 disk. My scaling harness for ePHPm even says so in capital letters in its README.

Clustered SQLite in ePHPm, Part 2: Adding Turso

How I decided a beta database engine was ready by testing it instead of reading its version number. ePHPm v0.7.0 replaces both rusqlite and sqld with Turso.

9-Minute Read

Bar chart of four SQLite lanes at 16 concurrent connections: rusqlite and Turso single node, sqld and Turso CDC clustered

Part 1 ended with ePHPm running two SQLite engines. rusqlite ran in-process for a single node, and an embedded sqld child process ran for a cluster. The sidecar fell off a cliff past four concurrent writers. The best fix I could ship was a semaphore set to 1. This post is about replacing both engines with one, and about how I decided that engine was ready.

Rolling Your Own KV Store Instead of Embedding Redis

Why ePHPm has its own Redis-compatible cache built in instead of running Redis. It also covers a bug that hid in it for four months and how it compares to real Redis on the same machine.

11-Minute Read

Bar chart of redis-benchmark GET and SET throughput against Redis 7.4, ePHPm v0.5.2 and ePHPm v0.2.0

Every PHP deployment I’ve run eventually grew a Redis. Sessions, the WordPress object cache, Laravel’s cache and locks, rate limiting. ePHPm is supposed to be one binary that replaces nginx + php-fpm + Redis, so it needed a cache in the box. I didn’t want to write one. This post covers why I did anyway and how it’s built. It also covers the bug that sat in its eviction code from March until last month. Then I compare it against a stock Redis on the same machine with…

Faking MySQL's Quirks Well Enough to Fool WordPress

Passing WordPress's install tests didn't mean litewire behaved like MySQL. This is the second round of fixes, for the small MySQL behaviors that WordPress and Laravel quietly depend on.

10-Minute Read

A table of MySQL behaviors WordPress relies on, with what litewire answered before and after the August fixes

Last month I wrote about running SQLite behind a MySQL socket with litewire. I ended on “passing an install suite is not the same thing as behaving like MySQL”. That turned out to be accurate. On August 10th a few things happened at once:

Requests That End Wrong: A 200 That Was Really a Crash

Three ePHPm bugs lived in how an embedded PHP request starts and ends. A bailout returned 200, $_POST came up empty and a SIGTERM corrupted the heap.

10-Minute Read

Diagram of a bare zend_bailout being absorbed by php_execute_script's own zend_try instead of reaching ePHPm's guard

The scariest response a server can send isn’t a 500. It’s a 200 that should have been a 500. A crash is loud. The pod restarts, an alert fires, somebody looks. A truncated page with a success status code is quiet. Your CDN caches it and the health check calls the box healthy. A client parses half a document as if it were the whole thing.

Database Connection Pooling with Smart Reset

PHP opens a new MySQL connection on every request, but ePHPm reuses them and cleans up only the ones that need it. Here's how that works, two bugs it had and how much load it takes off MySQL.

10-Minute Read

PHP request connecting to the ePHPm MySQL proxy, which lends it a pooled backend connection and resets it only if the session was dirtied

PHP’s share-nothing model is one of its best properties. It has one expensive side effect: a request can’t keep a database connection for the next request. php-fpm gives you PDO::ATTR_PERSISTENT and mysqli’s p: hosts. But those are per-worker-process. PDO’s keep whatever session state the last request left behind. mysqli at least resets the session with COM_CHANGE_USER on reuse. And almost nobody turns them on. So in practice every PHP request does a TCP connect, a MySQL…

Clustered SQLite in ePHPm, Part 1: Replication by Sidecar

ePHPm first made SQLite highly available by embedding sqld as a child process. This is how that worked and the list of things that went wrong with it.

9-Minute Read

Two ePHPm nodes, each running an embedded sqld child process, connected by gossip, a KV data plane and sqld gRPC

Most PHP sites I have run in the last twenty years did not need a MySQL server. They needed a database, and MySQL was the one every framework spoke. With ePHPm I have been trying to get rid of that server for the common case. The piece that makes it possible is litewire. PHP connects with plain pdo_mysql. litewire speaks the MySQL wire protocol, and the queries land in a SQLite file. WordPress and Laravel don’t know the difference. No mysqld, no child processes, just ephpm and a .db file.

Recent Posts

Categories

About

Luther is a 20+ year software veteran specializing in backend APIs and architecting products. Employed at SUSE as the Team Lead for the Containers on Windows Team working on Rancher + Kubernetes