Luther Monson, Staff Software Engineer at SUSE

Luther Monson

Go by way of PHP · Containers · VMs · Windows · Kubernetes

A Lock Is Only as Good as Its Release

SETNX gets all the attention, but most of the bugs in ePHPm's KV locks were on the release side. This is how blind deletes, a missing compare-and-delete and a cluster where two nodes could both win got fixed.

9-Minute Read

Timeline: request A's lock expires, B acquires it, A's blind DEL frees B's lock and C gets in

Every “build a lock on Redis” tutorial spends its time on acquisition: SET key value NX PX 30000, one winner, done. The release gets one line, DEL key. That line is where most of the lock bugs I’ve had in ePHPm lived. Over about three months the same mistake showed up in three places. First the native session handler, then two of the Composer wrapper packages, and finally the cluster. Each time it wore a slightly different costume.

Clustered SQLite in ePHPm, Part 3: CDC Over the Cluster Channel

ePHPm ships Turso's change data capture stream between nodes over one authenticated, multiplexed TCP connection. This covers how that works and the two failover bugs that showed up when the e2e suite killed the primary.

15-Minute Read

Two ePHPm nodes connected by gossip on UDP 7946, the KV data plane on TCP 7947 and the cluster channel on TCP 7948 carrying cdc, snapshot and sql streams

By the end of Part 2 ePHPm v0.7.0 ran exactly one SQLite engine in process. That was Turso, for single-node and clustered mode alike. The sidecar was gone. What I skipped over is how a replica actually gets the primary’s writes, and what happens when the primary dies with data on it. That’s this post.

Clustered ACME in Production: Three Bugs That Took TLS Down

Moving wildcard TLS onto a live 3-node ePHPm cluster found a leader election that wasn't, a certificate only two nodes could see, and an HTTP port speaking TLS.

10-Minute Read

Diagram comparing a sharded KV write that leaves one of three nodes without the certificate to a broadcast write that reaches all three

ePHPm gets its own certificates from Let’s Encrypt. In a cluster only one node should be talking to the CA. The nodes elect an ACME leader over the shared KV store. The leader orders the cert, and everybody else installs it from KV. That all had unit tests and had worked on single nodes for months. Then I tried to move wildcard TLS termination for our preview environment off a separate proxy and onto the 3-node cluster itself. In about two days it found three separate bugs that each took…

Clustered SQLite in ePHPm, Part 1: Replication by Sidecar

ePHPm first made SQLite highly available by embedding sqld as a child process. This is how that worked and the list of things that went wrong with it.

9-Minute Read

Two ePHPm nodes, each running an embedded sqld child process, connected by gossip, a KV data plane and sqld gRPC

Most PHP sites I have run in the last twenty years did not need a MySQL server. They needed a database, and MySQL was the one every framework spoke. With ePHPm I have been trying to get rid of that server for the common case. The piece that makes it possible is litewire. PHP connects with plain pdo_mysql. litewire speaks the MySQL wire protocol, and the queries land in a SQLite file. WordPress and Laravel don’t know the difference. No mysqld, no child processes, just ephpm and a .db file.

Gossip-Based Clustering for PHP Apps

How ePHPm servers find each other, share data and handle a server joining or leaving. It also measures how fast a write reaches the other servers and covers a bug I found while measuring it.

11-Minute Read

Three ePHPm nodes: small values replicated to every node over gossip, a large value stored on its owner and one replica over a TCP data plane

Most PHP developers never touch clustering primitives. That’s by design. PHP is share-nothing. Each request starts from zero, and anything shared lives somewhere else. Sessions go in Redis. Cache goes in Redis or Memcached. Locks go in Redis. When you run two app servers, the “cluster” is really two stateless boxes pointed at the same Redis. That works until Redis is the thing you’re trying not to run.

Recent Posts

Categories

About

Luther is a 20+ year software veteran specializing in backend APIs and architecting products. Employed at SUSE as the Team Lead for the Containers on Windows Team working on Rancher + Kubernetes