Clustered ACME in Production: Three Bugs That Took TLS Down

Wed, 02 Sep 2026 19:30:00 -0700

ePHPm gets its own certificates from Let’s Encrypt. In a cluster only one node should be talking to the CA. The nodes elect an ACME leader over the shared KV store. The leader orders the cert, and everybody else installs it from KV. That all had unit tests and had worked on single nodes for months. Then I tried to move wildcard TLS termination for our preview environment off a separate proxy and onto the 3-node cluster itself. In about two days it found three separate bugs that each took TLS down. None of them showed up in a unit test. All of them are the kind of thing you only learn by putting the feature on real machines.

Acme on Luther Monson

Clustered ACME in Production: Three Bugs That Took TLS Down