Clustered ACME in Production: Three Bugs That Took TLS Down
Moving wildcard TLS onto a live 3-node ePHPm cluster found a leader election that wasn't, a certificate only two nodes could see, and an HTTP port speaking TLS.
ePHPm gets its own certificates from Let’s Encrypt. In a cluster only one node should be talking to the CA. The nodes elect an ACME leader over the shared KV store. The leader orders the cert, and everybody else installs it from KV. That all had unit tests and had worked on single nodes for months. Then I tried to move wildcard TLS termination for our preview environment off a separate proxy and onto the 3-node cluster itself. In about two days it found three separate bugs that each took…
