Webhooks Arrive Out of Order: How My Scheduler Killed Live Builds
ephemerd's orphan sweep trusted webhook history over reality and destroyed a runner mid-build. Now the rules only nominate, and a live busy check gets a veto.
The scheduler in ephemerd is event-driven. GitHub sends a
workflow_job webhook when a job is queued, when it starts and when it completes. ephemerd creates and destroys runners
to match. For a while that worked well enough that I stopped thinking about it. Then a docker-build job failed with
“The self-hosted runner lost communication with the server”. I pulled the node’s journal. ephemerd had destroyed that
runner itself, on purpose, while it was running the build.
