Luther Monson, Staff Software Engineer at SUSE

Luther Monson

Go by way of PHP · Containers · VMs · Windows · Kubernetes

Webhooks Arrive Out of Order: How My Scheduler Killed Live Builds

ephemerd's orphan sweep trusted webhook history over reality and destroyed a runner mid-build. Now the rules only nominate, and a live busy check gets a veto.

9-Minute Read

Timeline of three same-label jobs where a runner executing a build is destroyed 2.3 seconds before its in_progress webhook arrives

The scheduler in ephemerd is event-driven. GitHub sends a workflow_job webhook when a job is queued, when it starts and when it completes. ephemerd creates and destroys runners to match. For a while that worked well enough that I stopped thinking about it. Then a docker-build job failed with “The self-hosted runner lost communication with the server”. I pulled the node’s journal. ephemerd had destroyed that runner itself, on purpose, while it was running the build.

Bind Mounts Across Mount Namespaces: Four PRs and a Container Escape

Making GitHub Actions container: jobs work through ephemerd's Docker API shim took four tries, then a TOCTOU race turned the working version into an escape.

8-Minute Read

Sequence diagram of the bind mount TOCTOU race and the openat2 plus staging fix

GitHub Actions has a container: key that runs a job’s steps inside an image of your choosing. It’s popular because it gives you a reproducible toolchain without building a custom runner image. On a self-hosted runner it works by having the runner drive Docker. On ephemerd, “Docker” is a shim that speaks the Docker API and creates containers in containerd. Getting bind mounts right through that shim took four PRs in a week. A couple of months later the version that worked…

shipmates Short Ones: 8 Minutes Extracting Go, and Three Voice-Loop Bugs

Four small stories from shipmates, starting with setup-go stuck extracting on Windows CI and Cloudflare's 100 second timeout. The other two are a voice loop that transcribed itself and events lost to a timestamp watermark.

5-Minute Read

setup-go on Windows CI: 7 seconds to download, 473 seconds extracting before the timeout

A few shipmates bugs don’t need a whole post each. They were too good to leave in commit messages, though.

ephemerd: One Daemon, a Fresh Runner Per Job, Every OS

A single Go binary that runs self-hosted GitHub Actions runners on Linux, Windows and macOS, with a new container or VM for every job.

6-Minute Read

ephemerd architecture across Linux, Windows and macOS hosts

I have CI that needs Linux amd64, Linux arm64, Windows and macOS. GitHub-hosted runners for all four get expensive fast, especially the arm64 and Mac minutes. Self-hosted runners fix the bill and create a different problem. A stock actions/runner on a box you own runs whatever the workflow says. It keeps whatever the last job left behind. And it’s one fork PR away from somebody running code on your LAN.

Escaping WSL: Booting My Own Linux VM Through Hyper-V's HCS API

WSL2 refuses to run under LocalSystem, so ephemerd boots an Alpine kernel directly through vmcompute.dll instead.

9-Minute Read

Diagram of ephemerd booting a Linux VM through HCS and reading its IP from the serial console

I’ve been building ephemerd, a daemon that runs self-hosted GitHub Actions runners with a fresh environment per job. On a Windows host, Windows jobs run in Hyper-V isolated containers. Linux jobs still need a Linux kernel from somewhere. The obvious answer on Windows in 2026 is WSL2, and that’s what I used first. It worked great from my terminal and fell over the moment ephemerd became a real Windows service.

Recent Posts

Categories

About

Luther is a 20+ year software veteran specializing in backend APIs and architecting products. Employed at SUSE as the Team Lead for the Containers on Windows Team working on Rancher + Kubernetes