Luther Monson, Staff Software Engineer at SUSE

Luther Monson

Go by way of PHP · Containers · VMs · Windows · Kubernetes

Docker-in-Docker in CI Without Handing Jobs Root

ephemerd runs Docker-in-Docker for container: jobs without a privileged container. Here is what a job could still reach, and the four PRs that closed each hole.

8-Minute Read

Four dind sandbox cracks and the ephemerd PR that closed each one

ephemerd runs a fresh GitHub Actions runner per job. Some of those jobs need Docker. A workflow builds an image, runs a service container, or uses the container: key. The usual way to give a self-hosted job Docker is Docker-in-Docker, or dind.

Cache Proxies for Ephemeral CI Runners

A fresh runner per job starts with an empty cache. Here is how ephemerd caches Go modules, Packagist, GitHub releases and the Actions cache on the node instead of re-downloading them every job.

10-Minute Read

A fresh runner per job points go mod, cargo, composer, release downloads and actions/cache at matching cache proxies on the ephemerd node, where the caches persist and only reach the internet on a miss

ephemerd gives every CI job a brand new runner. A new container on Linux, a new VM on macOS and Windows. The job runs, the runner is thrown away, and the next job starts from a clean image. That is the whole point. Nothing a job does can leak into the next one.

Ephemeral macOS Runners: A Fresh VM Per Job on Apple Silicon

How ephemerd runs each macOS CI job in a throwaway Virtualization.framework VM, and the slot leaks and disk limits I had to fix to make it reliable.

9-Minute Read

A macOS CI job lifecycle: clone the base image, boot and grow the disk, set up over SSH, run the job, then stop and sweep, with a band showing the single slot held the whole time

ephemerd gives every CI job a brand new environment and throws it away after. On Linux and Windows that environment is a container. On macOS it can’t be. Apple’s license wants macOS running on Apple hardware, so the Mac in my rack is the real thing. The isolation comes from a VM instead.

Webhooks Arrive Out of Order: How My Scheduler Killed Live Builds

ephemerd's orphan sweep trusted webhook history over reality and destroyed a runner mid-build. Now the rules only nominate, and a live busy check gets a veto.

9-Minute Read

Timeline of three same-label jobs where a runner executing a build is destroyed 2.3 seconds before its in_progress webhook arrives

The scheduler in ephemerd is event-driven. GitHub sends a workflow_job webhook when a job is queued, when it starts and when it completes. ephemerd creates and destroys runners to match. For a while that worked well enough that I stopped thinking about it. Then a docker-build job failed with “The self-hosted runner lost communication with the server”. I pulled the node’s journal. ephemerd had destroyed that runner itself, on purpose, while it was running the build.

Bind Mounts Across Mount Namespaces: Four PRs and a Container Escape

Making GitHub Actions container: jobs work through ephemerd's Docker API shim took four tries, then a TOCTOU race turned the working version into an escape.

8-Minute Read

Sequence diagram of the bind mount TOCTOU race and the openat2 plus staging fix

GitHub Actions has a container: key that runs a job’s steps inside an image of your choosing. It’s popular because it gives you a reproducible toolchain without building a custom runner image. On a self-hosted runner it works by having the runner drive Docker. On ephemerd, “Docker” is a shim that speaks the Docker API and creates containers in containerd. Getting bind mounts right through that shim took four PRs in a week. A couple of months later the version that worked…

Kata Containers for CI: Boot Cost Isn't the Problem, File I/O Is

Kata containers are safer for CI, but slower. Starting one takes 12x longer, which is fine. Reading small files is about 40x slower for the entire job, which isn't.

7-Minute Read

Bar chart of Kata slowdown versus runc for each measurement

Of the three platforms ephemerd runs jobs on, Linux has the weakest isolation. Windows jobs get a Hyper-V isolated container with its own kernel. macOS jobs get a whole VM. A Linux job is an ordinary runc container sharing the host kernel. A kernel escape from a Linux job is a host compromise. Kata Containers fixes that by putting each container in a lightweight VM with its own kernel. It plugs into containerd as just another runtime handler. So I added it in #155, measured it, and left it off…

My iptables Rule Matched 0 Packets in 9.8 GB

Container-to-host traffic goes through INPUT, not FORWARD, so ephemerd's carefully written allow-list never matched a single packet.

8-Minute Read

Netfilter flow showing container traffic to the host taking INPUT instead of FORWARD

After a week of Windows firewall rules that listed fine and enforced nothing, I went back to the Linux side of ephemerd feeling pretty good about it. iptables is the boring, well-understood one. Then I looked at the packet counters on a live node. I found a rule that had never matched a packet in its life. That node had pushed 9.8 GB through the same chain.

ephemerd: One Daemon, a Fresh Runner Per Job, Every OS

A single Go binary that runs self-hosted GitHub Actions runners on Linux, Windows and macOS, with a new container or VM for every job.

6-Minute Read

ephemerd architecture across Linux, Windows and macOS hosts

I have CI that needs Linux amd64, Linux arm64, Windows and macOS. GitHub-hosted runners for all four get expensive fast, especially the arm64 and Mac minutes. Self-hosted runners fix the bill and create a different problem. A stock actions/runner on a box you own runs whatever the workflow says. It keeps whatever the last job left behind. And it’s one fork PR away from somebody running code on your LAN.

Escaping WSL: Booting My Own Linux VM Through Hyper-V's HCS API

WSL2 refuses to run under LocalSystem, so ephemerd boots an Alpine kernel directly through vmcompute.dll instead.

9-Minute Read

Diagram of ephemerd booting a Linux VM through HCS and reading its IP from the serial console

I’ve been building ephemerd, a daemon that runs self-hosted GitHub Actions runners with a fresh environment per job. On a Windows host, Windows jobs run in Hyper-V isolated containers. Linux jobs still need a Linux kernel from somewhere. The obvious answer on Windows in 2026 is WSL2, and that’s what I used first. It worked great from my terminal and fell over the moment ephemerd became a real Windows service.

Recent Posts

Categories

About

Luther is a 20+ year software veteran specializing in backend APIs and architecting products. Employed at SUSE as the Team Lead for the Containers on Windows Team working on Rancher + Kubernetes