ephemerd on Luther Monson

Docker-in-Docker in CI Without Handing Jobs Root

Sat, 24 Oct 2026 10:00:00 -0700

ephemerd runs a fresh GitHub Actions runner per job. Some of those jobs need Docker. A workflow builds an image, runs a service container, or uses the container: key. The usual way to give a self-hosted job Docker is Docker-in-Docker, or dind.

The catch is what dind normally costs you. The standard recipe is a --privileged container. Privileged hands the job most of root on the host. For CI that runs fork pull requests, that is a non-starter. A fork PR is a stranger’s code.

Cache Proxies for Ephemeral CI Runners

Fri, 23 Oct 2026 15:00:00 -0700

ephemerd gives every CI job a brand new runner. A new container on Linux, a new VM on macOS and Windows. The job runs, the runner is thrown away, and the next job starts from a clean image. That is the whole point. Nothing a job does can leak into the next one.

Clean has a cost. A fresh runner has an empty cache. So every job downloads the world again. Go modules, crates, the Rust toolchain, Composer packages, release tarballs. On a busy repo that is gigabytes a day, pulled over and over, most of it byte-for-byte identical to what the last job pulled.

Ephemeral macOS Runners: A Fresh VM Per Job on Apple Silicon

Fri, 23 Oct 2026 11:00:00 -0700

ephemerd gives every CI job a brand new environment and throws it away after. On Linux and Windows that environment is a container. On macOS it can’t be. Apple’s license wants macOS running on Apple hardware, so the Mac in my rack is the real thing. The isolation comes from a VM instead.

Each macOS job gets a fresh Virtualization.framework VM cloned from a base image. The job runs inside it. The VM is destroyed when the job finishes. That’s the same contract as the container path. Getting there took four fixes that I only found by running the fleet. This post walks the VM lifecycle and the three ways it bit me:

Webhooks Arrive Out of Order: How My Scheduler Killed Live Builds

Wed, 07 Oct 2026 21:10:00 -0700

The scheduler in ephemerd is event-driven. GitHub sends a workflow_job webhook when a job is queued, when it starts and when it completes. ephemerd creates and destroys runners to match. For a while that worked well enough that I stopped thinking about it. Then a docker-build job failed with “The self-hosted runner lost communication with the server”. I pulled the node’s journal. ephemerd had destroyed that runner itself, on purpose, while it was running the build.

Bind Mounts Across Mount Namespaces: Four PRs and a Container Escape

Thu, 24 Sep 2026 21:02:00 -0700

GitHub Actions has a container: key that runs a job’s steps inside an image of your choosing. It’s popular because it gives you a reproducible toolchain without building a custom runner image. On a self-hosted runner it works by having the runner drive Docker. On ephemerd, “Docker” is a shim that speaks the Docker API and creates containers in containerd. Getting bind mounts right through that shim took four PRs in a week. A couple of months later the version that worked turned out to be an escape.

Kata Containers for CI: Boot Cost Isn't the Problem, File I/O Is

Thu, 03 Sep 2026 21:35:00 -0700

Of the three platforms ephemerd runs jobs on, Linux has the weakest isolation. Windows jobs get a Hyper-V isolated container with its own kernel. macOS jobs get a whole VM. A Linux job is an ordinary runc container sharing the host kernel. A kernel escape from a Linux job is a host compromise. Kata Containers fixes that by putting each container in a lightweight VM with its own kernel. It plugs into containerd as just another runtime handler. So I added it in #155, measured it, and left it off by default. This post is why.

My iptables Rule Matched 0 Packets in 9.8 GB

Wed, 26 Aug 2026 20:30:00 -0700

After a week of Windows firewall rules that listed fine and enforced nothing, I went back to the Linux side of ephemerd feeling pretty good about it. iptables is the boring, well-understood one. Then I looked at the packet counters on a live node. I found a rule that had never matched a packet in its life. That node had pushed 9.8 GB through the same chain.

The rules I thought I had

Job containers sit on a CNI bridge, 10.88.0.0/16, with the host on 10.88.0.1. ephemerd puts its egress policy in its own chain, EPHEMERD-FORWARD. FORWARD jumps to it for anything to or from the container subnet. From a live node:

ephemerd: One Daemon, a Fresh Runner Per Job, Every OS

Tue, 18 Aug 2026 21:05:00 -0700

I have CI that needs Linux amd64, Linux arm64, Windows and macOS. GitHub-hosted runners for all four get expensive fast, especially the arm64 and Mac minutes. Self-hosted runners fix the bill and create a different problem. A stock actions/runner on a box you own runs whatever the workflow says. It keeps whatever the last job left behind. And it’s one fork PR away from somebody running code on your LAN.

Escaping WSL: Booting My Own Linux VM Through Hyper-V's HCS API

Sun, 14 Jun 2026 20:12:00 -0700

I’ve been building ephemerd, a daemon that runs self-hosted GitHub Actions runners with a fresh environment per job. On a Windows host, Windows jobs run in Hyper-V isolated containers. Linux jobs still need a Linux kernel from somewhere. The obvious answer on Windows in 2026 is WSL2, and that’s what I used first. It worked great from my terminal and fell over the moment ephemerd became a real Windows service.

WSL2 doesn’t do LocalSystem

A Windows service that wants to create containers and Hyper-V VMs runs as LocalSystem. WSL2 doesn’t support that context at all. You get WSL_E_LOCAL_SYSTEM_NOT_SUPPORTED and that’s the end of the conversation. There’s no flag for it. WSL is a per-user feature and a service account isn’t a user.