Introduction

Ephemeral self-hosted runners have become a popular approach for teams looking to build scalable CI infrastructure.

The idea is simple:

• Create a runner when a workflow needs execution
• Process the workload
• Destroy the environment after completion

The expected flow:

Workflow Triggered

        ↓

Create Runner

        ↓

Register Runner

        ↓

Receive Job

        ↓

Execute Workflow

        ↓

Destroy Runner

However, many teams running ephemeral GitHub Actions runners encounter a confusing situation:

The runner is created successfully.

The runner registers successfully.

The runner appears online.

But the queued GitHub Actions job is still waiting.

Example:

✓ Connected to GitHub

✓ Runner successfully added

✓ Settings Saved

At the same time:

Waiting for a runner to pick up this job...

The infrastructure exists, but the workload has not started.

The Problem With Ephemeral Runner Scaling

Ephemeral runners introduce an additional coordination layer between infrastructure availability and job execution.

A workflow does not directly execute on a runner immediately after registration.

Multiple systems need to synchronize:

                 Workflow Queue

                        |

                        ↓

             GitHub Actions Scheduler

                        |

                        ↓

          Available Self-Hosted Runners

                        |

                        ↓

              Runner Listener

                        |

                        ↓

             Workflow Execution

A runner being online is only one part of the process.

The scheduler still needs to:

• detect the newly registered runner
• match runner labels
• verify workflow requirements
• assign the queued job

Until this happens, the workflow remains in the queue.

The Failure Scenario

A typical ephemeral runner lifecycle looks like this:

10:00:00

Developer pushes code

        ↓

10:00:05

Autoscaler creates runner

        ↓

10:00:35

Runner registers successfully

        ↓

10:00:36

Runner status: Online

        ↓

10:00:37

Job still waiting

        ↓

10:02:00

Runner receives no workload

From an infrastructure perspective, everything appears healthy:

• runner creation succeeded
• authentication succeeded
• registration succeeded

But the job assignment did not happen.

The gap between:

Runner Available

        ↓

Job Assigned

becomes hidden CI latency.

Runner Registration Does Not Guarantee Job Assignment

A common assumption is:

Runner Registered

        =

Job Can Start

However, these are separate events.

The actual sequence is:

Runner Created

        ↓

Runner Connects To GitHub

        ↓

Runner Registers

        ↓

GitHub Detects Runner Availability

        ↓

Scheduler Matches Workflow

        ↓

Job Assigned

        ↓

Workflow Starts

Successful registration only confirms that the runner can communicate with GitHub.

It does not guarantee that a queued workflow will immediately be assigned to that runner.

Why Does This Happen?

Ephemeral runners depend on coordination between several components:

• runner registration
• workflow queue
• GitHub Actions scheduler
• runner labels
• repository configuration

The system needs to match two independent events:

Available Runner

        +

Queued Workflow

        ↓

Job Assignment

If the synchronization between these components is delayed, a workflow can remain queued even when a runner is already online.

The Operational Challenges

1. Unpredictable Scaling Behavior

An autoscaler may correctly create runners based on queue size.

Example:

5 workflows waiting

        ↓

5 runners created

        ↓

5 runners registered

However, the workloads may not immediately start.

The infrastructure scales successfully, but execution still waits.

2. Idle Infrastructure Without Execution

A runner can reach a healthy state:

Runner Status:

Online ✓

Connected ✓

Executing ❌

The compute resource exists, but the developer workload is still blocked.

This creates wasted capacity and increases feedback time.

3. Difficult Debugging

The runner logs may look completely healthy:

Connected successfully

Registered successfully

Listening for jobs

But the workflow remains queued.

This makes troubleshooting difficult because the failure is not obvious.

Teams need to determine whether the issue is related to:

• runner provisioning
• runner registration
• job scheduling
• workflow configuration
• CI platform coordination

Why Ephemeral Runners Make This More Visible

Persistent runners usually have a long-running lifecycle:

Runner Exists

        ↓

Wait For Jobs

        ↓

Execute

They are already connected and available when workloads arrive.

Ephemeral runners have a much shorter lifecycle:

Create

↓

Register

↓

Receive Job

↓

Execute

↓

Terminate

Because the runner exists only for a specific workload, any delay between registration and job assignment becomes immediately visible.

The runner has already been created, but the developer is still waiting.

Building More Reliable CI Execution

Reliable CI infrastructure needs to reduce the gap between:

Runner Available

        ↓

Workflow Executing

The goal is not only creating more runners.

The goal is ensuring that execution environments can reliably accept workloads when they are needed.

Modern CI infrastructure needs to consider:

• runner readiness
• job assignment reliability
• startup consistency
• execution availability

A fast machine is not useful if the workload is waiting for infrastructure coordination.

Monk CI Perspective

At Monk CI, we focus on improving the reliability of the CI execution layer.

Runner availability is only one part of the problem.

The important question is:

When a workflow enters the queue, how quickly and reliably does execution begin?

Our infrastructure is designed around ready-to-execute environments that reduce dependency on runtime provisioning, runner registration delays, and coordination overhead before workloads start.

The objective is simple:

When developers push code, the CI system should be ready to execute — not preparing itself to execute.

Reliable CI is about predictable execution.

A pipeline should start when the developer needs feedback, not after additional infrastructure coordination.

Conclusion

Ephemeral self-hosted runners solve many scalability and isolation challenges, but they also introduce a new coordination layer between runner availability and workload execution.

A runner can be:

• created successfully
• registered successfully
• online and healthy

while workflows remain queued.

Understanding this gap is important for teams building scalable CI systems.

The future of CI is not only faster compute.

It is predictable execution infrastructure that starts workloads consistently.