runner.pool-reconcile-ec2-vm

Controller reconcile loop for warm-pool runner groups on EC2. Observes GitHub (via LTIP) + AWS EC2 + the engine DB every tick and converges the pool toward min_warm_idle / max_count. Replaces the event-driven scale-check + manual pool-sweep model. Idempotent; safe to run on a stable workflow_id per group.

Per-group warm-pool reconcile loop for raw EC2 VMs (controller).

Inputs:

• runner_group_uuid — the group to reconcile.
• queued_demand_hint — optional inline webhook demand breadcrumb.

Outputs (carried in state_data across the state machine):

• observation — snapshot of GH runners + EC2 instances + DB rows.
• transitions — per-instance state changes computed from it.
• action_taken — at most one of launch / terminate / drain / defer / noop chosen this tick.
• composition — final pool composition counts.
• observation_lag_ms — wall clock from observe start to snapshot.
• degraded — true if an essential observation source failed (GH throttled, EC2 transient error). Reconcile skips destructive transitions in degraded mode and waits for the next tick.

Overview

Property	Value
Workflow type	Linear
Library	App-runners-aws
Version	1.0

Input Schema

Field	Type	Required	Default	Description
runner_group_uuid	uuid	Yes	—	UUID of the RunnerGroup this reconcile tick operates on, picked from the organization's list of runner groups.
queued_demand_hint	json	No	—	Inline webhook demand hint from runner.dispatch-from-job-queued; used as a fallback when queued-job observation misses demand.

Output Schema

Field	Type	Required	Default	Description
runner_group_uuid	uuid	No	—	UUID of the RunnerGroup this reconcile tick operated on, echoed from input.
started_at	string	No	—	—
completed_at	string	No	—	—
warm_pool_enabled	boolean	No	—	—
queued_demand_hint	json	No	—	—
observation	json	No	—	—
degraded	boolean	No	—	—
observation_errors	list	No	—	—
queued_hint_count	integer	No	—	—
effective_queued_count	integer	No	—	—
transitions	list	No	—	—
stale_github_runners_deleted	integer	No	—	—
stale_github_runners_cleanup_errors	list	No	—	—
action_taken	string	No	—	—
queued_hint_consumed_count	integer	No	—	—
quota_cooldown_until	string	No	—	—
quota_failure_code	string	No	—	—
quota_failure_execution_uuid	uuid	No	—	—
launched_workflow_uuid	string	No	—	Engine run id of the runner.execution-launch-ec2-vm workflow started by this tick, when a launch was performed. Internal engine correlation id, not a business-entity foreign key — no list/query workflow backs it.
terminated_instance_uuid	uuid	No	—	UUID of the RunnerInstance terminated or drained during this reconcile tick, when one was taken.
desired_alive	integer	No	—	—
composition	json	No	—	—
observation_lag_ms	integer	No	—	—
failure_reason	string	No	—	—
failed_at	string	No	—	—
step_errors	list	No	—	—
error	string	No	—	—
error_type	string	No	—	—
failed_at_state	string	No	—	—
failed_layer	string	No	—	—
failed_step	string	No	—	—

States

State	Initial	Terminal	Success	Auto-advance	Description
pending	Yes	No	—	observe	—
emitting_snapshot	No	No	—	complete	—
enforcing_floor_cap	No	No	—	emit_snapshot	—
observing	No	No	—	diff_and_transition	—
transitioning	No	No	—	enforce_floor_cap	—
completed	No	Yes	Yes	—	—
failed	No	Yes	No	—	—

State Diagram

Transitions

From	Action	To	Description
pending	observe	observing	—
observing	diff_and_transition	transitioning	—
transitioning	enforce_floor_cap	enforcing_floor_cap	—
enforcing_floor_cap	emit_snapshot	emitting_snapshot	—
emitting_snapshot	complete	completed	—
* (any state)	fail	failed	—

API Usage

POST /api/workflows/start
Content-Type: application/json

{
  "workflow_type": "runner.pool-reconcile-ec2-vm",
  "initial_data": {
    "runner_group_uuid": "value"
  }
}