Status and lifecycle
Overview shows a single presence chip per runner (not a separate busy chip).
Runner status at a glance
Presence labels
| You see | Stored / computed status | Meaning |
|---|---|---|
| Idle | ONLINE, no active jobs | Ready for work |
| ✣Running 1 job / Running N jobs | ONLINE, activeJobs \> 0 | Executing |
| Offline | OFFLINE | No heartbeat for 90 seconds, or agent sent status: OFFLINE on shutdown |
| Pending | PENDING | Registered but no heartbeat yet |
| ⌕Maintenance | MAINTENANCE | Admin put the runner in maintenance |
| ⅡPaused | PAUSED | Same idea as maintenance (legacy label) |
| ◷Draining | DRAINING | Current jobs may finish; no new leases |
| ⊘Disabled | DISABLED | Hard-blocked from scheduling |
ONLINE does not mean “the process in my terminal is still running.” It means a valid axrun_\* token sent a heartbeat (or log ingest) within 90 seconds.
How ONLINE / OFFLINE is computed
- If status is
DISABLED,PAUSED,MAINTENANCE, orDRAINING→ show that. - Else if the last heartbeat payload said
OFFLINE→ Offline immediately. - Else if there is no
lastHeartbeatAt→ Pending. - Else if heartbeat is ≤ 90 seconds old → Online (Idle or Running).
- Else → Offline.
Default agent heartbeat interval is about 10 seconds. Restarting the API or UI does not reset these rows — they live in PostgreSQL.
Graceful stop vs hard kill
| Stop | What the UI does |
|---|---|
| SIGTERM / Ctrl+C / systemd stop | Agent sends a final OFFLINE heartbeat. UI shows Offline right away. Row stays registered by default. |
| SIGKILL / power loss | No final heartbeat. UI shows Offline after ~90 seconds. |
Deregister on shutdown
Default: the runner stays registered so the same axrun_\* token works on restart.
If the group’s internal pool has deregisterOnShutdown: true, a graceful stop calls POST /runner-agent/deregister and deletes the row. Use this for ephemeral CI runners, not long-lived systemd services.
Admin actions
| Action | Result |
|---|---|
| Enter maintenance | Status → Maintenance. No new jobs. |
| Resume / Exit maintenance / Stop draining | Back to Online when heartbeats are fresh. |
| Drain | Status → Draining. Finish in-flight jobs; accept none. |
| Disable | Status → Disabled. Blocked until Enable. |
| Enable | Clears Disabled. |
| Disconnect | Deletes the row. Agent must re-register with a new axreg_\* command. |
Disconnect is permanent for that identity. If the process is still running, stop it so it does not confuse operators.
Busy after a crash
If a runner stays Running after the process died:
- Wait ~15 seconds for stale-busy reconcile.
- If an execution lease is still active, wait up to ~3 minutes.
- Cancel the stuck deployment/workflow if needed, or restart the agent so heartbeats reset
activeJobs.
Managed (autoscaled) runners that crash without deregistering can be reclaimed after ~180 seconds.
Stale offline rows
Use Remove offline runners or the offline cleanup policy. The agent must re-register to come back.
More information
Deep dive: RUNNER_STATUS_AND_TROUBLESHOOTING