[PR #254] [CLOSED] Triage ARC runner outage: document current failure state and isolate runtime breakage #164

New issue

Closed

opened 2026-06-08 08:30:27 +00:00 by ryangr0 · 0 comments

ryangr0 commented

2026-06-08 08:30:27 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/webgrip/homelab-cluster/pull/254
Author: @Copilot
Created: 5/24/2026
Status: ❌ Closed

Base: main ← Head: copilot/triage-arc-runners-issue

📝 Commits (1)

10bd6a6 Initial plan

📄 Description

ARC-backed workflows are currently not executing: jobs targeting arc-runner-set remain queued and are never assigned a runner. This PR captures the current failure state and narrows scope to ARC runtime availability/registration (not workflow label wiring or chart digest integrity).

Scope
- Diagnosis-only update; no runtime/config fix is introduced in this change set.
- Consolidates evidence for why ARC is the active bottleneck.
Key findings captured
- GitHub Actions jobs with labels: ["arc-runner-set"] are persistently queued.
- Queued jobs show runner_id: 0 and empty runner_name, indicating no runner pickup.
- Job logs are unavailable (404) because jobs never start.
- Workflow/scale-set label wiring is consistent (runs-on: arc-runner-set ↔ runnerScaleSetName: arc-runner-set).
What was ruled out
- ARC OCI chart digest references validate successfully (verify-oci-digests.sh), so manifest source pins are not the failure point.
Current diagnosis
- ARC runners are not registering/available to serve arc-runner-set jobs.
- Remaining unknown is the in-cluster runtime cause (controller/listener/auth/session path), which requires live cluster runtime inspection.

# Workflow demand side (.github/workflows/renovate-review-publish.yml)
jobs:
  relay:
    runs-on: arc-runner-set

// GitHub job state (representative)
{
  "status": "queued",
  "labels": ["arc-runner-set"],
  "runner_id": 0,
  "runner_name": ""
}

Original prompt

I think my arc runners are broken, triage and diagnose. Don't fix yet, let me know exactly what is wrong currently

Created from VS Code.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/webgrip/homelab-cluster/pull/254 **Author:** [@Copilot](https://github.com/apps/copilot-swe-agent) **Created:** 5/24/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `copilot/triage-arc-runners-issue` --- ### 📝 Commits (1) - [`10bd6a6`](https://github.com/webgrip/homelab-cluster/commit/10bd6a691746eb9d25f74b3c511c5d0cc29b459c) Initial plan ### 📄 Description ARC-backed workflows are currently not executing: jobs targeting `arc-runner-set` remain queued and are never assigned a runner. This PR captures the current failure state and narrows scope to ARC runtime availability/registration (not workflow label wiring or chart digest integrity). - **Scope** - Diagnosis-only update; no runtime/config fix is introduced in this change set. - Consolidates evidence for why ARC is the active bottleneck. - **Key findings captured** - GitHub Actions jobs with `labels: ["arc-runner-set"]` are persistently queued. - Queued jobs show `runner_id: 0` and empty `runner_name`, indicating no runner pickup. - Job logs are unavailable (`404`) because jobs never start. - Workflow/scale-set label wiring is consistent (`runs-on: arc-runner-set` ↔ `runnerScaleSetName: arc-runner-set`). - **What was ruled out** - ARC OCI chart digest references validate successfully (`verify-oci-digests.sh`), so manifest source pins are not the failure point. - **Current diagnosis** - ARC runners are not registering/available to serve `arc-runner-set` jobs. - Remaining unknown is the in-cluster runtime cause (controller/listener/auth/session path), which requires live cluster runtime inspection. ```yaml # Workflow demand side (.github/workflows/renovate-review-publish.yml) jobs: relay: runs-on: arc-runner-set ``` ```json // GitHub job state (representative) { "status": "queued", "labels": ["arc-runner-set"], "runner_id": 0, "runner_name": "" } ```  <details> <summary>Original prompt</summary> > I think my arc runners are broken, triage and diagnose. Don't fix yet, let me know exactly what is wrong currently </details>  Created from [VS Code](https://code.visualstudio.com/docs/copilot/copilot-coding-agent). --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

ryangr0

2026-06-08 08:30:27 +00:00

closed this issue
added the
pull-request
label

ryangr0 referenced this issue

2026-06-08 08:30:48 +00:00

[PR #164] [CLOSED] chore(review): add Renovate image update risk assessment for PR #162 #226

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

webgrip/homelab-cluster#164

No description provided.

Rows
Columns