Yicong-Huang commented on code in PR #7172:
URL: https://github.com/apache/texera/pull/7172#discussion_r3741380362
##########
.github/workflows/precheck.yml:
##########
@@ -314,6 +341,169 @@ jobs:
runPyrightLanguageService = false;
}
+ // Runner tiers: merge queue entries default to the
+ // self-hosted scale sets (the queue is the latency-critical
+ // serialized path most hurt by hosted-pool congestion);
+ // PR events opt in via the ci:self-hosted label. Everything
+ // else (push / dispatch) stays GitHub-hosted. Tiers:
+ // arc-heavy-linux for the amber/sbt stacks, arc-medium-linux
+ // for platform and pyamber, arc-light-linux for the small
+ // service jobs.
+ //
+ // Fallback probing, two levels, evaluated per tier:
+ //
+ // Level 1 (availability): a set with no online runner (cluster
+ // down, scale set removed) falls back to GitHub-hosted so a
+ // labeled PR never queues forever. Relies on each set keeping
+ // warm runners (minRunners > 0) — an idle scale-set with
+ // minRunners: 0 registers no runners and would look
+ // unavailable.
+ //
+ // Level 2 (congestion): predict queueing before it happens by
+ // comparing this round's demand (the stack decisions above
+ // tell us how many jobs each tier is about to receive)
+ // against idle supply (runner busy flags) plus the backlog
+ // of already-queued jobs waiting for the same label. Blind
+ // spot: k8s-side capacity (Pending pods, scale-up headroom)
+ // is invisible to the GitHub API.
+ //
+ // Runner listing needs ARC_STATUS_TOKEN (administration:read);
+ // without it the label is trusted as-is. Any probe error
+ // degrades to trusting the label.
+ const selfHosted = eventName === "merge_group" ||
labels.includes("ci:self-hosted");
+ const tiers = {
+ heavy_runner: selfHosted ? "arc-heavy-linux" : "ubuntu-latest",
+ medium_runner: selfHosted ? "arc-medium-linux" : "ubuntu-latest",
+ light_runner: selfHosted ? "arc-light-linux" : "ubuntu-latest",
+ };
+ // Per-tier job counts this round. PLATFORM_SERVICES mirrors the
+ // service matrix size in build.yml — keep in sync.
+ const PLATFORM_SERVICES = 6;
+ const demand = {
+ "arc-heavy-linux":
+ (runAmber ? 1 : 0) + (runAmberIntegration ? 1 : 0) +
(runFrontend ? 1 : 0),
+ "arc-medium-linux":
+ (runPlatform ? PLATFORM_SERVICES : 0) +
+ (runPlatformIntegration ? PLATFORM_SERVICES : 0) +
+ (runPyamber ? 3 : 0),
+ "arc-light-linux":
+ (runInfra ? 1 : 0) +
+ (runAgentService ? 1 : 0) +
+ (runPyrightLanguageService ? 1 : 0),
+ };
+ // Kill switch: ARC_AVAILABLE=false (manual, or flipped by the
+ // runner-heartbeat watchdog) forces every tier to GitHub-hosted.
+ if (selfHosted && process.env.ARC_AVAILABLE === "false") {
+ for (const k of Object.keys(tiers)) tiers[k] = "ubuntu-latest";
+ core.warning("ARC_AVAILABLE=false; routing all Linux tiers to
GitHub-hosted runners.");
+ }
+ // Heartbeat fallback: runner-heartbeat.yml proves fleet liveness
+ // every 15 minutes. Stale last-success (with history present)
+ // means the fleet is down — fall back without needing any extra
+ // credential (plain GITHUB_TOKEN, actions: read).
+ if (selfHosted && tiers.light_runner !== "ubuntu-latest") {
+ try {
+ const STALE_MS = 35 * 60 * 1000;
+ const ok = await github.rest.actions.listWorkflowRuns({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ workflow_id: "runner-heartbeat.yml",
+ status: "success",
+ per_page: 1,
+ });
+ const any = await github.rest.actions.listWorkflowRuns({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ workflow_id: "runner-heartbeat.yml",
+ per_page: 1,
+ });
+ const hasHistory = (any.data.workflow_runs ?? []).length > 0;
+ const last = ok.data.workflow_runs?.[0];
+ const lastOk = last ? new Date(last.updated_at).getTime() : 0;
+ if (hasHistory && Date.now() - lastOk > STALE_MS) {
+ for (const k of Object.keys(tiers)) tiers[k] =
"ubuntu-latest";
+ core.warning("Runner heartbeat is stale; routing all Linux
tiers to GitHub-hosted runners.");
+ }
+ } catch (e) {
+ core.info(`Heartbeat check skipped (${e.message}).`);
+ }
+ }
+ if (selfHosted && process.env.ARC_STATUS_TOKEN) {
+ try {
+ const res = await fetch(
+
`https://api.github.com/repos/${context.repo.owner}/${context.repo.repo}/actions/runners?per_page=100`,
+ {
+ headers: {
+ authorization: `Bearer ${process.env.ARC_STATUS_TOKEN}`,
+ accept: "application/vnd.github+json",
+ },
+ }
Review Comment:
Not just sugar — `github.rest.*` injects auth, throws a structured
`HttpError`, and gives you `paginate` (plus `retries`, if the step asks for it).
What blocks it here is the token. `github` is bound to the workflow
`GITHUB_TOKEN`, and Octokit's auth hook overwrites the `authorization` header
on every request, so one client can't carry a second token. Listing self-hosted
runners needs `administration:read`, which only `ARC_STATUS_TOKEN` has — while
the backlog probe 20 lines below stays on the SDK because `actions:read` is
within the workflow token's reach. So the split follows token scope, not style;
I've pushed a comment at each raw call site saying exactly that.
The brittleness is bounded by the try/catch: any failure warns and falls
back to trusting the label, so the cost is losing the probe, not a failed job.
The clean fix is to move this probe into its own `github-script` step with
`github-token: ARC_STATUS_TOKEN`, which also restores `github.paginate` (the
raw call caps at 100 runners today). That's better as a follow-up than
restructuring precheck in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
