potiuk opened a new pull request, #77: URL: https://github.com/apache/airflow-ci-infra/pull/77
Between 2026-10-25 and 2026-10-27 the stack launched 4706 Spot and 4038 on-demand runners. The Spot quota was not the cause (peak 2332 of 3000 vCPU); the on-demand quota filled up instead (2004 of 2000 vCPU on 2026-10-26 16:00 UTC, 1653 `VcpuLimitExceeded`). The worker log records why each on-demand runner was chosen (`force_on_demand_reason`): | Reason | Runners | Cause | |---|---|---| | `spot_disabled` | 2641 | The default `SpotCircuitBreaker` `2/15/30`; none since #73 was applied (2026-10-26 20:29 UTC) | | `force_on_demand` | 678 | x64 Spot capacity misses: up to v3.4 one miss sends **every** job to on-demand for five minutes | | `spot_recovery_rerun` | 719 | Rerun of an interrupted job, on-demand by design | v3.5.0 sends only the affected runners to on-demand after a Spot capacity miss or an exhausted Spot quota, and v3.5.1 updates the control-plane image's CA bundle on top. Changes: - `runs_on_version` `v3.3.2` → `v3.5.1`, and `v3.5.1` added to `runs_on_versions` in `terraform/runs-on-access`. `v3.3.2` stays listed so rollback keeps working. - `runs-on.yml.example`: x64 presets also allow `m8a`, `m8i`, `m5a`, `m5`. Of 2156 Spot capacity errors in a day, 1959 were x64 and 197 arm64, and more families give the allocator more Spot pools. The live labels are in `apache/airflow` workflows and need the same change there (separate PR). - README: SHA-256 of the v3.5.1 template and a "Spot capacity" section with the numbers above. It also notes that the CloudFormation template is fixed to two AZs: Spot spread across zones (`RUNS_ON_SPOT_FORCE_ZONE_SPREAD`) exists only in RunsOn's Terraform module. Checked for the upgrade: - v3.5.1 creates the same IAM roles and managed policies as v3.3.2, so the service-role guardrails need no change. - New parameter `EnableWarmPools` (default `true`) only affects `pool=` labels; the stack uses dynamic runners. - Since v3.5.0, Linux jobs run under `/home/runner/work`: caches keyed on the absolute workspace path miss once. - v3.4.0 changed swap on small Linux runners; watch the first runs. Apply order: an admin applies `terraform/runs-on-access` first (the apply role only deploys allowlisted template URLs), then the `runs-on` stack through the usual workflow. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
