potiuk opened a new pull request, #75833: URL: https://github.com/apache/airflow/pull/75833
Most test jobs on the self-hosted RunsOn runners can't use the CPUs they get. Jobs that run with `--run-in-parallel` (DB, lowest-dependencies and special tests) start one serial pytest per test type, so a job never uses more CPUs than it has test types. Logs from canary run 40517104168 show this: - **Postgres core DB job (8 CPUs):** runs 3 test types (Core, Other, Serialization). Serialization finishes in 2 min, Other in about 21 min and Core in 23 min, so for most of the job only 2 of the 8 CPUs are busy. - **Lowest-dependencies provider jobs (32 CPUs):** about 20 providers per job, so at most about 60% of the CPUs can be used. - **Integration-core and system tests:** one pytest each, on 32-CPU / 128 GB runners. The core test types are split into two matrix jobs to keep the Non-DB xdist collection inside the 16 GB of a public runner. On RunsOn that job gets 128 GB. The DB halves each still last as long as their slowest test type, so on RunsOn the split only doubles the job count. It also ran the single system test once per slice. Changes, for RunsOn only (public runners are unchanged): - Core test types run in one matrix job instead of 2. - Lowest-dependencies provider tests run in 3 jobs instead of 5, which is about one provider per CPU of the 32-CPU runner. - Integration-core and system tests move to the `8+16` / `32+64` size class, the same as the provider integration tests. The selective-checks doc covers the new sizing and slicing rules, and a test covers the slice counts on both runner types. Checks run locally: the selective-checks test suite (passes), the fast static checks (pass, except the docker and network hooks the sandbox blocks), and a roborev review (no findings). This PR has the `canary` and `full tests needed` labels so the full matrix runs on RunsOn and the new job durations can be compared with the canary run above. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Opus 5 Generated-by: Claude Opus 5 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
