github-actions[bot] opened a new pull request, #75992: URL: https://github.com/apache/airflow/pull/75992
* Size CI test jobs on RunsOn runners to the parallelism they can use Jobs that run tests with --run-in-parallel start one serial pytest per test type, so they never use more CPUs than they have test types. On RunsOn, splitting core test types into two jobs only doubles the jobs: each half still lasts as long as its slowest test type, and the out-of-memory risk the split guards against on public runners does not apply to the larger RunsOn runners. The lowest-dependencies provider jobs left their 32 CPUs partly idle, and the integration-core and system tests ran a single pytest on 32-CPU runners. Generated-by: Claude Opus 5 * Run provider tests one provider per test type on RunsOn runners Grouping all providers but the long-running ones into one test type left the provider jobs waiting on that group's single serial pytest while the other CPUs idled. The Non-DB jobs run every provider in one xdist pytest, where splitting test types only adds entries, so they get the providers unsplit. Generated-by: Claude Opus 5 * Give docker on RunsOn runners enough address pools for parallel test runs Each test type of a parallel test run is a docker compose project with its own network. Docker's default address pools fit about 30 networks, so provider jobs running a test type per CPU on 32-CPU and larger runners failed with "all predefined address pools have been fully subnetted". Generated-by: Claude Opus 5 * Split the longest-running provider tests into several test types on RunsOn Once every provider runs as its own test type, a job lasts as long as its slowest provider, and amazon, google and standard each take far longer than the rest as a single serial pytest. Splitting them lets the large RunsOn runners work on their parts at once. Lowest-dependencies jobs are sized by test-type count, so a change touching a few providers keeps one job. Generated-by: Claude Opus 5 * Keep split provider test types out of compose project and artifact names Docker Compose project names and GitHub artifact names reject the colon that names a part of a split provider. Generated-by: Claude Opus 5 * Skip split provider test types whose provider is ignored A split provider part selects a folder inside the provider's tests, and pytest runs an explicitly selected path even when its provider folder is ignored, so a skipped or excluded provider's parts would still run. Generated-by: Claude Opus 5 * Fit RunsOn runner sizes to the work of each job and record their CPU and memory use Several jobs ran a single process, or a minute or two of work, on 32-CPU or 8-CPU runners, and fourteen Helm unit test jobs each spent more time preparing the image than running their tests. Charting each job's CPU and memory use lets the size classes be set from measurements rather than from how the jobs are expected to behave. Generated-by: Claude Opus 5 * Skip ignored test selections by the paths the test type selects The check for an empty run treated the first remaining argument, or any path-like argument, as a test selection, so option values such as the "8" of "-n 8" or a path passed to --rootdir decided whether tests ran. It now compares the ignored folders with the paths the test type itself selects, and passes paths the user names explicitly to pytest as they are. Generated-by: Claude Opus 5 * Reject split provider test types where all test types share one pytest Under xdist every test type runs in one pytest process, where the --ignore arguments of a provider's other part would silently be dropped or would remove its sibling parts, so the wrong tests would run. Generated-by: Claude Opus 5 * Count only test folders as the selections of a test type Test suites such as All-Postgres add pytest options with values, which were taken for selected test paths, so a run whose every test folder was ignored still started pytest with nothing selected. Generated-by: Claude Opus 5 * Run only its own test type in each worker of a parallel test run Each worker kept the whole list of test types, so with --skip-db-tests every worker selected the folders of all of them. Generated-by: Claude Opus 5 * Raise the AIO limit on RunsOn runners for parallel MySQL tests A few dozen MySQL servers running at once exhaust the default fs.aio-max-nr, and InnoDB then fails with "io_setup() failed with EAGAIN". Generated-by: Claude Opus 5 * Keep the provider compatibility tests on the grouped provider test types Each test type of a compatibility test installs an Airflow version and the providers before running, so listing every provider as its own test type multiplied those installs and the jobs ran out of time. (cherry picked from commit 083b82d7c4d6fe3864027d0de42fdba48bb1ad8b) Co-authored-by: Jarek Potiuk <[email protected]> Generated-by: Claude Opus 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
