bugraoz93 commented on code in PR #68814: URL: https://github.com/apache/airflow/pull/68814#discussion_r3477197963
########## dev/breeze/doc/ci/04_selective_checks.md: ########## @@ -573,6 +573,21 @@ That's why we do not base our `full tests needed` decision on changes in depende from the `provider.yaml` files, but on `generated/provider_dependencies.json` and `pyproject.toml` files being modified. This can be overridden by setting `full tests needed` label in the PR. +[2] Note on how `providers-test-types-list-as-strings-in-json` is split. + +The provider DB tests run as parallel test-type *groups* (`breeze testing providers-tests +--run-in-parallel`), but each group runs **without xdist** — its tests run serially. To keep all +parallel slots busy, the big "all other providers" group is **not** emitted as one +`Providers[-amazon,celery,google]` monolith (which would be the serial bottleneck while the other slots +idle); instead every provider except the few isolated big ones (`amazon`, `celery`, `google`) is split +into `NUMBER_OF_PARALLEL_PROVIDER_SLICES` balanced chunks (`Providers[a,b,c] Providers[d,e,f] …`). The Review Comment: Great split idea! I only concern numbers are distributed sparse. Asked AI to sort and distribute :) Just an idea as an improvement. It looks some of these 2 chunks A and S ones will need more time than others. We can also add some limits to how many we have and split those alone. Maybe it is over engineering even though there could be 2 big chunks similar to big provider split should he still okay. If looks okay, can follow up after this :) --- AI ``` The Airflow Registry currently lists 101 providers. Here is the count grouped by the first letter shown in the registry: Letter Providers A 24 C 7 D 6 E 3 F 3 G 4 H 2 I 4 J 2 K 2 M 4 N 1 O 7 P 7 Q 1 R 1 S 12 T 4 V 2 W 2 Y 2 Z 1 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
