Asquator commented on PR #55537: URL: https://github.com/apache/airflow/pull/55537#issuecomment-3324403325
### BENCHMARK: Workload with two concurrency limits #### Description **DAGs: 20 Tasks per DAG: 200 `max_active_tasks=5` Pools: 20 Slots in pool: 5 Tasks are uniformly distributed among pools Operator: `BashOperator` Weight rule: downstream (default)** -> [See DAG code](https://gist.github.com/Asquator/6e52baf6a6687c9edefa97ec5a99121d) <-- <details> <summary>Technical details</summary> DB: Postgres 13 CPU: 16 cores - D16s v3 (Azure) RAM: 64GB Executor: `LocalExecutor` Configurations: ``` AIRFLOW__CORE__PARALLELISM=128 AIRFLOW__SCHEDULER__MAX_TIS_PER_QUERY=16 AIRFLOW__CELERY__WORKER_AUTOSCALE=48,16 AIRFLOW__CELERY__OPERATION_TIMEOUT=60 ``` </details> #### Results | Selector strategy ➡️ <br> Measurement ⬇️| Old | New | Meaning |----------|----------|----------|----------| | Total iterations | 1443 | 336 | Less by ~4.3x | | Average scheduler loop duration | 739[ms] | 1625[ms] | Longer by ~2.2x | | Total time| ~1065[sec] | ~546[sec] | Throughput improvement ~x1.95 On the left: **old** selector On the right: **new** selector <img width="2484" height="1376" alt="pool benchmark" src="https://github.com/user-attachments/assets/ecd7cb04-b724-48ac-b806-62b3e172422d"; /> Note: _Due to Statsd+Prometheus approximations, sometimes we see metrics at 0 when they're small enough._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
