Asquator commented on PR #55537: URL: https://github.com/apache/airflow/pull/55537#issuecomment-3319698658
### Workload with one concurrency limit benchmark #### Description **DAGs: 20 Tasks per DAG: 200 `max_active_tasks=5` Operator: `BashOperator` Weight rule: downstream (default)** -> [See DAG code](https://gist.github.com/Asquator/eb25879e72d74143a7ac1003210f8cfb) <-- <details> <summary>Technical details</summary> DB: Postgres 13 CPU: 16 cores - D16s v3 (Azure) RAM: 64GB Executor: `LocalExecutor` Configurations: ``` AIRFLOW__CORE__PARALLELISM=128 AIRFLOW__SCHEDULER__MAX_TIS_PER_QUERY=16 AIRFLOW__CELERY__WORKER_AUTOSCALE=48,16 AIRFLOW__CELERY__OPERATION_TIMEOUT=60 ``` </details> #### Results | Selector strategy ➡️ <br> Category ⬇️| Old | New | Meaning |----------|----------|----------|----------| | Total iterations | 1401 | 331 | Less by ~4.2x | | Average scheduler loop duration | 678[ms] | 1663[ms] | Longer by ~2.45x | <img width="2456" height="1332" alt="Sproc benchmark" src="https://github.com/user-attachments/assets/b3757247-1ee6-40cd-aec1-e81c8ce4a78f"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
