GitHub user gudata added a comment to the discussion: Airflow unable to handle more than 16 tasks at once
We had the same problem on airflow 2.6 or 2.9 (don't remember) With a dedicated pool with 256 tasks. I start 50 ec2 machines with enough ram assigned to that pool and only 10 active tasks. The problem gone when we upgraded to the new airflow 3 without changing anything. The setup is: celery with airflow in containers from the example compose.yaml GitHub link: https://github.com/apache/airflow/discussions/32609#discussioncomment-14415538 ---- This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
