GitHub user potiuk added a comment to the discussion: Airflow[v2.11.0] Celery worker CPU starvation
You likely have some problem with the structure of your Dags (many parallel tasks) and scheduling after task execution which takes long time - you can disable "schedule after task execution" configuration - which was intended to be an optimization but it suffers from similar kind of issues in certain cases. This feature has been removed in Airflow 3, and we also highly recommend you to move to Airflow 3 as Airlfow 2.11 is in limited maintenance and will stop receiving patches (even security patches) in April. BTW. We are in a process of testing the RC candidate of 2.11.1 -> see https://github.com/apache/airflow/issues/62056 - we need people like you to test the release candidate and confirm that it works for them - this release contains a few security patches and is likely the last one we release for 2.11, so you have one of the last chances to report any serious/security related issues that we can fix in this release. GitHub link: https://github.com/apache/airflow/discussions/62065#discussioncomment-15836464 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
