Subham-KRLX commented on issue #67272: URL: https://github.com/apache/airflow/issues/67272#issuecomment-4508853900
Thanks for confirming with AWS this is a Celery fork bug affecting all Airflow 3.0.X versions the worker autoscale workaround didn't help because it doesn't fix the underlying fork deadlock issue. The waiter throttling fix in PR 67222 won't solve this either your jobs fail because Celery kills the process mid execution due to thread deadlock from fork your best options are to switch to deferrable tasks wait for Airflow 3.1 to see if they fixed the Celery fork issue, or stay on 2.7.2 until 3.1 is stable this is an Airflow core issue not an EMR provider issue so you should escalate to Airflow maintainers to prioritize the Celery fork fix for 3.1 or a patch release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
