Jorricks opened a new pull request #16718: URL: https://github.com/apache/airflow/pull/16718
Celery executor is currently adopting anything that has ever run before and has been cleared since then. We have a DAG that runs over 200 tasks but has a concurrency of 3 (because of the size of some jobs). Therefore, many tasks should be in scheduled for a bit. However, because of the current implementations, if these tasks ever run before they would get adopted by the schedulers executor instance and become stuck forever (which is fixed in another PR). Now this PR prevents the tasks that are in scheduled from being picked up as adopted and furthermore it makes sure that cleared tasks are fully reset. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
