Nataneljpwd commented on issue #45636: URL: https://github.com/apache/airflow/issues/45636#issuecomment-2798867988
After looking a little deeper, it looks like the solution recommended above would not solve this problem, all that is done there is to stay longer in the scheduler loop, it does not look as if it locks any additional tasks so each run mostly the same tasks will be returned from the query, and if it does lock, then the time spent in the critical section and amount of locked tasks increase, which both are things that we try to minimize, time in the critical section and amount of locked tasks. The problem does not seem to lie in the scheduler loop but in the prioritization and the way airflow chooses which tasks to run first, so maybe in addition to the window solution, the way prioritization works in airflow should change I think it might be a deeper problem and some interesting solutions could arise from a discussion -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
