Hello community, I want to spark again a discussion that was held in the past and delayed (due to the focus on releasing airflow 3.2), now that 3.2 is released, I think it might be a good Idea to bring up the discussion again.
Wiki: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-100+Eliminate+Scheduler+Starvation+On+Concurrency+Limits In the current situation, tasks may starve in airflow, and in large scale deployments (hundreds of thousands of tasks (or more) per day) we tend to experience severe starvation, where a group of tasks may starve other tasks, not allowing them to run, as described in the wiki. After the february devcall, where I have proposed the AIP, a few comments have arised, and so I had begun to research again about different scheduling algorithms and I have added to the considerations as part of the AIP. As of now, the state of the AIP is where there are a few ideas proposed (some of which are pretty similar to each other, while others are quite different), as the main concern from the devcall was that the approaches given might not be the best way to solve the issue, as it is a very hard problem to solve. After that, I have made some edits to the AIP and to the propositions, in order to help decide and clarify the advantages and disadvantages of each approach. The current "best approach" can be found here here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Currentbestproposition>, where the new proposed algorithms are defined here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Othernonagingalgorithmsother_algs> . In order to continue with the effort, a community consensus needs to be reached about the preferred solution/solutions, once this is done, it is possible to go on and implement + stress test the proposed solution/s. I would appreciate a review from community members, moreover, I would also appreciate any new propositions or improvements which can be done. Best regards Natanel.
