github-actions[bot] opened a new pull request, #54922: URL: https://github.com/apache/airflow/pull/54922
LocalExecutor would only actually begin execution for every second task submitted to it. This change fixes this behaviour. The number of queued tasks would need to exceed the number of currently running tasks for a new task to be executed, which is logically close to what is needed but not quite. This means if 4 tasks are running and 4 are pending none of the 4 tasks will begin, but if a 5th task is queued it will begin and there will be 5 tasks running and 4 waiting, it will take another two tasks submitted (for a total of 5 then 6 pending tasks) before the next is started, and so one. So in reality one of every two tasks is started. The logic is really as simple as: if we have any pending tasks and we are still within our parallelism limits, start those tasks running. This schedules tasks in accordance to user expectations and how Airflow 2.X scheduled tasks for the LocalExecutor (since this regressed during the migration from 2.X to 3.X for the LocalExecutor) (cherry picked from commit e36a809ebdaa04fbc7a76d546ff9fa1db2cb39d4) Co-authored-by: Niko Oliveira <[email protected]> Co-authored-by: Jarek Potiuk <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
