uranusjr commented on issue #49508: URL: https://github.com/apache/airflow/issues/49508#issuecomment-2864959991
I don’t think multiple schedulers would help since all those schedulers would pull runs with the same query in `get_queued_dag_runs_to_set_running`, and once you hit the point of starvation, they all get stuck on the same runs over and over, no matter how many schedulers you have. For multiple schedulers to help, we need to introduce additional discrimination logic between schedulers so they look at different dag runs to have a possibility to untangle the situation. Another thing I’m thinking now is, maybe we should just update `last_scheduling_decision` when a dag run is selected in the queued state? Currently we only update the field after the run enters the running state, so when no more runs can enter running, the flag never changes, and the query is stuck selecting the same runs over and over and not making progress. I think this can work…? Not sure. Anyone interested to help verify this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
