Asquator commented on PR #54284: URL: https://github.com/apache/airflow/pull/54284#issuecomment-3239277533
Anyway, before we move any further, we have to decide what priorities are. They indeed pose a weak limit, but in the current implementation it's not only weak, but also a very **vague** parameter. Current behavior respects priorities between scheduler's iterations but doesn't respect them between the critical section iterations (one or more). But the fact we may drop some tasks and schedule `< max_tis` of them, makes the parameter's behavior unpredictable (make sure you understand why), and I personally think (@Nataneljpwd will prbly agree with me) that we should decide on a new behavior. The example I cross-posted on slack is: Suppose we have a pool with 2/3 open slots. There are two tasks: Priority: 2, pool slots: 2 Priority: 1, pool slots: 1 The question is: Should we schedule the second task right away because we have enough pool slots, or would we like to wait for the first task because it's more prioritized, and it should run first in that pool? Except pools it also applies to DAG runs, DAG-wise and mapped-task limits. Current behavior is that we may schedule the second task anyway, but is it the right way? I mean, what if we happen to constantly have just one pool slot freed and schedule the less-prioritized task all the time? Isn't it considered starvation for the prioritized task? This decision is very important, including in the scope of what we're doing in https://github.com/apache/airflow/pull/53492. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
