dstandish commented on PR #54284: URL: https://github.com/apache/airflow/pull/54284#issuecomment-3179676189
OK after talking with @xBis7 yesterday about the priority weight issue, it occurred to me we can sidestep that problem by storing last_queueing_decision at the dag run level. So I moved the [new] column from task instance to dag run. So now, whenever we "see" a dag run in the query for `_executable_task_instances_to_queued`, we update that dag run's `last_queueing_decision`. Then the next time we run the query, in effect we prioritize the TIs in dag runs that we did not see last time. This would only have an effect if there are more "scheduled" TIs than we are allowed to query at once (by user setting). So essentially what this does is ensure that we don't get stuck querying the TIs for the same dag run every time -- it will ensure that the scheduler will rotate through the TIs for all dag runs. And within each dag run, it will still sort the highest priority tasks first. But, thinking about it now, I guess it still breaks the priority concept because it gives proirity to dag runs it hasn't seen in the query (and this might have just been because its tasks are low priority). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
