aeroyorch commented on PR #65932: URL: https://github.com/apache/airflow/pull/65932#issuecomment-4378472306
> This is "too specialized" and the logic about why it only applies to specific fields or in specific cases is not clear to me that it is the behaviour we want. > This is "too specialized" and the logic about why it only applies to specific fields or in specific cases is not clear to me that it is the behaviour we want. First of all, thanks for the review! I agree that the current approach is too specialized. The intent behind limiting the fields was to "optimize" the update so that we only refreshed the values that can be changed internally by `refresh_from_task`. That said, the desired fix in this PR is to correctly update fields such as priority, queue, etc., which, according to the Airflow docs and existing examples, can be changed per retry attempt. In fact, our specific use case is to use a different queue for retries, backed by KEDA autoscaling through the official Helm chart, which uses the queue SQL column to autoscale groups of workers. So my understanding is that dynamically modifying some scheduling options per try is still expected Airflow behavior, and not something considered legacy or unintended. Is that correct? I just want to confirm that I am not trying to fix behavior that we actually want to move away from. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
