tatiana commented on issue #75406: URL: https://github.com/apache/airflow/issues/75406#issuecomment-6342280136
That would be great, @amoghrajesh! When I opened the Cosmos PR, I hoped moving the backup to the task state store would improve performance and simplify the implementation. The performance side worked out (both due to the lack of a secrets backend and in large dbt projects with over 500 nodes). The simplification did not: because the retry callback cannot write to the store, we still have to keep the Variable path to flush the tail of the backup, so the implementation ended up more complex than before, not less. If the callback could run before the retry transition, we could drop the Variable fallback entirely on 3.3+. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
