robg-eb commented on PR #33718: URL: https://github.com/apache/airflow/pull/33718#issuecomment-1914788890
@dstandish and team - Thank you for creating this PR, as we are currently unable to properly use retries with Deferrable (Async) sensors, because of the inconsistency in retry / timeout behavior between sync and async sensors — Currently, a sync sensor will only apply retries up to the `timeout` , but an async sensor will apply `retries * timeout` total amount of waiting time. We tested this PR now and found that it DOES properly begin to resolve this difference - In the case where no interruption happens, and no retry is needed, then the async sensors on this PR do only apply retries up to the timeout interval, just like sync sensors do. This is great. However, we found another inconsistency with sync sensors here. In the case where an interruption does happen and a retry is needed, it seems with this implementation, the timeout interval will **restart** after the retry. To take an example - If we have an async sensor with a 12 hour timeout , and 5 hours into its run, it is interrupted (let’s say, for example, there was a network or metadata DB issue). At this point, when it goes into retry, it then **resets the timeout,** and starts waiting another 12 hours, meaning it would wait a total of 17 hours. This is not consistent with how sync sensors work, and how they are documented here - https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#timeouts > If the sensor fails due to other reasons such as network outages during the 3600 seconds interval, it can retry up to 2 times as defined by retries. Retrying does not reset the timeout. It will still have up to 3600 seconds in total for it to succeed. So, if we want async sensors to behave the same as sync sensors (and in line with this documentation about sync sensors), we still need to resolve this inconsistency. Thoughts on that? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
