Ohashiro commented on issue #44618: URL: https://github.com/apache/airflow/issues/44618#issuecomment-2573201358
> For the second case, in the operator, you should do in the execute method an xcom_pull to see if there is an existing dataset_refresh_id or not, and if not you know you have to execute the whole flow, if it's there then you know you should only trigger the second part. After a bit a investigation, I think this won't be possible to pass XCom messages between different task executions of the same operator (if this was what you meant). According to [Airflow XComs documentation](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/xcoms.html), "_If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent._". From what I understand, it is not possible for a 2nd operator (in a task retry) to access the 1st task XCom message. However, it should be possible to retry the operator "execute" function without retrying the task, as done in `DmsStartReplicationOperator`: https://github.com/apache/airflow/blob/9ba279d15f26088400b15281b3cc346e2d7a0e30/providers/src/airflow/providers/amazon/aws/operators/dms.py#L720-L723 Though I'm not sure if this is what you'd prefer. After looking at the codebase, I see retry mechanisms in some hooks (ex:`BaseDatabricksHook`, `LivyHook`, `GlueJobHook`, `EmrContainerHook`...), handlers (ex: `S3TaskHandler`) and in some operators as well (ex: the Kubernetes operators as your colleague pointed out, or some AWS operators such as `BedrockCustomizeModelOperator`, `DmsDeleteReplicationConfigOperator`...). I didn't look at all the files, there must be others. The retry can be used to wait for a status (ex: hook `ElastiCacheReplicationGroupHook`, method `wait_for_availability`, [link](https://github.com/apache/airflow/blob/9ba279d15f26088400b15281b3cc346e2d7a0e30/providers/src/airflow/providers/amazon/aws/hooks/elasticache_replication_group.py#L120)) or to wait for an API request to succeed. In our case, I think that we can fall in these cases: We could add a retry in `PowerBIHook.get_refresh_details_by_refresh_id` before raising the PowerBIDatasetRefreshException? Not sure which solution is best to be honest. What do you think? - Should I set the retry within the operator? with a `retry_execution` method? or a `while` loop in the `execute` method? (or else?) - Should I handle the retry in the existing `while` loop from the trigger? - Or should I set a retry in the hook directly? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
