dabla commented on PR #62922: URL: https://github.com/apache/airflow/pull/62922#issuecomment-5352779924
> A question about how iteration state survives a retry. > > If I'm reading `XComIterable` correctly, per item results land under `return_value_0`, `return_value_1` and so on with `map_index = -1`. Does resume after failure read those keys back to skip items that already finished? > > If it does, I do not think it can work today. On every task run that is not a deferral resume, the API server collects every xcom key for the task instance and sends them to the worker as `xcom_keys_to_clear` it. And the task sdk runner deletes them on a task start up. > > For a task running for 100 items, with crash on item 60, Attempt 1's `return_value_0` through `return_value_59` are gone before attempt 2 starts, and the retry repeats all 100 items. > > A suggestion: the xcom design for results looks right to me. Downstream tasks have to read them, and that is what XCom is for. > > Its the progress record that needs a different home, and `task_state_store` from AIP-103 is built for exactly this, since rows are scoped per task instance and deliberately survive retries. The AIP-104 wiki page already mentions AIP-103 for intermediate state, so this may just be the implementation catching up with the design. > > That split might also shrink #70223. If progress lives in task state and only final results go to XCom, the bulk-key read endpoint may not be needed. Indeed you're correct, that's why I was excited when AIP-103 was announced, as it will solve this issue, but I would like to do this in a separate PR once this PR is merged, otherwise the PR would become too big and it's already very big. Same for the XComIterable optimisation, I also created a dedicated [PR](https://github.com/apache/airflow/pull/70223) to address this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
