dabla commented on PR #62922: URL: https://github.com/apache/airflow/pull/62922#issuecomment-5865250109
Three more commits, following up on the earlier remark about suffixing on the pull side and a gap it uncovered on the state store: - f5deadde98 **Give each iteration its own task state store keys, as for XComs.** Every iteration runs under the same task instance, so a key one iteration stored through `context["task_state_store"]` was shared with its siblings, last writer wins. `IndexedTaskStateStoreAccessor` decorates the parent's accessor and suffixes keys with the index on get/set/delete, sync and async, the same rule `IndexedTaskInstance.xcom_push` applies to XComs. `clear()` is refused inside an iteration. The operator's own checkpoints keep their unsuffixed `_iterable_<index>` keys in the parent's store. Tested on the accessor, on `IndexedTaskInstance`, and end to end with a sync and an async operator that keep state. - 55c48ed508 **Build the sub-task's context in the runner.** The clone of the parent's context with the sub-task's ti, state store view and outlet events was built twice in the operator; it now lives in one place next to the task instance it belongs to, and the two `_run_operator` methods are gone. - 7dd2accadb **Rename `TaskExecutor` to `IndexedTaskRunner`.** It read like one of Airflow's executors; it runs one index of an iterated task, so it is named after `IndexedTaskInstance`. The four stacked PRs are restacked on top; each still one commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
