dstandish opened a new issue, #75904: URL: https://github.com/apache/airflow/issues/75904
### Apache Airflow Provider(s) cncf-kubernetes ### Versions - apache-airflow: 3.4.0 (main) - apache-airflow-providers-cncf-kubernetes: 10.25.0 (main) ### What happened `KubernetesPodOperator._get_ti_pod_labels` labels a pod with `dag_id`, `task_id`, `run_id`, `try_number`, and `map_index` (only when `ti.map_index >= 0`). Since dynamic regions (#74275, #74345, #74346), `ti.map_index` is the *public* map index (`public_map_index_expression`). It is the region index only for a task in its own mapped expansion. For a task inside a task loop, or any region owned by another node, it is `-1`, so KPO writes no `map_index` label at all. Different iterations of the same task therefore produce pods with identical labels, and nothing on the pod identifies which task instance created it. This affects two things: 1. **`find_pod` / reattach.** `_build_find_pod_label_selector` uses these labels without `try_number`. Per `loops.rst`, "the next iteration can start while the previous iteration's teardowns are still running", so a KPO teardown in iteration N+1 can run while the same teardown from iteration N still has an unchecked pod. On reattach, the selector matches both pods. That raises `FoundMoreThanOnePodFailure`, or with `reattach_on_restart=False` falls into `process_duplicate_label_pods`, possibly picking the other iteration's pod. A pod left unmarked by a crashed worker in an earlier iteration could be adopted the same way. I have not reproduced this end to end. It follows from the selector and the loop docs. 2. **External tools that map pods back to task instances**, for example to clean up orphaned pods. They can't tell iterations apart and have to guess. A pod labeled `try_number=1` with no `map_index` could be iteration 1's live attempt or a stale first attempt of iteration 2 that has since retried. ### What you think should happen instead Add labels that identify the task instance exactly: - **`ti_id`**: `ti.id`. On 3.4 every try is its own row with its own UUID (`prepare_db_for_next_try` assigns a new `uuid7()`), so this names exactly one attempt. External tools can then look up the row by primary key and check `working_set` / state, with no key reconstruction or `try_number` heuristics. - **`region_id` + `region_index`**: for `find_pod`. Reattach intentionally matches across tries, so `ti_id` can't be used there, but adding the region coordinates to the selector would keep loop iterations (and nested mapped instances) apart. All of these fit within the 63-character label value limit. The new labels would only be set when the TI has the attributes (Airflow 3.4+), so older Airflow versions are unaffected. ### How to reproduce Labels (code inspection): 1. Write a Dag with a `task_group.loop(...)` whose body contains a `KubernetesPodOperator`. 2. Run two or more iterations. 3. Every iteration's pod has the same `dag_id` / `task_id` / `run_id` labels and no `map_index` label. Only `try_number` can differ. Reattach collision (not yet reproduced): add a slow KPO teardown to the loop body so the iteration N+1 teardown starts while iteration N's is still running, then restart the worker running iteration N+1's teardown. ### Anything else Relevant code: - `providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py`: `_get_ti_pod_labels`, `_build_find_pod_label_selector`, `find_pod` - `airflow-core/src/airflow/models/task_coordinates.py`: `public_map_index_expression` ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's Code of Conduct -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
