amoghrajesh opened a new pull request, #77197:
URL: https://github.com/apache/airflow/pull/77197
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
### What
A durable `KubernetesPodOperator` is meant to let a retry pick up a pod that
is still running instead of starting the work again. That does not work when
the try ends because of `execution_timeout`, even with
`on_kill_action="keep_pod"`. Every retry starts a new pod from zero, so a long
job whose timeout is slightly too low can never finish.
### Current behaviour
When `execution_timeout` fires, the timeout exception leaves `execute_sync`,
and its `finally` block runs `cleanup` before the task runner calls `on_kill`.
At that point `_killed` is still False, so `cleanup`:
1. labels the pod as already checked, and
2. deletes it per `on_finish_action` (default `delete_pod`).
`on_kill_action` never gets a say. Even with `on_finish_action="keep_pod"`,
the "already checked" label makes the next try reject the pod:
```
Pod airflow/kpo-keep-pod-work-kehfcc39 from task state store was already
checked by a prior attempt, creating a fresh pod instead of reattaching.
```
### Proposed change
When `AirflowTaskTimeout` interrupts `execute_sync`, the operator is durable
(`reattach_on_restart`), and `on_kill_action="keep_pod"`, `cleanup` now leaves
the pod running and unlabelled. The next try then reattaches to it through the
task state store, gets a fresh `execution_timeout` window, and waits for the
remaining work. Every other path (success, pod failure, other exceptions,
default `on_kill_action`) is unchanged.
The `on_kill_action` docstring and the "Durable execution" section in
`operators.rst` describe the new behaviour and its limits.
### Changes of Note
- The decision is made in an `except AirflowTaskTimeout` in `execute_sync`,
because the task runner only calls `on_kill` after the exception propagates. By
then `cleanup` has already run.
- This is opt-in through `on_kill_action="keep_pod"`. Keeping pods past a
timeout by default would leave running pods behind whenever retries run out.
- `AirflowTaskTimeout` is imported from
`airflow.providers.common.compat.sdk`, so the provider's Airflow floor is
unchanged.
### User implications / backcompat
- Default behaviour is unchanged. A timeout still deletes the pod unless
`on_kill_action="keep_pod"` is set.
- Users who already set `on_kill_action="keep_pod"` with durable/reattach on
(the default) will now see the pod survive a timeout, not just a manual kill.
That includes Airflow `<` 3.3, where `reattach_on_restart` defaults to True and
the retry reattaches through the label search.
- If the last retry times out, its pod is left running and has to be cleaned
up manually. This is documented.
- For a timed-out try that keeps its pod, `on_finish_action`,
`on_pod_teardown` and the failure XCom push and event logging are skipped.
`on_pod_cleanup` callbacks still run.
### Testing
End to end on a kind cluster (`breeze k8s`, LocalExecutor). The task is a
pod that needs 240 s of work, with `execution_timeout=90s`, `retries=3` and
`durable=True`. The same Dag ran once with `on_kill_action="keep_pod"` and once
with `on_kill_action="delete_pod"` as a control.
#### Before
Both variants behaved the same:
| | `keep_pod` | `delete_pod` |
|---|---|---|
| Pods created | 4 (new pod every try) | 4 |
| Result | failed after 4 tries | failed after 4 tries |
All 4 tries failed, each on a new pod:
<img width="3200" height="2000" alt="01_before_task_tries"
src="https://github.com/user-attachments/assets/fd3cf502-e0ba-4c36-a720-549966fe8efa"
/>
Try 2 found the kept pod but rejected it as "already checked":
<img width="3200" height="2000" alt="02_before_try2_already_checked"
src="https://github.com/user-attachments/assets/4fe1747b-e0b6-4c51-905c-0d65e78e761a"
/>
#### After
| | `keep_pod` | `delete_pod` |
|---|---|---|
| Pods created | **1** | 4 |
| Tries 2 and 3 | `Found matching pod kpo-keep-pod-work-4s7uxflh`,
reattached | new pod |
| Result | **success on try 3**, 4 s after the pod finished its 240 s of
work | failed after 4 tries |
Try 1 timed out and kept the pod for the next try:
<img width="3200" height="2000" alt="04_after_try1_keeping_pod"
src="https://github.com/user-attachments/assets/71e942d5-83c5-48c2-9b36-79162d7f2404"
/>
Try 2 reattached to the same pod, timed out again, and kept it:
<img width="3200" height="2000" alt="05_after_try2_found_matching_pod"
src="https://github.com/user-attachments/assets/1aae8e9e-abef-45ad-be8d-6654109da97a"
/>
Try 3 reattached and succeeded once the pod finished (`started at 09:25:42`,
`finished at 09:29:42`):
<img width="3200" height="2000" alt="03_after_task_tries"
src="https://github.com/user-attachments/assets/cb0aa24e-91ef-4ce1-84ef-19aa8f93cdbd"
/>
A third run, watched with `kubectl get pods`. One `keep_pod` pod (yellow)
stays alive across all tries, while `delete_pod` (blue) gets a new pod every
try:
<img width="2172" height="1420" alt="06_pod_list"
src="https://github.com/user-attachments/assets/588f7d02-b463-47b1-ab03-932cb57c7320"
/>
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]