thc1006 opened a new issue, #71748:
URL: https://github.com/apache/airflow/issues/71748
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
**Issue Description**
This is a follow up of #71743, and sorry if I miss something here.
In #71744 the pod uid is check one time, when the identity is read back from
task state store. That close the reattach case. But after that check the
operator keep working by name and namespace only, so if the pod go away and
another pod take the same name, the later call can land on the other pod.
Two place I find:
1. `KubernetesPodTrigger` keep only `pod_name` and `pod_namespace`. The
constructor and `serialize()` both have no uid, and `_get_pod()` do
```python
pod = await self.hook.get_pod(name=self.pod_name,
namespace=self.pod_namespace)
```
on every poll. A deferrable task can wait very long, so this window is
not small.
2. `PodManager.delete_pod()` send
```python
self._client.delete_namespaced_pod(
pod.metadata.name, pod.metadata.namespace,
body=client.V1DeleteOptions()
)
```
with no preconditions. `KubernetesHook.delete_pod()` and `on_kill()` are
the same, they all delete by name.
So the order is like this:
```
operator check the uid, pod A is correct
pod A is gone
pod B take the same name
next poll / log read / delete work on pod B
```
**Steps to reproduce**
I do not race a real trigger poll, sorry. I only show the delete part, on
k8s v1.37.0-rc.0 with apache-airflow-providers-cncf-kubernetes 10.21.0. The
trigger part I read from the code.
```
the operator observed this pod and remembered it
kpo-uid-delete-test/my-pod uid = 4057eb7b-8483-4064-8779-ecc230861cf1
it goes away and something else takes the name
kpo-uid-delete-test/my-pod uid = 5284f5cd-b90b-4e4c-9f10-a039495a53ea
A. delete the way PodManager.delete_pod does it today
accepted -> the replacement is being deleted
pod gone. the deleted one was uid 5284f5cd-..., not the remembered
4057eb7b-...
B. same delete, but with the remembered uid as a precondition
another replacement appears, uid = c3c80f2a-54e9-4714-a147-d50f90514506
rejected 409 Conflict
still alive: uid c3c80f2a-54e9-4714-a147-d50f90514506
```
<details><summary>script I use</summary>
```python
import subprocess, time
from kubernetes import client as k8s, config as k8s_config
NS, POD = "kpo-uid-delete-test", "my-pod"
def body():
return {"apiVersion": "v1", "kind": "Pod",
"metadata": {"name": POD, "namespace": NS},
"spec": {"restartPolicy": "Never", "containers": [
{"name": "base", "image": "registry.k8s.io/pause:3.9"}]}}
def wait_gone(core):
while True:
try:
core.read_namespaced_pod(POD, NS); time.sleep(1)
except k8s.rest.ApiException:
return
subprocess.run(["kubectl", "delete", "ns", NS, "--ignore-not-found"])
subprocess.run(["kubectl", "create", "ns", NS])
k8s_config.load_kube_config()
core = k8s.CoreV1Api()
uid_a = core.create_namespaced_pod(NS, body()).metadata.uid
core.delete_namespaced_pod(POD, NS,
body=k8s.V1DeleteOptions(grace_period_seconds=0))
wait_gone(core)
uid_b = core.create_namespaced_pod(NS, body()).metadata.uid
print("remembered", uid_a, "/ now", uid_b)
# A: how delete_pod does it today
core.delete_namespaced_pod(POD, NS,
body=k8s.V1DeleteOptions(grace_period_seconds=0))
wait_gone(core)
# B: with the uid as a precondition
uid_c = core.create_namespaced_pod(NS, body()).metadata.uid
try:
core.delete_namespaced_pod(POD, NS, body=k8s.V1DeleteOptions(
grace_period_seconds=0,
preconditions=k8s.V1Preconditions(uid=uid_a)))
except k8s.rest.ApiException as e:
print("rejected", e.status, "/ still alive",
core.read_namespaced_pod(POD, NS).metadata.uid)
```
</details>
### What you think should happen instead?
Once the operator know which pod it work on, I think the uid can travel with
it, so every later call talk to the same pod.
For the trigger, `KubernetesPodTrigger` can take a `pod_uid`, put it in
`serialize()`, and check it after each `_get_pod()`. If the uid is not the
same, the pod we wait for is already gone, and that is a normal end condition,
not a new pod to watch.
For the delete, `V1DeleteOptions` already accept `preconditions`, and the
client we pin (kubernetes 36.0.3) already have `V1Preconditions` with a `uid`
field. A 409 from it just mean the pod we want to delete is not there any more,
so we can treat it like the 404 we already handle, and leave the other pod
alone. This is nicer than read first and delete after, because the api server
do the check for us.
But this change touch the sync path, the async path and the cleanup
together, and the trigger serialization is not a small thing to change, so I do
not want to guess. May I ask if this direction look ok to you before I try a
PR? If you already think about it and decide it is fine as it is now, please
tell me and I close this. Thank you very much.
Two more thing I am not sure about, maybe they belong here or maybe not:
- `get_or_create_pod()` persist the uid from the server answer, but it
`return pod_request_obj`, so `self.pod` carry no uid. If the uid become the
runtime identity then it need to live somewhere.
- #70140 list the pods first and delete by name later. It look like the same
shape to me, but it is not my PR so I only mention it.
### Operating System
Ubuntu 26.04 LTS
### Deployment
Virtualenv installation
### Apache Airflow Provider(s)
cncf-kubernetes
### Deployment details
kubeadm cluster, k8s v1.37.0-rc.0, single node.
apache-airflow==3.3.1, apache-airflow-providers-cncf-kubernetes==10.21.0,
kubernetes==36.0.3
### Anything else?
Thank you for #69914 and for the review on it, the durable execution is very
useful for us. This is only about the part after the first check, everything
else already work well.
Related: #71743 and #71744.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
