gregwhorley opened a new issue, #37369:
URL: https://github.com/apache/airflow/issues/37369
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==7.14.0
### Apache Airflow version
2.8.1
### Operating System
Amazon Linux 2
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
EKS/Kubernetes 1.25
### What happened
Our scheduler pods have been logging an "Unknown Error" from
`KubernetesJobWatcher` around the time of our scheduled DAG runs. The tasks run
successfully despite the error, but this is popping up in our logs pretty
frequently and making it difficult to troubleshoot other issues.
```
[2024-02-07T17:54:30.645+0000] {kubernetes_executor_utils.py:121} ERROR -
Unknown error in KubernetesJobWatcher. Failing
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py",
line 112, in run
self.resource_version = self._run(
^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py",
line 168, in _run
for event in self._pod_events(kube_client=kube_client,
query_kwargs=kwargs):
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/watch/watch.py",
line 182, in stream
raise client.rest.ApiException(
kubernetes.client.exceptions.ApiException: (410)
Reason: Expired: too old resource version: 725263658 (725300129)
Process KubernetesJobWatcher-8:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in
_bootstrap
self.run()
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py",
line 112, in run
self.resource_version = self._run(
^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py",
line 168, in _run
for event in self._pod_events(kube_client=kube_client,
query_kwargs=kwargs):
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/watch/watch.py",
line 182, in stream
raise client.rest.ApiException(
kubernetes.client.exceptions.ApiException: (410)
Reason: Expired: too old resource version: 725263658 (725300129)
[2024-02-07T17:54:31.158+0000] {kubernetes_executor_utils.py:359} ERROR -
Error while health checking kube watcher process for namespace
"our-airflow-dag-runner-namespace". Process died for unknown reasons
[2024-02-07T17:54:31.185+0000] {kubernetes_executor_utils.py:157} INFO -
Event: and now my watch begins starting at resource_version: 0
```
### What you think should happen instead
I looked at the code referenced in the Traceback and saw that `_pod_events`
is supposed to catch Kubernetes client API exceptions and, if its status code
is 410, retrieve resource versions again or bubble it up otherwise. In our
case, the status check fails and the exception is raised and caught by `run()`
then logged as an "unknown error in KubernetesJobWatcher".
```
def _pod_events(self, kube_client: client.CoreV1Api, query_kwargs: dict):
watcher = watch.Watch()
try:
if self.namespace == ALL_NAMESPACES:
return
watcher.stream(kube_client.list_pod_for_all_namespaces, **query_kwargs)
else:
return watcher.stream(kube_client.list_namespaced_pod,
self.namespace, **query_kwargs)
except ApiException as e:
if e.status == 410: # Resource version is too old
if self.namespace == ALL_NAMESPACES:
pods =
kube_client.list_pod_for_all_namespaces(watch=False)
else:
pods =
kube_client.list_namespaced_pod(namespace=self.namespace, watch=False)
resource_version = pods.metadata.resource_version
query_kwargs["resource_version"] = resource_version
return self._pod_events(kube_client=kube_client,
query_kwargs=query_kwargs)
else:
raise
```
### How to reproduce
Our DAGs are controlled by either `KubernetesPodOperator` or
`SparkSubmitOperator` and are run on different schedules where some are run
only once per day while others run every 8 hours. These errors are thrown
around the time of both sets of scheduled runs. Simply running the DAGs will
eventually produce this error for our case.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]