gregwhorley opened a new issue, #37369:
URL: https://github.com/apache/airflow/issues/37369

   ### Apache Airflow Provider(s)
   
   cncf-kubernetes
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-cncf-kubernetes==7.14.0
   
   ### Apache Airflow version
   
   2.8.1
   
   ### Operating System
   
   Amazon Linux 2
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   EKS/Kubernetes 1.25
   
   
   ### What happened
   
   Our scheduler pods have been logging an "Unknown Error" from 
`KubernetesJobWatcher` around the time of our scheduled DAG runs. The tasks run 
successfully despite the error, but this is popping up in our logs pretty 
frequently and making it difficult to troubleshoot other issues.
   
   ```
   [2024-02-07T17:54:30.645+0000] {kubernetes_executor_utils.py:121} ERROR - 
Unknown error in KubernetesJobWatcher. Failing
   Traceback (most recent call last):
     File 
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py",
 line 112, in run
       self.resource_version = self._run(
                               ^^^^^^^^^^
     File 
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py",
 line 168, in _run
       for event in self._pod_events(kube_client=kube_client, 
query_kwargs=kwargs):
     File 
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/watch/watch.py", 
line 182, in stream
       raise client.rest.ApiException(
   kubernetes.client.exceptions.ApiException: (410)
   Reason: Expired: too old resource version: 725263658 (725300129)
   
   Process KubernetesJobWatcher-8:
   Traceback (most recent call last):
     File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in 
_bootstrap
       self.run()
     File 
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py",
 line 112, in run
       self.resource_version = self._run(
                               ^^^^^^^^^^
     File 
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py",
 line 168, in _run
       for event in self._pod_events(kube_client=kube_client, 
query_kwargs=kwargs):
     File 
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/watch/watch.py", 
line 182, in stream
       raise client.rest.ApiException(
   kubernetes.client.exceptions.ApiException: (410)
   Reason: Expired: too old resource version: 725263658 (725300129)
   
   [2024-02-07T17:54:31.158+0000] {kubernetes_executor_utils.py:359} ERROR - 
Error while health checking kube watcher process for namespace 
"our-airflow-dag-runner-namespace". Process died for unknown reasons
   [2024-02-07T17:54:31.185+0000] {kubernetes_executor_utils.py:157} INFO - 
Event: and now my watch begins starting at resource_version: 0
   ```
   
   ### What you think should happen instead
   
   I looked at the code referenced in the Traceback and saw that `_pod_events` 
is supposed to catch Kubernetes client API exceptions and, if its status code 
is 410, retrieve resource versions again or bubble it up otherwise. In our 
case, the status check fails and the exception is raised and caught by `run()` 
then logged as an "unknown error in KubernetesJobWatcher".
   
   ```
   def _pod_events(self, kube_client: client.CoreV1Api, query_kwargs: dict):
           watcher = watch.Watch()
           try:
               if self.namespace == ALL_NAMESPACES:
                   return 
watcher.stream(kube_client.list_pod_for_all_namespaces, **query_kwargs)
               else:
                   return watcher.stream(kube_client.list_namespaced_pod, 
self.namespace, **query_kwargs)
           except ApiException as e:
               if e.status == 410:  # Resource version is too old
                   if self.namespace == ALL_NAMESPACES:
                       pods = 
kube_client.list_pod_for_all_namespaces(watch=False)
                   else:
                       pods = 
kube_client.list_namespaced_pod(namespace=self.namespace, watch=False)
                   resource_version = pods.metadata.resource_version
                   query_kwargs["resource_version"] = resource_version
                   return self._pod_events(kube_client=kube_client, 
query_kwargs=query_kwargs)
               else:
                   raise
   ```
   
   ### How to reproduce
   
   Our DAGs are controlled by either `KubernetesPodOperator` or 
`SparkSubmitOperator` and are run on different schedules where some are run 
only once per day while others run every 8 hours. These errors are thrown 
around the time of both sets of scheduled runs. Simply running the DAGs will 
eventually produce this error for our case.
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to