romanzdk opened a new issue, #35010:
URL: https://github.com/apache/airflow/issues/35010
### Apache Airflow version
2.7.2
### What happened
Yesterday we updated cncf kubernetes plugin to 7.7.0 and since then we get
following error when running KubernetesPodOperator
```
...
[2023-10-18, 06:45:27 UTC] {pod.py:824} INFO - Deleting pod:
some-pod-yv0coojk
[2023-10-18, 06:45:27 UTC] {taskinstance.py:1937} ERROR - Task failed with
exception
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/tenacity/__init__.py", line
382, in __call__
result = fn(*args, **kwargs)
File
"/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 411, in consume_logs
logs = self.read_pod_logs(
File "/usr/local/lib/python3.10/site-packages/tenacity/__init__.py", line
289, in wrapped_f
return self(f, *args, **kw)
File "/usr/local/lib/python3.10/site-packages/tenacity/__init__.py", line
379, in __call__
do = self.iter(retry_state=retry_state)
File "/usr/local/lib/python3.10/site-packages/tenacity/__init__.py", line
325, in iter
raise retry_exc.reraise()
File "/usr/local/lib/python3.10/site-packages/tenacity/__init__.py", line
158, in reraise
raise self.last_attempt.result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 451, in
result
return self.__get_result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in
__get_result
raise self._exception
File "/usr/local/lib/python3.10/site-packages/tenacity/__init__.py", line
382, in __call__
result = fn(*args, **kwargs)
File
"/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 632, in read_pod_logs
logs = self._client.read_namespaced_pod_log(
File
"/usr/local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py",
line 23747, in read_namespaced_pod_log
return self.read_namespaced_pod_log_with_http_info(name, namespace,
**kwargs) # noqa: E501
File
"/usr/local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py",
line 23866, in read_namespaced_pod_log_with_http_info
return self.api_client.call_api(
File
"/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line
348, in call_api
return self.__call_api(resource_path, method,
File
"/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line
180, in __call_api
response_data = self.request(
File
"/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line
373, in request
return self.rest_client.GET(url,
File "/usr/local/lib/python3.10/site-packages/kubernetes/client/rest.py",
line 240, in GET
return self.request("GET", url,
File "/usr/local/lib/python3.10/site-packages/kubernetes/client/rest.py",
line 234, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id':
'4ee17171-c49a-4215-ac99-da4ee8d9aed7', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'Date': 'Wed, 18 Oct 2023 06:45:27 GMT',
'Content-Length': '215'})
HTTP response body:
b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container
\\"base\\" in pod \\"abn-positions-validation-yv0coojk\\" is waiting to start:
PodInitializing","reason":"BadRequest","code":400}\n'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 600, in execute_sync
self.pod_manager.fetch_requested_container_logs(
File
"/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 504, in fetch_requested_container_logs
status = self.fetch_container_logs(
File
"/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 472, in fetch_container_logs
last_log_time, logs = consume_logs(
File "/usr/local/lib/python3.10/site-packages/tenacity/__init__.py", line
289, in wrapped_f
return self(f, *args, **kw)
File "/usr/local/lib/python3.10/site-packages/tenacity/__init__.py", line
379, in __call__
do = self.iter(retry_state=retry_state)
File "/usr/local/lib/python3.10/site-packages/tenacity/__init__.py", line
326, in iter
raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x7fbb44f29360 state=finished
raised ApiException>]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 581, in execute
return self.execute_sync(context)
File
"/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 620, in execute_sync
self.cleanup(
File
"/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 752, in cleanup
raise AirflowException(
airflow.exceptions.AirflowException: Pod some-pod-yv0coojk returned a
failure.
...
```
### What you think should happen instead
No error should be raised
### How to reproduce
Use Airflow 2.7.2 with cncf-kubernetes-provider 7.7.0 and run
KubernetesPodOperator
### Operating System
Debian GNU/Linux 10 (buster)
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.7.1
apache-airflow-providers-cncf-kubernetes==7.7.0
apache-airflow-providers-common-sql==1.7.2
apache-airflow-providers-ftp==3.5.2
apache-airflow-providers-http==4.5.2
apache-airflow-providers-imap==3.3.2
apache-airflow-providers-postgres==5.6.1
apache-airflow-providers-slack==8.1.0
apache-airflow-providers-sqlite==3.4.3
### Deployment
Other Docker-based deployment
### Deployment details
On-premise kubernetes cluster, LocalExecutor (3 replicas), 10 parallelism
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
