jmaicher opened a new issue, #37008:
URL: https://github.com/apache/airflow/issues/37008
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==7.13.0
kubernetes==23.6.0
### Apache Airflow version
2.8.1
### Operating System
debian 12 (bookworm)
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Official helm chart with official airflow image (2.8.1)
### What happened
When upgrading from 2.8.0 to 2.8.1 the airflow scheduler ended in a
CrashLoopBackOff with:
```
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2024-01-25T09:52:55.993+0000] {task_context_logger.py:63} INFO - Task
context logging is enabled
[2024-01-25T09:52:55.993+0000] {executor_loader.py:115} INFO - Loaded
executor: KubernetesExecutor
[2024-01-25T09:52:56.067+0000] {scheduler_job_runner.py:808} INFO - Starting
the scheduler
[2024-01-25T09:52:56.068+0000] {scheduler_job_runner.py:815} INFO -
Processing each file at most -1 times
[2024-01-25T09:52:56.069+0000] {kubernetes_executor.py:316} INFO - Start
Kubernetes executor
[2024-01-25T09:52:56.106+0000] {kubernetes_executor_utils.py:157} INFO -
Event: and now my watch begins starting at resource_version: 0
[2024-01-25T09:52:56.114+0000] {kubernetes_executor.py:237} INFO - Found 0
queued task instances
[2024-01-25T09:52:56.124+0000] {manager.py:169} INFO - Launched
DagFileProcessorManager with pid: 108
[2024-01-25T09:52:56.125+0000] {scheduler_job_runner.py:1619} INFO -
Adopting or resetting orphaned tasks for active dag runs
[2024-01-25T09:52:56.133+0000] {settings.py:60} INFO - Configured default
timezone UTC
[2024-01-25T09:52:56.149+0000] {settings.py:533} INFO - Loaded
airflow_local_settings from /opt/airflow/config/airflow_local_settings.py .
[2024-01-25T09:52:56.196+0000] {scheduler_job_runner.py:872} ERROR -
Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job_runner.py",
line 855, in _execute
self._run_scheduler_loop()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job_runner.py",
line 937, in _run_scheduler_loop
self.adopt_or_reset_orphaned_tasks()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py",
line 79, in wrapper
return func(*args, session=session, **kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job_runner.py",
line 1622, in adopt_or_reset_orphaned_tasks
for attempt in run_with_db_retries(logger=self.log):
File
"/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line
347, in __iter__
do = self.iter(retry_state=retry_state)
File
"/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line
314, in iter
return fut.result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 437, in
result
return self.__get_result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in
__get_result
raise self._exception
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job_runner.py",
line 1667, in adopt_or_reset_orphaned_tasks
to_reset =
self.job.executor.try_adopt_task_instances(tis_to_adopt_or_reset)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py",
line 587, in try_adopt_task_instances
self._adopt_completed_pods(kube_client)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py",
line 687, in _adopt_completed_pods
pod_list = self._list_pods(query_kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py",
line 171, in _list_pods
pod_resource = dynamic_client.resources.get(api_version="v1", kind="Pod")
File
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
line 199, in get
results = self.search(**kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
line 242, in search
results = self.__search(self.__build_search(**kwargs), self.__resources,
[])
File
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
line 290, in __search
matches.extend(self.__search([key] + parts[1:], resources, reqParams))
File
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
line 276, in __search
return self.__search(parts[1:], resourcePart, reqParams + [part] )
File
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
line 290, in __search
matches.extend(self.__search([key] + parts[1:], resources, reqParams))
File
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
line 276, in __search
return self.__search(parts[1:], resourcePart, reqParams + [part] )
File
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
line 265, in __search
resourcePart.resources = self.get_resources_for_api_version(
File
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
line 169, in get_resources_for_api_version
resource, name = subresource['name'].split('/')
ValueError: too many values to unpack (expected 2)
```
### What you think should happen instead
_No response_
### How to reproduce
Deployment on a kubernetes cluster with a sub-resource that is named
following the a/b/c naming scheme.
### Anything else
The problem was likely introduced with
https://github.com/apache/airflow/commit/b9c574c61ae42481b9d2c9ce7c42c93dc44b9507
and the bug has been already fixed in the python kubernetes client, see
https://github.com/kubernetes-client/python/issues/2091. An upgrade of the
kubernetes client is already planned
(https://github.com/apache/airflow/issues/36678), but we wanted to make this
issue visible as it likely breaks other deployments as well.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]