jmaicher opened a new issue, #37008:
URL: https://github.com/apache/airflow/issues/37008

   ### Apache Airflow Provider(s)
   
   cncf-kubernetes
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-cncf-kubernetes==7.13.0
   kubernetes==23.6.0
   
   ### Apache Airflow version
   
   2.8.1
   
   ### Operating System
   
   debian 12 (bookworm)
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   Official helm chart with official airflow image (2.8.1)
   
   ### What happened
   
   When upgrading from 2.8.0 to 2.8.1 the airflow scheduler ended in a 
CrashLoopBackOff with:
   ```
   
     ____________       _____________
    ____    |__( )_________  __/__  /________      __
   ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
   ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
    _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
   [2024-01-25T09:52:55.993+0000] {task_context_logger.py:63} INFO - Task 
context logging is enabled
   [2024-01-25T09:52:55.993+0000] {executor_loader.py:115} INFO - Loaded 
executor: KubernetesExecutor
   [2024-01-25T09:52:56.067+0000] {scheduler_job_runner.py:808} INFO - Starting 
the scheduler
   [2024-01-25T09:52:56.068+0000] {scheduler_job_runner.py:815} INFO - 
Processing each file at most -1 times
   [2024-01-25T09:52:56.069+0000] {kubernetes_executor.py:316} INFO - Start 
Kubernetes executor
   [2024-01-25T09:52:56.106+0000] {kubernetes_executor_utils.py:157} INFO - 
Event: and now my watch begins starting at resource_version: 0
   [2024-01-25T09:52:56.114+0000] {kubernetes_executor.py:237} INFO - Found 0 
queued task instances
   [2024-01-25T09:52:56.124+0000] {manager.py:169} INFO - Launched 
DagFileProcessorManager with pid: 108
   [2024-01-25T09:52:56.125+0000] {scheduler_job_runner.py:1619} INFO - 
Adopting or resetting orphaned tasks for active dag runs
   [2024-01-25T09:52:56.133+0000] {settings.py:60} INFO - Configured default 
timezone UTC
   [2024-01-25T09:52:56.149+0000] {settings.py:533} INFO - Loaded 
airflow_local_settings from /opt/airflow/config/airflow_local_settings.py .
   [2024-01-25T09:52:56.196+0000] {scheduler_job_runner.py:872} ERROR - 
Exception when executing SchedulerJob._run_scheduler_loop
   Traceback (most recent call last):
     File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job_runner.py",
 line 855, in _execute
       self._run_scheduler_loop()
     File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job_runner.py",
 line 937, in _run_scheduler_loop
       self.adopt_or_reset_orphaned_tasks()
     File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", 
line 79, in wrapper
       return func(*args, session=session, **kwargs)
     File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job_runner.py",
 line 1622, in adopt_or_reset_orphaned_tasks
       for attempt in run_with_db_retries(logger=self.log):
     File 
"/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 
347, in __iter__
       do = self.iter(retry_state=retry_state)
     File 
"/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 
314, in iter
       return fut.result()
     File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 437, in 
result
       return self.__get_result()
     File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in 
__get_result
       raise self._exception
     File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job_runner.py",
 line 1667, in adopt_or_reset_orphaned_tasks
       to_reset = 
self.job.executor.try_adopt_task_instances(tis_to_adopt_or_reset)
     File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py",
 line 587, in try_adopt_task_instances
       self._adopt_completed_pods(kube_client)
     File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py",
 line 687, in _adopt_completed_pods
       pod_list = self._list_pods(query_kwargs)
     File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py",
 line 171, in _list_pods
       pod_resource = dynamic_client.resources.get(api_version="v1", kind="Pod")
     File 
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
 line 199, in get
       results = self.search(**kwargs)
     File 
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
 line 242, in search
       results = self.__search(self.__build_search(**kwargs), self.__resources, 
[])
     File 
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
 line 290, in __search
       matches.extend(self.__search([key] + parts[1:], resources, reqParams))
     File 
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
 line 276, in __search
       return self.__search(parts[1:], resourcePart, reqParams + [part] )
     File 
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
 line 290, in __search
       matches.extend(self.__search([key] + parts[1:], resources, reqParams))
     File 
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
 line 276, in __search
       return self.__search(parts[1:], resourcePart, reqParams + [part] )
     File 
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
 line 265, in __search
       resourcePart.resources = self.get_resources_for_api_version(
     File 
"/home/airflow/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py",
 line 169, in get_resources_for_api_version
       resource, name = subresource['name'].split('/')
   ValueError: too many values to unpack (expected 2)
   ```
   
   ### What you think should happen instead
   
   _No response_
   
   ### How to reproduce
   
   Deployment on a kubernetes cluster with a sub-resource that is named 
following the a/b/c naming scheme.
   
   ### Anything else
   
   The problem was likely introduced with 
https://github.com/apache/airflow/commit/b9c574c61ae42481b9d2c9ce7c42c93dc44b9507
 and the bug has been already fixed in the python kubernetes client, see 
https://github.com/kubernetes-client/python/issues/2091. An upgrade of the 
kubernetes client is already planned 
(https://github.com/apache/airflow/issues/36678), but we wanted to make this 
issue visible as it likely breaks other deployments as well.
   
   
   
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to