vatsrahul1001 opened a new issue, #68827:
URL: https://github.com/apache/airflow/issues/68827
### Apache Airflow Provider(s)
cncf-kubernetes
### What happened
With `KubernetesExecutor` running **in-cluster**, any task that sets a
`V1Pod` `pod_override` in `executor_config` crashes the **scheduler** in a loop:
```
_pickle.PicklingError: Can't pickle local object
<function InClusterConfigLoader._set_config.<locals>._refresh_api_key at
0x...>
when serializing kubernetes.client.configuration.Configuration ...
when serializing dict item 'pod_override' ... ExecuteTask ...
File ".../executors/kubernetes_executor.py", line 230, in execute_async
self.task_queue.put(KubernetesJob(key, command, kube_executor_config,
pod_template_file))
```
`execute_async` puts the `KubernetesJob` (embedding the task's
`pod_override` `V1Pod`) onto a `multiprocessing.Manager` queue, whose `.put()`
pickles synchronously in the scheduler process — so a pickle failure takes the
scheduler down rather than failing the task.
### Root cause
In kubernetes-client/python **36.0.0**, the generated model `__init__`
changed how the default `local_vars_configuration` is set (PR
[kubernetes-client/python#2532](https://github.com/kubernetes-client/python/pull/2532),
upgrading to OpenAPI Generator v6.6.0 —
[OpenAPITools/openapi-generator#8500](https://github.com/OpenAPITools/openapi-generator/pull/8500)):
```python
# kubernetes 35.x
if local_vars_configuration is None:
local_vars_configuration = Configuration() # empty ->
picklable
# kubernetes 36.x
if local_vars_configuration is None:
local_vars_configuration = Configuration.get_default_copy() # copy of
the in-cluster default
```
In-cluster, the default `Configuration` carries `refresh_api_key_hook =
InClusterConfigLoader._set_config.<locals>._refresh_api_key`, a **local
closure** that `pickle` cannot serialize. So in 36.x every in-cluster `V1Pod`
inherits an unpicklable `Configuration`.
This is **independent of the Airflow version** (reproduces on 3.2.x and
3.3.x); it is gated on kubernetes client >=36 + an in-cluster scheduler + a
`V1Pod` `pod_override`. Note that downgrading the client below 36 is not an
option for deployments that need the 36.x [`no_proxy` fix
(#2520)](https://github.com/kubernetes-client/python/issues/2520) — the
counterpart to #67845, which is what enabled 36.x support in the first place.
### How to reproduce
In-cluster `KubernetesExecutor`, kubernetes client 36.x, run a task with a
`V1Pod` `pod_override`:
```python
from kubernetes.client import models as k8s
PythonOperator(
task_id="t",
python_callable=lambda: None,
executor_config={
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
resources=k8s.V1ResourceRequirements(
requests={"cpu": "100m", "memory": "384Mi"},
),
)
]
),
)
},
)
```
### Operating System
Any (Linux, in-cluster). Reproduced on Astro Runtime with Airflow 3.2.x and
3.3.x.
### Versions of Apache Airflow Providers
`apache-airflow-providers-cncf-kubernetes==10.18.0`, `kubernetes==36.0.0` /
`36.0.2`, in-cluster `KubernetesExecutor`.
### Deployment
Official Apache Airflow Helm Chart / Astronomer
### Deployment details
In-cluster `KubernetesExecutor` (scheduler loads config via
`load_incluster_config`).
### Proposed fix
#68819 — serialize the `pod_override` to a plain dict before queuing
(dropping the `Configuration`) and rebuild the `V1Pod` worker-side, so the
executor is robust regardless of the kubernetes client version. The worker side
already sanitizes pods with `sanitize_for_serialization`; this closes the same
gap on the scheduler -> queue boundary. The underlying client behavior should
also be reported upstream to kubernetes-client/python (root cause: #2532 /
openapi-generator#8500).
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR! (#68819)
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
---
Drafted-by: Claude Code (Opus 4.8); reviewed by @vatsrahul1001 before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
