sakethsomaraju opened a new issue, #71879:
URL: https://github.com/apache/airflow/issues/71879
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
### Apache Airflow version
3.3.1 and 2.11.2 ; reproduced in current `main` of
airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py
### What happened
`run_pod_async()` in the KubernetesExecutor calls `create_namespaced_pod()`
with no retry handling for transient network/connection failures. When the
call to the Kubernetes API server fails with a `RemoteDisconnected` /
`ProtocolError` (connection dropped without a response, a common transient
condition on managed clusters, e.g. AKS API server load-balancer resets),
the exception is caught only long enough to emit a stats metric and log,
then unconditionally re-raised:
except ApiException as e:
...
raise
except Exception as e:
...
raise e
This propagates out of `run_pod_async` → `run_next` → `sync()` →
`executor.heartbeat()` → `SchedulerJob._run_scheduler_loop()`, which is not
prepared to treat pod-creation failures as retryable. The scheduler process
then exits (non-zero) and has to be restarted by kubelet.
Observed traceback:
File ".../kubernetes_executor_utils.py", line 452, in run_pod_async
resp = self.kube_client.create_namespaced_pod(
...
urllib3.exceptions.ProtocolError: ('Connection aborted.',
RemoteDisconnected('Remote end closed connection without response'))
...
File ".../scheduler_job_runner.py", in _run_scheduler_loop
executor.heartbeat()
...
[scheduler process exits, kubelet restarts the pod]
I confirmed this same unguarded re-raise pattern is still present on current
`main`, including in the newer async batched pod-creation path
(`_create_pods_async`), which also has no retry, it just isolates the
failure to one job via `asyncio.gather(..., return_exceptions=True)` rather
than crashing the scheduler outright, which is an incidental side effect,
not a fix, and only applies when async batch creation is enabled.
This is a narrower, more specific case of #24748 (configurable retry policy
for the k8s ApiClient), which remains open with no merged fix. This issue is
scoped specifically to the pod-creation call path and its direct impact on
scheduler stability.
### How to reproduce
Inject a connection drop / RST on the path between the scheduler and the
Kubernetes API server while a task pod is being created (e.g. via a chaos
proxy, or simply during periods of API server load-balancer rotation on
AKS). The scheduler will crash on the next `create_namespaced_pod` call that
lands during the drop.
### Deployment details
KubernetesExecutor, standard Astro worker
pod template with git-sync sidecar.
### What you think should happen instead?
A single transient connection drop while creating a task pod should not be
fatal to the scheduler process. `run_pod_async` (and its async counterpart)
should retry `create_namespaced_pod` a small, bounded number of times with
backoff for connection-level errors (e.g. `urllib3.exceptions.ProtocolError`,
`http.client.RemoteDisconnected`, `ConnectionError`), before giving up and
raising.
Either approach should be configurable (max retries, backoff) so operators
running on cloud-managed API servers with occasional connection resets
(AKS, EKS, GKE) don't see routine scheduler restarts from single blips.
### Operating System
Linux (AKS)
### Deployment
Astronomer
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes (10.19.0); same gap confirmed
present in
airflow/providers/cncf/kubernetes `main` as of this report.
### Official Helm Chart version
Not Applicable
### Kubernetes Version
NA
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
