AlejandroMorgante commented on PR #71244:
URL: https://github.com/apache/airflow/pull/71244#issuecomment-5430379619
Thanks for the feedback. Airflow commonly acts as a client of
infrastructure it does not own: `SSHOperator` does not manage the server, SQL
operators do not manage the database, and `SsmRunCommandOperator` does not
manage the target instances. They own only the operation being executed. This
operator follows the same model: Airflow owns the exec session, while another
system owns the Pod lifecycle.
Although this can be implemented with `BashOperator` and `kubectl exec`,
that moves additional integration logic to every user: installing and
maintaining `kubectl` on the workers, distributing kubeconfig and cloud
authentication helpers, handling shell quoting, and mapping Kubernetes errors
and output into Airflow. This operator provides that behavior through
`KubernetesHook` and Airflow connections. The provider already uses the
`pods/exec` API internally, so this is not a new Kubernetes mechanism.
Keeping this provider-neutral also allows GKE- or EKS-specific operators to
reuse it and add only their cloud authentication and cluster configuration,
following the existing provider layering. I prefer keeping it separate from KPO
because KPO owns the complete Pod lifecycle, while this operator deliberately
treats the Pod as externally managed.
If the Pod disappears, restarts, or the connection fails, the task fails. A
retry creates a new exec session; Airflow never creates, replaces, or deletes
the Pod.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
