AlejandroMorgante commented on PR #71244:
URL: https://github.com/apache/airflow/pull/71244#issuecomment-5221917597
@kaxil
> What's the User story and real-world use cases for this?
Thanks for asking. The user story is:
> As an Airflow user who already manages a long-running Kubernetes Pod, I
want to execute commands in one
of its containers as Airflow tasks, without Airflow creating, restarting,
or deleting that Pod.
Our concrete use case is running dbt on EKS. Today, our workflow starts a
new Pod, initializes the dbt project and its dependencies, and then executes a
command such as `dbt run`. A significant part of the execution time is spent
scheduling the Pod and initializing the environment.
We want to manage that Pod separately and keep it warm, then use Airflow to
execute ad hoc commands such as `dbt run`, `dbt test`, or `dbt build`. This
avoids the repeated cold-start and initialization cost.
More generally, this is useful whenever a Pod’s lifecycle is managed outside
the task and its environment is already initialized. The operator exposes the
equivalent of `kubectl exec` as an Airflow task, including Airflow logging and
optional XCom output, without assuming ownership of the Pod lifecycle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
