Hi Hemanth, Fokko and Taylor have covered this question but just cc'ing myself in case you have any further questions about the k8s executor :).
On Wed, Aug 22, 2018 at 12:10 PM Taylor Edmiston <tedmis...@gmail.com> wrote: > Hemanth - > > To add to what Fokko mentioned... > > You can find the KubernetesExecutor implementation in KubernetesExecutor > < > https://github.com/apache/incubator-airflow/blob/7ea7dd2031c32a5dd14e4fd74f415f938a69e3ad/airflow/contrib/executors/kubernetes_executor.py#L465 > > > and AirflowKubernetesScheduler > < > https://github.com/apache/incubator-airflow/blob/7ea7dd2031c32a5dd14e4fd74f415f938a69e3ad/airflow/contrib/executors/kubernetes_executor.py#L292 > >. > There's also a usage example > < > https://github.com/apache/incubator-airflow/blob/272952a9dce932cb2c648f82c9f9f2cafd572ff1/airflow/example_dags/example_kubernetes_executor.py > > > . > > And here's the KubernetesPodOperator > < > https://github.com/apache/incubator-airflow/blob/9d516c7134eb22a3c2fc63cf96626ef6e8b247f2/airflow/contrib/operators/kubernetes_pod_operator.py#L33 > > > implementation > which has an example > < > https://github.com/apache/incubator-airflow/blob/9d516c7134eb22a3c2fc63cf96626ef6e8b247f2/airflow/example_dags/example_kubernetes_operator.py > > > as > well. > > There are brief docs > <https://airflow.readthedocs.io/en/latest/kubernetes.html> available for > both as well. > > And a shameless plug — we've been building an open source Airflow on > Kubernetes stack at Astronomer (Helm charts > <https://github.com/astronomerio/helm.astronomer.io>, Docker images > <https://github.com/astronomerio/astronomer>). Some people use this to > avoid having to rebuild all of the containerization. Currently we use > CeleryExecutor but are already migrating toward the KubernetesExecutor. > Our goal is to provide the best experience for running Airflow on > Kubernetes. Feel free to message me off list if this is interesting. > > Best, > Taylor > > *Taylor Edmiston* > Blog <https://blog.tedmiston.com/> | CV > <https://stackoverflow.com/cv/taylor> | LinkedIn > <https://www.linkedin.com/in/tedmiston/> | AngelList > <https://angel.co/taylor> | Stack Overflow > <https://stackoverflow.com/users/149428/taylor-edmiston> > > > > On Tue, Aug 21, 2018 at 10:15 AM Driesprong, Fokko <fo...@driesprong.frl> > wrote: > > > Hi Hemanth, > > > > Thanks for the question. There is the KubernetesExecutor which just uses > a > > container with Airflow to schedule a task on the kubernetes cluster. This > > container will then execute a single task, and then the Executor checks > the > > exitcode of the container. > > > > There is also the KubernetesOperator which allows you to kick of a new > > container on the kubernetes cluster. Hope this helps. > > > > Cheers, Fokko > > > > Op di 21 aug. 2018 om 15:54 schreef Musunuru, Hemanth < > > hemanth.musun...@nike.com> > > > > > Hi Team, > > > > > > Does Kubernetes executor in 1.10 create new pod for every task ? > (Which > > > means won’t there be any active workers running except for new Tasks) > > > > > > We are trying to containerize our existing to airflow environment, your > > > reply Will help us to understand it more. > > > > > > Thanks in Advance . > > > > > > Thanks > > > Hemanth > > >
