Hi everyone! I'm new here and I've read some of the discussions about K8s + Airflow. And I would be very interested to join this meeting.
But I'm not quiet sure what is this integration. Is it to make it easy to deploy Airflow to K8S and have scheduler, webserver and other airflow components inside K8S pods or is it to be able to run dags/tasks in isolated pod environments to allow temporary isolated environments that will be destroyed as soon as the task instance / dag run is over? At the company I work on here in Brazil, we have deployed Airflow with K8S and we are thinking in a way to run dags/tasks in temporary isolated pods to allow, mainly, any powerful operator like PythonOperator and BashOperator. Something using this https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#what-is-a-job . Ps: Sorry if this wasn't the right thread to discuss about ir. If that is true, message me and I'll take this message to the right place. On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <maximebeauche...@gmail.com> wrote: > +1! > > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu <fen...@google.com.invalid> > wrote: > > > Would like to join, please kindly invite me. > > Thanks! > > > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman < > > daniel.imber...@gmail.com> > > wrote: > > > > > Hello everyone, > > > > > > Recently there's been a fair amount of discussion regarding the > > integration > > > of airflow with kubernetes. If there is interest I would love to host > an > > > e-meeting to discuss this integration. I can go over the architecture > as > > it > > > stands right now and would love feedback on > > improvements/features/design. I > > > could also attempt to get one or two members of google's kubernetes > team > > to > > > join to discuss best practices. > > > > > > I'm currently thinking that next Thursday at 11AM PST over zoom.us, > > though > > > if there's strong opinions otherwise I'd be glad to propose other > times. > > > > > > Cheers! > > > > > > Daniel > > > > > >
