Re: Airflow + Kubernetes discussion

[Daniel Imberman]

Hi Guys, sorry about the technical difficulties. I will be switching this
call to zoom.us and send out a link shortly

On Thu, Jul 27, 2017 at 11:07 AM Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:

> I see the same thing, "The call is full"
>
> On Thu, Jul 27, 2017 at 11:04 AM, Wilson Lian 
> wrote:
>
> > The call is full :(
> >
> > On Thu, Jul 27, 2017 at 11:02 AM, Daniel Imberman <
> > daniel.imber...@gmail.com
> > > wrote:
> >
> > > Hello everyone,
> > >
> > > The kubernetes discussion is beginning now. I am including a link
> below,
> > > but it can also be found in the calendar event itself.
> > >
> > > https://hangouts.google.com/hangouts/_/calendar/
> > >
> ZGFuaWVsLmltYmVybWFuQGdtYWlsLmNvbQ.3jcahq63njusjbnc2l4po6t53a?authuser=0
> > >
> > > On Thu, Jul 27, 2017 at 10:44 AM Marc Bollinger 
> > > wrote:
> > >
> > > > Awesome, thanks!
> > > >
> > > > On Thu, Jul 27, 2017 at 10:38 AM, Daniel Imberman <
> > > > daniel.imber...@gmail.com
> > > > > wrote:
> > > >
> > > > > Hi Marc,
> > > > >
> > > > > Sorry for the confusion. The meeting will take place over the
> google
> > > > > hangout link on the event (zoom was just an early suggestion
> before I
> > > > > realized we could just create the google hangout).
> > > > >
> > > > > Best,
> > > > > Daniel
> > > > >
> > > > > On Thu, Jul 27, 2017 at 10:11 AM Marc Bollinger <
> m...@lumoslabs.com>
> > > > > wrote:
> > > > >
> > > > > > Is this still happening over a Zoom link to be distributed, or on
> > the
> > > > > > Google Hangouts link already associated with the invite?
> > > > > >
> > > > > > On Sun, Jul 23, 2017 at 12:53 PM, Daniel Imberman <
> > > > > > daniel.imber...@gmail.com
> > > > > > > wrote:
> > > > > >
> > > > > > > Hi Victor,
> > > > > > >
> > > > > > > This implementation is about allowing native kubernetes
> > deployment
> > > of
> > > > > > tasks
> > > > > > > on airflow. There are existing deployments/helm scripts with
> all
> > > the
> > > > > > > components needed to launch an airflow cluster, but these
> > clusters
> > > > > > > basically consist of having a set up static workers run with
> > celery
> > > > to
> > > > > > > communicate between pods.
> > > > > > >
> > > > > > > The KubernetesExecutor will dynamically create pods for each
> task
> > > > > you're
> > > > > > > running. Your cluster will be significantly more elastic and it
> > > won't
> > > > > > waste
> > > > > > > resources when jobs aren't running.
> > > > > > >
> > > > > > > We have also created a k8s operator which will allow people to
> > > build
> > > > > pods
> > > > > > > by linking to images/settings. This will give users a lot more
> > > > > > flexibility
> > > > > > > in creating custom operators that launch single pods or entire
> > > > clusters
> > > > > > to
> > > > > > > complete a task.
> > > > > > >
> > > > > > > I think our implementation is similar to what you're proposing,
> > but
> > > > if
> > > > > > > there's any improvements you can see I'd be glad to speak
> further
> > > > over
> > > > > DM
> > > > > > > or during the meet-up this Thursday :).
> > > > > > >
> > > > > > > Here is the proposal for a high level overview and the WIP PR
> for
> > > > > > > reference:
> > > > > > >
> > > > > > >
> > > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > > action?pageId=71013666
> > > > > > > https://github.com/apache/incubator-airflow/pull/2414
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Daniel
> > > > > > >
> > > > > > > On Sat, Jul 22, 2017 at 2:24 PM Amit Kulkarni  >
> > > > wrote:
> > > > > > >
> > > > > > > > I would like to join as well.
> > > > > > > >
> > > > > > > >
> > > > > > > > Amit Kulkarni
> > > > > > > > Site Reliability Engineer
> > > > > > > > Mobile:  (716)-352-3270 <(716)%20352-3270>
> <(716)%20352-3270> <(716)%20352-3270>
> > > > <(716)%20352-3270>
> > > > > > > >
> > > > > > > > Payments partner to the platform economy
> > > > > > > >
> > > > > > > > On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro <
> > > > > > > victor.monte...@ubee.in>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi everyone!
> > > > > > > > >
> > > > > > > > > I'm new here and I've read some of the discussions about
> K8s
> > +
> > > > > > Airflow.
> > > > > > > > And
> > > > > > > > > I would be very interested to join this meeting.
> > > > > > > > >
> > > > > > > > > But I'm not quiet sure what is this integration. Is it to
> > make
> > > it
> > > > > > easy
> > > > > > > to
> > > > > > > > > deploy Airflow to K8S and have scheduler, webserver and
> other
> > > > > airflow
> > > > > > > > > components inside K8S pods or is it to be able to run
> > > dags/tasks
> > > > in
> > > > > > > > > isolated pod environments to allow temporary isolated
> > > > environments
> > > > > > that
> > > > > > > > > will be destroyed as soon as the task instance / dag run is
> > > over?
> > > > > > > > >
> > > > > > > > > At the company I work on here in 

Re: Airflow + Kubernetes discussion

[Maxime Beauchemin]

I see the same thing, "The call is full"

On Thu, Jul 27, 2017 at 11:04 AM, Wilson Lian 
wrote:

> The call is full :(
>
> On Thu, Jul 27, 2017 at 11:02 AM, Daniel Imberman <
> daniel.imber...@gmail.com
> > wrote:
>
> > Hello everyone,
> >
> > The kubernetes discussion is beginning now. I am including a link below,
> > but it can also be found in the calendar event itself.
> >
> > https://hangouts.google.com/hangouts/_/calendar/
> > ZGFuaWVsLmltYmVybWFuQGdtYWlsLmNvbQ.3jcahq63njusjbnc2l4po6t53a?authuser=0
> >
> > On Thu, Jul 27, 2017 at 10:44 AM Marc Bollinger 
> > wrote:
> >
> > > Awesome, thanks!
> > >
> > > On Thu, Jul 27, 2017 at 10:38 AM, Daniel Imberman <
> > > daniel.imber...@gmail.com
> > > > wrote:
> > >
> > > > Hi Marc,
> > > >
> > > > Sorry for the confusion. The meeting will take place over the google
> > > > hangout link on the event (zoom was just an early suggestion before I
> > > > realized we could just create the google hangout).
> > > >
> > > > Best,
> > > > Daniel
> > > >
> > > > On Thu, Jul 27, 2017 at 10:11 AM Marc Bollinger 
> > > > wrote:
> > > >
> > > > > Is this still happening over a Zoom link to be distributed, or on
> the
> > > > > Google Hangouts link already associated with the invite?
> > > > >
> > > > > On Sun, Jul 23, 2017 at 12:53 PM, Daniel Imberman <
> > > > > daniel.imber...@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > Hi Victor,
> > > > > >
> > > > > > This implementation is about allowing native kubernetes
> deployment
> > of
> > > > > tasks
> > > > > > on airflow. There are existing deployments/helm scripts with all
> > the
> > > > > > components needed to launch an airflow cluster, but these
> clusters
> > > > > > basically consist of having a set up static workers run with
> celery
> > > to
> > > > > > communicate between pods.
> > > > > >
> > > > > > The KubernetesExecutor will dynamically create pods for each task
> > > > you're
> > > > > > running. Your cluster will be significantly more elastic and it
> > won't
> > > > > waste
> > > > > > resources when jobs aren't running.
> > > > > >
> > > > > > We have also created a k8s operator which will allow people to
> > build
> > > > pods
> > > > > > by linking to images/settings. This will give users a lot more
> > > > > flexibility
> > > > > > in creating custom operators that launch single pods or entire
> > > clusters
> > > > > to
> > > > > > complete a task.
> > > > > >
> > > > > > I think our implementation is similar to what you're proposing,
> but
> > > if
> > > > > > there's any improvements you can see I'd be glad to speak further
> > > over
> > > > DM
> > > > > > or during the meet-up this Thursday :).
> > > > > >
> > > > > > Here is the proposal for a high level overview and the WIP PR for
> > > > > > reference:
> > > > > >
> > > > > >
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=71013666
> > > > > > https://github.com/apache/incubator-airflow/pull/2414
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Daniel
> > > > > >
> > > > > > On Sat, Jul 22, 2017 at 2:24 PM Amit Kulkarni 
> > > wrote:
> > > > > >
> > > > > > > I would like to join as well.
> > > > > > >
> > > > > > >
> > > > > > > Amit Kulkarni
> > > > > > > Site Reliability Engineer
> > > > > > > Mobile:  (716)-352-3270 <(716)%20352-3270> <(716)%20352-3270>
> > > <(716)%20352-3270>
> > > > > > >
> > > > > > > Payments partner to the platform economy
> > > > > > >
> > > > > > > On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro <
> > > > > > victor.monte...@ubee.in>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone!
> > > > > > > >
> > > > > > > > I'm new here and I've read some of the discussions about K8s
> +
> > > > > Airflow.
> > > > > > > And
> > > > > > > > I would be very interested to join this meeting.
> > > > > > > >
> > > > > > > > But I'm not quiet sure what is this integration. Is it to
> make
> > it
> > > > > easy
> > > > > > to
> > > > > > > > deploy Airflow to K8S and have scheduler, webserver and other
> > > > airflow
> > > > > > > > components inside K8S pods or is it to be able to run
> > dags/tasks
> > > in
> > > > > > > > isolated pod environments to allow temporary isolated
> > > environments
> > > > > that
> > > > > > > > will be destroyed as soon as the task instance / dag run is
> > over?
> > > > > > > >
> > > > > > > > At the company I work on here in Brazil, we have deployed
> > Airflow
> > > > > with
> > > > > > > K8S
> > > > > > > > and we are thinking in a way to run dags/tasks in temporary
> > > > isolated
> > > > > > pods
> > > > > > > > to allow, mainly, any powerful operator like PythonOperator
> and
> > > > > > > > BashOperator. Something using this
> > > > > > > >
> > > > > https://kubernetes.io/docs/concepts/workloads/
> > controllers/jobs-run-to-
> > > > > > > > completion/#what-is-a-job
> > > > > > > > .
> > > > > > > >
> > > > > > > > Ps: Sorry if this wasn't the 

Re: Airflow + Kubernetes discussion

[Marc Bollinger]

Awesome, thanks!

On Thu, Jul 27, 2017 at 10:38 AM, Daniel Imberman  wrote:

> Hi Marc,
>
> Sorry for the confusion. The meeting will take place over the google
> hangout link on the event (zoom was just an early suggestion before I
> realized we could just create the google hangout).
>
> Best,
> Daniel
>
> On Thu, Jul 27, 2017 at 10:11 AM Marc Bollinger 
> wrote:
>
> > Is this still happening over a Zoom link to be distributed, or on the
> > Google Hangouts link already associated with the invite?
> >
> > On Sun, Jul 23, 2017 at 12:53 PM, Daniel Imberman <
> > daniel.imber...@gmail.com
> > > wrote:
> >
> > > Hi Victor,
> > >
> > > This implementation is about allowing native kubernetes deployment of
> > tasks
> > > on airflow. There are existing deployments/helm scripts with all the
> > > components needed to launch an airflow cluster, but these clusters
> > > basically consist of having a set up static workers run with celery to
> > > communicate between pods.
> > >
> > > The KubernetesExecutor will dynamically create pods for each task
> you're
> > > running. Your cluster will be significantly more elastic and it won't
> > waste
> > > resources when jobs aren't running.
> > >
> > > We have also created a k8s operator which will allow people to build
> pods
> > > by linking to images/settings. This will give users a lot more
> > flexibility
> > > in creating custom operators that launch single pods or entire clusters
> > to
> > > complete a task.
> > >
> > > I think our implementation is similar to what you're proposing, but if
> > > there's any improvements you can see I'd be glad to speak further over
> DM
> > > or during the meet-up this Thursday :).
> > >
> > > Here is the proposal for a high level overview and the WIP PR for
> > > reference:
> > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=71013666
> > > https://github.com/apache/incubator-airflow/pull/2414
> > >
> > > Cheers,
> > >
> > > Daniel
> > >
> > > On Sat, Jul 22, 2017 at 2:24 PM Amit Kulkarni  wrote:
> > >
> > > > I would like to join as well.
> > > >
> > > >
> > > > Amit Kulkarni
> > > > Site Reliability Engineer
> > > > Mobile:  (716)-352-3270 <(716)%20352-3270> <(716)%20352-3270>
> > > >
> > > > Payments partner to the platform economy
> > > >
> > > > On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro <
> > > victor.monte...@ubee.in>
> > > > wrote:
> > > >
> > > > > Hi everyone!
> > > > >
> > > > > I'm new here and I've read some of the discussions about K8s +
> > Airflow.
> > > > And
> > > > > I would be very interested to join this meeting.
> > > > >
> > > > > But I'm not quiet sure what is this integration. Is it to make it
> > easy
> > > to
> > > > > deploy Airflow to K8S and have scheduler, webserver and other
> airflow
> > > > > components inside K8S pods or is it to be able to run dags/tasks in
> > > > > isolated pod environments to allow temporary isolated environments
> > that
> > > > > will be destroyed as soon as the task instance / dag run is over?
> > > > >
> > > > > At the company I work on here in Brazil, we have deployed Airflow
> > with
> > > > K8S
> > > > > and we are thinking in a way to run dags/tasks in temporary
> isolated
> > > pods
> > > > > to allow, mainly, any powerful operator like PythonOperator and
> > > > > BashOperator. Something using this
> > > > >
> > https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-
> > > > > completion/#what-is-a-job
> > > > > .
> > > > >
> > > > > Ps: Sorry if this wasn't the right thread to discuss about ir. If
> > that
> > > is
> > > > > true, message me and I'll take this message to the right place.
> > > > > On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <
> > > > maximebeauche...@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > +1!
> > > > > >
> > > > > > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu
>  > >
> > > > > > wrote:
> > > > > >
> > > > > > > Would like to join, please kindly invite me.
> > > > > > > Thanks!
> > > > > > >
> > > > > > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > > > > > > daniel.imber...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > Recently there's been a fair amount of discussion regarding
> the
> > > > > > > integration
> > > > > > > > of airflow with kubernetes. If there is interest I would love
> > to
> > > > host
> > > > > > an
> > > > > > > > e-meeting to discuss this integration. I can go over the
> > > > architecture
> > > > > > as
> > > > > > > it
> > > > > > > > stands right now and would love feedback on
> > > > > > > improvements/features/design. I
> > > > > > > > could also attempt to get one or two members of google's
> > > kubernetes
> > > > > > team
> > > > > > > to
> > > > > > > > join to discuss best practices.
> > > > > > > >
> > > > > > > > I'm currently thinking that next Thursday at 11AM PST over
> > > 

Re: Airflow + Kubernetes discussion

[Daniel Imberman]

Hi Victor,

This implementation is about allowing native kubernetes deployment of tasks
on airflow. There are existing deployments/helm scripts with all the
components needed to launch an airflow cluster, but these clusters
basically consist of having a set up static workers run with celery to
communicate between pods.

The KubernetesExecutor will dynamically create pods for each task you're
running. Your cluster will be significantly more elastic and it won't waste
resources when jobs aren't running.

We have also created a k8s operator which will allow people to build pods
by linking to images/settings. This will give users a lot more flexibility
in creating custom operators that launch single pods or entire clusters to
complete a task.

I think our implementation is similar to what you're proposing, but if
there's any improvements you can see I'd be glad to speak further over DM
or during the meet-up this Thursday :).

Here is the proposal for a high level overview and the WIP PR for reference:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71013666
https://github.com/apache/incubator-airflow/pull/2414

Cheers,

Daniel

On Sat, Jul 22, 2017 at 2:24 PM Amit Kulkarni  wrote:

> I would like to join as well.
>
>
> Amit Kulkarni
> Site Reliability Engineer
> Mobile:  (716)-352-3270 <(716)%20352-3270>
>
> Payments partner to the platform economy
>
> On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro 
> wrote:
>
> > Hi everyone!
> >
> > I'm new here and I've read some of the discussions about K8s + Airflow.
> And
> > I would be very interested to join this meeting.
> >
> > But I'm not quiet sure what is this integration. Is it to make it easy to
> > deploy Airflow to K8S and have scheduler, webserver and other airflow
> > components inside K8S pods or is it to be able to run dags/tasks in
> > isolated pod environments to allow temporary isolated environments that
> > will be destroyed as soon as the task instance / dag run is over?
> >
> > At the company I work on here in Brazil, we have deployed Airflow with
> K8S
> > and we are thinking in a way to run dags/tasks in temporary isolated pods
> > to allow, mainly, any powerful operator like PythonOperator and
> > BashOperator. Something using this
> > https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-
> > completion/#what-is-a-job
> > .
> >
> > Ps: Sorry if this wasn't the right thread to discuss about ir. If that is
> > true, message me and I'll take this message to the right place.
> > On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <
> maximebeauche...@gmail.com
> > >
> > wrote:
> >
> > > +1!
> > >
> > > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu 
> > > wrote:
> > >
> > > > Would like to join, please kindly invite me.
> > > > Thanks!
> > > >
> > > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > > > daniel.imber...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > Recently there's been a fair amount of discussion regarding the
> > > > integration
> > > > > of airflow with kubernetes. If there is interest I would love to
> host
> > > an
> > > > > e-meeting to discuss this integration. I can go over the
> architecture
> > > as
> > > > it
> > > > > stands right now and would love feedback on
> > > > improvements/features/design. I
> > > > > could also attempt to get one or two members of google's kubernetes
> > > team
> > > > to
> > > > > join to discuss best practices.
> > > > >
> > > > > I'm currently thinking that next Thursday at 11AM PST over zoom.us
> ,
> > > > though
> > > > > if there's strong opinions otherwise I'd be glad to propose other
> > > times.
> > > > >
> > > > > Cheers!
> > > > >
> > > > > Daniel
> > > > >
> > > >
> > >
> >
>

Re: Airflow + Kubernetes discussion

[Amit Kulkarni]

I would like to join as well.


Amit Kulkarni
Site Reliability Engineer
Mobile:  (716)-352-3270

Payments partner to the platform economy

On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro 
wrote:

> Hi everyone!
>
> I'm new here and I've read some of the discussions about K8s + Airflow. And
> I would be very interested to join this meeting.
>
> But I'm not quiet sure what is this integration. Is it to make it easy to
> deploy Airflow to K8S and have scheduler, webserver and other airflow
> components inside K8S pods or is it to be able to run dags/tasks in
> isolated pod environments to allow temporary isolated environments that
> will be destroyed as soon as the task instance / dag run is over?
>
> At the company I work on here in Brazil, we have deployed Airflow with K8S
> and we are thinking in a way to run dags/tasks in temporary isolated pods
> to allow, mainly, any powerful operator like PythonOperator and
> BashOperator. Something using this
> https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-
> completion/#what-is-a-job
> .
>
> Ps: Sorry if this wasn't the right thread to discuss about ir. If that is
> true, message me and I'll take this message to the right place.
> On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin  >
> wrote:
>
> > +1!
> >
> > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu 
> > wrote:
> >
> > > Would like to join, please kindly invite me.
> > > Thanks!
> > >
> > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > > daniel.imber...@gmail.com>
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > Recently there's been a fair amount of discussion regarding the
> > > integration
> > > > of airflow with kubernetes. If there is interest I would love to host
> > an
> > > > e-meeting to discuss this integration. I can go over the architecture
> > as
> > > it
> > > > stands right now and would love feedback on
> > > improvements/features/design. I
> > > > could also attempt to get one or two members of google's kubernetes
> > team
> > > to
> > > > join to discuss best practices.
> > > >
> > > > I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> > > though
> > > > if there's strong opinions otherwise I'd be glad to propose other
> > times.
> > > >
> > > > Cheers!
> > > >
> > > > Daniel
> > > >
> > >
> >
>

Re: Airflow + Kubernetes discussion

[Victor Monteiro]

Hi everyone!

I'm new here and I've read some of the discussions about K8s + Airflow. And
I would be very interested to join this meeting.

But I'm not quiet sure what is this integration. Is it to make it easy to
deploy Airflow to K8S and have scheduler, webserver and other airflow
components inside K8S pods or is it to be able to run dags/tasks in
isolated pod environments to allow temporary isolated environments that
will be destroyed as soon as the task instance / dag run is over?

At the company I work on here in Brazil, we have deployed Airflow with K8S
and we are thinking in a way to run dags/tasks in temporary isolated pods
to allow, mainly, any powerful operator like PythonOperator and
BashOperator. Something using this
https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#what-is-a-job
.

Ps: Sorry if this wasn't the right thread to discuss about ir. If that is
true, message me and I'll take this message to the right place.
On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin 
wrote:

> +1!
>
> On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu 
> wrote:
>
> > Would like to join, please kindly invite me.
> > Thanks!
> >
> > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > daniel.imber...@gmail.com>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > Recently there's been a fair amount of discussion regarding the
> > integration
> > > of airflow with kubernetes. If there is interest I would love to host
> an
> > > e-meeting to discuss this integration. I can go over the architecture
> as
> > it
> > > stands right now and would love feedback on
> > improvements/features/design. I
> > > could also attempt to get one or two members of google's kubernetes
> team
> > to
> > > join to discuss best practices.
> > >
> > > I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> > though
> > > if there's strong opinions otherwise I'd be glad to propose other
> times.
> > >
> > > Cheers!
> > >
> > > Daniel
> > >
> >
>

Re: Airflow + Kubernetes discussion

[Maxime Beauchemin]

+1!

On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu  wrote:

> Would like to join, please kindly invite me.
> Thanks!
>
> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> daniel.imber...@gmail.com>
> wrote:
>
> > Hello everyone,
> >
> > Recently there's been a fair amount of discussion regarding the
> integration
> > of airflow with kubernetes. If there is interest I would love to host an
> > e-meeting to discuss this integration. I can go over the architecture as
> it
> > stands right now and would love feedback on
> improvements/features/design. I
> > could also attempt to get one or two members of google's kubernetes team
> to
> > join to discuss best practices.
> >
> > I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> though
> > if there's strong opinions otherwise I'd be glad to propose other times.
> >
> > Cheers!
> >
> > Daniel
> >
>

Re: Airflow + Kubernetes discussion

[Daniel Imberman]

Glad to hear that people are interested! I've created a google calendar
event and messaged everyone in this thread, if anyone else would like to
join please let me know!

On Thu, Jul 20, 2017 at 1:04 PM Bolke de Bruin  wrote:

> Invite would be nice, I will try to join!
>
> > On 20 Jul 2017, at 20:36, Gerard Toonstra  wrote:
> >
> > send me an invite too!
> >
> > On Thu, Jul 20, 2017 at 8:17 PM, Jeremiah Lowin 
> wrote:
> >
> >> I'm interested as well.
> >>
> >> On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger 
> wrote:
> >>
> >>> +1 We're in the middle of moving some services to k8s, and have had our
> >>> eye on Airflow.
> >>>
>  On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari <
> sumeet.ma...@gmail.com
> >>>
> >>> wrote:
> 
>  I would join as well for sure.
> 
>  Thanks,
>  Sumit Maheshwari
>  cell. 9632202950
> 
> 
>  On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini <
> >> criccom...@apache.org
> 
>  wrote:
> 
> > I would definitely be up to joining. We're interested in the K8s work
> > that's going on. That time works for me.
> >
> > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > daniel.imber...@gmail.com>
> > wrote:
> >
> >> Hello everyone,
> >>
> >> Recently there's been a fair amount of discussion regarding the
> > integration
> >> of airflow with kubernetes. If there is interest I would love to
> host
> >>> an
> >> e-meeting to discuss this integration. I can go over the
> architecture
> >>> as
> > it
> >> stands right now and would love feedback on
> > improvements/features/design. I
> >> could also attempt to get one or two members of google's kubernetes
> >>> team
> > to
> >> join to discuss best practices.
> >>
> >> I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> > though
> >> if there's strong opinions otherwise I'd be glad to propose other
> >>> times.
> >>
> >> Cheers!
> >>
> >> Daniel
> >>
> >
> >>>
> >>
>
>

Re: Airflow + Kubernetes discussion

[Bolke de Bruin]

Invite would be nice, I will try to join!

> On 20 Jul 2017, at 20:36, Gerard Toonstra  wrote:
> 
> send me an invite too!
> 
> On Thu, Jul 20, 2017 at 8:17 PM, Jeremiah Lowin  wrote:
> 
>> I'm interested as well.
>> 
>> On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger  wrote:
>> 
>>> +1 We're in the middle of moving some services to k8s, and have had our
>>> eye on Airflow.
>>> 
 On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari >> 
>>> wrote:
 
 I would join as well for sure.
 
 Thanks,
 Sumit Maheshwari
 cell. 9632202950
 
 
 On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini <
>> criccom...@apache.org
 
 wrote:
 
> I would definitely be up to joining. We're interested in the K8s work
> that's going on. That time works for me.
> 
> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> daniel.imber...@gmail.com>
> wrote:
> 
>> Hello everyone,
>> 
>> Recently there's been a fair amount of discussion regarding the
> integration
>> of airflow with kubernetes. If there is interest I would love to host
>>> an
>> e-meeting to discuss this integration. I can go over the architecture
>>> as
> it
>> stands right now and would love feedback on
> improvements/features/design. I
>> could also attempt to get one or two members of google's kubernetes
>>> team
> to
>> join to discuss best practices.
>> 
>> I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> though
>> if there's strong opinions otherwise I'd be glad to propose other
>>> times.
>> 
>> Cheers!
>> 
>> Daniel
>> 
> 
>>> 
>> 

Re: Airflow + Kubernetes discussion

[Gerard Toonstra]

send me an invite too!

On Thu, Jul 20, 2017 at 8:17 PM, Jeremiah Lowin  wrote:

> I'm interested as well.
>
> On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger  wrote:
>
> > +1 We're in the middle of moving some services to k8s, and have had our
> > eye on Airflow.
> >
> > > On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari  >
> > wrote:
> > >
> > > I would join as well for sure.
> > >
> > > Thanks,
> > > Sumit Maheshwari
> > > cell. 9632202950
> > >
> > >
> > > On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini <
> criccom...@apache.org
> > >
> > > wrote:
> > >
> > >> I would definitely be up to joining. We're interested in the K8s work
> > >> that's going on. That time works for me.
> > >>
> > >> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > >> daniel.imber...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hello everyone,
> > >>>
> > >>> Recently there's been a fair amount of discussion regarding the
> > >> integration
> > >>> of airflow with kubernetes. If there is interest I would love to host
> > an
> > >>> e-meeting to discuss this integration. I can go over the architecture
> > as
> > >> it
> > >>> stands right now and would love feedback on
> > >> improvements/features/design. I
> > >>> could also attempt to get one or two members of google's kubernetes
> > team
> > >> to
> > >>> join to discuss best practices.
> > >>>
> > >>> I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> > >> though
> > >>> if there's strong opinions otherwise I'd be glad to propose other
> > times.
> > >>>
> > >>> Cheers!
> > >>>
> > >>> Daniel
> > >>>
> > >>
> >
>

Re: Airflow + Kubernetes discussion

[Jeremiah Lowin]

I'm interested as well.

On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger  wrote:

> +1 We're in the middle of moving some services to k8s, and have had our
> eye on Airflow.
>
> > On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari 
> wrote:
> >
> > I would join as well for sure.
> >
> > Thanks,
> > Sumit Maheshwari
> > cell. 9632202950
> >
> >
> > On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini  >
> > wrote:
> >
> >> I would definitely be up to joining. We're interested in the K8s work
> >> that's going on. That time works for me.
> >>
> >> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> >> daniel.imber...@gmail.com>
> >> wrote:
> >>
> >>> Hello everyone,
> >>>
> >>> Recently there's been a fair amount of discussion regarding the
> >> integration
> >>> of airflow with kubernetes. If there is interest I would love to host
> an
> >>> e-meeting to discuss this integration. I can go over the architecture
> as
> >> it
> >>> stands right now and would love feedback on
> >> improvements/features/design. I
> >>> could also attempt to get one or two members of google's kubernetes
> team
> >> to
> >>> join to discuss best practices.
> >>>
> >>> I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> >> though
> >>> if there's strong opinions otherwise I'd be glad to propose other
> times.
> >>>
> >>> Cheers!
> >>>
> >>> Daniel
> >>>
> >>
>

Re: Airflow + Kubernetes discussion

[Marc Bollinger]

+1 We're in the middle of moving some services to k8s, and have had our eye on 
Airflow.

> On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari  wrote:
> 
> I would join as well for sure.
> 
> Thanks,
> Sumit Maheshwari
> cell. 9632202950
> 
> 
> On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini 
> wrote:
> 
>> I would definitely be up to joining. We're interested in the K8s work
>> that's going on. That time works for me.
>> 
>> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
>> daniel.imber...@gmail.com>
>> wrote:
>> 
>>> Hello everyone,
>>> 
>>> Recently there's been a fair amount of discussion regarding the
>> integration
>>> of airflow with kubernetes. If there is interest I would love to host an
>>> e-meeting to discuss this integration. I can go over the architecture as
>> it
>>> stands right now and would love feedback on
>> improvements/features/design. I
>>> could also attempt to get one or two members of google's kubernetes team
>> to
>>> join to discuss best practices.
>>> 
>>> I'm currently thinking that next Thursday at 11AM PST over zoom.us,
>> though
>>> if there's strong opinions otherwise I'd be glad to propose other times.
>>> 
>>> Cheers!
>>> 
>>> Daniel
>>> 
>> 

Re: Airflow + Kubernetes discussion

[Sumit Maheshwari]

I would join as well for sure.

Thanks,
Sumit Maheshwari
cell. 9632202950


On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini 
wrote:

> I would definitely be up to joining. We're interested in the K8s work
> that's going on. That time works for me.
>
> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> daniel.imber...@gmail.com>
> wrote:
>
> > Hello everyone,
> >
> > Recently there's been a fair amount of discussion regarding the
> integration
> > of airflow with kubernetes. If there is interest I would love to host an
> > e-meeting to discuss this integration. I can go over the architecture as
> it
> > stands right now and would love feedback on
> improvements/features/design. I
> > could also attempt to get one or two members of google's kubernetes team
> to
> > join to discuss best practices.
> >
> > I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> though
> > if there's strong opinions otherwise I'd be glad to propose other times.
> >
> > Cheers!
> >
> > Daniel
> >
>