subject:"Airflow Developers Meeting \- 08\/03 Notes"

Re: Airflow Developers Meeting - 08/03 Notes

2016-08-23 Thread Jakob Homan

ASF provides per-projects blog infrastructure:
http://www.apache.org/dev/project-blogs

-Jakob


On 23 August 2016 at 15:07, Maxime Beauchemin
 wrote:
> I'm planning on working on a minimal apache release this week. If someone
> can point to the commit or PRs I can add them to `branch-1.7.2-apache` and
> start moving towards the release.
>
> Max
>
> On Tue, Aug 23, 2016 at 11:13 AM, Alex Van Boxel  wrote:
>
>> All stuff is in master (I'm running pre-production on master), if it needs
>> to be cherry-picked to a branch I can always test it. I would be grate if
>> it's in the next release so I can start blogging about it.
>>
>> On Tue, Aug 23, 2016 at 8:05 PM Bolke de Bruin  wrote:
>>
>> > (Sorry for the late response, I was on holiday)
>> >
>> > I think the G* operators just need to be cherry picked. This will make us
>> > deviate slightly from the
>> > previous release, but makes sure we don’t have to ‘fix’ history
>> afterwards.
>> >
>> > Anyone against this?
>> >
>> > - B.
>> >
>> > > Op 8 aug. 2016, om 14:21 heeft Alex Van Boxel  het
>> > volgende geschreven:
>> > >
>> > > Sorry, Bolke. What needs to be done for Google Cloud operators/hooks?
>> > >
>> > > If you bring me up-to-speed I can do this. I'm currently working an a
>> CI
>> > > setup for the Google stuff).
>> > >
>> > > On Sat, Aug 6, 2016 at 3:56 PM Bolke de Bruin 
>> wrote:
>> > >
>> > >> I have indeed a branch with cherry picked commits. This branch is
>> > >> available in the apache repo. The branch is available as
>> > >> “branch-1.7.2-apache”. What is left to do is to decide weather we do
>> > >> upgrade the google cloud operators/hooks as part of this release as
>> the
>> > >> licenses were only added after these changes came in. I prefer this
>> > >> approach as we stay away from custom commits as much as possible then.
>> > >>
>> > >> - Bolke
>> > >>
>> > >>> Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli <
>> gurer.kira...@airbnb.com
>> > .INVALID>
>> > >> het volgende geschreven:
>> > >>>
>> > >>> Agenda
>> > >>>
>> > >>>
>> > >>>  -
>> > >>>
>> > >>>  Committers sync-up: progress and plans
>> > >>>  -
>> > >>>
>> > >>> Max to do a recap since last release
>> > >>> -
>> > >>>
>> > >>> Airbnb
>> > >>> -
>> > >>>
>> > >>> Anyone else?
>> > >>> -
>> > >>>
>> > >>>  Cooperation Best Practices
>> > >>>  -
>> > >>>
>> > >>>  Solicit for feedback for Impersonation Design Review
>> > >>>  -
>> > >>>
>> > >>>  Release Schedule, Management
>> > >>>  -
>> > >>>
>> > >>>  Roadmap discussion
>> > >>>
>> > >>>
>> > >>> Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?),
>> Maxime
>> > >>> Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur
>> > Wiedmer,
>> > >>> Gurer Kiratli
>> > >>>
>> > >>> Notes
>> > >>>
>> > >>>
>> > >>>  -
>> > >>>
>> > >>>  Recap for the overall project
>> > >>>  -
>> > >>>
>> > >>> [Max]
>> > >>>
>> > >> https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
>> > >>> -
>> > >>>
>> > >>>  Airbnb Recap
>> > >>>  -
>> > >>>
>> > >>> [Paul] Airbnb is continuing work on features related to migrating
>> > >>> more jobs onto Airflow - DB connection scalability,
>> impersonation,
>> > >>> refreshing web UI, DAG git versioning, task resource isolation
>> > >>> -
>> > >>>
>> > >>>  Other Recaps
>> > >>>  -
>> > >>>
>> > >>> [Chris] Improvements to Import functionality. Joy did some work
>> > >>> around CLIs and mark success in collaboration with Bolke.
>> > >>> -
>> > >>>
>> > >>> [Jeremiah] Came up with the Merge tool. Focusing on the work
>> around
>> > >>> configuration.
>> > >>> -
>> > >>>
>> > >>>  Meeting cadence
>> > >>>  -
>> > >>>
>> > >>> [Max] We can have monthly meetings. Let’s start with monthly we
>> can
>> > >>> change the cadence. We should invite all contributors. We should
>> > >>> specify in
>> > >>> the invite that everyone is welcome. Currently posting this on
>> > >>> the mailing
>> > >>> list and wiki.
>> > >>> -
>> > >>>
>> > >>> *Action Item*  Gurer to schedule monthly meetings next one at
>> > >> Airbnb.
>> > >>> Airbnb is welcoming folks to come onsite
>> > >>> -
>> > >>>
>> > >>>  Blog/Documentation
>> > >>>  -
>> > >>>
>> > >>> [Max, Chris, Jeremiah] Having a blog would be useful. We can
>> > >>> structure our thoughts.
>> > >>> -
>> > >>>
>> > >>> [Andrew] Blog should have better language with screenshots and
>> all.
>> > >>> This raises the level of effort bar and this might mean that
>> > >>> nobody really
>> > >>> does it.
>> > >>> -
>> > >>>
>> > >>> [Max] Maybe use Medium? We can try it out.
>> > >>> -
>> > >>>
>> > >>> [Max] Haven’t looked at the documentation for a long time. I
>> don’t
>> > >>> really know about its quality.
>> > >>> -
>> > >>>
>> > >>> [Arthur] Looking at the email list responses, seems

Re: Airflow Developers Meeting - 08/03 Notes

2016-08-23 Thread Maxime Beauchemin

I'm planning on working on a minimal apache release this week. If someone
can point to the commit or PRs I can add them to `branch-1.7.2-apache` and
start moving towards the release.

Max

On Tue, Aug 23, 2016 at 11:13 AM, Alex Van Boxel  wrote:

> All stuff is in master (I'm running pre-production on master), if it needs
> to be cherry-picked to a branch I can always test it. I would be grate if
> it's in the next release so I can start blogging about it.
>
> On Tue, Aug 23, 2016 at 8:05 PM Bolke de Bruin  wrote:
>
> > (Sorry for the late response, I was on holiday)
> >
> > I think the G* operators just need to be cherry picked. This will make us
> > deviate slightly from the
> > previous release, but makes sure we don’t have to ‘fix’ history
> afterwards.
> >
> > Anyone against this?
> >
> > - B.
> >
> > > Op 8 aug. 2016, om 14:21 heeft Alex Van Boxel  het
> > volgende geschreven:
> > >
> > > Sorry, Bolke. What needs to be done for Google Cloud operators/hooks?
> > >
> > > If you bring me up-to-speed I can do this. I'm currently working an a
> CI
> > > setup for the Google stuff).
> > >
> > > On Sat, Aug 6, 2016 at 3:56 PM Bolke de Bruin 
> wrote:
> > >
> > >> I have indeed a branch with cherry picked commits. This branch is
> > >> available in the apache repo. The branch is available as
> > >> “branch-1.7.2-apache”. What is left to do is to decide weather we do
> > >> upgrade the google cloud operators/hooks as part of this release as
> the
> > >> licenses were only added after these changes came in. I prefer this
> > >> approach as we stay away from custom commits as much as possible then.
> > >>
> > >> - Bolke
> > >>
> > >>> Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli <
> gurer.kira...@airbnb.com
> > .INVALID>
> > >> het volgende geschreven:
> > >>>
> > >>> Agenda
> > >>>
> > >>>
> > >>>  -
> > >>>
> > >>>  Committers sync-up: progress and plans
> > >>>  -
> > >>>
> > >>> Max to do a recap since last release
> > >>> -
> > >>>
> > >>> Airbnb
> > >>> -
> > >>>
> > >>> Anyone else?
> > >>> -
> > >>>
> > >>>  Cooperation Best Practices
> > >>>  -
> > >>>
> > >>>  Solicit for feedback for Impersonation Design Review
> > >>>  -
> > >>>
> > >>>  Release Schedule, Management
> > >>>  -
> > >>>
> > >>>  Roadmap discussion
> > >>>
> > >>>
> > >>> Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?),
> Maxime
> > >>> Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur
> > Wiedmer,
> > >>> Gurer Kiratli
> > >>>
> > >>> Notes
> > >>>
> > >>>
> > >>>  -
> > >>>
> > >>>  Recap for the overall project
> > >>>  -
> > >>>
> > >>> [Max]
> > >>>
> > >> https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
> > >>> -
> > >>>
> > >>>  Airbnb Recap
> > >>>  -
> > >>>
> > >>> [Paul] Airbnb is continuing work on features related to migrating
> > >>> more jobs onto Airflow - DB connection scalability,
> impersonation,
> > >>> refreshing web UI, DAG git versioning, task resource isolation
> > >>> -
> > >>>
> > >>>  Other Recaps
> > >>>  -
> > >>>
> > >>> [Chris] Improvements to Import functionality. Joy did some work
> > >>> around CLIs and mark success in collaboration with Bolke.
> > >>> -
> > >>>
> > >>> [Jeremiah] Came up with the Merge tool. Focusing on the work
> around
> > >>> configuration.
> > >>> -
> > >>>
> > >>>  Meeting cadence
> > >>>  -
> > >>>
> > >>> [Max] We can have monthly meetings. Let’s start with monthly we
> can
> > >>> change the cadence. We should invite all contributors. We should
> > >>> specify in
> > >>> the invite that everyone is welcome. Currently posting this on
> > >>> the mailing
> > >>> list and wiki.
> > >>> -
> > >>>
> > >>> *Action Item*  Gurer to schedule monthly meetings next one at
> > >> Airbnb.
> > >>> Airbnb is welcoming folks to come onsite
> > >>> -
> > >>>
> > >>>  Blog/Documentation
> > >>>  -
> > >>>
> > >>> [Max, Chris, Jeremiah] Having a blog would be useful. We can
> > >>> structure our thoughts.
> > >>> -
> > >>>
> > >>> [Andrew] Blog should have better language with screenshots and
> all.
> > >>> This raises the level of effort bar and this might mean that
> > >>> nobody really
> > >>> does it.
> > >>> -
> > >>>
> > >>> [Max] Maybe use Medium? We can try it out.
> > >>> -
> > >>>
> > >>> [Max] Haven’t looked at the documentation for a long time. I
> don’t
> > >>> really know about its quality.
> > >>> -
> > >>>
> > >>> [Arthur] Looking at the email list responses, seems like the
> > >>> documentation is missing certain elements.
> > >>> -
> > >>>
> > >>> [Max] Maybe we can interview people who just started and identify
> > >>> gaps.
> > >>> -
> > >>>
> > >>> *Action Item* [Jeremiah] We can also ask the mailing list about
> > >> this.
> > >>> He will send this out.
> > >>> -
>

Re: Airflow Developers Meeting - 08/03 Notes

2016-08-23 Thread Alex Van Boxel

All stuff is in master (I'm running pre-production on master), if it needs
to be cherry-picked to a branch I can always test it. I would be grate if
it's in the next release so I can start blogging about it.

On Tue, Aug 23, 2016 at 8:05 PM Bolke de Bruin  wrote:

> (Sorry for the late response, I was on holiday)
>
> I think the G* operators just need to be cherry picked. This will make us
> deviate slightly from the
> previous release, but makes sure we don’t have to ‘fix’ history afterwards.
>
> Anyone against this?
>
> - B.
>
> > Op 8 aug. 2016, om 14:21 heeft Alex Van Boxel  het
> volgende geschreven:
> >
> > Sorry, Bolke. What needs to be done for Google Cloud operators/hooks?
> >
> > If you bring me up-to-speed I can do this. I'm currently working an a CI
> > setup for the Google stuff).
> >
> > On Sat, Aug 6, 2016 at 3:56 PM Bolke de Bruin  wrote:
> >
> >> I have indeed a branch with cherry picked commits. This branch is
> >> available in the apache repo. The branch is available as
> >> “branch-1.7.2-apache”. What is left to do is to decide weather we do
> >> upgrade the google cloud operators/hooks as part of this release as the
> >> licenses were only added after these changes came in. I prefer this
> >> approach as we stay away from custom commits as much as possible then.
> >>
> >> - Bolke
> >>
> >>> Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli  .INVALID>
> >> het volgende geschreven:
> >>>
> >>> Agenda
> >>>
> >>>
> >>>  -
> >>>
> >>>  Committers sync-up: progress and plans
> >>>  -
> >>>
> >>> Max to do a recap since last release
> >>> -
> >>>
> >>> Airbnb
> >>> -
> >>>
> >>> Anyone else?
> >>> -
> >>>
> >>>  Cooperation Best Practices
> >>>  -
> >>>
> >>>  Solicit for feedback for Impersonation Design Review
> >>>  -
> >>>
> >>>  Release Schedule, Management
> >>>  -
> >>>
> >>>  Roadmap discussion
> >>>
> >>>
> >>> Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?), Maxime
> >>> Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur
> Wiedmer,
> >>> Gurer Kiratli
> >>>
> >>> Notes
> >>>
> >>>
> >>>  -
> >>>
> >>>  Recap for the overall project
> >>>  -
> >>>
> >>> [Max]
> >>>
> >> https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
> >>> -
> >>>
> >>>  Airbnb Recap
> >>>  -
> >>>
> >>> [Paul] Airbnb is continuing work on features related to migrating
> >>> more jobs onto Airflow - DB connection scalability, impersonation,
> >>> refreshing web UI, DAG git versioning, task resource isolation
> >>> -
> >>>
> >>>  Other Recaps
> >>>  -
> >>>
> >>> [Chris] Improvements to Import functionality. Joy did some work
> >>> around CLIs and mark success in collaboration with Bolke.
> >>> -
> >>>
> >>> [Jeremiah] Came up with the Merge tool. Focusing on the work around
> >>> configuration.
> >>> -
> >>>
> >>>  Meeting cadence
> >>>  -
> >>>
> >>> [Max] We can have monthly meetings. Let’s start with monthly we can
> >>> change the cadence. We should invite all contributors. We should
> >>> specify in
> >>> the invite that everyone is welcome. Currently posting this on
> >>> the mailing
> >>> list and wiki.
> >>> -
> >>>
> >>> *Action Item*  Gurer to schedule monthly meetings next one at
> >> Airbnb.
> >>> Airbnb is welcoming folks to come onsite
> >>> -
> >>>
> >>>  Blog/Documentation
> >>>  -
> >>>
> >>> [Max, Chris, Jeremiah] Having a blog would be useful. We can
> >>> structure our thoughts.
> >>> -
> >>>
> >>> [Andrew] Blog should have better language with screenshots and all.
> >>> This raises the level of effort bar and this might mean that
> >>> nobody really
> >>> does it.
> >>> -
> >>>
> >>> [Max] Maybe use Medium? We can try it out.
> >>> -
> >>>
> >>> [Max] Haven’t looked at the documentation for a long time. I don’t
> >>> really know about its quality.
> >>> -
> >>>
> >>> [Arthur] Looking at the email list responses, seems like the
> >>> documentation is missing certain elements.
> >>> -
> >>>
> >>> [Max] Maybe we can interview people who just started and identify
> >>> gaps.
> >>> -
> >>>
> >>> *Action Item* [Jeremiah] We can also ask the mailing list about
> >> this.
> >>> He will send this out.
> >>> -
> >>>
> >>>  Cooperation guidelines
> >>>  -
> >>>
> >>> [Max, Arthur]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Community+Guidelines
> >>> -
> >>>
> >>> [Chris] Let’s link the guidelines to contributor md . This should
> be
> >>> discoverable
> >>> -
> >>>
> >>>  Release
> >>>  -
> >>>
> >>> [Chris, Max] Bolke might have branch with High charts and Licensing
> >>> cherry picked. We can use this as a dry run. We can test the
> >> process.
> >>> -
> >>>
> >>> [Max] Big release is probably around September.
> >>> -
> >>>
> >>>

Airflow Developers Meeting - 08/03 Notes

2016-08-04 Thread Gurer Kiratli

Agenda


   -

   Committers sync-up: progress and plans
   -

  Max to do a recap since last release
  -

  Airbnb
  -

  Anyone else?
  -

   Cooperation Best Practices
   -

   Solicit for feedback for Impersonation Design Review
   -

   Release Schedule, Management
   -

   Roadmap discussion


Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?), Maxime
Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur Wiedmer,
Gurer Kiratli

Notes


   -

   Recap for the overall project
   -

  [Max]
  https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
  -

   Airbnb Recap
   -

  [Paul] Airbnb is continuing work on features related to migrating
  more jobs onto Airflow - DB connection scalability, impersonation,
  refreshing web UI, DAG git versioning, task resource isolation
  -

   Other Recaps
   -

  [Chris] Improvements to Import functionality. Joy did some work
  around CLIs and mark success in collaboration with Bolke.
  -

  [Jeremiah] Came up with the Merge tool. Focusing on the work around
  configuration.
  -

   Meeting cadence
   -

  [Max] We can have monthly meetings. Let’s start with monthly we can
  change the cadence. We should invite all contributors. We should
specify in
  the invite that everyone is welcome. Currently posting this on
the mailing
  list and wiki.
  -

  *Action Item*  Gurer to schedule monthly meetings next one at Airbnb.
  Airbnb is welcoming folks to come onsite
  -

   Blog/Documentation
   -

  [Max, Chris, Jeremiah] Having a blog would be useful. We can
  structure our thoughts.
  -

  [Andrew] Blog should have better language with screenshots and all.
  This raises the level of effort bar and this might mean that
nobody really
  does it.
  -

  [Max] Maybe use Medium? We can try it out.
  -

  [Max] Haven’t looked at the documentation for a long time. I don’t
  really know about its quality.
  -

  [Arthur] Looking at the email list responses, seems like the
  documentation is missing certain elements.
  -

  [Max] Maybe we can interview people who just started and identify
  gaps.
  -

  *Action Item* [Jeremiah] We can also ask the mailing list about this.
  He will send this out.
  -

   Cooperation guidelines
   -

  [Max, Arthur]
  https://cwiki.apache.org/confluence/display/AIRFLOW/Community+Guidelines
  -

  [Chris] Let’s link the guidelines to contributor md . This should be
  discoverable
  -

   Release
   -

  [Chris, Max] Bolke might have branch with High charts and Licensing
  cherry picked. We can use this as a dry run. We can test the process.
  -

  [Max] Big release is probably around September.
  -

  [Andrew] If we want to have a true practice, we need to be diligently
  following Apache guidelines.
  -

  [Max] We might want to revive the email thread about how we should do
  this.
  -

  [Chris] It would be great for someone who hasn’t done this before to
  do this.
  -

  [Andrew] Building a script for this is smart otherwise the manual
  work is painful from experience.
  https://cwiki.apache.org/confluence/display/JCLOUDS/Releasing+jclouds


https://cwiki.apache.org/confluence/display/JCLOUDS/Validate+a+Release

   -

  *Action Item* [Max] Max will own this. This will happen by 08/19.
  -

  [Arthur] For the main release around mid September. We need to
  confirm this date.
  -

   Roadmap
   -

  [Max] Maybe each company puts their own section? Or should we have a
  more cohesive roadmap?
  -

  *Action Item* Gurer to schedule a roadmap discussion
  -

   Misc
   -

  [Max] For executor thinking about sticking with Celery and maybe
  having Docker implemented on top of it.
  -

  [Max] We might want to explore Sphinx for generating documentation
  from code.
  - [Max] Please review the impersonation design review. This is pretty
  simple and reversible.

Re: Airflow Developers Meeting - 08/03 Notes

Re: Airflow Developers Meeting - 08/03 Notes

Re: Airflow Developers Meeting - 08/03 Notes

Airflow Developers Meeting - 08/03 Notes

4 matches

Site Navigation

Mail list logo

Footer information