Podling Report Reminder - January 2018

[johndament]

Dear podling,

This email was sent by an automated system on behalf of the Apache
Incubator PMC. It is an initial reminder to give you plenty of time to
prepare your quarterly board report.

The board meeting is scheduled for Wed, 17 January 2018, 10:30 am PDT.
The report for your podling will form a part of the Incubator PMC
report. The Incubator PMC requires your report to be submitted 2 weeks
before the board meeting, to allow sufficient time for review and
submission (Wed, January 03).

Please submit your report with sufficient time to allow the Incubator
PMC, and subsequently board members to review and digest. Again, the
very latest you should submit your report is 2 weeks prior to the board
meeting.

Thanks,

The Apache Incubator PMC

Submitting your Report

--

Your report should contain the following:

*   Your project name
*   A brief description of your project, which assumes no knowledge of
the project or necessarily of its field
*   A list of the three most important issues to address in the move
towards graduation.
*   Any issues that the Incubator PMC or ASF Board might wish/need to be
aware of
*   How has the community developed since the last report
*   How has the project developed since the last report.
*   How does the podling rate their own maturity.

This should be appended to the Incubator Wiki page at:

https://wiki.apache.org/incubator/January2018

Note: This is manually populated. You may need to wait a little before
this page is created from a template.

Mentors
---

Mentors should review reports for their project(s) and sign them off on
the Incubator wiki page. Signing off reports shows that you are
following the project - projects that are not signed may raise alarms
for the Incubator PMC.

Incubator PMC

Re: 4/17 Airflow Meetup Slides

[Dan Davydov]

Thanks Sid!

On Wed, Dec 27, 2017 at 3:13 PM Sid Anand  wrote:

> Thx for posting the slides : I've added them to the Announcements page :
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-Nov1,2017
>
> On Thu, Dec 7, 2017 at 12:09 PM, Dan Davydov  .invalid
> > wrote:
>
> > Here  >
> > are the presentations from the 4/17 Airflow meetup at Airbnb.
> >
> > Unfortunately we weren't able to get the recording software working for
> the
> > talk.
> >
> > Thank you all for coming!
> >
>

Re: [RESULT][VOTE] Airflow 1.9.0rc8

[Jakob Homan]

Sent out a call to arms...

-jg


On 27 December 2017 at 10:22, Sid Anand  wrote:
> Ugh... you're right. Hmn... what about our other 2 mentors?
>
> -s
>
> On Wed, Dec 27, 2017 at 4:14 AM, Bolke de Bruin  wrote:
>
>> Did you count 3? I only counted 2?
>>
>> Sent from my iPhone
>>
>> > On 27 Dec 2017, at 02:48, Sid Anand  wrote:
>> >
>> > I see that we received 3 +1 binding votes (assuming all IPMC members).
>> Per
>> > https://incubator.apache.org/guides/releasemanagement.html, shouldn't
>> that
>> > be sufficient?
>> >
>> > In the podling report, should I mention that we did our 4th release or
>> > still working toward it?
>> > -s
>> >
>> > On Tue, Dec 19, 2017 at 9:50 AM, Chris Riccomini 
>> > wrote:
>> >
>> >> I am waiting until incubator actually approves the release. Otherwise, I
>> >> might have to redo the tag if they reject us and we have to re-vote a
>> new
>> >> RC.
>> >>
>> >>> On Tue, Dec 19, 2017 at 8:19 AM, Bolke de Bruin 
>> wrote:
>> >>>
>> >>> Chris,
>> >>>
>> >>> Can you please tag 1.9.0 correctly (ie 1.9.0)? I see reports on gitter
>> >>> people have issues with it.
>> >>>
>> >>> Cheers,
>> >>> Bolke.
>> >>>
>>  On 18 Dec 2017, at 21:49, Chris Riccomini 
>> >> wrote:
>> 
>>  Thanks to all for helping validate it. :)
>> 
>>  On Mon, Dec 18, 2017 at 12:48 PM, Ash Berlin-Taylor <
>>  ash_airflowl...@firemirror.com> wrote:
>> 
>> > Woot! Great news, and thanks for all the time you've spent preparing
>> >> all
>> > the RCs.
>> >
>> > -ash
>> >
>> >> On 18 Dec 2017, at 20:13, Chris Riccomini 
>> >>> wrote:
>> >>
>> >> Hello,
>> >>
>> >> Apache Airflow (incubating) 1.9.0 (based on RC8) has been accepted.
>> >>
>> >> 5 “+1” binding votes received:
>> >>
>> >> - Maxime Beauchemin (binding)
>> >> - Chris Riccomini (binding)
>> >> - Bolke de Bruin (binding)
>> >> - Joy Gao (binding)
>> >> - Fokko Driesprong (binding)
>> >>
>> >> My next step is to open a thread with the IPMC.
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> On Mon, Dec 18, 2017 at 11:26 AM, Chris Riccomini <
>> >>> criccom...@apache.org
>> >>
>> >> wrote:
>> >>
>> >>> I believe so, yes.
>> >>>
>> >>> I will proceed with release. :)
>> >>>
>> >>> On Mon, Dec 18, 2017 at 11:20 AM, Bolke de Bruin <
>> bdbr...@gmail.com
>> >>>
>> >>> wrote:
>> >>>
>>  But it doesn’t stop us from releasing now right? You can just
>> >> update
>> > the
>>  tag. It is not connected to the artifacts.
>> 
>>  Bolke.
>> 
>>  Sent from my iPhone
>> 
>> > On 18 Dec 2017, at 18:51, Chris Riccomini > >
>>  wrote:
>> >
>> > https://github.com/apache/incubator-airflow/pull/2889
>> >
>> > On Mon, Dec 18, 2017 at 9:44 AM, Chris Riccomini <
>> > criccom...@apache.org
>> >
>> > wrote:
>> >
>> >> Sigh. There's a catch 22 here. If we tag an RC as 1.9.0, we'll
>> >> have
>> > to
>> >> keep editing the tag every time there's a new RC. I agree that
>>  removing the
>> >> check seems like the best course of action.
>> >>
>> >> On Sun, Dec 17, 2017 at 12:01 AM, Bolke de Bruin <
>> >>> bdbr...@gmail.com>
>> >> wrote:
>> >>
>> >>> This is just matter of setting the tag in the repo right?
>> >>>
>> >>> We should remove that check or make it not fail at least. It is
>> >>> ridiculous.
>> >>>
>> >>> B.
>> >>>
>> >>> Verstuurd vanaf mijn iPad
>> >>>
>>  Op 17 dec. 2017 om 07:32 heeft Joy Gao  het
>> > volgende
>> >>> geschreven:
>> 
>>  Ahh, tested the build on a fresh virtualenv, which succeeded
>> >> *the
>>  first
>>  time* given gitpython was not installed and it skipped the
>> > assertion
>> >>> check
>>  > >>> setup.py#L68-L73>.
>>  Build fails on re-installs :(
>> 
>> > On Fri, Dec 15, 2017 at 6:01 PM, Feng Lu
>> > > >
>> >>> wrote:
>> >
>> > +0.5 (non-binding)
>> >
>> > Looks like the version(1.9.0) and tag(1.9.0rc8) is
>> mismatched,
>> > > >>> setup.py#L87>
>> > which will cause the installation (pip install or python
>> >> setup)
>> >>> to
>> >>> error
>> > out and fail.
>> > nit: mind also updating the release log "
>> > https://github.com/apache/incubator-airflow/blob/1.9.0rc8/
>> >>> CHANGELOG.txt"
>> >
>> >
>> > On Fri, Dec 15, 2017 at 3:21 PM, Driesprong, Fokko
>> >>> 
>> > wrot

Re: 4/17 Airflow Meetup Slides

[Sid Anand]

Thx for posting the slides : I've added them to the Announcements page :
https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-Nov1,2017

On Thu, Dec 7, 2017 at 12:09 PM, Dan Davydov  wrote:

> Here 
> are the presentations from the 4/17 Airflow meetup at Airbnb.
>
> Unfortunately we weren't able to get the recording software working for the
> talk.
>
> Thank you all for coming!
>

Re: Airflow documentation update

[Sid Anand]

Thx Andy. The process for building this documentation might still be in
Max's hands.

Max,
Can you bring the other committers/contributors/community up to speed on
this process by outlining the process on our Airflow wiki?
-s

On Wed, Dec 27, 2017 at 7:37 AM, Andy Hadjigeorgiou 
wrote:

> Hey Sid,
>
> I'm an Airflow contributor, working on a number of Airflow 2.0 roadmap
>  items,
> and I was wondering if you'd have some insight for one of them.
>
> https://issues.apache.org/jira/browse/AIRFLOW-1924
>
> This ticket describes an issue we are having with our documentation at
> airflow.apache.org, where it is not updating along with updates to our
> codebase. I'm not sure if these Sphinx docs need to be manually rebuilt, or
> if there is a configuration on our end that is broken.
>
> Is this something that you'd have visibility or experience with?
>
> I really appreciate the help.
>
> Best,
> Andy
>

Re: Apache Airflow git mirror sync issue

[Sid Anand]

it's working now! Closing the INFRA Jira.
-s

On Tue, Dec 26, 2017 at 4:57 PM, Sid Anand  wrote:

> FYI,
> I've opened the infra ticket below when I noticed that our git mirror
> stopped syncing.
>
> https://issues.apache.org/jira/browse/INFRA-15725
>
> -s
>

Re: [RESULT][VOTE] Airflow 1.9.0rc8

[Sid Anand]

Ugh... you're right. Hmn... what about our other 2 mentors?

-s

On Wed, Dec 27, 2017 at 4:14 AM, Bolke de Bruin  wrote:

> Did you count 3? I only counted 2?
>
> Sent from my iPhone
>
> > On 27 Dec 2017, at 02:48, Sid Anand  wrote:
> >
> > I see that we received 3 +1 binding votes (assuming all IPMC members).
> Per
> > https://incubator.apache.org/guides/releasemanagement.html, shouldn't
> that
> > be sufficient?
> >
> > In the podling report, should I mention that we did our 4th release or
> > still working toward it?
> > -s
> >
> > On Tue, Dec 19, 2017 at 9:50 AM, Chris Riccomini 
> > wrote:
> >
> >> I am waiting until incubator actually approves the release. Otherwise, I
> >> might have to redo the tag if they reject us and we have to re-vote a
> new
> >> RC.
> >>
> >>> On Tue, Dec 19, 2017 at 8:19 AM, Bolke de Bruin 
> wrote:
> >>>
> >>> Chris,
> >>>
> >>> Can you please tag 1.9.0 correctly (ie 1.9.0)? I see reports on gitter
> >>> people have issues with it.
> >>>
> >>> Cheers,
> >>> Bolke.
> >>>
>  On 18 Dec 2017, at 21:49, Chris Riccomini 
> >> wrote:
> 
>  Thanks to all for helping validate it. :)
> 
>  On Mon, Dec 18, 2017 at 12:48 PM, Ash Berlin-Taylor <
>  ash_airflowl...@firemirror.com> wrote:
> 
> > Woot! Great news, and thanks for all the time you've spent preparing
> >> all
> > the RCs.
> >
> > -ash
> >
> >> On 18 Dec 2017, at 20:13, Chris Riccomini 
> >>> wrote:
> >>
> >> Hello,
> >>
> >> Apache Airflow (incubating) 1.9.0 (based on RC8) has been accepted.
> >>
> >> 5 “+1” binding votes received:
> >>
> >> - Maxime Beauchemin (binding)
> >> - Chris Riccomini (binding)
> >> - Bolke de Bruin (binding)
> >> - Joy Gao (binding)
> >> - Fokko Driesprong (binding)
> >>
> >> My next step is to open a thread with the IPMC.
> >>
> >> Cheers,
> >> Chris
> >>
> >> On Mon, Dec 18, 2017 at 11:26 AM, Chris Riccomini <
> >>> criccom...@apache.org
> >>
> >> wrote:
> >>
> >>> I believe so, yes.
> >>>
> >>> I will proceed with release. :)
> >>>
> >>> On Mon, Dec 18, 2017 at 11:20 AM, Bolke de Bruin <
> bdbr...@gmail.com
> >>>
> >>> wrote:
> >>>
>  But it doesn’t stop us from releasing now right? You can just
> >> update
> > the
>  tag. It is not connected to the artifacts.
> 
>  Bolke.
> 
>  Sent from my iPhone
> 
> > On 18 Dec 2017, at 18:51, Chris Riccomini  >
>  wrote:
> >
> > https://github.com/apache/incubator-airflow/pull/2889
> >
> > On Mon, Dec 18, 2017 at 9:44 AM, Chris Riccomini <
> > criccom...@apache.org
> >
> > wrote:
> >
> >> Sigh. There's a catch 22 here. If we tag an RC as 1.9.0, we'll
> >> have
> > to
> >> keep editing the tag every time there's a new RC. I agree that
>  removing the
> >> check seems like the best course of action.
> >>
> >> On Sun, Dec 17, 2017 at 12:01 AM, Bolke de Bruin <
> >>> bdbr...@gmail.com>
> >> wrote:
> >>
> >>> This is just matter of setting the tag in the repo right?
> >>>
> >>> We should remove that check or make it not fail at least. It is
> >>> ridiculous.
> >>>
> >>> B.
> >>>
> >>> Verstuurd vanaf mijn iPad
> >>>
>  Op 17 dec. 2017 om 07:32 heeft Joy Gao  het
> > volgende
> >>> geschreven:
> 
>  Ahh, tested the build on a fresh virtualenv, which succeeded
> >> *the
>  first
>  time* given gitpython was not installed and it skipped the
> > assertion
> >>> check
>   >>> setup.py#L68-L73>.
>  Build fails on re-installs :(
> 
> > On Fri, Dec 15, 2017 at 6:01 PM, Feng Lu
> >  >
> >>> wrote:
> >
> > +0.5 (non-binding)
> >
> > Looks like the version(1.9.0) and tag(1.9.0rc8) is
> mismatched,
> >  >>> setup.py#L87>
> > which will cause the installation (pip install or python
> >> setup)
> >>> to
> >>> error
> > out and fail.
> > nit: mind also updating the release log "
> > https://github.com/apache/incubator-airflow/blob/1.9.0rc8/
> >>> CHANGELOG.txt"
> >
> >
> > On Fri, Dec 15, 2017 at 3:21 PM, Driesprong, Fokko
> >>> 
> > wrote:
> >
> >> +1 binding
> >>
> >> Op vr 15 dec. 2017 om 23:39 schreef Bolke de Bruin <
>  bdbr...@gmail.com
> 
> >>
> >>> +1, binding
> >>>
> >>>

Re: Custom BashSensor

[Diogo Franco]

Could someone look into merging this?
https://github.com/apache/incubator-airflow/pull/2489

I've been actively maintaining it conflict free for some months.

Thanks!
Cheers,
Diogo

On 4 August 2017 at 15:50, Diogo Franco  wrote:

> A PR implementing this is here https://github.com/
> apache/incubator-airflow/pull/2489
>
> Cheers,
> Diogo
>
> On 30 July 2017 at 14:29, Diogo Franco  wrote:
>
>> Hi Maxime,
>>
>> Awesome, I also believe PythonSensor makes sense in the core package of
>> course. These also allow someone trying out airflow to easily build dag's
>> with sensors without setting up connections, for example.
>>
>> I'll open a different ticket for PythonSensor and do it separately, if
>> that is ok.
>>
>> Cheers.
>>
>> On 28 July 2017 at 17:46, Maxime Beauchemin 
>> wrote:
>>
>>> BashSensor would be a good contribution.
>>>
>>> I'm guessing that the reason why it doesn't exist or (PythonSensor
>>> doesn't
>>> exist for that matter) is that it's pretty easy to derive
>>> BaseSensorOperator and make your own. It still would be neat to have it
>>> in
>>> the core package and in the docs.
>>>
>>> Max
>>>
>>> On Fri, Jul 28, 2017 at 7:12 AM, Diogo Franco >> >
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > I was looking for a sensor that allowed me to customize the sensing
>>> logic.
>>> > I couldn't find any way of doing this and wrote a BashSensor, which
>>> returns
>>> > True or False based on a command/script's return code. The
>>> implementation
>>> > is similar to the BashOperator, and it seems to me that it would fit
>>> pretty
>>> > naturally within airflow.
>>> >
>>> > Is there another way to achieve this that I don't know about? If you
>>> think
>>> > it is worthwhile, I'm happy to work on some tests and a PR.
>>> >
>>> > Thank you!
>>> >
>>>
>>
>>
>

Re: [RESULT][VOTE] Airflow 1.9.0rc8

[Bolke de Bruin]

Did you count 3? I only counted 2?

Sent from my iPhone

> On 27 Dec 2017, at 02:48, Sid Anand  wrote:
> 
> I see that we received 3 +1 binding votes (assuming all IPMC members). Per
> https://incubator.apache.org/guides/releasemanagement.html, shouldn't that
> be sufficient?
> 
> In the podling report, should I mention that we did our 4th release or
> still working toward it?
> -s
> 
> On Tue, Dec 19, 2017 at 9:50 AM, Chris Riccomini 
> wrote:
> 
>> I am waiting until incubator actually approves the release. Otherwise, I
>> might have to redo the tag if they reject us and we have to re-vote a new
>> RC.
>> 
>>> On Tue, Dec 19, 2017 at 8:19 AM, Bolke de Bruin  wrote:
>>> 
>>> Chris,
>>> 
>>> Can you please tag 1.9.0 correctly (ie 1.9.0)? I see reports on gitter
>>> people have issues with it.
>>> 
>>> Cheers,
>>> Bolke.
>>> 
 On 18 Dec 2017, at 21:49, Chris Riccomini 
>> wrote:
 
 Thanks to all for helping validate it. :)
 
 On Mon, Dec 18, 2017 at 12:48 PM, Ash Berlin-Taylor <
 ash_airflowl...@firemirror.com> wrote:
 
> Woot! Great news, and thanks for all the time you've spent preparing
>> all
> the RCs.
> 
> -ash
> 
>> On 18 Dec 2017, at 20:13, Chris Riccomini 
>>> wrote:
>> 
>> Hello,
>> 
>> Apache Airflow (incubating) 1.9.0 (based on RC8) has been accepted.
>> 
>> 5 “+1” binding votes received:
>> 
>> - Maxime Beauchemin (binding)
>> - Chris Riccomini (binding)
>> - Bolke de Bruin (binding)
>> - Joy Gao (binding)
>> - Fokko Driesprong (binding)
>> 
>> My next step is to open a thread with the IPMC.
>> 
>> Cheers,
>> Chris
>> 
>> On Mon, Dec 18, 2017 at 11:26 AM, Chris Riccomini <
>>> criccom...@apache.org
>> 
>> wrote:
>> 
>>> I believe so, yes.
>>> 
>>> I will proceed with release. :)
>>> 
>>> On Mon, Dec 18, 2017 at 11:20 AM, Bolke de Bruin >> 
>>> wrote:
>>> 
 But it doesn’t stop us from releasing now right? You can just
>> update
> the
 tag. It is not connected to the artifacts.
 
 Bolke.
 
 Sent from my iPhone
 
> On 18 Dec 2017, at 18:51, Chris Riccomini 
 wrote:
> 
> https://github.com/apache/incubator-airflow/pull/2889
> 
> On Mon, Dec 18, 2017 at 9:44 AM, Chris Riccomini <
> criccom...@apache.org
> 
> wrote:
> 
>> Sigh. There's a catch 22 here. If we tag an RC as 1.9.0, we'll
>> have
> to
>> keep editing the tag every time there's a new RC. I agree that
 removing the
>> check seems like the best course of action.
>> 
>> On Sun, Dec 17, 2017 at 12:01 AM, Bolke de Bruin <
>>> bdbr...@gmail.com>
>> wrote:
>> 
>>> This is just matter of setting the tag in the repo right?
>>> 
>>> We should remove that check or make it not fail at least. It is
>>> ridiculous.
>>> 
>>> B.
>>> 
>>> Verstuurd vanaf mijn iPad
>>> 
 Op 17 dec. 2017 om 07:32 heeft Joy Gao  het
> volgende
>>> geschreven:
 
 Ahh, tested the build on a fresh virtualenv, which succeeded
>> *the
 first
 time* given gitpython was not installed and it skipped the
> assertion
>>> check
 >> setup.py#L68-L73>.
 Build fails on re-installs :(
 
> On Fri, Dec 15, 2017 at 6:01 PM, Feng Lu
>  
>>> wrote:
> 
> +0.5 (non-binding)
> 
> Looks like the version(1.9.0) and tag(1.9.0rc8) is mismatched,
> >> setup.py#L87>
> which will cause the installation (pip install or python
>> setup)
>>> to
>>> error
> out and fail.
> nit: mind also updating the release log "
> https://github.com/apache/incubator-airflow/blob/1.9.0rc8/
>>> CHANGELOG.txt"
> 
> 
> On Fri, Dec 15, 2017 at 3:21 PM, Driesprong, Fokko
>>> 
> wrote:
> 
>> +1 binding
>> 
>> Op vr 15 dec. 2017 om 23:39 schreef Bolke de Bruin <
 bdbr...@gmail.com
 
>> 
>>> +1, binding
>>> 
>>> Checked sigs, version, source is there (did not check
>> build),
> bin
 is
>> there.
>>> 
>>> Bolke
>>> 
>>> Verstuurd vanaf mijn iPad
>>> 
 Op 15 dec. 2017 om 23:31 heeft Joy Gao 
>> het
>>> volgende
>>> geschreven:
 
 +1, binding
 
 Tha