Re: [VOTE] Release Airflow 1.9.0

[Chris Riccomini]

> Please don't use the ".sha" extension ; use ".sha1", ".sha256", or
".sha512" instead for :

Done.

> If development of 1.8.0-incubating (and/or 1.8.1-incubating) has stopped
(because superseded by 1.8.2-incubating), please remove it from
/dist/release/incubator/airflow/.

Done.

> The dist.apache.org URL is not for use by the general public; it is only
intended as the staging area for Airflow developers during a release.

@sebb, this is where I'm confused. Since you guys haven't approved the
release yet, I was under the impression that we were still staging the
release. Are you saying I should move the rc8 artifacts over to
https://dist.apache.org/repos/dist/release/incubator/airflow/ now, before
the IPMC has +1'd the release?

On Tue, Dec 19, 2017 at 8:35 AM, sebb <seb...@gmail.com> wrote:

> On 18 December 2017 at 20:32, Chris Riccomini <criccom...@apache.org>
> wrote:
> > Hello Incubator PMC’ers,
> >
> > The Apache Airflow community has voted and approved the proposal to
> release
> > Apache Airflow 1.9.0 (incubating) based on 1.9.0 Release Candidate 8. We
> > now kindly request the Incubator PMC members to review and vote on this
> > incubator release.
> >
> > Airflow is a platform to programmatically author, schedule, and monitor
> > workflows. Use Airflow to author workflows as directed acyclic graphs
> > (DAGs) of tasks. The airflow scheduler executes your tasks on an array of
> > workers while following the specified dependencies. Rich command line
> > utilities make performing complex surgeries on DAGs a snap. The rich user
> > interface makes it easy to visualize pipelines running in production,
> > monitor progress, and troubleshoot issues when needed. When workflows are
> > defined as code, they become more maintainable, versionable, testable,
> and
> > collaborative.
> >
> > Artifacts are available at:
> >
> > https://dist.apache.org/repos/dist/dev/incubator/airflow
>
> The dist.apache.org URL is not for use by the general public; it is
> only intended as the staging area for Airflow developers during a
> release.
>
> Releases MUST use the ASF 3rd party mirror system:
>
> https://www.apache.org/dyn/closer.cgi/incubator/airflow
> See also:
> http://www.apache.org/dev/release-download-pages.html#links
>
> > Public keys are available at:
> >
> > https://dist.apache.org/repos/dist/release/incubator/airflow
>
> Again, that is not allowed; only reference KEYS, sigs and hashes from
>
> https://www.apache.org/dist/incubator/airflow
>
> > apache-airflow-1.9.0rc8+incubating-source.tar.gz
> > <
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
> 9.0rc8/apache-airflow-1.9.0rc8+incubating-source.tar.gz
>
> This must use closer.cgi as above
>
> >>
> > is a source release that comes with INSTALL instructions. Along with it,
> for
> > convenience, find the binary Python "sdist" as
> > apache-airflow-1.9.0rc8+incubating-bin.tar.gz
> > <
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
> 9.0rc8/apache-airflow-1.9.0rc8+incubating-bin.tar.gz
> >>
> >
> > Vote thread:
> > https://mail-archives.apache.org/mod_mbox/incubator-
> airflow-dev/201712.mbox/%3CCABYbY7ecu8F8sp%3DgvZrvW3knj%3DQpdCUZu-
> Jh02KngjGOezhBKA%40mail.gmail.com%3E
> >
> > Git tag:
> > https://github.com/apache/incubator-airflow/releases/tag/1.9.0rc8
> >
> > The vote will be open for at least 72 hours or until necessary number of
> > votes are reached.
>
> Nitpick: you cannot close the vote until 72 hours have elapsed even if
> the required number of votes have already been cast.
>
> > Members please be sure to indicate "(Binding)" with your vote which will
> > help in tallying the vote(s).
> >
> > * Here is my +1 (non-binding) *
> >
> > Cheers,
> > Chris
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

[VOTE] Release Airflow 1.9.0

[Chris Riccomini]

Hello Incubator PMC’ers,

The Apache Airflow community has voted and approved the proposal to release
Apache Airflow 1.9.0 (incubating) based on 1.9.0 Release Candidate 8. We
now kindly request the Incubator PMC members to review and vote on this
incubator release.

Airflow is a platform to programmatically author, schedule, and monitor
workflows. Use Airflow to author workflows as directed acyclic graphs
(DAGs) of tasks. The airflow scheduler executes your tasks on an array of
workers while following the specified dependencies. Rich command line
utilities make performing complex surgeries on DAGs a snap. The rich user
interface makes it easy to visualize pipelines running in production,
monitor progress, and troubleshoot issues when needed. When workflows are
defined as code, they become more maintainable, versionable, testable, and
collaborative.

Artifacts are available at:

https://dist.apache.org/repos/dist/dev/incubator/airflow

Public keys are available at:

https://dist.apache.org/repos/dist/release/incubator/airflow

apache-airflow-1.9.0rc8+incubating-source.tar.gz
<
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0rc8/apache-airflow-1.9.0rc8+incubating-source.tar.gz
>
is a source release that comes with INSTALL instructions. Along with it, for
convenience, find the binary Python "sdist" as
apache-airflow-1.9.0rc8+incubating-bin.tar.gz
<
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0rc8/apache-airflow-1.9.0rc8+incubating-bin.tar.gz
>

Vote thread:
https://mail-archives.apache.org/mod_mbox/incubator-airflow-dev/201712.mbox/%3CCABYbY7ecu8F8sp%3DgvZrvW3knj%3DQpdCUZu-Jh02KngjGOezhBKA%40mail.gmail.com%3E

Git tag:
https://github.com/apache/incubator-airflow/releases/tag/1.9.0rc8

The vote will be open for at least 72 hours or until necessary number of
votes are reached.

Members please be sure to indicate "(Binding)" with your vote which will
help in tallying the vote(s).

* Here is my +1 (non-binding) *

Cheers,
Chris

Re: [RESULT] [VOTE] Release Apache Airflow 1.8.1 (incubating)

[Chris Riccomini]

NOTE: Jakob Homan is an IPMC member, and +1'd the release during the
Airflow community vote. Thanks!

On Tue, May 9, 2017 at 9:52 AM, Chris Riccomini <criccom...@apache.org>
wrote:

> Hey all,
>
> The vote has passed after 72+ hours.
>
> There were two +1 votes:
>
>   Josh Elser, John D. Ament
>
> No 0/-1 votes. I'm going to wrap up the release.
>
> @John, thanks for the feedback. We'll take this into account on the next
> release. Some items have actually already been fixed, just not
> cherry-picked into the 1.8.1 branch.
>
> Cheers,
> Chris
>
> On Sun, May 7, 2017 at 2:08 PM, John D. Ament <johndam...@apache.org>
> wrote:
>
>> +1, however there's a few issues with the LICENSE file:
>>
>> - Would be good to list out the locations of each file (or path to a group
>> of files) (some have this, and others do not so its hard to follow)
>> - There's errant /* .. */ around each license declaration, which should be
>> removed.
>> - Missing license bodies for FooTable v2, jQuery Clock Plugin,
>>
>> Likewise, your NOTICE has copyright 2011-2017, however Airflow hasn't been
>> incubating that long.  If you like, you can give origination notices to
>> the
>> original creators here to specify the original copyright dates.
>>
>> I would challenge the podling to see if there's a way to simplify their
>> LICENSE by instead using npm or some other javascript packaging tool to
>> build a distribution, rather than shipping the dependencies in the source
>> release, makes it much easier to use.
>>
>> As the podling matures, would be good to see information about the author
>> switch from an individual to a community (in setup.cfg, its already in
>> setup.py so may have been a miss)
>>
>> It would be great to see a binary distribution in the next vote to see how
>> that may work, its not clear how to build it from this.  Likewise, don't
>> hesitate to clean up your old release artifacts, I downloaded the wrong
>> artifact at first.
>>
>> John
>>
>> On Fri, May 5, 2017 at 4:44 PM Chris Riccomini <criccom...@apache.org>
>> wrote:
>>
>> > Hello Incubator PMC’ers,
>> >
>> > The Apache Airflow community has voted and approved the proposal to
>> release
>> > Apache Airflow 1.8.1 (incubating) based on 1.8.1 Release Candidate 2. We
>> > now kindly request the Incubator PMC members to review and vote on this
>> > incubator release. If the vote is successful we will rename release
>> > candidate 2 to final.
>> >
>> > Airflow is a platform to programmatically author, schedule and monitor
>> > workflows. Use airflow to author workflows as directed acyclic graphs
>> > (DAGs) of tasks. The airflow scheduler executes your tasks on an array
>> of
>> > workers while following the specified dependencies. Rich command line
>> > utilities make performing complex surgeries on DAGs a snap. The rich
>> user
>> > interface makes it easy to visualize pipelines running in production,
>> > monitor progress, and troubleshoot issues when needed. When workflows
>> are
>> > defined as code, they become more maintainable, versionable, testable,
>> and
>> > collaborative.
>> >
>> > The Apache Airflow-1.8.1-incubating release candidate is now available
>> with
>> > the following artifacts for a project vote:
>> >
>> > * [VOTE] Thread:*
>> >
>> > https://mail-archives.apache.org/mod_mbox/incubator-airflow-
>> dev/201705.mbox/%3CCABYbY7eZk2SYHs3ni31h94uwwRdcn4dkJwumYtUM
>> sfFLq-gwjg%40mail.gmail.com%3E
>> >
>> > *[RESULT] [VOTE] Thread:*
>> >
>> > https://mail-archives.apache.org/mod_mbox/incubator-airflow-
>> dev/201705.mbox/%3CCABYbY7dBAnjcvb-w_4Hdy3ZUM3CGn6Y3eUvZctuF
>> CfdAfPGAXQ%40mail.gmail.com%3E
>> >
>> > *The release candidate(s) to be voted on is available at:*
>> > https://dist.apache.org/repos/dist/dev/incubator/airflow/ or
>> >
>> > https://dist.apache.org/repos/dist/dev/incubator/airflow/apa
>> che-airflow-1.8.1rc2+incubating.tar.gz
>> >
>> > *Git branch*
>> > https://github.com/apache/incubator-airflow/tree/v1-8-test or
>> >
>> > https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.
>> git;a=tree;h=refs/heads/v1-8-stable;hb=refs/heads/v1-8-test
>> >
>> > *Git tag*
>> >
>> > https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.
>> git;a=shortlog;h=af2d0b4b5cb1ef30a065b1af66f90a01a953e2be
>> >
>> > 

Re: [RESULT] [VOTE] Release Apache Airflow 1.8.1 (incubating)

[Chris Riccomini]

Just to close the loop, we had a +1 during the Airflow community vote from
Jakob Homan, who is an IPMC member:

https://home.apache.org/phonebook.html?uid=jghoman

I'm going to proceed with the release.

On Tue, May 9, 2017 at 10:23 AM, Chris Riccomini <criccom...@apache.org>
wrote:

> > Were there mentor +1 votes on the dev@ vote thread that can be carried
> over?
>
> Yes, Jakob Homan's vote can be carried over, I believe.
>
> On Tue, May 9, 2017 at 10:18 AM, P. Taylor Goetz <ptgo...@gmail.com>
> wrote:
>
>> Hi Chris,
>>
>> Unless I’m missing something, you’re one vote shy of the requirement of
>> having 3 IPMC +1 votes [1].
>>
>> Were there mentor +1 votes on the dev@ vote thread that can be carried
>> over?
>>
>> -Taylor
>>
>> [1] http://incubator.apache.org/incubation/Incubation_Policy.htm
>> l#Releases
>>
>>
>>
>> > On May 9, 2017, at 12:52 PM, Chris Riccomini <criccom...@apache.org>
>> wrote:
>> >
>> > Hey all,
>> >
>> > The vote has passed after 72+ hours.
>> >
>> > There were two +1 votes:
>> >
>> >  Josh Elser, John D. Ament
>> >
>> > No 0/-1 votes. I'm going to wrap up the release.
>> >
>> > @John, thanks for the feedback. We'll take this into account on the next
>> > release. Some items have actually already been fixed, just not
>> > cherry-picked into the 1.8.1 branch.
>> >
>> > Cheers,
>> > Chris
>> >
>> > On Sun, May 7, 2017 at 2:08 PM, John D. Ament <johndam...@apache.org>
>> wrote:
>> >
>> >> +1, however there's a few issues with the LICENSE file:
>> >>
>> >> - Would be good to list out the locations of each file (or path to a
>> group
>> >> of files) (some have this, and others do not so its hard to follow)
>> >> - There's errant /* .. */ around each license declaration, which
>> should be
>> >> removed.
>> >> - Missing license bodies for FooTable v2, jQuery Clock Plugin,
>> >>
>> >> Likewise, your NOTICE has copyright 2011-2017, however Airflow hasn't
>> been
>> >> incubating that long.  If you like, you can give origination notices
>> to the
>> >> original creators here to specify the original copyright dates.
>> >>
>> >> I would challenge the podling to see if there's a way to simplify their
>> >> LICENSE by instead using npm or some other javascript packaging tool to
>> >> build a distribution, rather than shipping the dependencies in the
>> source
>> >> release, makes it much easier to use.
>> >>
>> >> As the podling matures, would be good to see information about the
>> author
>> >> switch from an individual to a community (in setup.cfg, its already in
>> >> setup.py so may have been a miss)
>> >>
>> >> It would be great to see a binary distribution in the next vote to see
>> how
>> >> that may work, its not clear how to build it from this.  Likewise,
>> don't
>> >> hesitate to clean up your old release artifacts, I downloaded the wrong
>> >> artifact at first.
>> >>
>> >> John
>> >>
>> >> On Fri, May 5, 2017 at 4:44 PM Chris Riccomini <criccom...@apache.org>
>> >> wrote:
>> >>
>> >>> Hello Incubator PMC’ers,
>> >>>
>> >>> The Apache Airflow community has voted and approved the proposal to
>> >> release
>> >>> Apache Airflow 1.8.1 (incubating) based on 1.8.1 Release Candidate 2.
>> We
>> >>> now kindly request the Incubator PMC members to review and vote on
>> this
>> >>> incubator release. If the vote is successful we will rename release
>> >>> candidate 2 to final.
>> >>>
>> >>> Airflow is a platform to programmatically author, schedule and monitor
>> >>> workflows. Use airflow to author workflows as directed acyclic graphs
>> >>> (DAGs) of tasks. The airflow scheduler executes your tasks on an
>> array of
>> >>> workers while following the specified dependencies. Rich command line
>> >>> utilities make performing complex surgeries on DAGs a snap. The rich
>> user
>> >>> interface makes it easy to visualize pipelines running in production,
>> >>> monitor progress, and troubleshoot issues when needed. When workflows
>> are
>> >>> defined as code,

Re: [RESULT] [VOTE] Release Apache Airflow 1.8.1 (incubating)

[Chris Riccomini]

> Were there mentor +1 votes on the dev@ vote thread that can be carried
over?

Yes, Jakob Homan's vote can be carried over, I believe.

On Tue, May 9, 2017 at 10:18 AM, P. Taylor Goetz <ptgo...@gmail.com> wrote:

> Hi Chris,
>
> Unless I’m missing something, you’re one vote shy of the requirement of
> having 3 IPMC +1 votes [1].
>
> Were there mentor +1 votes on the dev@ vote thread that can be carried
> over?
>
> -Taylor
>
> [1] http://incubator.apache.org/incubation/Incubation_Policy.html#Releases
>
>
>
> > On May 9, 2017, at 12:52 PM, Chris Riccomini <criccom...@apache.org>
> wrote:
> >
> > Hey all,
> >
> > The vote has passed after 72+ hours.
> >
> > There were two +1 votes:
> >
> >  Josh Elser, John D. Ament
> >
> > No 0/-1 votes. I'm going to wrap up the release.
> >
> > @John, thanks for the feedback. We'll take this into account on the next
> > release. Some items have actually already been fixed, just not
> > cherry-picked into the 1.8.1 branch.
> >
> > Cheers,
> > Chris
> >
> > On Sun, May 7, 2017 at 2:08 PM, John D. Ament <johndam...@apache.org>
> wrote:
> >
> >> +1, however there's a few issues with the LICENSE file:
> >>
> >> - Would be good to list out the locations of each file (or path to a
> group
> >> of files) (some have this, and others do not so its hard to follow)
> >> - There's errant /* .. */ around each license declaration, which should
> be
> >> removed.
> >> - Missing license bodies for FooTable v2, jQuery Clock Plugin,
> >>
> >> Likewise, your NOTICE has copyright 2011-2017, however Airflow hasn't
> been
> >> incubating that long.  If you like, you can give origination notices to
> the
> >> original creators here to specify the original copyright dates.
> >>
> >> I would challenge the podling to see if there's a way to simplify their
> >> LICENSE by instead using npm or some other javascript packaging tool to
> >> build a distribution, rather than shipping the dependencies in the
> source
> >> release, makes it much easier to use.
> >>
> >> As the podling matures, would be good to see information about the
> author
> >> switch from an individual to a community (in setup.cfg, its already in
> >> setup.py so may have been a miss)
> >>
> >> It would be great to see a binary distribution in the next vote to see
> how
> >> that may work, its not clear how to build it from this.  Likewise, don't
> >> hesitate to clean up your old release artifacts, I downloaded the wrong
> >> artifact at first.
> >>
> >> John
> >>
> >> On Fri, May 5, 2017 at 4:44 PM Chris Riccomini <criccom...@apache.org>
> >> wrote:
> >>
> >>> Hello Incubator PMC’ers,
> >>>
> >>> The Apache Airflow community has voted and approved the proposal to
> >> release
> >>> Apache Airflow 1.8.1 (incubating) based on 1.8.1 Release Candidate 2.
> We
> >>> now kindly request the Incubator PMC members to review and vote on this
> >>> incubator release. If the vote is successful we will rename release
> >>> candidate 2 to final.
> >>>
> >>> Airflow is a platform to programmatically author, schedule and monitor
> >>> workflows. Use airflow to author workflows as directed acyclic graphs
> >>> (DAGs) of tasks. The airflow scheduler executes your tasks on an array
> of
> >>> workers while following the specified dependencies. Rich command line
> >>> utilities make performing complex surgeries on DAGs a snap. The rich
> user
> >>> interface makes it easy to visualize pipelines running in production,
> >>> monitor progress, and troubleshoot issues when needed. When workflows
> are
> >>> defined as code, they become more maintainable, versionable, testable,
> >> and
> >>> collaborative.
> >>>
> >>> The Apache Airflow-1.8.1-incubating release candidate is now available
> >> with
> >>> the following artifacts for a project vote:
> >>>
> >>> * [VOTE] Thread:*
> >>>
> >>> https://mail-archives.apache.org/mod_mbox/incubator-
> >> airflow-dev/201705.mbox/%3CCABYbY7eZk2SYHs3ni31h94uwwRd
> >> cn4dkJwumYtUMsfFLq-gwjg%40mail.gmail.com%3E
> >>>
> >>> *[RESULT] [VOTE] Thread:*
> >>>
> >>> https://mail-archives.apache.org/mod_mbox/incubator-
> >> airflow-dev/201

Re: [VOTE] Release Apache Airflow 1.8.1 (incubating)

[Chris Riccomini]

Jakob Homan already +1'd the release as part of the Airflow community vote.
Does that count towards the IPMC vote, or does he have to +1 it again?

On Tue, May 9, 2017 at 10:03 AM, Chris Riccomini <criccom...@apache.org>
wrote:

> Argh, forgot that we need three votes for incubator release. Can someone
> else please +1 this?
>
> On Sun, May 7, 2017 at 2:08 PM, John D. Ament <johndam...@apache.org>
> wrote:
>
>> +1, however there's a few issues with the LICENSE file:
>>
>> - Would be good to list out the locations of each file (or path to a group
>> of files) (some have this, and others do not so its hard to follow)
>> - There's errant /* .. */ around each license declaration, which should be
>> removed.
>> - Missing license bodies for FooTable v2, jQuery Clock Plugin,
>>
>> Likewise, your NOTICE has copyright 2011-2017, however Airflow hasn't been
>> incubating that long.  If you like, you can give origination notices to
>> the
>> original creators here to specify the original copyright dates.
>>
>> I would challenge the podling to see if there's a way to simplify their
>> LICENSE by instead using npm or some other javascript packaging tool to
>> build a distribution, rather than shipping the dependencies in the source
>> release, makes it much easier to use.
>>
>> As the podling matures, would be good to see information about the author
>> switch from an individual to a community (in setup.cfg, its already in
>> setup.py so may have been a miss)
>>
>> It would be great to see a binary distribution in the next vote to see how
>> that may work, its not clear how to build it from this.  Likewise, don't
>> hesitate to clean up your old release artifacts, I downloaded the wrong
>> artifact at first.
>>
>> John
>>
>> On Fri, May 5, 2017 at 4:44 PM Chris Riccomini <criccom...@apache.org>
>> wrote:
>>
>> > Hello Incubator PMC’ers,
>> >
>> > The Apache Airflow community has voted and approved the proposal to
>> release
>> > Apache Airflow 1.8.1 (incubating) based on 1.8.1 Release Candidate 2. We
>> > now kindly request the Incubator PMC members to review and vote on this
>> > incubator release. If the vote is successful we will rename release
>> > candidate 2 to final.
>> >
>> > Airflow is a platform to programmatically author, schedule and monitor
>> > workflows. Use airflow to author workflows as directed acyclic graphs
>> > (DAGs) of tasks. The airflow scheduler executes your tasks on an array
>> of
>> > workers while following the specified dependencies. Rich command line
>> > utilities make performing complex surgeries on DAGs a snap. The rich
>> user
>> > interface makes it easy to visualize pipelines running in production,
>> > monitor progress, and troubleshoot issues when needed. When workflows
>> are
>> > defined as code, they become more maintainable, versionable, testable,
>> and
>> > collaborative.
>> >
>> > The Apache Airflow-1.8.1-incubating release candidate is now available
>> with
>> > the following artifacts for a project vote:
>> >
>> > * [VOTE] Thread:*
>> >
>> > https://mail-archives.apache.org/mod_mbox/incubator-airflow-
>> dev/201705.mbox/%3CCABYbY7eZk2SYHs3ni31h94uwwRdcn4dkJwumYtUM
>> sfFLq-gwjg%40mail.gmail.com%3E
>> >
>> > *[RESULT] [VOTE] Thread:*
>> >
>> > https://mail-archives.apache.org/mod_mbox/incubator-airflow-
>> dev/201705.mbox/%3CCABYbY7dBAnjcvb-w_4Hdy3ZUM3CGn6Y3eUvZctuF
>> CfdAfPGAXQ%40mail.gmail.com%3E
>> >
>> > *The release candidate(s) to be voted on is available at:*
>> > https://dist.apache.org/repos/dist/dev/incubator/airflow/ or
>> >
>> > https://dist.apache.org/repos/dist/dev/incubator/airflow/apa
>> che-airflow-1.8.1rc2+incubating.tar.gz
>> >
>> > *Git branch*
>> > https://github.com/apache/incubator-airflow/tree/v1-8-test or
>> >
>> > https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.
>> git;a=tree;h=refs/heads/v1-8-stable;hb=refs/heads/v1-8-test
>> >
>> > *Git tag*
>> >
>> > https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.
>> git;a=shortlog;h=af2d0b4b5cb1ef30a065b1af66f90a01a953e2be
>> >
>> > *PGP signature*
>> >
>> > https://dist.apache.org/repos/dist/dev/incubator/airflow/apa
>> che-airflow-1.8.1rc2+incubating.tar.gz.asc
>> >
>> > *MD5/SHA Hashes*
>> >
>> > https://dist.apache.org/repos/dist

Re: [VOTE] Release Apache Airflow 1.8.1 (incubating)

[Chris Riccomini]

Argh, forgot that we need three votes for incubator release. Can someone
else please +1 this?

On Sun, May 7, 2017 at 2:08 PM, John D. Ament <johndam...@apache.org> wrote:

> +1, however there's a few issues with the LICENSE file:
>
> - Would be good to list out the locations of each file (or path to a group
> of files) (some have this, and others do not so its hard to follow)
> - There's errant /* .. */ around each license declaration, which should be
> removed.
> - Missing license bodies for FooTable v2, jQuery Clock Plugin,
>
> Likewise, your NOTICE has copyright 2011-2017, however Airflow hasn't been
> incubating that long.  If you like, you can give origination notices to the
> original creators here to specify the original copyright dates.
>
> I would challenge the podling to see if there's a way to simplify their
> LICENSE by instead using npm or some other javascript packaging tool to
> build a distribution, rather than shipping the dependencies in the source
> release, makes it much easier to use.
>
> As the podling matures, would be good to see information about the author
> switch from an individual to a community (in setup.cfg, its already in
> setup.py so may have been a miss)
>
> It would be great to see a binary distribution in the next vote to see how
> that may work, its not clear how to build it from this.  Likewise, don't
> hesitate to clean up your old release artifacts, I downloaded the wrong
> artifact at first.
>
> John
>
> On Fri, May 5, 2017 at 4:44 PM Chris Riccomini <criccom...@apache.org>
> wrote:
>
> > Hello Incubator PMC’ers,
> >
> > The Apache Airflow community has voted and approved the proposal to
> release
> > Apache Airflow 1.8.1 (incubating) based on 1.8.1 Release Candidate 2. We
> > now kindly request the Incubator PMC members to review and vote on this
> > incubator release. If the vote is successful we will rename release
> > candidate 2 to final.
> >
> > Airflow is a platform to programmatically author, schedule and monitor
> > workflows. Use airflow to author workflows as directed acyclic graphs
> > (DAGs) of tasks. The airflow scheduler executes your tasks on an array of
> > workers while following the specified dependencies. Rich command line
> > utilities make performing complex surgeries on DAGs a snap. The rich user
> > interface makes it easy to visualize pipelines running in production,
> > monitor progress, and troubleshoot issues when needed. When workflows are
> > defined as code, they become more maintainable, versionable, testable,
> and
> > collaborative.
> >
> > The Apache Airflow-1.8.1-incubating release candidate is now available
> with
> > the following artifacts for a project vote:
> >
> > * [VOTE] Thread:*
> >
> > https://mail-archives.apache.org/mod_mbox/incubator-
> airflow-dev/201705.mbox/%3CCABYbY7eZk2SYHs3ni31h94uwwRd
> cn4dkJwumYtUMsfFLq-gwjg%40mail.gmail.com%3E
> >
> > *[RESULT] [VOTE] Thread:*
> >
> > https://mail-archives.apache.org/mod_mbox/incubator-
> airflow-dev/201705.mbox/%3CCABYbY7dBAnjcvb-w_
> 4Hdy3ZUM3CGn6Y3eUvZctuFCfdAfPGAXQ%40mail.gmail.com%3E
> >
> > *The release candidate(s) to be voted on is available at:*
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/ or
> >
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/
> apache-airflow-1.8.1rc2+incubating.tar.gz
> >
> > *Git branch*
> > https://github.com/apache/incubator-airflow/tree/v1-8-test or
> >
> > https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.
> git;a=tree;h=refs/heads/v1-8-stable;hb=refs/heads/v1-8-test
> >
> > *Git tag*
> >
> > https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.
> git;a=shortlog;h=af2d0b4b5cb1ef30a065b1af66f90a01a953e2be
> >
> > *PGP signature*
> >
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/
> apache-airflow-1.8.1rc2+incubating.tar.gz.asc
> >
> > *MD5/SHA Hashes*
> >
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/
> apache-airflow-1.8.1rc2+incubating.tar.gz.md5
> >
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/
> apache-airflow-1.8.1rc2+incubating.tar.gz.sha
> >
> > *Keys to verify the signature of the release artifacts are available at:*
> > https://dist.apache.org/repos/dist/release/incubator/airflow/
> >
> > * RAT License checks*
> >
> > RAT is executed as part of the CI process (e.g.
> > https://travis-ci.org/apache/incubator-airflow/builds/226568140) but can
> > also be run manually by issuing “sh scripts/ci/check-license.sh” from the
> > t

[RESULT] [VOTE] Release Apache Airflow 1.8.1 (incubating)

[Chris Riccomini]

Hey all,

The vote has passed after 72+ hours.

There were two +1 votes:

  Josh Elser, John D. Ament

No 0/-1 votes. I'm going to wrap up the release.

@John, thanks for the feedback. We'll take this into account on the next
release. Some items have actually already been fixed, just not
cherry-picked into the 1.8.1 branch.

Cheers,
Chris

On Sun, May 7, 2017 at 2:08 PM, John D. Ament <johndam...@apache.org> wrote:

> +1, however there's a few issues with the LICENSE file:
>
> - Would be good to list out the locations of each file (or path to a group
> of files) (some have this, and others do not so its hard to follow)
> - There's errant /* .. */ around each license declaration, which should be
> removed.
> - Missing license bodies for FooTable v2, jQuery Clock Plugin,
>
> Likewise, your NOTICE has copyright 2011-2017, however Airflow hasn't been
> incubating that long.  If you like, you can give origination notices to the
> original creators here to specify the original copyright dates.
>
> I would challenge the podling to see if there's a way to simplify their
> LICENSE by instead using npm or some other javascript packaging tool to
> build a distribution, rather than shipping the dependencies in the source
> release, makes it much easier to use.
>
> As the podling matures, would be good to see information about the author
> switch from an individual to a community (in setup.cfg, its already in
> setup.py so may have been a miss)
>
> It would be great to see a binary distribution in the next vote to see how
> that may work, its not clear how to build it from this.  Likewise, don't
> hesitate to clean up your old release artifacts, I downloaded the wrong
> artifact at first.
>
> John
>
> On Fri, May 5, 2017 at 4:44 PM Chris Riccomini <criccom...@apache.org>
> wrote:
>
> > Hello Incubator PMC’ers,
> >
> > The Apache Airflow community has voted and approved the proposal to
> release
> > Apache Airflow 1.8.1 (incubating) based on 1.8.1 Release Candidate 2. We
> > now kindly request the Incubator PMC members to review and vote on this
> > incubator release. If the vote is successful we will rename release
> > candidate 2 to final.
> >
> > Airflow is a platform to programmatically author, schedule and monitor
> > workflows. Use airflow to author workflows as directed acyclic graphs
> > (DAGs) of tasks. The airflow scheduler executes your tasks on an array of
> > workers while following the specified dependencies. Rich command line
> > utilities make performing complex surgeries on DAGs a snap. The rich user
> > interface makes it easy to visualize pipelines running in production,
> > monitor progress, and troubleshoot issues when needed. When workflows are
> > defined as code, they become more maintainable, versionable, testable,
> and
> > collaborative.
> >
> > The Apache Airflow-1.8.1-incubating release candidate is now available
> with
> > the following artifacts for a project vote:
> >
> > * [VOTE] Thread:*
> >
> > https://mail-archives.apache.org/mod_mbox/incubator-
> airflow-dev/201705.mbox/%3CCABYbY7eZk2SYHs3ni31h94uwwRd
> cn4dkJwumYtUMsfFLq-gwjg%40mail.gmail.com%3E
> >
> > *[RESULT] [VOTE] Thread:*
> >
> > https://mail-archives.apache.org/mod_mbox/incubator-
> airflow-dev/201705.mbox/%3CCABYbY7dBAnjcvb-w_
> 4Hdy3ZUM3CGn6Y3eUvZctuFCfdAfPGAXQ%40mail.gmail.com%3E
> >
> > *The release candidate(s) to be voted on is available at:*
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/ or
> >
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/
> apache-airflow-1.8.1rc2+incubating.tar.gz
> >
> > *Git branch*
> > https://github.com/apache/incubator-airflow/tree/v1-8-test or
> >
> > https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.
> git;a=tree;h=refs/heads/v1-8-stable;hb=refs/heads/v1-8-test
> >
> > *Git tag*
> >
> > https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.
> git;a=shortlog;h=af2d0b4b5cb1ef30a065b1af66f90a01a953e2be
> >
> > *PGP signature*
> >
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/
> apache-airflow-1.8.1rc2+incubating.tar.gz.asc
> >
> > *MD5/SHA Hashes*
> >
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/
> apache-airflow-1.8.1rc2+incubating.tar.gz.md5
> >
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/
> apache-airflow-1.8.1rc2+incubating.tar.gz.sha
> >
> > *Keys to verify the signature of the release artifacts are available at:*
> > https://dist.apache.org/repos/dist/release/incubator/airflow/
> >
> > * RAT License checks*
&g

Re: [VOTE] Release Apache Airflow 1.8.1 (incubating)

[Chris Riccomini]

Sigh, that was meant to be signed:

Cheers,
Chris

On Fri, May 5, 2017 at 1:44 PM, Chris Riccomini <criccom...@apache.org>
wrote:

> Hello Incubator PMC’ers,
>
> The Apache Airflow community has voted and approved the proposal to
> release Apache Airflow 1.8.1 (incubating) based on 1.8.1 Release Candidate
> 2. We now kindly request the Incubator PMC members to review and vote on
> this incubator release. If the vote is successful we will rename release
> candidate 2 to final.
>
> Airflow is a platform to programmatically author, schedule and monitor
> workflows. Use airflow to author workflows as directed acyclic graphs
> (DAGs) of tasks. The airflow scheduler executes your tasks on an array of
> workers while following the specified dependencies. Rich command line
> utilities make performing complex surgeries on DAGs a snap. The rich user
> interface makes it easy to visualize pipelines running in production,
> monitor progress, and troubleshoot issues when needed. When workflows are
> defined as code, they become more maintainable, versionable, testable, and
> collaborative.
>
> The Apache Airflow-1.8.1-incubating release candidate is now available
> with the following artifacts for a project vote:
>
> * [VOTE] Thread:*
> https://mail-archives.apache.org/mod_mbox/incubator-
> airflow-dev/201705.mbox/%3CCABYbY7eZk2SYHs3ni31h94uwwRd
> cn4dkJwumYtUMsfFLq-gwjg%40mail.gmail.com%3E
>
> *[RESULT] [VOTE] Thread:*
> https://mail-archives.apache.org/mod_mbox/incubator-
> airflow-dev/201705.mbox/%3CCABYbY7dBAnjcvb-w_
> 4Hdy3ZUM3CGn6Y3eUvZctuFCfdAfPGAXQ%40mail.gmail.com%3E
>
> *The release candidate(s) to be voted on is available at:*
> https://dist.apache.org/repos/dist/dev/incubator/airflow/ or
> https://dist.apache.org/repos/dist/dev/incubator/airflow/
> apache-airflow-1.8.1rc2+incubating.tar.gz
>
> *Git branch*
> https://github.com/apache/incubator-airflow/tree/v1-8-test or
> https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.
> git;a=tree;h=refs/heads/v1-8-stable;hb=refs/heads/v1-8-test
>
> *Git tag*
> https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.
> git;a=shortlog;h=af2d0b4b5cb1ef30a065b1af66f90a01a953e2be
>
> *PGP signature*
> https://dist.apache.org/repos/dist/dev/incubator/airflow/
> apache-airflow-1.8.1rc2+incubating.tar.gz.asc
>
> *MD5/SHA Hashes*
> https://dist.apache.org/repos/dist/dev/incubator/airflow/
> apache-airflow-1.8.1rc2+incubating.tar.gz.md5
> https://dist.apache.org/repos/dist/dev/incubator/airflow/
> apache-airflow-1.8.1rc2+incubating.tar.gz.sha
>
> *Keys to verify the signature of the release artifacts are available at:*
> https://dist.apache.org/repos/dist/release/incubator/airflow/
>
> * RAT License checks*
>
> RAT is executed as part of the CI process (e.g.
> https://travis-ci.org/apache/incubator-airflow/builds/226568140) but can
> also be run manually by issuing “sh scripts/ci/check-license.sh” from the
> top level.
>
> Source code is always included, i.e. there is no binary release.
> Compilation and installation will happen by standard Python practices, e.g.
> pip install <> or python setup.py install.
>
> The vote will be open for at least 72 hours or until necessary number of
> votes are reached.
>
> Members please be sure to indicate "(Binding)" with your vote which will
> help in tallying the vote(s).
>
> [ ] +1  approve
> [ ] +0  no opinion
> [ ] -1  disapprove (and reason why)
>
> *Here is my +1 (non-binding)*
>
> Cheers,
> Bolke
>
>

[VOTE] Release Apache Airflow 1.8.1 (incubating)

[Chris Riccomini]

Hello Incubator PMC’ers,

The Apache Airflow community has voted and approved the proposal to release
Apache Airflow 1.8.1 (incubating) based on 1.8.1 Release Candidate 2. We
now kindly request the Incubator PMC members to review and vote on this
incubator release. If the vote is successful we will rename release
candidate 2 to final.

Airflow is a platform to programmatically author, schedule and monitor
workflows. Use airflow to author workflows as directed acyclic graphs
(DAGs) of tasks. The airflow scheduler executes your tasks on an array of
workers while following the specified dependencies. Rich command line
utilities make performing complex surgeries on DAGs a snap. The rich user
interface makes it easy to visualize pipelines running in production,
monitor progress, and troubleshoot issues when needed. When workflows are
defined as code, they become more maintainable, versionable, testable, and
collaborative.

The Apache Airflow-1.8.1-incubating release candidate is now available with
the following artifacts for a project vote:

* [VOTE] Thread:*
https://mail-archives.apache.org/mod_mbox/incubator-airflow-dev/201705.mbox/%3CCABYbY7eZk2SYHs3ni31h94uwwRdcn4dkJwumYtUMsfFLq-gwjg%40mail.gmail.com%3E

*[RESULT] [VOTE] Thread:*
https://mail-archives.apache.org/mod_mbox/incubator-airflow-dev/201705.mbox/%3CCABYbY7dBAnjcvb-w_4Hdy3ZUM3CGn6Y3eUvZctuFCfdAfPGAXQ%40mail.gmail.com%3E

*The release candidate(s) to be voted on is available at:*
https://dist.apache.org/repos/dist/dev/incubator/airflow/ or
https://dist.apache.org/repos/dist/dev/incubator/airflow/apache-airflow-1.8.1rc2+incubating.tar.gz

*Git branch*
https://github.com/apache/incubator-airflow/tree/v1-8-test or
https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;a=tree;h=refs/heads/v1-8-stable;hb=refs/heads/v1-8-test

*Git tag*
https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;a=shortlog;h=af2d0b4b5cb1ef30a065b1af66f90a01a953e2be

*PGP signature*
https://dist.apache.org/repos/dist/dev/incubator/airflow/apache-airflow-1.8.1rc2+incubating.tar.gz.asc

*MD5/SHA Hashes*
https://dist.apache.org/repos/dist/dev/incubator/airflow/apache-airflow-1.8.1rc2+incubating.tar.gz.md5
https://dist.apache.org/repos/dist/dev/incubator/airflow/apache-airflow-1.8.1rc2+incubating.tar.gz.sha

*Keys to verify the signature of the release artifacts are available at:*
https://dist.apache.org/repos/dist/release/incubator/airflow/

* RAT License checks*

RAT is executed as part of the CI process (e.g.
https://travis-ci.org/apache/incubator-airflow/builds/226568140) but can
also be run manually by issuing “sh scripts/ci/check-license.sh” from the
top level.

Source code is always included, i.e. there is no binary release.
Compilation and installation will happen by standard Python practices, e.g.
pip install <> or python setup.py install.

The vote will be open for at least 72 hours or until necessary number of
votes are reached.

Members please be sure to indicate "(Binding)" with your vote which will
help in tallying the vote(s).

[ ] +1  approve
[ ] +0  no opinion
[ ] -1  disapprove (and reason why)

*Here is my +1 (non-binding)*

Cheers,
Bolke

Airflow voting on release artifacts

[Chris Riccomini]

Hey all,

We've had a question arise in the Airflow project. We're currently cutting
release candidates (RCs), and voting on those. These RCs contain an
artifact with the suffix -rc0, -rc1, etc. The problem with this is that the
final RC that passes the vote still has an -rcX in its version number. To
get around this, the release manager has just been rebuilding the .asc,
.sha, etc, and publishing this new artifact as the final release.

Hitesh recently raised the issue that the artifact that passes the vote
MUST be the one that we actually release. The problem with this is that it
would force us to publish multiple RCs with the exact same version number.
This is a terrible experience from a developer and user point of view.
Airflow is a Python project, and many users pip install various versions of
Airflow to test things out. Having multiple RCs with the same version
number is going to make it really difficult to keep things straight, both
during the VOTE, and afterwards.

Based on a cursory look, it seems that other projects handle this problem
in one of three ways:

* Double vote (the final RC gets voted AGAIN, this time, with proper naming)
* Rename/rebuild after final vote (This is what Airflow is doing, and Beam
does this, I believe)
* All RCs have the exact same version number

Most Java projects are just publishing multiple RCs with the exact same
version number. This seems bad from a general software engineering
perspective, since it's going to cause build cache issues (e.g. .mvn
already contains version 1.2.3, even though a new RC with the same version
should be installed). Beam does seem to be using -SNAPSHOT releases in Java
for RCs, which is CLOSER to what we've been doing in Airflow (-rc0, -rc1,
etc), and they seem to be renaming after the release, just like we were.

What's the guidance here? We seem to have two requirements that are at odds
with eachtother:

1) The RC artifact that we vote on is the one that we release
2) We have real version numbers for RCs which are independent of one
another so build systems, dependency managements systems, and package
management systems can properly handle them.

Any help or thoughts would be appreciated.

Cheers,
Chris

Re: [VOTE] Release Apache Airflow 1.8.0 (incubating)

[Chris Riccomini]

+1 (non-binding)

On Thu, Mar 16, 2017 at 9:32 AM, Bolke de Bruin  wrote:

> Hello Incubator PMC’ers,
>
> The Apache Airflow community has voted and approved the proposal to
> release Apache Airflow 1.8.0 (incubating) based on 1.8.0 Release Candidate
> 5. We now kindly request the Incubator PMC members to review and vote on
> this incubator release. If the vote is successful we will rename release
> candidate 4 to final.
>
> Airflow is a platform to programmatically author, schedule and monitor
> workflows. Use airflow to author workflows as directed acyclic graphs
> (DAGs) of tasks. The airflow scheduler executes your tasks on an array of
> workers while following the specified dependencies. Rich command line
> utilities make performing complex surgeries on DAGs a snap. The rich user
> interface makes it easy to visualize pipelines running in production,
> monitor progress, and troubleshoot issues when needed. When workflows are
> defined as code, they become more maintainable, versionable, testable, and
> collaborative.
>
> The Apache Airflow-1.8.0-incubating release candidate is now available
> with the following artefacts for a project vote:
>
> * [VOTE] Thread:*
> http://mail-archives.apache.org/mod_mbox/incubator-
> airflow-dev/201703.mbox/%3cB1833A3A-05FB-4112-B395-
> 135caf930...@gmail.com%3e  org/mod_mbox/incubator-airflow-dev/201703.mbox/%3CB1833A3A-05FB-4112-B395-
> 135caf930...@gmail.com%3E>
>
> *[RESULT][VOTE] Thread:*
> http://mail-archives.apache.org/mod_mbox/incubator-
> airflow-dev/201703.mbox/%3c59BC8C2B-12E2-4DE3-9555-
> b2273660a...@gmail.com%3e  org/mod_mbox/incubator-airflow-dev/201703.mbox/%3c59BC8C2B-12E2-4DE3-9555-
> b2273660a...@gmail.com%3e>
>
> *The release candidate(s) to be voted on is available at:*
> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/> or
> https://dist.apache.org/repos/dist/dev/incubator/airflow/
> airflow-1.8.0rc5+apache.incubating.tar.gz  repos/dist/dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz
> >
>
> *Git branch*
> https://github.com/apache/incubator-airflow/tree/v1-8-stable <
> https://github.com/apache/incubator-airflow/tree/v1-8-stable> or
> https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.
> git;a=tree;h=refs/heads/v1-8-stable;hb=refs/heads/v1-8-stable <
> https://git-wip-us.apache.org/repos/asf?p=incubator-
> airflow.git;a=tree;h=refs/heads/v1-8-stable;hb=refs/heads/v1-8-stable>
>
> *Git tag*
> https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.
> git;a=shortlog;h=f4760c320a29be62469799355e76efa42d0b6bb2 <
> https://git-wip-us.apache.org/repos/asf?p=incubator-
> airflow.git;a=shortlog;h=f4760c320a29be62469799355e76efa42d0b6bb2>
>
> PGP signature
> https://dist.apache.org/repos/dist/dev/incubator/airflow/
> airflow-1.8.0rc5+apache.incubating.tar.gz.asc  repos/dist/dev/incubator/airflow/airflow-1.8.0rc5+
> apache.incubating.tar.gz.asc>
>
> MD5/SHA Hashes:
> https://dist.apache.org/repos/dist/dev/incubator/airflow/
> airflow-1.8.0rc5+apache.incubating.tar.gz.md5  repos/dist/dev/incubator/airflow/airflow-1.8.0rc5+
> apache.incubating.tar.gz.md5>
> https://dist.apache.org/repos/dist/dev/incubator/airflow/
> airflow-1.8.0rc5+apache.incubating.tar.gz.sha  repos/dist/dev/incubator/airflow/airflow-1.8.0rc5+
> apache.incubating.tar.gz.sha>
>
> *Keys to verify the signature of the release artifacts are available at:*
> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>
> * RAT License checks*
>
> RAT is executed as part of the CI process (e.g.
> https://travis-ci.org/apache/incubator-airflow/builds/203106493 <
> https://travis-ci.org/apache/incubator-airflow/builds/203106493>) but can
> also be run manually by issuing “sh scripts/ci/check-license.sh” from the
> top level.
>
> Source code is always included, i.e. there is no binary release.
> Compilation and installation will happen by standard Python practices, e.g.
> pip install <> or python setup.py install.
>
> The vote will be open for at least 72 hours or until necessary number of
> votes are reached.
>
> Members please be sure to indicate "(Binding)" with your vote which will
> help in tallying the vote(s).
>
> [ ] +1  approve
>
> [ ] +0  no opinion
>
> [ ] -1  disapprove (and reason why)
>
>
> *Here is my +1 (non-binding)*
>
> Cheers,
> Bolke
>

Re: [VOTE] Release Apache Airflow 1.8.0 (incubating)

[Chris Riccomini]

+1 non-binding

On Mon, Feb 20, 2017 at 11:50 PM, Bolke de Bruin  wrote:

> Hello Incubator PMC’ers,
>
> The Apache Airflow community has voted and approved the proposal to
> release Apache Airflow 1.8.0 (incubating) based on 1.8.0 Release Candidate
> 4. We now kindly request the Incubator PMC members to review and vote on
> this incubator release. If the vote is successful we will rename release
> candidate 4 to final.
>
> Airflow is a platform to programmatically author, schedule and monitor
> workflows. Use airflow to author workflows as directed acyclic graphs
> (DAGs) of tasks. The airflow scheduler executes your tasks on an array of
> workers while following the specified dependencies. Rich command line
> utilities make performing complex surgeries on DAGs a snap. The rich user
> interface makes it easy to visualize pipelines running in production,
> monitor progress, and troubleshoot issues when needed. When workflows are
> defined as code, they become more maintainable, versionable, testable, and
> collaborative.
>
> The Apache Airflow-1.8.0-incubating release candidate is now available
> with the following artefacts for a project vote:
>
> * [VOTE] Thread:*
> http://mail-archives.apache.org/mod_mbox/incubator-
> airflow-dev/201702.mbox/%3cD360D9BE-C358-42A1-9188-
> 6c92c31a2...@gmail.com%3e  org/mod_mbox/incubator-airflow-dev/201702.mbox/%3CD360D9BE-C358-42A1-9188-
> 6c92c31a2...@gmail.com%3E>
>
> *[RESULT][VOTE] Thread:*
> http://mail-archives.apache.org/mod_mbox/incubator-
> airflow-dev/201702.mbox/%3c0C96110D-2074-468A-BCAC-
> a7feee331...@gmail.com%3e  org/mod_mbox/incubator-airflow-dev/201702.mbox/%3C0C96110D-2074-468A-BCAC-
> a7feee331...@gmail.com%3E>
>
> *The release candidate(s) to be voted on is available at:*
> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/> or
> https://dist.apache.org/repos/dist/dev/incubator/airflow/
> airflow-1.8.0rc4+apache.incubating.tar.gz  repos/dist/dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz
> >
>
> *Git branch*
> https://github.com/apache/incubator-airflow/tree/v1-8-stable <
> https://github.com/apache/incubator-airflow/tree/v1-8-stable> or
> https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.
> git;a=tree;h=refs/heads/v1-8-stable;hb=refs/heads/v1-8-stable <
> https://git-wip-us.apache.org/repos/asf?p=incubator-
> airflow.git;a=tree;h=refs/heads/v1-8-stable;hb=refs/heads/v1-8-stable>
>
> PGP signature
> https://dist.apache.org/repos/dist/dev/incubator/airflow/
> airflow-1.8.0rc4+apache.incubating.tar.gz.asc  repos/dist/dev/incubator/airflow/airflow-1.8.0rc4+
> apache.incubating.tar.gz.asc>
>
> MD5/SHA Hashes:
> https://dist.apache.org/repos/dist/dev/incubator/airflow/
> airflow-1.8.0rc4+apache.incubating.tar.gz.md5  repos/dist/dev/incubator/airflow/airflow-1.8.0rc4+
> apache.incubating.tar.gz.md5>
> https://dist.apache.org/repos/dist/dev/incubator/airflow/
> airflow-1.8.0rc4+apache.incubating.tar.gz.sha  repos/dist/dev/incubator/airflow/airflow-1.8.0rc4+
> apache.incubating.tar.gz.sha>
>
> *Keys to verify the signature of the release artifacts are available at:*
> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>
> * RAT License checks*
>
> RAT is executed as part of the CI process (e.g.
> https://travis-ci.org/apache/incubator-airflow/builds/203106493 <
> https://travis-ci.org/apache/incubator-airflow/builds/203106493>) but can
> also be run manually by issuing “sh scripts/ci/check-license.sh” from the
> top level.
>
> Source code is always included, i.e. there is no binary release.
> Compilation and installation will happen by standard Python practices, e.g.
> pip install <> or python setup.py install.
>
> The vote will be open for at least 72 hours or until necessary number of
> votes are reached.
>
> Members please be sure to indicate "(Binding)" with your vote which will
> help in tallying the vote(s).
>
> [ ] +1  approve
>
> [ ] +0  no opinion
>
> [ ] -1  disapprove (and reason why)
>
>
> *Here is my +1 (non-binding)*
>
> Cheers,
> Bolke
>
>
>
>
>

[jira] [Commented] (INCUBATOR-131) airflow installation issue

[Chris Riccomini (JIRA)]

[ 
https://issues.apache.org/jira/browse/INCUBATOR-131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477293#comment-15477293
 ] 

Chris Riccomini commented on INCUBATOR-131:
---

You've opened an INCUBATOR JIRA. You need to open an AIRFLOW JIRA. 
https://issues.apache.org/jira/browse/AIRFLOW

> airflow installation issue
> --
>
> Key: INCUBATOR-131
> URL: https://issues.apache.org/jira/browse/INCUBATOR-131
> Project: Incubator
>  Issue Type: Bug
>Reporter: Afroz Hussain
>
> Before I tell you about my issue, let me describe my Airflow environment:
> Python Version: Python 2.6.6(tried on 2.7)
> Operating System: RedHat 6.5(tried on RedHat 7.5)
> Python packages: argparse==1.2.1 aws-cfn-bootstrap==1.3 
> backports.ssl-match-hostname==3.4.0.2 boto==2.34.0 cas==0.15 chardet==2.0.1 
> Cheetah==2.4.1 cloud-init==0.7.2 configobj==4.6.0 ethtool==0.6 
> freeipa==2.0.0a0 iniparse==0.3.1 iotop==0.3.2 ipapython==3.0.0 iwlib==1.0 
> kerberos==1.1 lockfile==0.9.1 lxml==2.2.3 M2Crypto==0.20.2 Markdown==2.0.1 
> matplotlib==0.99.1.1 netaddr==0.7.5 nose==0.10.4 numpy==1.4.1 
> ordereddict==1.2 paramiko==1.7.5 policycoreutils-default-encoding==0.1 
> prettytable==0.6.1 pycparser==2.14 pycrypto==2.0.1 pycurl==7.19.0 
> Pygments==1.1.1 pygpgme==0.1 pyOpenSSL==0.10 pystache==0.5.4 
> python-daemon==1.5.5 python-dateutil==1.4.1 python-default-encoding==0.1 
> python-dmidecode==3.10.13 python-ldap==2.3.10 python-nss==0.16.0 pytz===2010h 
> PyYAML==3.10 requests==1.1.0 rhnlib==2.5.22 rhsm==1.12.5 setools==1.0 
> simplejson==2.0.9 six==1.9.0 SSSDConfig==1.12.4 urlgrabber==3.9.1 
> urllib3==1.5 yum-metadata-parser==1.1.2
> I am unable to install airflow using "pip install airflow", getting errors as:
> RHEL 6
> Complete output from command python setup.py egg_info:
> Traceback (most recent call last):
> File "", line 1, in 
> File "/tmp/pip-build-bI9hiv/pandas/setup.py", line 631, in 
> **setuptools_kwargs)
> File "/usr/lib64/python2.6/distutils/core.py", line 113, in setup
> setup_distribution = dist = klass(attrs)
> File "/usr/lib/python2.6/site-packages/setuptools/dist.py", line 221, in 
> __init_
> self.fetch_build_eggs(attrs.pop('setup_requires'))
> File "/usr/lib/python2.6/site-packages/setuptools/dist.py", line 245, in 
> fetch_build_eggs
> parse_requirements(requires), installer=self.fetch_build_egg
> File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 550, in resolve
> raise VersionConflict(dist,req) # XXX put more info here
> pkg_resources.VersionConflict: (numpy 1.4.1 
> (/usr/lib64/python2.6/site-packages), Requirement.parse('numpy>=1.7.0'))
> 
> Command "python setup.py egg_info" failed with error code 1 in 
> /tmp/pip-build-bI9hiv/pandas
> RHEL 7:
> Complete output from command python setup.py egg_info:
> Running from numpy source directory.
> /tmp/easy_install-9Yjxgb/numpy-1.11.0/setup.py:327: UserWarning: Unrecognized 
> setuptools command, proceeding with generating Cython sources and expanding 
> templates
> warnings.warn("Unrecognized setuptools command, proceeding with "
> /tmp/easy_install-9Yjxgb/numpy-1.11.0/numpy/distutils/system_info.py:1640: 
> UserWarning:
> Atlas (http://math-atlas.sourceforge.net/) libraries not found.
> Directories to search for the libraries can be specified in the
> numpy/distutils/site.cfg file (section [atlas]) or by setting
> the ATLAS environment variable.
> warnings.warn(AtlasNotFoundError.doc)
> /tmp/easy_install-9Yjxgb/numpy-1.11.0/numpy/distutils/system_info.py:1649: 
> UserWarning:
> Blas (http://www.netlib.org/blas/) libraries not found.
> Directories to search for the libraries can be specified in the
> numpy/distutils/site.cfg file (section [blas]) or by setting
> the BLAS environment variable.
> warnings.warn(BlasNotFoundError.doc)
> /tmp/easy_install-9Yjxgb/numpy-1.11.0/numpy/distutils/system_info.py:1652: 
> UserWarning:
> Blas (http://www.netlib.org/blas/) sources not found.
> Directories to search for the sources can be specified in the
> numpy/distutils/site.cfg file (section [blas_src]) or by setting
> the BLAS_SRC environment variable.
> warnings.warn(BlasSrcNotFoundError.doc)
> /tmp/easy_install-9Yjxgb/numpy-1.11.0/numpy/distutils/system_info.py:1542: 
> UserWarning:
> Atlas (http://math-atlas.sourceforge.net/) libraries not found.
> Directories to search for the libraries can be specified in the
> numpy/distutils/site.cfg file (section [atlas]) or by setting
> the ATLAS environment variable.
> warnings.warn(A

Airflow podling name search

[Chris Riccomini]

Hey all,

Could someone please have a look at:

  https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-110

Has been open for 2 weeks with no activity.

Thanks!
Chris

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: In-flight release in Incubator

[Chris Riccomini]

> If I remember correctly, I believe non-ASF release artifacts have to be
hosted externally (i.e. not on ASF hardware).

Sounds good. They will be.

Thanks all!

On Fri, May 6, 2016 at 11:07 AM, P. Taylor Goetz <ptgo...@gmail.com> wrote:

> If I remember correctly, I believe non-ASF release artifacts have to be
> hosted externally (i.e. not on ASF hardware).
>
> When we had to do this for Storm, we added the following note to our
> download page:
>
> "NOTE: The following releases are neither a top-level Apache releases nor
> Apache Incubator releases. They are not endorsed by the Apache Software
> Foundation, nor hosted by Apache.”
>
> -Taylor
>
> > On May 6, 2016, at 12:45 PM, Ted Dunning <ted.dunn...@gmail.com> wrote:
> >
> > If you need a release before being ready for an Apache release, an
> > old-style non-Apache release is a fine thing to do.
> >
> > Also, keep in mind that the purpose of first Apache release is to iron
> out
> > process and IP issues. It should not necessarily intended as a full
> quality
> > release. So while an external release may be necessary for product and
> > quality reasons trumping Apache process, you should also push forward
> with
> > an Apache release with less than normal regard for product and quality.
> > That will mean that you will be ready when the project is ready to put
> out
> > a qualified release to also make it be an Apache release (incubating).
> >
> > The fact that quality/bugs/features and Apache IP checking are
> essentially
> > orthogonal issues is often not obvious to incoming projects.
> >
> >
> >
> > On Thu, May 5, 2016 at 6:22 PM, Shane Curcuru <a...@shanecurcuru.org>
> wrote:
> >
> >> Chris Riccomini wrote on 5/5/16 6:26 PM:
> >>> Hey all,
> >>>
> >>> Airflow was in the middle of a release prior to joining Apache
> Incubator,
> >>> and had several RCs out. The release has already been delayed by
> several
> >>> weeks due to bugs we've discovered. We don't want to tie this release
> to
> >>> the full-blown Apache release cycle, since we're not yet ready to tick
> >> off
> >>> all the boxes there. Is it possible for us to finish off our prior
> >> release
> >>> outside of Apache (i.e. tag it in Git, and put a package in PyPI)?
> >>>
> >>> Cheers,
> >>> Chris
> >>
> >> I don't see why not.  Just don't call it an Apache project or release -
> >> call it whatever you used to call it, using whatever release platform
> >> the project used in the past.
> >>
> >> You can certainly email the dev@ list noting the release, just clearly
> >> noting that it's a release outside of Apache.
> >>
> >> All the code is open source licensed, so people are free to use it as
> >> they wish, just not using "Apache" with it (yet).
> >>
> >> - Shane
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> >> For additional commands, e-mail: general-h...@incubator.apache.org
> >>
> >>
>
>

In-flight release in Incubator

[Chris Riccomini]

Hey all,

Airflow was in the middle of a release prior to joining Apache Incubator,
and had several RCs out. The release has already been delayed by several
weeks due to bugs we've discovered. We don't want to tie this release to
the full-blown Apache release cycle, since we're not yet ready to tick off
all the boxes there. Is it possible for us to finish off our prior release
outside of Apache (i.e. tag it in Git, and put a package in PyPI)?

Cheers,
Chris

[RESULT] [VOTE] Accept Airflow into the Apache Incubator

[Chris Riccomini]

Hi All,

Thanks to all who voted! The vote has passed with the following tallies:

+1 Binding (7 total)
Jacques Nadeau
Suresh Marru
Chris Nauroth
Jakob Homan
Seetharam Venkatesh
Hitesh Shah
Sergio Fernández

+1 Non-binding (9 total)
Jeremiah Lowin
Timothy Chen
Moon Soo Lee
Felix Cheung
Stian Soiland-Reyes
Arthur Wiedmer
Chris Riccomini
Maxime Beauchemin
Stian Soiland-Reyes

We'll work with the mentors to get the project bootstrapped.

Thanks!
Chris

Re: [VOTE] Accept Airflow into the Incubator

[Chris Riccomini]

+1 non-binding :)

On Thu, Mar 24, 2016 at 8:00 PM, Siddharth Anand <san...@agari.com.invalid>
wrote:

> Following the discussion earlier:
> https://s.apache.org/AirflowDiscussion
>
> I would like to call a VOTE for accepting Airflow as a new incubator
> project.
>
> The proposal is available at:
> https://wiki.apache.org/incubator/AirflowProposal
>
> The proposal is also included at the bottom of this email.
>
> Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT
> [ ] +1 accept Airflow into the Apache Incubator
> [ ] ±0
> [ ] -1 because...
>
> +1 (non-binding)
>
> Thanks,
> -s (Sid)
>
>
> == Abstract ==
>
> Airflow is a workflow automation and scheduling system that can be
> used to author and manage data pipelines.
>
> == Proposal ==
>
> Airflow provides a system for authoring and managing workflows a.k.a.
> data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer
> authors DAGs in Python using an Airflow-provided framework. He/She
> then executes the DAG using Airflow’s scheduler or registers the DAG
> for event-based execution. A web-based UI provides the developer with
> a range of options for managing and viewing his/her data pipelines.
> Background
>
> Airflow was developed at Airbnb to enable easier authorship and
> management of DAGs than were possible with existing solutions such as
> Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or
> more XML or property files to be bundled together to define a
> workflow. This separation of code and config can present a challenge
> to understanding the DAG - in Azkaban, a DAG’s structure is reflected
> by its file system tree and one can find himself/herself traversing
> the file system when inspecting or changing the structure of the DAG.
> Airflow workflows, on the other hand, are simply and elegantly defined
> in Python code, often a single file. Airflow merges the powerful
> Web-based management aspects of projects like Azkaban and Oozie with
> the simplicity and elegance of defining workflows in Python. Airflow,
> less than a year old in terms of its Open Source launch, is currently
> used in production environments in more than 30 companies and boasts
> an active contributor list of more than 100 developers, the vast
> majority of which (>95%) are outside of Airbnb.
>
> We would like to share it with the ASF and begin developing a
> community of developers and users within Apache.
>
> == Rationale ==
>
> Many organizations (>30) already benefit from running Airflow to
> manage data pipelines. Our 100+ contributors continue to provide
> integrations with 3rd party systems through the implementation of new
> hooks and operators, both of which are used in defining the tasks that
> compose workflows.
>
> == Current Status ==
>
> === Meritocracy ===
>
> Our intent with this incubator proposal is to start building a diverse
> developer community around Airflow following the Apache meritocracy
> model. Since Airflow was open-sourced in mid-2015, we have had fast
> adoption and contributions by multiple organizations the world over.
> We plan to continue to support new contributors and we will work to
> actively promote those who contribute significantly to the project to
> committers.
>
> === Community ===
>
> Airflow is currently being used in over 30 companies. We hope to
> extend our contributor base significantly and invite all those who are
> interested in building large-scale distributed systems to participate.
>
> === Core Developers ===
>
> Airflow is currently being developed by four engineers: Maxime
> Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini.
> Chris is a member of the Apache Samza PMC and a contributor to various
> Apache projects, including Apache Kafka and Apache YARN. Maxime,
> Siddharth, and Bolke have contributed to Airflow.
>
> === Alignment ===
> The ASF is the natural choice to host the Airflow project as its goal
> of encouraging community-driven open-source projects fits with our
> vision for Airflow.
>
> == Known Risks ==
>
> === Orphaned Products ===
>
> The core developers plan to work part time on the project. There is
> very little risk of Airflow being abandoned as all of our companies
> rely on it.
>
> === Inexperience with Open Source ===
>
> All of the core developers have experience with open source
> development. Chris is a member of the Apache Samza PMC and a
> contributor to various Apache projects, including Apache Kafka and
> Apache YARN. Bolke is contributor on multiple open source projects and
> a few Apache projects as well, including Apache Hive, Apache Hadoop,
> and Apache Ranger.
>
> === Homo

Re: [PROPOSAL] : Airflow

[Chris Riccomini]

@Jake, may I join the IPMC? :)

On Thu, Mar 17, 2016 at 9:06 AM, Jake Farrell <jfarr...@apache.org> wrote:

> Hey Chris
> As an officer you can be a champion if your project you chair (Samza) is
> sponsoring the incoming podling with the intent of merging the two
> communities - "Where the Champion is not a Member of the Foundation (i.e.
> is an Officer only), the Champion shall be a member of the PMC of the
> Sponsor.", did not think this was the case based on the proposal.
>
> Makes sense about the committers and developing the community during
> incubation, was just curious because there where some people with 20+
> commits not in the initial committers list. Was not looking for a change,
> more that there had been some thought behind this exclusion and how the
> project intended to start developing that community
>
> -Jake
>
> On Thu, Mar 17, 2016 at 11:58 AM, Chris Riccomini <criccom...@apache.org>
> wrote:
>
> > > The proposal has the incubator listed as the sponsor and a quick check
> > shows Chris Riccomini is not on the IPMC or a member currently, details
> on
> > roles are available at [1].
> >
> > I was under the impression that I could sponsor because I am an officer:
> >
> > "A candidate project shall be sponsored by an Officer
> > <http://www.apache.org/foundation/index.html> or Member
> > <http://www.apache.org/foundation/members.html> of the Foundation"
> >
> > The officer page lists me as a VP of Apache Samza. Perhaps I'm
> misreading?
> >
> > > Mailing lists are in the old format
> >
> > Fixed.
> >
> > > Github shows 109 contributors, but the proposal lists only 6 initial
> > committers, where any of the other existing contributors considered for
> the
> > proposal?
> >
> > In terms of the high ratio of contributors to committers (109:6), the
> > reason is as follows. Most contributions are to the fringes of the system
> > and are in the form of 3rd party contributed hooks/operators (for 3rd
> party
> > integrations). For example, if you want to use Airflow to speak to
> > Cassandra, you may want to contribute a Cassandra Hook and CRUD
> operators.
> > Airflow allows for this extensibility and allows for our users to adapt
> > Airflow to their needs. However, the committer list is limited to folks
> who
> > will touch the internals of the system (e.g. scheduler, metadata
> > management, etc...). Overtime, we would like other contributors to get
> > familiar with the internals to the point where they can also become
> > maintainers, but they don't currently have the expertise.
> >
> >
> > On Thu, Mar 17, 2016 at 8:16 AM, Scott Deboy <scott.de...@gmail.com>
> > wrote:
> >
> > > Ll
> > > On Mar 17, 2016 6:51 AM, "Jake Farrell" <jfarr...@apache.org> wrote:
> > >
> > > > Hi Siddharth
> > > > Thanks for drafting a proposal and looking to bring Airflow to the
> > Apache
> > > > Incubator. Overall the proposal looks good, just a couple comments.
> The
> > > > proposal has the incubator listed as the sponsor and a quick check
> > shows
> > > > Chris Riccomini is not on the IPMC or a member currently, details on
> > > roles
> > > > are available at [1]. Not a blocker as you have members listed in
> your
> > > > group of mentors and can ask that they step up as the champion.
> > > >
> > > > Mailing lists are in the old format, they should be
> > > > - priv...@airflow.incubator.apache.org (moderated)
> > > > - d...@airflow.incubator.apache.org
> > > > - comm...@airflow.incubator.apache.org
> > > >
> > > > Github shows 109 contributors, but the proposal lists only 6 initial
> > > > committers, where any of the other existing contributors considered
> for
> > > the
> > > > proposal?
> > > >
> > > > When you submit the proposal please make sure to include the entire
> > > > proposal in the email. If you have any questions please let us know
> > > >
> > > > -Jake
> > > >
> > > >
> > > > [1]:
> > > >
> > > >
> > >
> >
> http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Champion
> > > >
> > > > On Wed, Mar 16, 2016 at 8:28 PM, Siddharth Anand
> > > <san...@agari.com.invalid
> > > > >
> > > > wrote:
> > > >
> > > > > https://wiki.apache.org/incubator/AirflowProposal
> > > > >
> > > > > Thoughts and comments are welcome!
> > > > > -s (Sid)
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] : Airflow

[Chris Riccomini]

> The proposal has the incubator listed as the sponsor and a quick check
shows Chris Riccomini is not on the IPMC or a member currently, details on
roles are available at [1].

I was under the impression that I could sponsor because I am an officer:

"A candidate project shall be sponsored by an Officer
<http://www.apache.org/foundation/index.html> or Member
<http://www.apache.org/foundation/members.html> of the Foundation"

The officer page lists me as a VP of Apache Samza. Perhaps I'm misreading?

> Mailing lists are in the old format

Fixed.

> Github shows 109 contributors, but the proposal lists only 6 initial
committers, where any of the other existing contributors considered for the
proposal?

In terms of the high ratio of contributors to committers (109:6), the
reason is as follows. Most contributions are to the fringes of the system
and are in the form of 3rd party contributed hooks/operators (for 3rd party
integrations). For example, if you want to use Airflow to speak to
Cassandra, you may want to contribute a Cassandra Hook and CRUD operators.
Airflow allows for this extensibility and allows for our users to adapt
Airflow to their needs. However, the committer list is limited to folks who
will touch the internals of the system (e.g. scheduler, metadata
management, etc...). Overtime, we would like other contributors to get
familiar with the internals to the point where they can also become
maintainers, but they don't currently have the expertise.


On Thu, Mar 17, 2016 at 8:16 AM, Scott Deboy <scott.de...@gmail.com> wrote:

> Ll
> On Mar 17, 2016 6:51 AM, "Jake Farrell" <jfarr...@apache.org> wrote:
>
> > Hi Siddharth
> > Thanks for drafting a proposal and looking to bring Airflow to the Apache
> > Incubator. Overall the proposal looks good, just a couple comments. The
> > proposal has the incubator listed as the sponsor and a quick check shows
> > Chris Riccomini is not on the IPMC or a member currently, details on
> roles
> > are available at [1]. Not a blocker as you have members listed in your
> > group of mentors and can ask that they step up as the champion.
> >
> > Mailing lists are in the old format, they should be
> > - priv...@airflow.incubator.apache.org (moderated)
> > - d...@airflow.incubator.apache.org
> > - comm...@airflow.incubator.apache.org
> >
> > Github shows 109 contributors, but the proposal lists only 6 initial
> > committers, where any of the other existing contributors considered for
> the
> > proposal?
> >
> > When you submit the proposal please make sure to include the entire
> > proposal in the email. If you have any questions please let us know
> >
> > -Jake
> >
> >
> > [1]:
> >
> >
> http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Champion
> >
> > On Wed, Mar 16, 2016 at 8:28 PM, Siddharth Anand
> <san...@agari.com.invalid
> > >
> > wrote:
> >
> > > https://wiki.apache.org/incubator/AirflowProposal
> > >
> > > Thoughts and comments are welcome!
> > > -s (Sid)
> > >
> >
>

Re: Wiki access

[Chris Riccomini]

Hey Jake,

Cool, I *think* I've added Martin as an admin. Martin, could you confirm?

Thanks!
Chris


On Tue, Jul 1, 2014 at 12:49 PM, Jake Farrell jfarr...@apache.org wrote:

 Hey Chris
 You will have to have access to edit the contributors group and add Nick to
 that list

 https://wiki.apache.org/samza/ContributorsGroup

 -Jake



 On Tue, Jul 1, 2014 at 3:36 PM, Chris Riccomini criccomini@gmail.com
 wrote:

  Hey Nick,
 
  I am a Samza committer with edit access to this wiki. How can I add
 Martin?
 
  Cheers,
  Chris
 
 
  On Tue, Jul 1, 2014 at 12:26 PM, Nick Burch apa...@gagravarr.org
 wrote:
 
   On Tue, 1 Jul 2014, Martin Kleppmann wrote:
  
   Thanks! Could you please also give me write access on the Samza wiki (
   https://wiki.apache.org/samza/FrontPage)? Sorry for the spam, I
 didn't
   realise each wiki has a separate user account.
  
  
   I don't have karma on that wiki, sorry
  
   https://wiki.apache.org/samza/ContributorsGroup suggests you need to
  drop
   a note to the samza dev list to request access there
  
   Nick
  
   -
   To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
   For additional commands, e-mail: general-h...@incubator.apache.org
  
  
 

Re: Hoya Proposal

[Chris Riccomini]

Hey Guys,

I'd like to +1 non-binding Hoya (or whatever you want to call it). It's
going to be really useful for the YARN ecosystem to have a project like
this out there, and it makes a lot of sense to put it in Apache.

Hopefully we can move past the naming stuff, and get the project in.

Cheers,
Chris


On Mon, Jan 20, 2014 at 12:32 PM, Tommaso Teofili tommaso.teof...@gmail.com
 wrote:

 I just can speak for the Hama-to-Yarn use case (with my newbie hat on) and
 it seems Hoya may be really helpful as the entry point to address it as it
 provides a set of simple and useful tools.
 For the sake of the names I think Hoya would be good (while Hyena would
 not).

 Just my 0.02 cents,
 Tommaso


 2014/1/17 Steve Loughran ste...@hortonworks.com

  On 16 January 2014 12:41, Tommaso Teofili tommaso.teof...@gmail.com
  wrote:
 
   Hi all,
  
   @Steve I'm curious to know if also Apache Hama may leverage Hoya to be
   deployed on YARN.
  
  
  If you are going to make a 100% commit to running in YARN -which, if you
  aren't ready yet, Hama should be able to do in some point in the future-
  then I'd say go for it! and then point you at Twill to get you started.
 
  If all you are trying to do is deploy the BSP daemons in a YARN cluster,
  and they don't need any persistent localfs storage, then Hoya should
 work.
  You already use ZK for the binding mechanism, don't you?
 
 
  If yes, that'd hopefully bring some more people involved from Hama as
 well.
  
 
 
  That'd be great!
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
 to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 

Re: Welcome to the Incubator, Samza community!

[Chris Riccomini]

Hey All,

Thanks for the info! Great to be here.

Cheers,
Chris


On Thu, Aug 15, 2013 at 9:01 AM, Ross Gardler rgard...@opendirective.comwrote:

 Good catch Daniel

 Samza community, be aware that the incubator is attempting to be more
 responsive to the occasional blocker on a podlings progress. You are the
 first of our podlings to receive this welcome email and as such you are
 also exposed to its bugs.  In all likelihood this bug will not affect
 you in practice.

 This discussion about the ombud is going around in circles and we need to
 close it. As an interim step I have edited the wiki to remove reference to
 the non-existent ombud. If the IPMC fails to deal with a podlings issue
 effectively then it is a failure of the IPMC and therefore a board issue.

 I've updated the para in the doc to say:

 In certain circumstances, there are specific people charged with
 specific responsibilities. Over time you can expect to learn who they
 are, and where they hang out. Your mentors should be able to guide you
 while you do learn. If, for whatever reason, they are unable or
 unwilling to help you can ask on the incubator general list
 (gene...@incubartor.apache.org). If the topic is
 too sensitive to discuss in public (eg a potential committer) you may
 contact the incubator PMC at priv...@incubator.apache.org. In the
 unlikely event that you are still unable to find a resolution then you
 can escalate to the Apache Board via your regular board reports.
 Should the matter be urgent feel free to contact the board via
 bo...@apache.org at any time. Please note that
 all these channels are concerned with the community and infrastructure
 aspects of your project, they have no interest in implementation issues
 you may face - your on your own with those.

 Ross
 Ross Gardler (@rgardler)
 Senior Technology Evangelist
 Microsoft Open Technologies, Inc.
 A subsidiary of Microsoft Corporation





 On 15 August 2013 08:27, Daniel Shahaf d...@daniel.shahaf.name wrote:

  Marvin Humphrey wrote on Wed, Aug 14, 2013 at 19:02:54 -0700:
   *   See http://wiki.apache.org/incubator/WhatToExpect for a
 practical
   introduction to the Incubator experience.
 
  My favourite part of this is how it first warns that some of the
  documentation may be out-of-date and if you ask about it you may get
  a totally different response, and then gives an email address for the
  ombudsman (and then warns that there's in fact no ombudsman)...
 
  Irony aside, that's at best a bug in WhatToExpect.  (and at worst the
  canary of that page growing all the same problems that existing
  documentations have)
 
  -
  To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
  For additional commands, e-mail: general-h...@incubator.apache.org
 
 

Re: [PROPOSAL] Samza Proposal

[Chris Riccomini]

Hey Guys,

Jakob (the project Champion) is in the process of getting all of the
resources requested in our proposal (JIRA, Hudson, webspace, etc).

As soon as we have webspace allocated, we'll put the Samza site up, which
has all of these docs on it. Henry, as you said, I'll follow up with this
thread when they're up.

Cheers,
Chris


On Wed, Jul 31, 2013 at 2:04 PM, Henry Saputra henry.sapu...@gmail.comwrote:

 Well, usually VOTE is conducted after discussion had calmed down. Looks
 like this time the VOTE starts even though there were some question about
 the proposal.

 Would be great to actually add links to the comparisons in the thread even
 though the VOTE had concluded.

 - Henry


 On Wed, Jul 31, 2013 at 1:29 PM, Phillip Rhodes
 motley.crue@gmail.comwrote:

  Same here.  Not that it matters as far as admission to the incubator
  (that vote is over now anyway), but I think a lot of people (including
  potential users of Samza) would like to see more about how it compares
   contrasts with other stream oriented systems.
 
 
  Phil
  This message optimized for indexing by NSA PRISM
 
 
  On Fri, Jul 26, 2013 at 8:27 PM, Alex Karasulu akaras...@apache.org
  wrote:
   +1
  
   I would love to see the documents comparing and contrasting Samza with
   MUPD8 and Storm.
  
  
   On Sat, Jul 27, 2013 at 2:53 AM, Enis Söztutar e...@apache.org
 wrote:
  
   +1 on incubation.
  
   Enis
  
  
   On Tue, Jul 23, 2013 at 7:17 PM, Chris Riccomini
   criccomini@gmail.comwrote:
  
Hey Henry and Debo,
   
Thanks for calling this out. Samza's feature set includes:
   
   - *Simpe API:* Unlike most low-level messaging system APIs, Samza
   provides a very simple call-back based process message API that
should be
   familiar to anyone that's used Map/Reduce.
   - *Managed state:* Samza manages snapshotting and restoration of
 a
   stream processor's state. Samza will restore a stream processor's
   state
to
   a snapshot consistent with the processor's last read messages
 when
  the
   processor is restarted.
   - *Fault tolerance:* Samza will work with YARN to restart your
  stream
   processor if there is a machine or processor failure.
   - Durability: Samza uses Kafka to guarantee that no messages will
  ever
   be lost.
   - *Scalability:* Samza is partitioned and distributed at every
  level.
   Kafka provides ordered, partitioned, replayable, fault-tolerant
   streams.
   YARN provides a distributed environment for Samza containers to
 run
   in.
   - *Pluggable:* Though Samza works out of the box with Kafka and
  YARN,
   Samza provides a pluggable API that lets you run Samza with other
messaging
   systems and execution environments.
   - *Processor isolation:* Samza works with Apache YARN, which
  supports
   processor security through Hadoop's security model, and resource
isolation
   through Linux CGroups.
   
Some of these feature are available in S4, and some are not. The
 same
   holds
true for Storm.
   
The open source stream processing systems that are available are
  actually
quite young, and no single system offers a complete solution.
 Problems
   like
how a stream processor's state (e.g. counts) should be managed,
  whether a
stream should be buffered remotely on disk or not, what to do when
duplicate messages are received or messages are lost, and how to
 model
underlying messaging systems are all pretty new.
   
Samza's main differentiators are:
   
   - State is modeled as a stream. When a processor fails and is
   restarted,
   the state stream is entirely replayed to restore it.
   - Streams are ordered, partitioned, replayable, and fault
 tolerant.
   - YARN is used for processor isolation, security, and fault
  tolerance.
   - All streams are materialized to Kafka.
   
If you guys are interested, I have much more in-depth documents
  comparing
and contrasting Samza with MUPD8 and Storm.
   
Cheers,
Chris
   
   
On Tue, Jul 23, 2013 at 6:48 PM, Henry Saputra 
  henry.sapu...@gmail.com
wrote:
   
 Looks like this is similar to S4 (http://incubator.apache.org/s4/
 )
   which
 allow stream and real time data processing via DAG?


 - Henry


 On Tue, Jul 23, 2013 at 10:47 AM, Chris Ricco 
   criccomini@gmail.com
 wrote:

  Hey All,
 
  Sending along an incubator proposal for Samza.
 
  Thanks!
  Chris
 
  https://wiki.apache.org/incubator/SamzaProposal
 
  
 
  == Abstract ==
 
  Samza is a stream processing system for running continuous
   computation
on
  infinite streams of data.
 
  == Proposal ==
 
  Samza provides a system for processing stream data from
publish-subscribe
  systems such as Apache Kafka. The developer writes

Re: [PROPOSAL] Samza Proposal

[Chris Riccomini]

Hey Marvin,

I think we pretty much agree with everything you've said. :)

We're definitely sensitive to the discuss-in-person issue. We'll pay
attention to it, and try to move conversation to the list when we see it
happening. The same holds true for the risk assessment comment. We're
sensitive to it. We'll try our best.

I talked with Jakob, and we'll hold off on the -user mailing list initially.

Thanks for the feedback!

Cheers,
Chris


On Thu, Jul 25, 2013 at 8:17 AM, Marvin Humphrey mar...@rectangular.comwrote:

 Hi,

 The current core developers are all from LinkedIn. However, we hope to
 establish a developer community that includes contributors from several
 corporations and we actively encouraging new contributors via the
 mailing
 lists and public presentations of Samza.

 Collective experience in the Incubator suggests that keeping discussions
 on-list will be an important challenge for the prospective Samza podling.
 It's much more efficient to just discuss things in person when everyone
 works
 in the same office -- especially difficult architectural issues requiring a
 high-level perspective on the complete project.  However, given the history
 both the individual contributors and LinkedIn itself, at least the podling
 ought to start off with a strong understanding of the challenge.

 The core developers plan to work full time on the project. There is
 very
 little risk of Samza being abandoned as it is part of LinkedIn's
 internal
 infrastructure.

 This risk assessment seems a tad optimistic to me ;) since I only trust
 commercial entities to do what's in the interest of the bottom line and
 that
 can change very rapidly.  (Witness IBM pulling all of its developers off of
 Harmony.)  But I certainly don't see any problem that ought to block entry
 into the Incubator.

 * samza-private for private PMC discussions (with moderated
 subscriptions)
 * samza-dev
 * samza-commits
 * samza-user

 I'd suggest foregoing the user list for now.  (See
 http://wiki.apache.org/incubator/MailingListOptions.)  You want highly
 engaged users who are likely to become developers.  It's best to keep
 everyone
 on the same list until dev traffic levels become burdensome to users; you
 don't have that problem yet.

 Nominated Mentors

 * Arun C Murthy acmurthy at apache dot org
 * Chris Douglas cdouglas at apache dot org
 * Roman Shaposhnik rvs at apache dot org

 The core developers may all work for LinkedIn, but that's a good, diverse
 list
 of Mentors in terms of affiliation.  (Hortonworks, Microsoft, and Cloudera,
 repectively.)

 Marvin Humphrey

 -
 To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
 For additional commands, e-mail: general-h...@incubator.apache.org

Re: [PROPOSAL] Samza Proposal

[Chris Riccomini]

 ===
 
  Our intent with this incubator proposal is to start building a diverse
  developer community around Samza following the Apache meritocracy model.
  Since Samza was initially developed in late 2011, we have had fast
 adoption
  and contributions by multiple teams at LinkedIn. We plan to continue
  support for new contributors and work with those who contribute
  significantly to the project to make them committers.
 
  === Community ===
 
  Samza is currently being used internally at LinkedIn. We hope to extend
 our
  contributor base significantly and invite all those who are interested in
  building large-scale distributed systems to participate.
 
  === Core Developers ===
 
  Samza is currently being developed by four engineers at LinkedIn: Jay
  Kreps, Jakob Homan, Sriram Subramanian, and Chris Riccomini. Jakob is an
  ASF Member, Incubator PMC member and PMC member on Apache Hadoop, Kafka
 and
  Giraph. Jay is a member of the Apache Kafka PMC and contributor to
 various
  Apache projects. Chris has been an active contributor for several
 projects
  including Apache Kafka and Apache YARN. Sriram has contributed to Samza,
 as
  well as Apache Kafka.
 
  === Alignment ===
 
  The ASF is the natural choice to host the Samza project as its goal of
  encouraging community-driven open-source projects fits with our vision
 for
  Samza. Additionally, many other projects with which we are familiar with
  and expect Samza to integrate with, such as Apache ZooKeeper, YARN, HDFS
  and log4j are hosted by the ASF and we will benefit and provide benefit
 by
  close proximity to them.
 
  == Known Risks ==
 
  === Orphaned Products ===
 
  The core developers plan to work full time on the project. There is very
  little risk of Samza being abandoned as it is part of LinkedIn's internal
  infrastructure.
 
  === Inexperience with Open Source ===
 
  All of the core developers have experience with open source development.
  Jay and Chris has been involved with several open source projects
 released
  by LinkedIn, and Jay is a committer on Apache Kafka. Jakob has been
  actively involved with the ASF as a full-time Hadoop committer and PMC
  member. Sriram is a contributor to Apache Kafka.
 
  === Homogeneous Developers ===
 
  The current core developers are all from LinkedIn. However, we hope to
  establish a developer community that includes contributors from several
  corporations and we actively encouraging new contributors via the mailing
  lists and public presentations of Samza.
 
  === Reliance on Salaried Developers ===
 
  Currently, the developers are paid to do work on Samza. However, once the
  project has a community built around it, we expect to get committers,
  developers and community from outside the current core developers.
 However,
  because LinkedIn relies on Samza internally, the reliance on salaried
  developers is unlikely to change.
 
  === Relationships with Other Apache Products ===
 
  Samza is deeply integrated with Apache products. Samza uses Apache Kafka
 as
  its underlying message passing system. Samza also uses Apache YARN for
 task
  scheduling. Both YARN and Kafka, in turn, rely on Apache ZooKeeper for
  coordination. In addition, we hope to integrate with Apache HDFS in the
  near future.
 
  === An Excessive Fascination with the Apache Brand ===
 
  While we respect the reputation of the Apache brand and have no doubts
 that
  it will attract contributors and users, our interest is primarily to give
  Samza a solid home as an open source project following an established
  development model. We have also given reasons in the Rationale and
  Alignment sections.
 
  == Documentation ==
 
  http://wiki.apache.org/incubator/SamzaProposal
 
  == Initial Source ==
 
  Available upon request.
 
  == External Dependencies ==
 
  The dependencies all have Apache compatible licenses.
 
   * metrics (Apache 2.0)
   * zkclient (Apache 2.0)
   * zookeeper (Apache 2.0)
   * jetty (Apache 2.0)
   * jackson (Apache 2.0)
   * commons-httpclient (Apache 2.0)
   * slf4j (MIT)
   * avro (Apache 2.0)
   * hadoop (Apache 2.0)
   * junit (Common Public License)
   * grizzled-slf4j (BSD)
   * scalatra (https://github.com/scalatra/scalatra/blob/develop/LICENSE)
   * scala (http://www.scala-lang.org/node/146)
   * joptsimple (MIT)
   * kafka (Apache 2.0)
   * scalate (Apache 2.0)
   * leveldb jni (BSD)
 
  == Cryptography ==
 
  Samza will depend on secure Hadoop, which can optionally use Kerberos.
 
  == Required Resources ==
 
  === Mailing Lists ===
 
  samza-private for private PMC discussions (with moderated subscriptions)
  samza-dev
  samza-commits
  samza-user
 
  === Subversion Directory ===
 
  Git is the preferred source control system: git://git.apache.org/samza
 
  === Issue Tracking ===
 
  JIRA Samza (SAMZA)
 
  === Other Resources ===
 
  The existing code already has unit tests, so we would like a Hudson
  instance to run them whenever a new patch is submitted. This can be added
  after