Re: Subsequent Airflow Meetup: 2017/01/11

[Kevin Mandich]

Hi George,

Confirmed - would like give a talk. Thanks,

Kevin Mandich

On Tue, Jan 3, 2017 at 5:40 AM, Dan Davydov 
wrote:

> Confirmed.
>
> On Sun, Jan 1, 2017 at 9:16 PM, George Leslie-Waksman <
> geo...@cloverhealth.com.invalid> wrote:
>
> > Sorry for the delayed response, end of year and holidays stole my
> attention
> > for a bit.
> >
> > With the new year, I was just looking to pick things back up and solicit
> > presenters for the meetup. Given we're looking for two more, and
> > Dan(Airbnb) and Kevin(Agari) have already expressed interest, I'd be
> happy
> > to give them the spots.
> >
> > I hope the delay in my response isn't too much of an inconvenience for
> > anyone. Dan, Kevin: confirm and I'll add you to the line up.
> >
> > --George
> >
> > On Sun, Nov 20, 2016 at 8:44 PM siddharth anand 
> wrote:
> >
> > > I suspect Clover Health is extremely busy with all of the benefit
> > > enrollments going on right now..
> > >
> > > George,
> > > When you come up for air, it looks like both Dan(Airbnb) and
> Kevin(Agari)
> > > have talk ideas.
> > >
> > > -s
> > >
> > > On Wed, Nov 16, 2016 at 11:50 PM, Dan Davydov <
> > > dan.davy...@airbnb.com.invalid> wrote:
> > >
> > > > Based on chatting with a couple of people today at the Airflow
> meet-up
> > I
> > > > think there has been some demand for an airflow operations talk,
> > > > specifically around monitoring/alerting. If there is still room I can
> > > give
> > > > a talk about this, let me know George.
> > > >
> > > > On Thu, Nov 10, 2016 at 10:17 AM, siddharth anand  >
> > > > wrote:
> > > >
> > > > > Kevin,
> > > > > Here's a link to the 1Q17 meet-up.
> > > > >
> > > https://www.meetup.com/Bay-Area-Apache-Airflow-
> Incubating-Meetup/events/
> > > > > 235259523/
> > > > >
> > > > > Both upcoming meet-ups (next week at WePay and 1Q17 at Clover
> Health)
> > > can
> > > > > be found on http://www.meetup.com/Bay-Area-Apache-Airflow-
> > > > > Incubating-Meetup/
> > > > >
> > > > > -s
> > > > >
> > > > >
> > > > > On Wed, Nov 9, 2016 at 4:24 PM, Kevin Mandich <
> > kevinmand...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi George,
> > > > > >
> > > > > > If there is still room, I'd like to give a talk about how we use
> > > > Airflow
> > > > > at
> > > > > > my company, Agari. We are a data company that is working to
> > eliminate
> > > > > > inbound, targeted e-mail attacks to our customers
> > (spear-phishing). I
> > > > am
> > > > > > currently working as a data scientist who is also responsible for
> > > > > shipping
> > > > > > my work to production.
> > > > > >
> > > > > > We currently use Airflow to build models from our telemetry data
> > > which
> > > > > are
> > > > > > then used for scoring in our near-real-time pipeline. I'd like to
> > > talk
> > > > > > about some of the DAGs we've set up to do this.
> > > > > >
> > > > > > Please let me know if this sounds reasonable. Thank you,
> > > > > >
> > > > > > Kevin Mandich
> > > > > > Agari Data, Inc.
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 31, 2016 at 11:27 PM, George Leslie-Waksman <
> > > > > > geo...@cloverhealth.com.invalid> wrote:
> > > > > >
> > > > > > > I know it's a bit far in advance, but to make sure there's
> space
> > > (and
> > > > > > food
> > > > > > > and drink), I've scheduled and booked the subsequent meetup for
> > > > January
> > > > > > > 11th at Clover Health in SF.
> > > > > > >
> > > > > > > If anyone wants to volunteer to talk, let me know, otherwise
> I'll
> > > > > > probably
> > > > > > > start bugging folks sometime after Thanksgiving and before the
> > > > December
> > > > > > > holidays.
> > > > > > >
> > > > > > > --George Leslie-Waksman
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Airflow 1.8.0 Alpha 1

[Chris Riccomini]

Hey Bolke,

Thanks for taking this on. I'm definitely up for running stuff in our
environments to verify everything is working.

Can I ask that you create a 1.8 alpha 1 branch in the git repo? This will
make it easier for us to track what changes are getting cherry picked into
the branch, and will also make it easier for users to pip install, if they
want to do so via github.

Also, yea, when we switch to beta, we need to stop merging anything other
than bug fixes into the release branch.

Cheers,
Chris

On Tue, Jan 3, 2017 at 10:31 AM, Dan Davydov  wrote:

> All very reasonable to me, one reason we may not have hit the bugs in our
> production is because we are running off a different merge base and our
> cherries aren't 1-1 with what we are running in production (we still test
> them but we can't run them in production), that being said I don't think I
> authored the commits you are referring to so I don't have full context.
>
> On Tue, Jan 3, 2017 at 1:27 PM, Bolke de Bruin  wrote:
>
> > Hi Dan et al,
> >
> > That sounds good to me, however I will be pretty critical of the changes
> > in the scheduler and the cleanliness of the patches. This is due to the
> > fact I have been chasing quite some bugs in master that were pretty hard
> to
> > track down even with a debugger at hand. I’m surprised that those didn’t
> > pop up in your production or maybe I am concerned ;-). Anyways, I hope
> you
> > understand I might be a bit picky in understanding and needing (design)
> > documentation for some of the changes.
> >
> > What I would like to suggest is that for the Alpha versions we still
> > accept “new” features so these PRs can get in, but from Beta we will not
> > accept new features anymore. For new features in the area of the
> scheduler
> > an integration DummyDag should be supplied, so others can test the
> > behaviour. Does this sound ok?
> >
> > My list of open code items for a release looks now like this:
> >
> > Blockers
> > * one_failed not honoured
> > * Alex’s sensor issue
> >
> > New features:
> > * Schedule all pending DAGs in a single loop
> > * Add support for backfill true/false
> > * Impersonation
> > * CGroups
> > * Add Cloud Storage updated sensor
> >
> > Alpha2 I will package tomorrow. Packages are signed now by my apache.org
> <
> > http://apache.org/> key. Please verify and let me know if something is
> > off. I’m still waiting for access to the incubating dist repository.
> >
> > Bolke
> >
> >
> > > On 3 Jan 2017, at 14:38, Dan Davydov 
> > wrote:
> > >
> > > I have also started on this effort, recently Alex Guziel and I have
> been
> > > pushing Airbnb's custom cherries onto master to get Airbnb back onto
> > master
> > > in order for us to do a release.
> > >
> > > I think it might make sense to wait for these two commits to get merged
> > in
> > > since they would be quite nice to have for all Airflow users and seem
> > like
> > > they will be merged soon:
> > > Schedule all pending DAG runs in a single scheduler loop -
> > > https://github.com/apache/incubator-airflow/pull/1906 <
> > https://github.com/apache/incubator-airflow/pull/1906>
> > > Add Support for dag.backfill=(True|False) Option -
> > > https://github.com/apache/incubator-airflow/pull/1830 <
> > https://github.com/apache/incubator-airflow/pull/1830>
> > > Impersonation Support + Cgroups - https://github.com/apache/ <
> > https://github.com/apache/>
> > > incubator-airflow/pull/1934 (this is kind of important from the Airbnb
> > side
> > > so that we can help test the new master without having to cherrypick
> this
> > > PR on top of it which would make the testing unreliable for others).
> > >
> > > If there are PRs that affect the core of Airflow that other committers
> > > think are important to merge we could include these too. I can commit
> to
> > > pushing out the Impersonation/Cgroups PR this week pending PR comments.
> > > What do you think Bolke?
> > >
> > > On Tue, Jan 3, 2017 at 4:26 AM, Bolke de Bruin  > > wrote:
> > >
> > >> Hey Alex,
> > >>
> > >> I have noticed the same, and it is also the reason why we have Alpha
> > >> versions. For now I have noticed the following:
> > >>
> > >> * Tasks can get in limbo between scheduler and executor:
> > >> https://github.com/apache/incubator-airflow/pull/1948 <
> > https://github.com/apache/incubator-airflow/pull/1948> <
> > >> https://github.com/apache/incubator-airflow/pull/1948 <
> > https://github.com/apache/incubator-airflow/pull/1948>>
> > >> * Try_number not increased due to reset in LocalTaskJob:
> > >> https://github.com/apache/incubator-airflow/pull/1969 <
> > https://github.com/apache/incubator-airflow/pull/1969> <
> > >> https://github.com/apache/incubator-airflow/pull/1969 <
> > https://github.com/apache/incubator-airflow/pull/1969>>
> > >> * one_failed trigger not executed
> > >>
> > >> My idea is to move to a Samba style 

Re: (INFRA-13207) Please create Airflow in /repos/dist/release/incubator/

[Chris Riccomini]

Hey Bolke,

I've created it:

  https://dist.apache.org/repos/dist/release/incubator/airflow

I've also added a KEYS file with my PGP key in it.

Cheers,
Chris

On Sun, Jan 1, 2017 at 3:08 AM, Bolke de Bruin  wrote:

> Jakob, Chris,
>
> Can you please create create ‘airflow’ in /repos/dist/release/incubator ?
> It is required for an official release.
>
> It would also be nice to know how to access this by svn (URI, What
> credentials).
>
> Cheers!
> Bolke
>
> Begin forwarded message:
>
> *From: *"Daniel Gruno (JIRA)" 
> *Subject: **[jira] [Resolved] (INFRA-13207) Please create Airflow in
> /repos/dist/release/incubator/*
> *Date: *1 January 2017 at 11:32:58 GMT+1
> *To: *bdbr...@gmail.com
>
>
> [ https://issues.apache.org/jira/browse/INFRA-13207?page=
> com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Daniel Gruno resolved INFRA-13207.
> --
>Resolution: Fixed
>  Assignee: Daniel Gruno
>
> Please have an IPMC member (mentor or champion) create this space for you,
> they have the credentials to do so.
>
> Please create Airflow in /repos/dist/release/incubator/
> ---
>
>Key: INFRA-13207
>URL: https://issues.apache.org/jira/browse/INFRA-13207
>Project: Infrastructure
> Issue Type: Task
> Components: Dists, SvnPubSub
>   Reporter: Bolke de Bruin
>   Assignee: Daniel Gruno
>
> There is not yet a directory for podling Airflow. Can this be created
> please? Thanks!
>
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>
>
>

Re: Airflow 1.8.0 Alpha 1

[Dan Davydov]

All very reasonable to me, one reason we may not have hit the bugs in our
production is because we are running off a different merge base and our
cherries aren't 1-1 with what we are running in production (we still test
them but we can't run them in production), that being said I don't think I
authored the commits you are referring to so I don't have full context.

On Tue, Jan 3, 2017 at 1:27 PM, Bolke de Bruin  wrote:

> Hi Dan et al,
>
> That sounds good to me, however I will be pretty critical of the changes
> in the scheduler and the cleanliness of the patches. This is due to the
> fact I have been chasing quite some bugs in master that were pretty hard to
> track down even with a debugger at hand. I’m surprised that those didn’t
> pop up in your production or maybe I am concerned ;-). Anyways, I hope you
> understand I might be a bit picky in understanding and needing (design)
> documentation for some of the changes.
>
> What I would like to suggest is that for the Alpha versions we still
> accept “new” features so these PRs can get in, but from Beta we will not
> accept new features anymore. For new features in the area of the scheduler
> an integration DummyDag should be supplied, so others can test the
> behaviour. Does this sound ok?
>
> My list of open code items for a release looks now like this:
>
> Blockers
> * one_failed not honoured
> * Alex’s sensor issue
>
> New features:
> * Schedule all pending DAGs in a single loop
> * Add support for backfill true/false
> * Impersonation
> * CGroups
> * Add Cloud Storage updated sensor
>
> Alpha2 I will package tomorrow. Packages are signed now by my apache.org <
> http://apache.org/> key. Please verify and let me know if something is
> off. I’m still waiting for access to the incubating dist repository.
>
> Bolke
>
>
> > On 3 Jan 2017, at 14:38, Dan Davydov 
> wrote:
> >
> > I have also started on this effort, recently Alex Guziel and I have been
> > pushing Airbnb's custom cherries onto master to get Airbnb back onto
> master
> > in order for us to do a release.
> >
> > I think it might make sense to wait for these two commits to get merged
> in
> > since they would be quite nice to have for all Airflow users and seem
> like
> > they will be merged soon:
> > Schedule all pending DAG runs in a single scheduler loop -
> > https://github.com/apache/incubator-airflow/pull/1906 <
> https://github.com/apache/incubator-airflow/pull/1906>
> > Add Support for dag.backfill=(True|False) Option -
> > https://github.com/apache/incubator-airflow/pull/1830 <
> https://github.com/apache/incubator-airflow/pull/1830>
> > Impersonation Support + Cgroups - https://github.com/apache/ <
> https://github.com/apache/>
> > incubator-airflow/pull/1934 (this is kind of important from the Airbnb
> side
> > so that we can help test the new master without having to cherrypick this
> > PR on top of it which would make the testing unreliable for others).
> >
> > If there are PRs that affect the core of Airflow that other committers
> > think are important to merge we could include these too. I can commit to
> > pushing out the Impersonation/Cgroups PR this week pending PR comments.
> > What do you think Bolke?
> >
> > On Tue, Jan 3, 2017 at 4:26 AM, Bolke de Bruin  > wrote:
> >
> >> Hey Alex,
> >>
> >> I have noticed the same, and it is also the reason why we have Alpha
> >> versions. For now I have noticed the following:
> >>
> >> * Tasks can get in limbo between scheduler and executor:
> >> https://github.com/apache/incubator-airflow/pull/1948 <
> https://github.com/apache/incubator-airflow/pull/1948> <
> >> https://github.com/apache/incubator-airflow/pull/1948 <
> https://github.com/apache/incubator-airflow/pull/1948>>
> >> * Try_number not increased due to reset in LocalTaskJob:
> >> https://github.com/apache/incubator-airflow/pull/1969 <
> https://github.com/apache/incubator-airflow/pull/1969> <
> >> https://github.com/apache/incubator-airflow/pull/1969 <
> https://github.com/apache/incubator-airflow/pull/1969>>
> >> * one_failed trigger not executed
> >>
> >> My idea is to move to a Samba style of releases eventually, but for now
> I
> >> would like to get master into a state that we understand and therefore
> not
> >> accept any patches that do not address any bugs.
> >>
> >> If you (or anyone else) can review the above PRs and add your own as
> well
> >> then I can create another Alpha version. I’ll be on gitter as much as I
> can
> >> so we can speed up if needed.
> >>
> >> - Bolke
> >>
> >>> On 3 Jan 2017, at 08:51, Alex Van Boxel  wrote:
> >>>
> >>> Hey Bolke,
> >>>
> >>> thanks for getting this moving. But I already have some blockers,
> since I
> >>> moved up master to this release (moved from end November to now)
> >> stability
> >>> has gone down (certainly on Celary). I'm trying to identify the core
> >>> problems and see if I can fix them.
> >>>
> >>> 

Re: Airflow 1.8.0 Alpha 1

[Bolke de Bruin]

Hi Dan et al,

That sounds good to me, however I will be pretty critical of the changes in the 
scheduler and the cleanliness of the patches. This is due to the fact I have 
been chasing quite some bugs in master that were pretty hard to track down even 
with a debugger at hand. I’m surprised that those didn’t pop up in your 
production or maybe I am concerned ;-). Anyways, I hope you understand I might 
be a bit picky in understanding and needing (design) documentation for some of 
the changes.

What I would like to suggest is that for the Alpha versions we still accept 
“new” features so these PRs can get in, but from Beta we will not accept new 
features anymore. For new features in the area of the scheduler an integration 
DummyDag should be supplied, so others can test the behaviour. Does this sound 
ok?

My list of open code items for a release looks now like this:

Blockers
* one_failed not honoured
* Alex’s sensor issue

New features:
* Schedule all pending DAGs in a single loop
* Add support for backfill true/false
* Impersonation
* CGroups
* Add Cloud Storage updated sensor

Alpha2 I will package tomorrow. Packages are signed now by my apache.org 
 key. Please verify and let me know if something is off. 
I’m still waiting for access to the incubating dist repository.

Bolke


> On 3 Jan 2017, at 14:38, Dan Davydov  wrote:
> 
> I have also started on this effort, recently Alex Guziel and I have been
> pushing Airbnb's custom cherries onto master to get Airbnb back onto master
> in order for us to do a release.
> 
> I think it might make sense to wait for these two commits to get merged in
> since they would be quite nice to have for all Airflow users and seem like
> they will be merged soon:
> Schedule all pending DAG runs in a single scheduler loop -
> https://github.com/apache/incubator-airflow/pull/1906 
> 
> Add Support for dag.backfill=(True|False) Option -
> https://github.com/apache/incubator-airflow/pull/1830 
> 
> Impersonation Support + Cgroups - https://github.com/apache/ 
> 
> incubator-airflow/pull/1934 (this is kind of important from the Airbnb side
> so that we can help test the new master without having to cherrypick this
> PR on top of it which would make the testing unreliable for others).
> 
> If there are PRs that affect the core of Airflow that other committers
> think are important to merge we could include these too. I can commit to
> pushing out the Impersonation/Cgroups PR this week pending PR comments.
> What do you think Bolke?
> 
> On Tue, Jan 3, 2017 at 4:26 AM, Bolke de Bruin  > wrote:
> 
>> Hey Alex,
>> 
>> I have noticed the same, and it is also the reason why we have Alpha
>> versions. For now I have noticed the following:
>> 
>> * Tasks can get in limbo between scheduler and executor:
>> https://github.com/apache/incubator-airflow/pull/1948 
>>  <
>> https://github.com/apache/incubator-airflow/pull/1948 
>> >
>> * Try_number not increased due to reset in LocalTaskJob:
>> https://github.com/apache/incubator-airflow/pull/1969 
>>  <
>> https://github.com/apache/incubator-airflow/pull/1969 
>> >
>> * one_failed trigger not executed
>> 
>> My idea is to move to a Samba style of releases eventually, but for now I
>> would like to get master into a state that we understand and therefore not
>> accept any patches that do not address any bugs.
>> 
>> If you (or anyone else) can review the above PRs and add your own as well
>> then I can create another Alpha version. I’ll be on gitter as much as I can
>> so we can speed up if needed.
>> 
>> - Bolke
>> 
>>> On 3 Jan 2017, at 08:51, Alex Van Boxel  wrote:
>>> 
>>> Hey Bolke,
>>> 
>>> thanks for getting this moving. But I already have some blockers, since I
>>> moved up master to this release (moved from end November to now)
>> stability
>>> has gone down (certainly on Celary). I'm trying to identify the core
>>> problems and see if I can fix them.
>>> 
>>> On Sat, Dec 31, 2016 at 9:52 PM Bolke de Bruin > >> wrote:
>>> 
>>> Dear All,
>>> 
>>> On the verge of the New Year, I decided to be a little bit cheeky and to
>>> make available an Airflow 1.8.0 Alpha 1. We have been talking about it
>> for
>>> a long time now and by doing this I wanted bootstrap the process. It
>> should
>>> by no means be considered an Apache release yet. This is for testing
>>> purposes in the dev community around Airflow, nothing else.
>>> 
>>> The build is exactly the same as the state of master (git 410736d) 

Re: Airflow 2.0

[Maxime Beauchemin]

Right, this conversation turned into more of a wishlist / roadmap
brainstorm which is also very useful. Let's treat this thread for the value
it has from that perspective

Seems like very few things needed are breaking changes. Re-packaging
(breaking Airflow into smaller python packages) would be the main breaking
change and I don't feel like it's a priority at this point. Long live 1.x!

Max

On Sun, Jan 1, 2017 at 6:57 PM, George Leslie-Waksman <
geo...@cloverhealth.com.invalid> wrote:

> There is a bit of a numbering question that we seem to be assuming an
> answer to when we talk about 2.0; does 2.0 *necessarily* follow 1.9, or
> will there be a 1.10, perhaps 1.11, etc.
>
> A lot of the things that folks are proposing for 2.0 wouldn't really break
> the world for people on 1.8, so maybe they can be introduced before 2.0.
>
> For things that we want in a 2.0, we might be willing to break some parts
> of the world, and continue to maintain (for a time) a 1.x and a 2.x branch.
>
> On Tue, Dec 13, 2016 at 7:04 AM Bolke de Bruin  wrote:
>
> > Hey Gurer,
> >
> > Thanks for the summary. I have updated the format a little bit and added
> > some items of my own. I left the old style in tact for now, if that is a
> > more convenient format after all.
> >
> > Bolke
> >
> > > Op 12 dec. 2016, om 17:04 heeft Gurer Kiratli <
> gurer.kira...@airbnb.com.INVALID>
> > het volgende geschreven:
> > >
> > > Hi folks,
> > >
> > > Here is the list
> > >  2017+Roadmap+Items>
> > of
> > > possible roadmap items for 2017. I think that clubbing deliverables
> into
> > > 1.9 or 2.0 is orthogonal to our high level 2017 planning so I went with
> > > this approach.
> > >
> > > Please take a look at the wiki and see if there is something missing or
> > > needs further clarification by the end of the week and I will send out
> a
> > > survey next week to get a sense of priorities across the community.
> > >
> > > Let me know if you have any questions.
> > >
> > > Cheers,
> > >
> > > Gurer
> > >
> > > On Tue, Dec 6, 2016 at 11:15 PM, Maxime Beauchemin <
> > > maximebeauche...@gmail.com> wrote:
> > >
> > >> I spoke with Gurer yesterday and he's going to summarize and send a
> > survey.
> > >> It should be out this week.
> > >>
> > >> Max
> > >>
> > >> On Tue, Dec 6, 2016 at 7:24 PM, siddharth anand 
> > wrote:
> > >>
> > >>> Max,
> > >>> Do you have time to summarize this thread? Perhaps, publish it on the
> > >> Wiki!
> > >>> -s
> > >>>
> > >>> On Thu, Dec 1, 2016 at 12:27 PM, Van Klaveren, Brian N. <
> > >>> b...@slac.stanford.edu> wrote:
> > >>>
> >  With the announcement of AWS Batch (https://aws.amazon.com/batch/),
> > >> and
> >  my own selfish needs, I think it'd be really great to generally
> > support
> >  Batch systems like AWS Batch, Slurm, and Torque as executors,
> > >> potentially
> >  with an extension of the BashOperator, but I think it might actually
> > be
> >  flexible enough to not need a dedicated BatchOperator.
> > 
> >  Brian
> > 
> > 
> >  On Nov 24, 2016, at 7:40 AM, Maycock, Luke  >  oliverwyman.com>
> > wrote:
> > 
> >  Add FK to dag_run to the task_instance table on Postgres so that
> >  task_instances can be uniquely attributed to dag runs.
> > 
> > 
> >  + 1
> > 
> > 
> >  Also, I believe xcoms would need to be addressed in the same way at
> > the
> >  same time - I have added a comment to that affect on
> >  https://issues.apache.org/jira/browse/AIRFLOW-642
> > 
> > 
> >  I believe this would be implemented for all supported back-ends, not
> > >> just
> >  PostgreSQL.
> > 
> > 
> >  Cheers,
> >  Luke Maycock
> >  OLIVER WYMAN
> >  luke.mayc...@affiliate.oliverwyman.com >  mayc...@affiliate.oliverwyman.com> >  affiliate.oliverwyman.com>
> >  www.oliverwyman.com >  www.oliverwyman.com/>
> > 
> > 
> > 
> >  
> >  From: Arunprasad Venkatraman  @uber.com
> > >>
> >  Sent: 21 November 2016 18:16
> >  To: dev@airflow.incubator.apache.org >  incubator.apache.org>
> >  Subject: Re: Airflow 2.0
> > 
> >  Add FK to dag_run to the task_instance table on Postgres so that
> >  task_instances can be uniquely attributed to dag runs.
> >  Ensure scheduler can be run continuously without needing restarts.
> >  Ensure scheduler can handle tens of thousands of active workflows
> > 
> >  +1
> > 
> >  We are planning to run around 40,000 tasks a day using airflow and
> > some
> > >>> of
> >  them are critical to give quick feedback to developers. Currently
> > >> having
> >  execution 

Re: Jan 2017 Airflow Podling Report Posted

[Chris Nauroth]

Thank you for putting together the report.  I have entered my mentor
sign-off.

On a side note, it's very exciting to see the push for an Apache release as
the new year begins!

Chris Nauroth

On Sat, Dec 31, 2016 at 12:58 PM, siddharth anand  wrote:

> Hi Folks!
> Here's the quarterly Podling Report for Apache Airflow. Feel free to
> suggest edits. If you are a committer/maintainer, you can directly edit it.
>
> https://wiki.apache.org/incubator/January2017
>
> -s
>

Re: Airflow 1.8.0 Alpha 1

[Maxime Beauchemin]

@bolke thanks for taking the lead on this. Let's get these cherries in
master and move forward with a release.

Happy new year everyone!

Max

On Tue, Jan 3, 2017 at 7:30 AM, Alex Van Boxel  wrote:

> If they should make the first alpha, maybe they should be rebased so they
> can be merged in.
>
> On Tue, Jan 3, 2017 at 2:39 PM Dan Davydov  invalid>
> wrote:
>
> > I have also started on this effort, recently Alex Guziel and I have been
> > pushing Airbnb's custom cherries onto master to get Airbnb back onto
> master
> > in order for us to do a release.
> >
> > I think it might make sense to wait for these two commits to get merged
> in
> > since they would be quite nice to have for all Airflow users and seem
> like
> > they will be merged soon:
> > Schedule all pending DAG runs in a single scheduler loop -
> > https://github.com/apache/incubator-airflow/pull/1906
> > Add Support for dag.backfill=(True|False) Option -
> > https://github.com/apache/incubator-airflow/pull/1830
> > Impersonation Support + Cgroups - https://github.com/apache/
> > incubator-airflow/pull/1934 (this is kind of important from the Airbnb
> side
> > so that we can help test the new master without having to cherrypick this
> > PR on top of it which would make the testing unreliable for others).
> >
> > If there are PRs that affect the core of Airflow that other committers
> > think are important to merge we could include these too. I can commit to
> > pushing out the Impersonation/Cgroups PR this week pending PR comments.
> > What do you think Bolke?
> >
> > On Tue, Jan 3, 2017 at 4:26 AM, Bolke de Bruin 
> wrote:
> >
> > > Hey Alex,
> > >
> > > I have noticed the same, and it is also the reason why we have Alpha
> > > versions. For now I have noticed the following:
> > >
> > > * Tasks can get in limbo between scheduler and executor:
> > > https://github.com/apache/incubator-airflow/pull/1948 <
> > > https://github.com/apache/incubator-airflow/pull/1948>
> > > * Try_number not increased due to reset in LocalTaskJob:
> > > https://github.com/apache/incubator-airflow/pull/1969 <
> > > https://github.com/apache/incubator-airflow/pull/1969>
> > > * one_failed trigger not executed
> > >
> > > My idea is to move to a Samba style of releases eventually, but for
> now I
> > > would like to get master into a state that we understand and therefore
> > not
> > > accept any patches that do not address any bugs.
> > >
> > > If you (or anyone else) can review the above PRs and add your own as
> well
> > > then I can create another Alpha version. I’ll be on gitter as much as I
> > can
> > > so we can speed up if needed.
> > >
> > > - Bolke
> > >
> > > > On 3 Jan 2017, at 08:51, Alex Van Boxel  wrote:
> > > >
> > > > Hey Bolke,
> > > >
> > > > thanks for getting this moving. But I already have some blockers,
> > since I
> > > > moved up master to this release (moved from end November to now)
> > > stability
> > > > has gone down (certainly on Celary). I'm trying to identify the core
> > > > problems and see if I can fix them.
> > > >
> > > > On Sat, Dec 31, 2016 at 9:52 PM Bolke de Bruin  > > > wrote:
> > > >
> > > > Dear All,
> > > >
> > > > On the verge of the New Year, I decided to be a little bit cheeky and
> > to
> > > > make available an Airflow 1.8.0 Alpha 1. We have been talking about
> it
> > > for
> > > > a long time now and by doing this I wanted bootstrap the process. It
> > > should
> > > > by no means be considered an Apache release yet. This is for testing
> > > > purposes in the dev community around Airflow, nothing else.
> > > >
> > > > The build is exactly the same as the state of master (git 410736d)
> plus
> > > the
> > > > change to version “1.8.0.alpha1” in version.py.
> > > >
> > > > I am dedicating quite some time next week and beyond to get a release
> > > out.
> > > > Hopefully we can get some help with testing, changelog etc. To make
> > this
> > > > possible I would like to propose a freeze to adding new features for
> at
> > > > least two weeks - say until Jan 15.
> > > >
> > > > You can find the tar here: http://people.apache.org/~bolke/ <
> > > > http://people.apache.org/~bolke/ >
> .
> > > It isn’t signed. Following versions
> > > > will be. SHA is available.
> > > >
> > > > Lastly, Alpha 1 does not have the fix for retries yet. So we will get
> > an
> > > > Alpha 2 :-). @Max / @Dan / @Paul: a potential fix is in
> > > > https://github.com/apache/incubator-airflow/pull/1948 <
> > > https://github.com/apache/incubator-airflow/pull/1948> <
> > > > https://github.com/apache/incubator-airflow/pull/1948 <
> > > https://github.com/apache/incubator-airflow/pull/1948>> , but your
> > > feedback
> > > > is required as it is entrenched in new processing code that you are
> > > running
> > > > in production afaik - so I wonder what happens in your fork.
> > > >
> > > > 

Re: Jan 2017 Airflow Podling Report Posted

[Maxime Beauchemin]

Thanks for taking the time to do this while most were on the holiday break!

Max

On Sat, Dec 31, 2016 at 12:58 PM, siddharth anand  wrote:

> Hi Folks!
> Here's the quarterly Podling Report for Apache Airflow. Feel free to
> suggest edits. If you are a committer/maintainer, you can directly edit it.
>
> https://wiki.apache.org/incubator/January2017
>
> -s
>

Re: Airflow 1.8.0 Alpha 1

[Alex Van Boxel]

If they should make the first alpha, maybe they should be rebased so they
can be merged in.

On Tue, Jan 3, 2017 at 2:39 PM Dan Davydov 
wrote:

> I have also started on this effort, recently Alex Guziel and I have been
> pushing Airbnb's custom cherries onto master to get Airbnb back onto master
> in order for us to do a release.
>
> I think it might make sense to wait for these two commits to get merged in
> since they would be quite nice to have for all Airflow users and seem like
> they will be merged soon:
> Schedule all pending DAG runs in a single scheduler loop -
> https://github.com/apache/incubator-airflow/pull/1906
> Add Support for dag.backfill=(True|False) Option -
> https://github.com/apache/incubator-airflow/pull/1830
> Impersonation Support + Cgroups - https://github.com/apache/
> incubator-airflow/pull/1934 (this is kind of important from the Airbnb side
> so that we can help test the new master without having to cherrypick this
> PR on top of it which would make the testing unreliable for others).
>
> If there are PRs that affect the core of Airflow that other committers
> think are important to merge we could include these too. I can commit to
> pushing out the Impersonation/Cgroups PR this week pending PR comments.
> What do you think Bolke?
>
> On Tue, Jan 3, 2017 at 4:26 AM, Bolke de Bruin  wrote:
>
> > Hey Alex,
> >
> > I have noticed the same, and it is also the reason why we have Alpha
> > versions. For now I have noticed the following:
> >
> > * Tasks can get in limbo between scheduler and executor:
> > https://github.com/apache/incubator-airflow/pull/1948 <
> > https://github.com/apache/incubator-airflow/pull/1948>
> > * Try_number not increased due to reset in LocalTaskJob:
> > https://github.com/apache/incubator-airflow/pull/1969 <
> > https://github.com/apache/incubator-airflow/pull/1969>
> > * one_failed trigger not executed
> >
> > My idea is to move to a Samba style of releases eventually, but for now I
> > would like to get master into a state that we understand and therefore
> not
> > accept any patches that do not address any bugs.
> >
> > If you (or anyone else) can review the above PRs and add your own as well
> > then I can create another Alpha version. I’ll be on gitter as much as I
> can
> > so we can speed up if needed.
> >
> > - Bolke
> >
> > > On 3 Jan 2017, at 08:51, Alex Van Boxel  wrote:
> > >
> > > Hey Bolke,
> > >
> > > thanks for getting this moving. But I already have some blockers,
> since I
> > > moved up master to this release (moved from end November to now)
> > stability
> > > has gone down (certainly on Celary). I'm trying to identify the core
> > > problems and see if I can fix them.
> > >
> > > On Sat, Dec 31, 2016 at 9:52 PM Bolke de Bruin  > > wrote:
> > >
> > > Dear All,
> > >
> > > On the verge of the New Year, I decided to be a little bit cheeky and
> to
> > > make available an Airflow 1.8.0 Alpha 1. We have been talking about it
> > for
> > > a long time now and by doing this I wanted bootstrap the process. It
> > should
> > > by no means be considered an Apache release yet. This is for testing
> > > purposes in the dev community around Airflow, nothing else.
> > >
> > > The build is exactly the same as the state of master (git 410736d) plus
> > the
> > > change to version “1.8.0.alpha1” in version.py.
> > >
> > > I am dedicating quite some time next week and beyond to get a release
> > out.
> > > Hopefully we can get some help with testing, changelog etc. To make
> this
> > > possible I would like to propose a freeze to adding new features for at
> > > least two weeks - say until Jan 15.
> > >
> > > You can find the tar here: http://people.apache.org/~bolke/ <
> > > http://people.apache.org/~bolke/ > .
> > It isn’t signed. Following versions
> > > will be. SHA is available.
> > >
> > > Lastly, Alpha 1 does not have the fix for retries yet. So we will get
> an
> > > Alpha 2 :-). @Max / @Dan / @Paul: a potential fix is in
> > > https://github.com/apache/incubator-airflow/pull/1948 <
> > https://github.com/apache/incubator-airflow/pull/1948> <
> > > https://github.com/apache/incubator-airflow/pull/1948 <
> > https://github.com/apache/incubator-airflow/pull/1948>> , but your
> > feedback
> > > is required as it is entrenched in new processing code that you are
> > running
> > > in production afaik - so I wonder what happens in your fork.
> > >
> > > Happy New Year!
> > >
> > > Bolke
> > >
> > >
> > >
> > > --
> > >  _/
> > > _/ Alex Van Boxel
> >
> >
>
-- 
  _/
_/ Alex Van Boxel

Re: Airflow Github Mirror is not synchronizing

[siddharth anand]

The repo mirror is syncing now.
-s
On Sat, Dec 31, 2016 at 11:22 AM siddharth anand  wrote:

> FYI!
> I've reopened my earlier JIRA issue. It looks like multiple Apache
> Projects are reporting the same.
> https://issues.apache.org/jira/browse/INFRA-12949
>
> Newly merged changes won't be available to contributors/users until the
> mirroring issue is fixed.
>
>
>

Re: Python 3

[Conor Nash]

We are using Airflow with Python 3.5 and RabbitMQ (CeleryExecutor). We're
using Celery version 3.1.24. For the BROKER_URL connection string we use
"pyamqp://:@host:5671/vhost" and for CELERY_RESULT_BACKEND we
use "rpc://".

Is Redis the preferred/better supported backend?

Cheers,

Conor Nash
Data Science Consulting
www.conornash.com | @conornash 

On Sun, Jan 1, 2017 at 9:12 PM, George Leslie-Waksman <
geo...@cloverhealth.com.invalid> wrote:

> We are using Airflow with Python3, PostgreSQL and Redis; it works just
> fine.
>
> On Thu, Dec 1, 2016 at 10:30 AM Maycock, Luke <
> luke.mayc...@affiliate.oliverwyman.com> wrote:
>
> > Thanks Dennis. We will perhaps try Redis as there has been a ticket to
> > make librabbitmq Python 3 compatible since 2012.
> >
> >
> > Cheers,
> > Luke Maycock
> > OLIVER WYMAN
> > luke.mayc...@affiliate.oliverwyman.com > luke.mayc...@affiliate.oliverwyman.com>
> > www.oliverwyman.com
> >
> >
> >
> > 
> > From: Dennis O'Brien 
> > Sent: 01 December 2016 04:45
> > To: dev@airflow.incubator.apache.org
> > Subject: Re: Python 3
> >
> > Hi Luke,
> >
> > I've been running Airflow on Python 3.5 with Celery.  I'm using Redis as
> > the message broker (and ElastiCache Redis in production).  I haven't
> tried
> > RabbitMQ so I can't speak to its compatibility with Python 3.5.
> >
> > On Mon, Nov 28, 2016 at 9:18 AM Maycock, Luke <
> > luke.mayc...@affiliate.oliverwyman.com> wrote:
> >
> > > I thought it would be worth following up on this given our experience
> of
> > > testing Python 3 compatibility.
> > >
> > >
> > > We are using Celery and RabbitMQ. Celery uses librabbitmq for
> interaction
> > > with RabbitMQ but librabbitmq is not compatible with Python 3 (see
> > > https://github.com/celery/librabbitmq/issues/13). Has anyone else been
> > > able to get Airflow/Python 3/Celery working?
> > >
> > >
> > >
> > > Cheers,
> > > Luke Maycock
> > > OLIVER WYMAN
> > > luke.mayc...@affiliate.oliverwyman.com > > luke.mayc...@affiliate.oliverwyman.com>
> > > www.oliverwyman.com
> > >
> > >
> > >
> > > 
> > > From: Maycock, Luke 
> > > Sent: 28 November 2016 12:33
> > > To: dev@airflow.incubator.apache.org
> > > Subject: Re: Python 3
> > >
> > > Thank you Dmitriy and Li.
> > >
> > > Luke Maycock
> > > OLIVER WYMAN
> > > luke.mayc...@affiliate.oliverwyman.com > > luke.mayc...@affiliate.oliverwyman.com>
> > > www.oliverwyman.com
> > >
> > >
> > >
> > > 
> > > From: Li Xuan Ji 
> > > Sent: 28 November 2016 12:20
> > > To: dev@airflow.incubator.apache.org
> > > Subject: Re: Python 3
> > >
> > > Airflow is supposed to be Python 3 compatible and any
> > > incompatibilities are bugs on our side.
> > >
> > > I haven't heard of anyone running airflow on windows, but I'm not sure
> > > what the official stance is.
> > >
> > > On 28 November 2016 at 07:01, Dmitriy Krasnikov <
> dkrasni...@hotmail.com>
> > > wrote:
> > > > I have been using it and developing plugins for it in Python 3.5 with
> > no
> > > problems.
> > > >
> > > >
> > > > -Original Message-
> > > > From: Maycock, Luke [mailto:luke.mayc...@affiliate.oliverwyman.com]
> > > > Sent: Monday, November 28, 2016 6:58 AM
> > > > To: dev@airflow.incubator.apache.org
> > > > Subject: Python 3
> > > >
> > > > Hi All,
> > > >
> > > >
> > > > I have been using Airflow for a while now and have a couple of
> > questions
> > > that I am hoping someone knows the answer to:
> > > >
> > > >  1. Is Airflow now Python 3 compatible? The documentation used to
> state
> > > that Airflow was only compatible with Python 2.7 (<
> > >
> > https://web.archive.org/web/20160310111640/http://
> pythonhosted.org/airflow/start.html
> > > >
> > >
> > https://web.archive.org/web/20160502024539/http://
> pythonhosted.org/airflow/start.html
> > )
> > > but that statement has since been removed.
> > > >  2.  I know Gunicorn does not run on Windows - is this the only
> > > component of Airflow that does not run on Windows?
> > > >
> > > >
> > > > Cheers,
> > > > Luke Maycock
> > > > OLIVER WYMAN
> > > > luke.mayc...@affiliate.oliverwyman.com > > luke.mayc...@affiliate.oliverwyman.com>
> > > > www.oliverwyman.com
> > > >
> > > >
> > > > 
> > > > This e-mail and any attachments may be confidential or legally
> > > privileged. If you received this message in error or are not the
> intended
> > > recipient, you should destroy the e-mail message and any attachments or
> > > copies, and you are prohibited from retaining, distributing, disclosing
> > or
> > > using any information contained herein. Please inform us of the
> erroneous
> > > delivery by return e-mail. Thank you for your cooperation.
> > >
> > >
> > >
> > > --
> > > 

Re: Subsequent Airflow Meetup: 2017/01/11

[Dan Davydov]

Confirmed.

On Sun, Jan 1, 2017 at 9:16 PM, George Leslie-Waksman <
geo...@cloverhealth.com.invalid> wrote:

> Sorry for the delayed response, end of year and holidays stole my attention
> for a bit.
>
> With the new year, I was just looking to pick things back up and solicit
> presenters for the meetup. Given we're looking for two more, and
> Dan(Airbnb) and Kevin(Agari) have already expressed interest, I'd be happy
> to give them the spots.
>
> I hope the delay in my response isn't too much of an inconvenience for
> anyone. Dan, Kevin: confirm and I'll add you to the line up.
>
> --George
>
> On Sun, Nov 20, 2016 at 8:44 PM siddharth anand  wrote:
>
> > I suspect Clover Health is extremely busy with all of the benefit
> > enrollments going on right now..
> >
> > George,
> > When you come up for air, it looks like both Dan(Airbnb) and Kevin(Agari)
> > have talk ideas.
> >
> > -s
> >
> > On Wed, Nov 16, 2016 at 11:50 PM, Dan Davydov <
> > dan.davy...@airbnb.com.invalid> wrote:
> >
> > > Based on chatting with a couple of people today at the Airflow meet-up
> I
> > > think there has been some demand for an airflow operations talk,
> > > specifically around monitoring/alerting. If there is still room I can
> > give
> > > a talk about this, let me know George.
> > >
> > > On Thu, Nov 10, 2016 at 10:17 AM, siddharth anand 
> > > wrote:
> > >
> > > > Kevin,
> > > > Here's a link to the 1Q17 meet-up.
> > > >
> > https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/
> > > > 235259523/
> > > >
> > > > Both upcoming meet-ups (next week at WePay and 1Q17 at Clover Health)
> > can
> > > > be found on http://www.meetup.com/Bay-Area-Apache-Airflow-
> > > > Incubating-Meetup/
> > > >
> > > > -s
> > > >
> > > >
> > > > On Wed, Nov 9, 2016 at 4:24 PM, Kevin Mandich <
> kevinmand...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi George,
> > > > >
> > > > > If there is still room, I'd like to give a talk about how we use
> > > Airflow
> > > > at
> > > > > my company, Agari. We are a data company that is working to
> eliminate
> > > > > inbound, targeted e-mail attacks to our customers
> (spear-phishing). I
> > > am
> > > > > currently working as a data scientist who is also responsible for
> > > > shipping
> > > > > my work to production.
> > > > >
> > > > > We currently use Airflow to build models from our telemetry data
> > which
> > > > are
> > > > > then used for scoring in our near-real-time pipeline. I'd like to
> > talk
> > > > > about some of the DAGs we've set up to do this.
> > > > >
> > > > > Please let me know if this sounds reasonable. Thank you,
> > > > >
> > > > > Kevin Mandich
> > > > > Agari Data, Inc.
> > > > >
> > > > >
> > > > > On Mon, Oct 31, 2016 at 11:27 PM, George Leslie-Waksman <
> > > > > geo...@cloverhealth.com.invalid> wrote:
> > > > >
> > > > > > I know it's a bit far in advance, but to make sure there's space
> > (and
> > > > > food
> > > > > > and drink), I've scheduled and booked the subsequent meetup for
> > > January
> > > > > > 11th at Clover Health in SF.
> > > > > >
> > > > > > If anyone wants to volunteer to talk, let me know, otherwise I'll
> > > > > probably
> > > > > > start bugging folks sometime after Thanksgiving and before the
> > > December
> > > > > > holidays.
> > > > > >
> > > > > > --George Leslie-Waksman
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Airflow 1.8.0 Alpha 1

[Dan Davydov]

I have also started on this effort, recently Alex Guziel and I have been
pushing Airbnb's custom cherries onto master to get Airbnb back onto master
in order for us to do a release.

I think it might make sense to wait for these two commits to get merged in
since they would be quite nice to have for all Airflow users and seem like
they will be merged soon:
Schedule all pending DAG runs in a single scheduler loop -
https://github.com/apache/incubator-airflow/pull/1906
Add Support for dag.backfill=(True|False) Option -
https://github.com/apache/incubator-airflow/pull/1830
Impersonation Support + Cgroups - https://github.com/apache/
incubator-airflow/pull/1934 (this is kind of important from the Airbnb side
so that we can help test the new master without having to cherrypick this
PR on top of it which would make the testing unreliable for others).

If there are PRs that affect the core of Airflow that other committers
think are important to merge we could include these too. I can commit to
pushing out the Impersonation/Cgroups PR this week pending PR comments.
What do you think Bolke?

On Tue, Jan 3, 2017 at 4:26 AM, Bolke de Bruin  wrote:

> Hey Alex,
>
> I have noticed the same, and it is also the reason why we have Alpha
> versions. For now I have noticed the following:
>
> * Tasks can get in limbo between scheduler and executor:
> https://github.com/apache/incubator-airflow/pull/1948 <
> https://github.com/apache/incubator-airflow/pull/1948>
> * Try_number not increased due to reset in LocalTaskJob:
> https://github.com/apache/incubator-airflow/pull/1969 <
> https://github.com/apache/incubator-airflow/pull/1969>
> * one_failed trigger not executed
>
> My idea is to move to a Samba style of releases eventually, but for now I
> would like to get master into a state that we understand and therefore not
> accept any patches that do not address any bugs.
>
> If you (or anyone else) can review the above PRs and add your own as well
> then I can create another Alpha version. I’ll be on gitter as much as I can
> so we can speed up if needed.
>
> - Bolke
>
> > On 3 Jan 2017, at 08:51, Alex Van Boxel  wrote:
> >
> > Hey Bolke,
> >
> > thanks for getting this moving. But I already have some blockers, since I
> > moved up master to this release (moved from end November to now)
> stability
> > has gone down (certainly on Celary). I'm trying to identify the core
> > problems and see if I can fix them.
> >
> > On Sat, Dec 31, 2016 at 9:52 PM Bolke de Bruin  > wrote:
> >
> > Dear All,
> >
> > On the verge of the New Year, I decided to be a little bit cheeky and to
> > make available an Airflow 1.8.0 Alpha 1. We have been talking about it
> for
> > a long time now and by doing this I wanted bootstrap the process. It
> should
> > by no means be considered an Apache release yet. This is for testing
> > purposes in the dev community around Airflow, nothing else.
> >
> > The build is exactly the same as the state of master (git 410736d) plus
> the
> > change to version “1.8.0.alpha1” in version.py.
> >
> > I am dedicating quite some time next week and beyond to get a release
> out.
> > Hopefully we can get some help with testing, changelog etc. To make this
> > possible I would like to propose a freeze to adding new features for at
> > least two weeks - say until Jan 15.
> >
> > You can find the tar here: http://people.apache.org/~bolke/ <
> > http://people.apache.org/~bolke/ > .
> It isn’t signed. Following versions
> > will be. SHA is available.
> >
> > Lastly, Alpha 1 does not have the fix for retries yet. So we will get an
> > Alpha 2 :-). @Max / @Dan / @Paul: a potential fix is in
> > https://github.com/apache/incubator-airflow/pull/1948 <
> https://github.com/apache/incubator-airflow/pull/1948> <
> > https://github.com/apache/incubator-airflow/pull/1948 <
> https://github.com/apache/incubator-airflow/pull/1948>> , but your
> feedback
> > is required as it is entrenched in new processing code that you are
> running
> > in production afaik - so I wonder what happens in your fork.
> >
> > Happy New Year!
> >
> > Bolke
> >
> >
> >
> > --
> >  _/
> > _/ Alex Van Boxel
>
>

Re: Airflow 1.8.0 Alpha 1

[Bolke de Bruin]

Hey Alex,

I have noticed the same, and it is also the reason why we have Alpha versions. 
For now I have noticed the following:

* Tasks can get in limbo between scheduler and executor: 
https://github.com/apache/incubator-airflow/pull/1948 

* Try_number not increased due to reset in LocalTaskJob: 
https://github.com/apache/incubator-airflow/pull/1969 

* one_failed trigger not executed

My idea is to move to a Samba style of releases eventually, but for now I would 
like to get master into a state that we understand and therefore not accept any 
patches that do not address any bugs.

If you (or anyone else) can review the above PRs and add your own as well then 
I can create another Alpha version. I’ll be on gitter as much as I can so we 
can speed up if needed.

- Bolke

> On 3 Jan 2017, at 08:51, Alex Van Boxel  wrote:
> 
> Hey Bolke,
> 
> thanks for getting this moving. But I already have some blockers, since I
> moved up master to this release (moved from end November to now) stability
> has gone down (certainly on Celary). I'm trying to identify the core
> problems and see if I can fix them.
> 
> On Sat, Dec 31, 2016 at 9:52 PM Bolke de Bruin  > wrote:
> 
> Dear All,
> 
> On the verge of the New Year, I decided to be a little bit cheeky and to
> make available an Airflow 1.8.0 Alpha 1. We have been talking about it for
> a long time now and by doing this I wanted bootstrap the process. It should
> by no means be considered an Apache release yet. This is for testing
> purposes in the dev community around Airflow, nothing else.
> 
> The build is exactly the same as the state of master (git 410736d) plus the
> change to version “1.8.0.alpha1” in version.py.
> 
> I am dedicating quite some time next week and beyond to get a release out.
> Hopefully we can get some help with testing, changelog etc. To make this
> possible I would like to propose a freeze to adding new features for at
> least two weeks - say until Jan 15.
> 
> You can find the tar here: http://people.apache.org/~bolke/ <
> http://people.apache.org/~bolke/ > . It 
> isn’t signed. Following versions
> will be. SHA is available.
> 
> Lastly, Alpha 1 does not have the fix for retries yet. So we will get an
> Alpha 2 :-). @Max / @Dan / @Paul: a potential fix is in
> https://github.com/apache/incubator-airflow/pull/1948 
>  <
> https://github.com/apache/incubator-airflow/pull/1948 
> > , but your feedback
> is required as it is entrenched in new processing code that you are running
> in production afaik - so I wonder what happens in your fork.
> 
> Happy New Year!
> 
> Bolke
> 
> 
> 
> -- 
>  _/
> _/ Alex Van Boxel