Airflow 1.8.0 Release Candidate 1

[Bolke de Bruin]

Hi All,

I have made the (first) RELEASE CANDIDATE of Airflow 1.8.0 available at: 
https://dist.apache.org/repos/dist/dev/incubator/airflow/ , public keys are 
available at https://dist.apache.org/repos/dist/release/incubator/airflow/ . It 
is tagged with a local version “apache.incubating” so it allows upgrading from 
earlier releases. This should be considered of  release quality, but not yet 
officially vetted as a release yet.

Issues fixed:
* Use static nvd3 and d3
* Python 3 incompatibilities
* CLI API trigger dag issue

As the difference between beta 5 and the release candidate is relatively small 
I hope to start the VOTE for releasing 1.8.0 quite soon (2 days?), if the vote 
passes also a vote needs to happen at the IPMC mailinglist. As this is our 
first Apache release I expect some comments and required changes and probably a 
RC 2.

Furthermore, we now have a “v1-8-stable” branch. This has version “1.8.0rc1” 
and will graduate to “1.8.0” when we release. The “v1-8-test” branch now has 
version “1.8.1alpha0” as version and “master” has version “1.9.0dev0”. Note 
that “v1-8-stable” is now closed. This means that, per release guidelines, 
patches accompanied with an ASSIGNED Jira and a sign-off from a committer. Only 
then the release manager applies the patch to stable (In this case that would 
be me). The release manager then closes the bug when the patches have landed in 
the appropriate branches. For more information please see: 
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Release+Planning+and+Supported+Release+Lifetime
 

 .

Any questions or suggestions don’t hesitate to ask!

Cheers
Bolke

Re: Airflow 1.8.0 Release Candidate 1

[Arthur Wiedmer]

Bolke,

Thank you again for leading this effort. This has been quite the journey.

Best,
Arthur

On Thu, Feb 2, 2017 at 11:50 AM, Bolke de Bruin  wrote:

> Hi All,
>
> I have made the (first) RELEASE CANDIDATE of Airflow 1.8.0 available at:
> https://dist.apache.org/repos/dist/dev/incubator/airflow/ , public keys
> are available at https://dist.apache.org/repos/dist/release/incubator/
> airflow/ . It is tagged with a local version “apache.incubating” so it
> allows upgrading from earlier releases. This should be considered of
> release quality, but not yet officially vetted as a release yet.
>
> Issues fixed:
> * Use static nvd3 and d3
> * Python 3 incompatibilities
> * CLI API trigger dag issue
>
> As the difference between beta 5 and the release candidate is relatively
> small I hope to start the VOTE for releasing 1.8.0 quite soon (2 days?), if
> the vote passes also a vote needs to happen at the IPMC mailinglist. As
> this is our first Apache release I expect some comments and required
> changes and probably a RC 2.
>
> Furthermore, we now have a “v1-8-stable” branch. This has version
> “1.8.0rc1” and will graduate to “1.8.0” when we release. The “v1-8-test”
> branch now has version “1.8.1alpha0” as version and “master” has version
> “1.9.0dev0”. Note that “v1-8-stable” is now closed. This means that, per
> release guidelines, patches accompanied with an ASSIGNED Jira and a
> sign-off from a committer. Only then the release manager applies the patch
> to stable (In this case that would be me). The release manager then closes
> the bug when the patches have landed in the appropriate branches. For more
> information please see: https://cwiki.apache.org/
> confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> Supported+Release+Lifetime  confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> Supported+Release+Lifetime> .
>
> Any questions or suggestions don’t hesitate to ask!
>
> Cheers
> Bolke

Re: Airflow 1.8.0 Release Candidate 1

[Dan Davydov]

+1, this requires a lot more work than appears on the surface.

On Thu, Feb 2, 2017 at 12:43 PM, Arthur Wiedmer 
wrote:

> Bolke,
>
> Thank you again for leading this effort. This has been quite the journey.
>
> Best,
> Arthur
>
> On Thu, Feb 2, 2017 at 11:50 AM, Bolke de Bruin  wrote:
>
> > Hi All,
> >
> > I have made the (first) RELEASE CANDIDATE of Airflow 1.8.0 available at:
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/ , public keys
> > are available at https://dist.apache.org/repos/dist/release/incubator/
> > airflow/ . It is tagged with a local version “apache.incubating” so it
> > allows upgrading from earlier releases. This should be considered of
> > release quality, but not yet officially vetted as a release yet.
> >
> > Issues fixed:
> > * Use static nvd3 and d3
> > * Python 3 incompatibilities
> > * CLI API trigger dag issue
> >
> > As the difference between beta 5 and the release candidate is relatively
> > small I hope to start the VOTE for releasing 1.8.0 quite soon (2 days?),
> if
> > the vote passes also a vote needs to happen at the IPMC mailinglist. As
> > this is our first Apache release I expect some comments and required
> > changes and probably a RC 2.
> >
> > Furthermore, we now have a “v1-8-stable” branch. This has version
> > “1.8.0rc1” and will graduate to “1.8.0” when we release. The “v1-8-test”
> > branch now has version “1.8.1alpha0” as version and “master” has version
> > “1.9.0dev0”. Note that “v1-8-stable” is now closed. This means that, per
> > release guidelines, patches accompanied with an ASSIGNED Jira and a
> > sign-off from a committer. Only then the release manager applies the
> patch
> > to stable (In this case that would be me). The release manager then
> closes
> > the bug when the patches have landed in the appropriate branches. For
> more
> > information please see: https://cwiki.apache.org/
> > confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> > Supported+Release+Lifetime  > confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> > Supported+Release+Lifetime> .
> >
> > Any questions or suggestions don’t hesitate to ask!
> >
> > Cheers
> > Bolke
>

Re: Airflow 1.8.0 Release Candidate 1

[Boris Tyukin]

Bolke, you are our hero! I am sure you put a lot of your time to make it
happen

On Thu, Feb 2, 2017 at 2:50 PM, Bolke de Bruin  wrote:

> Hi All,
>
> I have made the (first) RELEASE CANDIDATE of Airflow 1.8.0 available at:
> https://dist.apache.org/repos/dist/dev/incubator/airflow/ , public keys
> are available at https://dist.apache.org/repos/dist/release/incubator/
> airflow/ . It is tagged with a local version “apache.incubating” so it
> allows upgrading from earlier releases. This should be considered of
> release quality, but not yet officially vetted as a release yet.
>
> Issues fixed:
> * Use static nvd3 and d3
> * Python 3 incompatibilities
> * CLI API trigger dag issue
>
> As the difference between beta 5 and the release candidate is relatively
> small I hope to start the VOTE for releasing 1.8.0 quite soon (2 days?), if
> the vote passes also a vote needs to happen at the IPMC mailinglist. As
> this is our first Apache release I expect some comments and required
> changes and probably a RC 2.
>
> Furthermore, we now have a “v1-8-stable” branch. This has version
> “1.8.0rc1” and will graduate to “1.8.0” when we release. The “v1-8-test”
> branch now has version “1.8.1alpha0” as version and “master” has version
> “1.9.0dev0”. Note that “v1-8-stable” is now closed. This means that, per
> release guidelines, patches accompanied with an ASSIGNED Jira and a
> sign-off from a committer. Only then the release manager applies the patch
> to stable (In this case that would be me). The release manager then closes
> the bug when the patches have landed in the appropriate branches. For more
> information please see: https://cwiki.apache.org/
> confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> Supported+Release+Lifetime  confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> Supported+Release+Lifetime> .
>
> Any questions or suggestions don’t hesitate to ask!
>
> Cheers
> Bolke

Re: Airflow 1.8.0 Release Candidate 1

[Maxime Beauchemin]

The Apache mailing doesn't support images so here's a link:

http://i.imgur.com/DUkpjZu.png
​

On Thu, Feb 2, 2017 at 12:52 PM, Boris Tyukin  wrote:

> Bolke, you are our hero! I am sure you put a lot of your time to make it
> happen
>
> On Thu, Feb 2, 2017 at 2:50 PM, Bolke de Bruin  wrote:
>
> > Hi All,
> >
> > I have made the (first) RELEASE CANDIDATE of Airflow 1.8.0 available at:
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/ , public keys
> > are available at https://dist.apache.org/repos/dist/release/incubator/
> > airflow/ . It is tagged with a local version “apache.incubating” so it
> > allows upgrading from earlier releases. This should be considered of
> > release quality, but not yet officially vetted as a release yet.
> >
> > Issues fixed:
> > * Use static nvd3 and d3
> > * Python 3 incompatibilities
> > * CLI API trigger dag issue
> >
> > As the difference between beta 5 and the release candidate is relatively
> > small I hope to start the VOTE for releasing 1.8.0 quite soon (2 days?),
> if
> > the vote passes also a vote needs to happen at the IPMC mailinglist. As
> > this is our first Apache release I expect some comments and required
> > changes and probably a RC 2.
> >
> > Furthermore, we now have a “v1-8-stable” branch. This has version
> > “1.8.0rc1” and will graduate to “1.8.0” when we release. The “v1-8-test”
> > branch now has version “1.8.1alpha0” as version and “master” has version
> > “1.9.0dev0”. Note that “v1-8-stable” is now closed. This means that, per
> > release guidelines, patches accompanied with an ASSIGNED Jira and a
> > sign-off from a committer. Only then the release manager applies the
> patch
> > to stable (In this case that would be me). The release manager then
> closes
> > the bug when the patches have landed in the appropriate branches. For
> more
> > information please see: https://cwiki.apache.org/
> > confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> > Supported+Release+Lifetime  > confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> > Supported+Release+Lifetime> .
> >
> > Any questions or suggestions don’t hesitate to ask!
> >
> > Cheers
> > Bolke
>

Re: Airflow 1.8.0 Release Candidate 1

[Boris Tyukin]

LOL awesome!

On Thu, Feb 2, 2017 at 4:00 PM, Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:

> The Apache mailing doesn't support images so here's a link:
>
> http://i.imgur.com/DUkpjZu.png
> ​
>
> On Thu, Feb 2, 2017 at 12:52 PM, Boris Tyukin 
> wrote:
>
> > Bolke, you are our hero! I am sure you put a lot of your time to make it
> > happen
> >
> > On Thu, Feb 2, 2017 at 2:50 PM, Bolke de Bruin 
> wrote:
> >
> > > Hi All,
> > >
> > > I have made the (first) RELEASE CANDIDATE of Airflow 1.8.0 available
> at:
> > > https://dist.apache.org/repos/dist/dev/incubator/airflow/ , public
> keys
> > > are available at https://dist.apache.org/repos/dist/release/incubator/
> > > airflow/ . It is tagged with a local version “apache.incubating” so it
> > > allows upgrading from earlier releases. This should be considered of
> > > release quality, but not yet officially vetted as a release yet.
> > >
> > > Issues fixed:
> > > * Use static nvd3 and d3
> > > * Python 3 incompatibilities
> > > * CLI API trigger dag issue
> > >
> > > As the difference between beta 5 and the release candidate is
> relatively
> > > small I hope to start the VOTE for releasing 1.8.0 quite soon (2
> days?),
> > if
> > > the vote passes also a vote needs to happen at the IPMC mailinglist. As
> > > this is our first Apache release I expect some comments and required
> > > changes and probably a RC 2.
> > >
> > > Furthermore, we now have a “v1-8-stable” branch. This has version
> > > “1.8.0rc1” and will graduate to “1.8.0” when we release. The
> “v1-8-test”
> > > branch now has version “1.8.1alpha0” as version and “master” has
> version
> > > “1.9.0dev0”. Note that “v1-8-stable” is now closed. This means that,
> per
> > > release guidelines, patches accompanied with an ASSIGNED Jira and a
> > > sign-off from a committer. Only then the release manager applies the
> > patch
> > > to stable (In this case that would be me). The release manager then
> > closes
> > > the bug when the patches have landed in the appropriate branches. For
> > more
> > > information please see: https://cwiki.apache.org/
> > > confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> > > Supported+Release+Lifetime  > > confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> > > Supported+Release+Lifetime> .
> > >
> > > Any questions or suggestions don’t hesitate to ask!
> > >
> > > Cheers
> > > Bolke
> >
>

Re: Airflow 1.8.0 Release Candidate 1

[Bolke de Bruin]

Now I am blushing :-)

Sent from my iPhone

> On 2 Feb 2017, at 22:05, Boris Tyukin  wrote:
> 
> LOL awesome!
> 
> On Thu, Feb 2, 2017 at 4:00 PM, Maxime Beauchemin <
> maximebeauche...@gmail.com> wrote:
> 
>> The Apache mailing doesn't support images so here's a link:
>> 
>> http://i.imgur.com/DUkpjZu.png
>> ​
>> 
>> On Thu, Feb 2, 2017 at 12:52 PM, Boris Tyukin 
>> wrote:
>> 
>>> Bolke, you are our hero! I am sure you put a lot of your time to make it
>>> happen
>>> 
>>> On Thu, Feb 2, 2017 at 2:50 PM, Bolke de Bruin 
>> wrote:
>>> 
 Hi All,
 
 I have made the (first) RELEASE CANDIDATE of Airflow 1.8.0 available
>> at:
 https://dist.apache.org/repos/dist/dev/incubator/airflow/ , public
>> keys
 are available at https://dist.apache.org/repos/dist/release/incubator/
 airflow/ . It is tagged with a local version “apache.incubating” so it
 allows upgrading from earlier releases. This should be considered of
 release quality, but not yet officially vetted as a release yet.
 
 Issues fixed:
 * Use static nvd3 and d3
 * Python 3 incompatibilities
 * CLI API trigger dag issue
 
 As the difference between beta 5 and the release candidate is
>> relatively
 small I hope to start the VOTE for releasing 1.8.0 quite soon (2
>> days?),
>>> if
 the vote passes also a vote needs to happen at the IPMC mailinglist. As
 this is our first Apache release I expect some comments and required
 changes and probably a RC 2.
 
 Furthermore, we now have a “v1-8-stable” branch. This has version
 “1.8.0rc1” and will graduate to “1.8.0” when we release. The
>> “v1-8-test”
 branch now has version “1.8.1alpha0” as version and “master” has
>> version
 “1.9.0dev0”. Note that “v1-8-stable” is now closed. This means that,
>> per
 release guidelines, patches accompanied with an ASSIGNED Jira and a
 sign-off from a committer. Only then the release manager applies the
>>> patch
 to stable (In this case that would be me). The release manager then
>>> closes
 the bug when the patches have landed in the appropriate branches. For
>>> more
 information please see: https://cwiki.apache.org/
 confluence/display/AIRFLOW/Airflow+Release+Planning+and+
 Supported+Release+Lifetime  .
 
 Any questions or suggestions don’t hesitate to ask!
 
 Cheers
 Bolke
>>> 
>> 

Re: Airflow 1.8.0 Release Candidate 1

[Jeremiah Lowin]

Fantastic work on this Bolke, thank you!

We've deployed the RC and will report if there are any issues...

On Thu, Feb 2, 2017 at 4:32 PM Bolke de Bruin  wrote:

> Now I am blushing :-)
>
> Sent from my iPhone
>
> > On 2 Feb 2017, at 22:05, Boris Tyukin  wrote:
> >
> > LOL awesome!
> >
> > On Thu, Feb 2, 2017 at 4:00 PM, Maxime Beauchemin <
> > maximebeauche...@gmail.com> wrote:
> >
> >> The Apache mailing doesn't support images so here's a link:
> >>
> >> http://i.imgur.com/DUkpjZu.png
> >> ​
> >>
> >> On Thu, Feb 2, 2017 at 12:52 PM, Boris Tyukin 
> >> wrote:
> >>
> >>> Bolke, you are our hero! I am sure you put a lot of your time to make
> it
> >>> happen
> >>>
> >>> On Thu, Feb 2, 2017 at 2:50 PM, Bolke de Bruin 
> >> wrote:
> >>>
>  Hi All,
> 
>  I have made the (first) RELEASE CANDIDATE of Airflow 1.8.0 available
> >> at:
>  https://dist.apache.org/repos/dist/dev/incubator/airflow/ , public
> >> keys
>  are available at
> https://dist.apache.org/repos/dist/release/incubator/
>  airflow/ . It is tagged with a local version “apache.incubating” so it
>  allows upgrading from earlier releases. This should be considered of
>  release quality, but not yet officially vetted as a release yet.
> 
>  Issues fixed:
>  * Use static nvd3 and d3
>  * Python 3 incompatibilities
>  * CLI API trigger dag issue
> 
>  As the difference between beta 5 and the release candidate is
> >> relatively
>  small I hope to start the VOTE for releasing 1.8.0 quite soon (2
> >> days?),
> >>> if
>  the vote passes also a vote needs to happen at the IPMC mailinglist.
> As
>  this is our first Apache release I expect some comments and required
>  changes and probably a RC 2.
> 
>  Furthermore, we now have a “v1-8-stable” branch. This has version
>  “1.8.0rc1” and will graduate to “1.8.0” when we release. The
> >> “v1-8-test”
>  branch now has version “1.8.1alpha0” as version and “master” has
> >> version
>  “1.9.0dev0”. Note that “v1-8-stable” is now closed. This means that,
> >> per
>  release guidelines, patches accompanied with an ASSIGNED Jira and a
>  sign-off from a committer. Only then the release manager applies the
> >>> patch
>  to stable (In this case that would be me). The release manager then
> >>> closes
>  the bug when the patches have landed in the appropriate branches. For
> >>> more
>  information please see: https://cwiki.apache.org/
>  confluence/display/AIRFLOW/Airflow+Release+Planning+and+
>  Supported+Release+Lifetime   confluence/display/AIRFLOW/Airflow+Release+Planning+and+
>  Supported+Release+Lifetime> .
> 
>  Any questions or suggestions don’t hesitate to ask!
> 
>  Cheers
>  Bolke
> >>>
> >>
>

Re: Airflow 1.8.0 Release Candidate 1

[Jayesh Senjaliya]

Thank You Bolke for all the efforts you are putting in !!

I have deployed this RC now.

On Thu, Feb 2, 2017 at 3:02 PM, Jeremiah Lowin  wrote:

> Fantastic work on this Bolke, thank you!
>
> We've deployed the RC and will report if there are any issues...
>
> On Thu, Feb 2, 2017 at 4:32 PM Bolke de Bruin  wrote:
>
> > Now I am blushing :-)
> >
> > Sent from my iPhone
> >
> > > On 2 Feb 2017, at 22:05, Boris Tyukin  wrote:
> > >
> > > LOL awesome!
> > >
> > > On Thu, Feb 2, 2017 at 4:00 PM, Maxime Beauchemin <
> > > maximebeauche...@gmail.com> wrote:
> > >
> > >> The Apache mailing doesn't support images so here's a link:
> > >>
> > >> http://i.imgur.com/DUkpjZu.png
> > >> ​
> > >>
> > >> On Thu, Feb 2, 2017 at 12:52 PM, Boris Tyukin 
> > >> wrote:
> > >>
> > >>> Bolke, you are our hero! I am sure you put a lot of your time to make
> > it
> > >>> happen
> > >>>
> > >>> On Thu, Feb 2, 2017 at 2:50 PM, Bolke de Bruin 
> > >> wrote:
> > >>>
> >  Hi All,
> > 
> >  I have made the (first) RELEASE CANDIDATE of Airflow 1.8.0 available
> > >> at:
> >  https://dist.apache.org/repos/dist/dev/incubator/airflow/ , public
> > >> keys
> >  are available at
> > https://dist.apache.org/repos/dist/release/incubator/
> >  airflow/ . It is tagged with a local version “apache.incubating” so
> it
> >  allows upgrading from earlier releases. This should be considered of
> >  release quality, but not yet officially vetted as a release yet.
> > 
> >  Issues fixed:
> >  * Use static nvd3 and d3
> >  * Python 3 incompatibilities
> >  * CLI API trigger dag issue
> > 
> >  As the difference between beta 5 and the release candidate is
> > >> relatively
> >  small I hope to start the VOTE for releasing 1.8.0 quite soon (2
> > >> days?),
> > >>> if
> >  the vote passes also a vote needs to happen at the IPMC mailinglist.
> > As
> >  this is our first Apache release I expect some comments and required
> >  changes and probably a RC 2.
> > 
> >  Furthermore, we now have a “v1-8-stable” branch. This has version
> >  “1.8.0rc1” and will graduate to “1.8.0” when we release. The
> > >> “v1-8-test”
> >  branch now has version “1.8.1alpha0” as version and “master” has
> > >> version
> >  “1.9.0dev0”. Note that “v1-8-stable” is now closed. This means that,
> > >> per
> >  release guidelines, patches accompanied with an ASSIGNED Jira and a
> >  sign-off from a committer. Only then the release manager applies the
> > >>> patch
> >  to stable (In this case that would be me). The release manager then
> > >>> closes
> >  the bug when the patches have landed in the appropriate branches.
> For
> > >>> more
> >  information please see: https://cwiki.apache.org/
> >  confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> >  Supported+Release+Lifetime  >  confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> >  Supported+Release+Lifetime> .
> > 
> >  Any questions or suggestions don’t hesitate to ask!
> > 
> >  Cheers
> >  Bolke
> > >>>
> > >>
> >
>

Re: Airflow 1.8.0 Release Candidate 1

[Jeremiah Lowin]

For what it's worth -- everything running smoothly after 24+ hours in a
production(ish) environment.

On Thu, Feb 2, 2017 at 11:25 PM Jayesh Senjaliya 
wrote:

> Thank You Bolke for all the efforts you are putting in !!
>
> I have deployed this RC now.
>
> On Thu, Feb 2, 2017 at 3:02 PM, Jeremiah Lowin  wrote:
>
> > Fantastic work on this Bolke, thank you!
> >
> > We've deployed the RC and will report if there are any issues...
> >
> > On Thu, Feb 2, 2017 at 4:32 PM Bolke de Bruin  wrote:
> >
> > > Now I am blushing :-)
> > >
> > > Sent from my iPhone
> > >
> > > > On 2 Feb 2017, at 22:05, Boris Tyukin  wrote:
> > > >
> > > > LOL awesome!
> > > >
> > > > On Thu, Feb 2, 2017 at 4:00 PM, Maxime Beauchemin <
> > > > maximebeauche...@gmail.com> wrote:
> > > >
> > > >> The Apache mailing doesn't support images so here's a link:
> > > >>
> > > >> http://i.imgur.com/DUkpjZu.png
> > > >> ​
> > > >>
> > > >> On Thu, Feb 2, 2017 at 12:52 PM, Boris Tyukin <
> bo...@boristyukin.com>
> > > >> wrote:
> > > >>
> > > >>> Bolke, you are our hero! I am sure you put a lot of your time to
> make
> > > it
> > > >>> happen
> > > >>>
> > > >>> On Thu, Feb 2, 2017 at 2:50 PM, Bolke de Bruin 
> > > >> wrote:
> > > >>>
> > >  Hi All,
> > > 
> > >  I have made the (first) RELEASE CANDIDATE of Airflow 1.8.0
> available
> > > >> at:
> > >  https://dist.apache.org/repos/dist/dev/incubator/airflow/ ,
> public
> > > >> keys
> > >  are available at
> > > https://dist.apache.org/repos/dist/release/incubator/
> > >  airflow/ . It is tagged with a local version “apache.incubating”
> so
> > it
> > >  allows upgrading from earlier releases. This should be considered
> of
> > >  release quality, but not yet officially vetted as a release yet.
> > > 
> > >  Issues fixed:
> > >  * Use static nvd3 and d3
> > >  * Python 3 incompatibilities
> > >  * CLI API trigger dag issue
> > > 
> > >  As the difference between beta 5 and the release candidate is
> > > >> relatively
> > >  small I hope to start the VOTE for releasing 1.8.0 quite soon (2
> > > >> days?),
> > > >>> if
> > >  the vote passes also a vote needs to happen at the IPMC
> mailinglist.
> > > As
> > >  this is our first Apache release I expect some comments and
> required
> > >  changes and probably a RC 2.
> > > 
> > >  Furthermore, we now have a “v1-8-stable” branch. This has version
> > >  “1.8.0rc1” and will graduate to “1.8.0” when we release. The
> > > >> “v1-8-test”
> > >  branch now has version “1.8.1alpha0” as version and “master” has
> > > >> version
> > >  “1.9.0dev0”. Note that “v1-8-stable” is now closed. This means
> that,
> > > >> per
> > >  release guidelines, patches accompanied with an ASSIGNED Jira and
> a
> > >  sign-off from a committer. Only then the release manager applies
> the
> > > >>> patch
> > >  to stable (In this case that would be me). The release manager
> then
> > > >>> closes
> > >  the bug when the patches have landed in the appropriate branches.
> > For
> > > >>> more
> > >  information please see: https://cwiki.apache.org/
> > >  confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> > >  Supported+Release+Lifetime  > >  confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> > >  Supported+Release+Lifetime> .
> > > 
> > >  Any questions or suggestions don’t hesitate to ask!
> > > 
> > >  Cheers
> > >  Bolke
> > > >>>
> > > >>
> > >
> >
>

Re: Airflow 1.8.0 Release Candidate 1

[Chris Riccomini]

Upgraded to RC1 in all environments this morning. So far so good.

On Fri, Feb 3, 2017 at 6:04 PM, Jeremiah Lowin  wrote:

> For what it's worth -- everything running smoothly after 24+ hours in a
> production(ish) environment.
>
> On Thu, Feb 2, 2017 at 11:25 PM Jayesh Senjaliya 
> wrote:
>
> > Thank You Bolke for all the efforts you are putting in !!
> >
> > I have deployed this RC now.
> >
> > On Thu, Feb 2, 2017 at 3:02 PM, Jeremiah Lowin 
> wrote:
> >
> > > Fantastic work on this Bolke, thank you!
> > >
> > > We've deployed the RC and will report if there are any issues...
> > >
> > > On Thu, Feb 2, 2017 at 4:32 PM Bolke de Bruin 
> wrote:
> > >
> > > > Now I am blushing :-)
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On 2 Feb 2017, at 22:05, Boris Tyukin 
> wrote:
> > > > >
> > > > > LOL awesome!
> > > > >
> > > > > On Thu, Feb 2, 2017 at 4:00 PM, Maxime Beauchemin <
> > > > > maximebeauche...@gmail.com> wrote:
> > > > >
> > > > >> The Apache mailing doesn't support images so here's a link:
> > > > >>
> > > > >> http://i.imgur.com/DUkpjZu.png
> > > > >> ​
> > > > >>
> > > > >> On Thu, Feb 2, 2017 at 12:52 PM, Boris Tyukin <
> > bo...@boristyukin.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Bolke, you are our hero! I am sure you put a lot of your time to
> > make
> > > > it
> > > > >>> happen
> > > > >>>
> > > > >>> On Thu, Feb 2, 2017 at 2:50 PM, Bolke de Bruin <
> bdbr...@gmail.com>
> > > > >> wrote:
> > > > >>>
> > > >  Hi All,
> > > > 
> > > >  I have made the (first) RELEASE CANDIDATE of Airflow 1.8.0
> > available
> > > > >> at:
> > > >  https://dist.apache.org/repos/dist/dev/incubator/airflow/ ,
> > public
> > > > >> keys
> > > >  are available at
> > > > https://dist.apache.org/repos/dist/release/incubator/
> > > >  airflow/ . It is tagged with a local version “apache.incubating”
> > so
> > > it
> > > >  allows upgrading from earlier releases. This should be
> considered
> > of
> > > >  release quality, but not yet officially vetted as a release yet.
> > > > 
> > > >  Issues fixed:
> > > >  * Use static nvd3 and d3
> > > >  * Python 3 incompatibilities
> > > >  * CLI API trigger dag issue
> > > > 
> > > >  As the difference between beta 5 and the release candidate is
> > > > >> relatively
> > > >  small I hope to start the VOTE for releasing 1.8.0 quite soon (2
> > > > >> days?),
> > > > >>> if
> > > >  the vote passes also a vote needs to happen at the IPMC
> > mailinglist.
> > > > As
> > > >  this is our first Apache release I expect some comments and
> > required
> > > >  changes and probably a RC 2.
> > > > 
> > > >  Furthermore, we now have a “v1-8-stable” branch. This has
> version
> > > >  “1.8.0rc1” and will graduate to “1.8.0” when we release. The
> > > > >> “v1-8-test”
> > > >  branch now has version “1.8.1alpha0” as version and “master” has
> > > > >> version
> > > >  “1.9.0dev0”. Note that “v1-8-stable” is now closed. This means
> > that,
> > > > >> per
> > > >  release guidelines, patches accompanied with an ASSIGNED Jira
> and
> > a
> > > >  sign-off from a committer. Only then the release manager applies
> > the
> > > > >>> patch
> > > >  to stable (In this case that would be me). The release manager
> > then
> > > > >>> closes
> > > >  the bug when the patches have landed in the appropriate
> branches.
> > > For
> > > > >>> more
> > > >  information please see: https://cwiki.apache.org/
> > > >  confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> > > >  Supported+Release+Lifetime  > > >  confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> > > >  Supported+Release+Lifetime> .
> > > > 
> > > >  Any questions or suggestions don’t hesitate to ask!
> > > > 
> > > >  Cheers
> > > >  Bolke
> > > > >>>
> > > > >>
> > > >
> > >
> >
>

Re: Airflow 1.8.0 Release Candidate 1

[Dan Davydov]

On the Airbnb side we should be good once https://github.com/apache/
incubator-airflow/pull/2057/ is merged.

On Mon, Feb 6, 2017 at 9:23 AM, Chris Riccomini 
wrote:

> Upgraded to RC1 in all environments this morning. So far so good.
>
> On Fri, Feb 3, 2017 at 6:04 PM, Jeremiah Lowin  wrote:
>
> > For what it's worth -- everything running smoothly after 24+ hours in a
> > production(ish) environment.
> >
> > On Thu, Feb 2, 2017 at 11:25 PM Jayesh Senjaliya 
> > wrote:
> >
> > > Thank You Bolke for all the efforts you are putting in !!
> > >
> > > I have deployed this RC now.
> > >
> > > On Thu, Feb 2, 2017 at 3:02 PM, Jeremiah Lowin 
> > wrote:
> > >
> > > > Fantastic work on this Bolke, thank you!
> > > >
> > > > We've deployed the RC and will report if there are any issues...
> > > >
> > > > On Thu, Feb 2, 2017 at 4:32 PM Bolke de Bruin 
> > wrote:
> > > >
> > > > > Now I am blushing :-)
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On 2 Feb 2017, at 22:05, Boris Tyukin 
> > wrote:
> > > > > >
> > > > > > LOL awesome!
> > > > > >
> > > > > > On Thu, Feb 2, 2017 at 4:00 PM, Maxime Beauchemin <
> > > > > > maximebeauche...@gmail.com> wrote:
> > > > > >
> > > > > >> The Apache mailing doesn't support images so here's a link:
> > > > > >>
> > > > > >> http://i.imgur.com/DUkpjZu.png
> > > > > >> ​
> > > > > >>
> > > > > >> On Thu, Feb 2, 2017 at 12:52 PM, Boris Tyukin <
> > > bo...@boristyukin.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Bolke, you are our hero! I am sure you put a lot of your time
> to
> > > make
> > > > > it
> > > > > >>> happen
> > > > > >>>
> > > > > >>> On Thu, Feb 2, 2017 at 2:50 PM, Bolke de Bruin <
> > bdbr...@gmail.com>
> > > > > >> wrote:
> > > > > >>>
> > > > >  Hi All,
> > > > > 
> > > > >  I have made the (first) RELEASE CANDIDATE of Airflow 1.8.0
> > > available
> > > > > >> at:
> > > > >  https://dist.apache.org/repos/dist/dev/incubator/airflow/ ,
> > > public
> > > > > >> keys
> > > > >  are available at
> > > > > https://dist.apache.org/repos/dist/release/incubator/
> > > > >  airflow/ . It is tagged with a local version
> “apache.incubating”
> > > so
> > > > it
> > > > >  allows upgrading from earlier releases. This should be
> > considered
> > > of
> > > > >  release quality, but not yet officially vetted as a release
> yet.
> > > > > 
> > > > >  Issues fixed:
> > > > >  * Use static nvd3 and d3
> > > > >  * Python 3 incompatibilities
> > > > >  * CLI API trigger dag issue
> > > > > 
> > > > >  As the difference between beta 5 and the release candidate is
> > > > > >> relatively
> > > > >  small I hope to start the VOTE for releasing 1.8.0 quite soon
> (2
> > > > > >> days?),
> > > > > >>> if
> > > > >  the vote passes also a vote needs to happen at the IPMC
> > > mailinglist.
> > > > > As
> > > > >  this is our first Apache release I expect some comments and
> > > required
> > > > >  changes and probably a RC 2.
> > > > > 
> > > > >  Furthermore, we now have a “v1-8-stable” branch. This has
> > version
> > > > >  “1.8.0rc1” and will graduate to “1.8.0” when we release. The
> > > > > >> “v1-8-test”
> > > > >  branch now has version “1.8.1alpha0” as version and “master”
> has
> > > > > >> version
> > > > >  “1.9.0dev0”. Note that “v1-8-stable” is now closed. This means
> > > that,
> > > > > >> per
> > > > >  release guidelines, patches accompanied with an ASSIGNED Jira
> > and
> > > a
> > > > >  sign-off from a committer. Only then the release manager
> applies
> > > the
> > > > > >>> patch
> > > > >  to stable (In this case that would be me). The release manager
> > > then
> > > > > >>> closes
> > > > >  the bug when the patches have landed in the appropriate
> > branches.
> > > > For
> > > > > >>> more
> > > > >  information please see: https://cwiki.apache.org/
> > > > >  confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> > > > >  Supported+Release+Lifetime  > > > >  confluence/display/AIRFLOW/Airflow+Release+Planning+and+
> > > > >  Supported+Release+Lifetime> .
> > > > > 
> > > > >  Any questions or suggestions don’t hesitate to ask!
> > > > > 
> > > > >  Cheers
> > > > >  Bolke
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Re: Airflow 1.8.0 Release Candidate 1

[siddharth anand]

I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
https://dist.apache.org/repos/dist/dev/incubator/airflow/airflow-1.8.0rc1+apache.incubating.tar.gz
and
then running airflow upgradedb didn't quite work. First, I thought it
completed successfully, then saw errors some tables were indeed missing. I
ran it again and encountered the following exception :

DB: postgresql://app_coust...@db-cousteau.ep.stage.agari.com:5432/airflow

[2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.

INFO  [alembic.runtime.migration] Will assume transactional DDL.

INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
211e584da130, add TI state index

INFO  [alembic.runtime.migration] Running upgrade 211e584da130 ->
64de9cddf6c9, add task fails journal table

INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
f2ca10b85618, add dag_stats table

INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
4addfa1236f1, Add fractional seconds to mysql tables

INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
8504051e801b, xcom dag task indices

INFO  [alembic.runtime.migration] Running upgrade 8504051e801b ->
5e7d17757c7a, add pid field to TaskInstance

INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
127d2bf2dfa7, Add dag_id/state index on dag_run table

/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
SAWarning: Column 'dag_stats.dag_id' is marked as a member of the primary
key for table 'dag_stats', but has no Python-side or server-side default
generator indicated, nor does it indicate 'autoincrement=True' or
'nullable=True', and no explicit value is passed.  Primary key columns
typically may not store NULL. Note that as of SQLAlchemy 1.1,
'autoincrement=True' must be indicated explicitly for composite (e.g.
multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is
expected for one of the columns in the primary key. CREATE TABLE statements
are impacted by this change as well on most backends.

Re: Airflow 1.8.0 Release Candidate 1

[siddharth anand]

Actually, I see the error is further down..

  File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line
469, in do_execute

cursor.execute(statement, parameters)

sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
column "dag_id" violates not-null constraint

DETAIL:  Failing row contains (null, running, 1, f).

 [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
%(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running',
'dirty': False}]

It looks like an autoincrement is missing for this table.


I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any version
greater than 0.9.8

-s



On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand  wrote:

> I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
> https://dist.apache.org/repos/dist/dev/incubator/airflow/
> airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
> upgradedb didn't quite work. First, I thought it completed successfully,
> then saw errors some tables were indeed missing. I ran it again and
> encountered the following exception :
>
> DB: postgresql://app_coust...@db-cousteau.ep.stage.agari.com:5432/airflow
>
> [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables
>
> INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
>
> INFO  [alembic.runtime.migration] Will assume transactional DDL.
>
> INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
> 211e584da130, add TI state index
>
> INFO  [alembic.runtime.migration] Running upgrade 211e584da130 ->
> 64de9cddf6c9, add task fails journal table
>
> INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
> f2ca10b85618, add dag_stats table
>
> INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
> 4addfa1236f1, Add fractional seconds to mysql tables
>
> INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
> 8504051e801b, xcom dag task indices
>
> INFO  [alembic.runtime.migration] Running upgrade 8504051e801b ->
> 5e7d17757c7a, add pid field to TaskInstance
>
> INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
> 127d2bf2dfa7, Add dag_id/state index on dag_run table
>
> /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
> SAWarning: Column 'dag_stats.dag_id' is marked as a member of the primary
> key for table 'dag_stats', but has no Python-side or server-side default
> generator indicated, nor does it indicate 'autoincrement=True' or
> 'nullable=True', and no explicit value is passed.  Primary key columns
> typically may not store NULL. Note that as of SQLAlchemy 1.1,
> 'autoincrement=True' must be indicated explicitly for composite (e.g.
> multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is
> expected for one of the columns in the primary key. CREATE TABLE statements
> are impacted by this change as well on most backends.
>

Re: Airflow 1.8.0 Release Candidate 1

[siddharth anand]

  Table "public.dag_stats"

 Column |  Type  | Modifiers

++---

 dag_id | character varying(250) | not null

 state  | character varying(50)  | not null

 count  | integer| not null

 dirty  | boolean| not null

Indexes:

"dag_stats_pkey" PRIMARY KEY, btree (dag_id, state)


The PKEY is a combination of 2 provided columns, so I'm wondering why
Alembic is complaining here.

On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand  wrote:

> Actually, I see the error is further down..
>
>   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
> line 469, in do_execute
>
> cursor.execute(statement, parameters)
>
> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
> column "dag_id" violates not-null constraint
>
> DETAIL:  Failing row contains (null, running, 1, f).
>
>  [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
> %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running',
> 'dirty': False}]
>
> It looks like an autoincrement is missing for this table.
>
>
> I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any
> version greater than 0.9.8
>
> -s
>
>
>
> On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand  wrote:
>
>> I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/air
>> flow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
>> upgradedb didn't quite work. First, I thought it completed successfully,
>> then saw errors some tables were indeed missing. I ran it again and
>> encountered the following exception :
>>
>> DB: postgresql://app_coust...@db-cousteau.ep.stage.agari.com:5432/airflow
>>
>> [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables
>>
>> INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
>>
>> INFO  [alembic.runtime.migration] Will assume transactional DDL.
>>
>> INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
>> 211e584da130, add TI state index
>>
>> INFO  [alembic.runtime.migration] Running upgrade 211e584da130 ->
>> 64de9cddf6c9, add task fails journal table
>>
>> INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
>> f2ca10b85618, add dag_stats table
>>
>> INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
>> 4addfa1236f1, Add fractional seconds to mysql tables
>>
>> INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
>> 8504051e801b, xcom dag task indices
>>
>> INFO  [alembic.runtime.migration] Running upgrade 8504051e801b ->
>> 5e7d17757c7a, add pid field to TaskInstance
>>
>> INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
>> 127d2bf2dfa7, Add dag_id/state index on dag_run table
>>
>> /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
>> SAWarning: Column 'dag_stats.dag_id' is marked as a member of the primary
>> key for table 'dag_stats', but has no Python-side or server-side default
>> generator indicated, nor does it indicate 'autoincrement=True' or
>> 'nullable=True', and no explicit value is passed.  Primary key columns
>> typically may not store NULL. Note that as of SQLAlchemy 1.1,
>> 'autoincrement=True' must be indicated explicitly for composite (e.g.
>> multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is
>> expected for one of the columns in the primary key. CREATE TABLE statements
>> are impacted by this change as well on most backends.
>>
>
>

Re: Airflow 1.8.0 Release Candidate 1

[Dan Davydov]

Bolke, attached is the patch for the cgroups fix. Let me know which
branches you would like me to merge it to. If anyone has complaints about
the patch let me know (but it does not touch the core of airflow, only the
new cgroups task runner).

On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand  wrote:

> Actually, I see the error is further down..
>
>   File
> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
> line
> 469, in do_execute
>
> cursor.execute(statement, parameters)
>
> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
> column "dag_id" violates not-null constraint
>
> DETAIL:  Failing row contains (null, running, 1, f).
>
>  [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
> %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running',
> 'dirty': False}]
>
> It looks like an autoincrement is missing for this table.
>
>
> I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any version
> greater than 0.9.8
>
> -s
>
>
>
> On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand  wrote:
>
> > I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/
> > airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
> > upgradedb didn't quite work. First, I thought it completed successfully,
> > then saw errors some tables were indeed missing. I ran it again and
> > encountered the following exception :
> >
> > DB: postgresql://app_coust...@db-cousteau.ep.stage.agari.com:
> 5432/airflow
> >
> > [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables
> >
> > INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
> >
> > INFO  [alembic.runtime.migration] Will assume transactional DDL.
> >
> > INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
> > 211e584da130, add TI state index
> >
> > INFO  [alembic.runtime.migration] Running upgrade 211e584da130 ->
> > 64de9cddf6c9, add task fails journal table
> >
> > INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
> > f2ca10b85618, add dag_stats table
> >
> > INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
> > 4addfa1236f1, Add fractional seconds to mysql tables
> >
> > INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
> > 8504051e801b, xcom dag task indices
> >
> > INFO  [alembic.runtime.migration] Running upgrade 8504051e801b ->
> > 5e7d17757c7a, add pid field to TaskInstance
> >
> > INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
> > 127d2bf2dfa7, Add dag_id/state index on dag_run table
> >
> > /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
> > SAWarning: Column 'dag_stats.dag_id' is marked as a member of the primary
> > key for table 'dag_stats', but has no Python-side or server-side default
> > generator indicated, nor does it indicate 'autoincrement=True' or
> > 'nullable=True', and no explicit value is passed.  Primary key columns
> > typically may not store NULL. Note that as of SQLAlchemy 1.1,
> > 'autoincrement=True' must be indicated explicitly for composite (e.g.
> > multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is
> > expected for one of the columns in the primary key. CREATE TABLE
> statements
> > are impacted by this change as well on most backends.
> >
>

Re: Airflow 1.8.0 Release Candidate 1

[siddharth anand]

I did get 1.8.0 installed and running at Agari.

I did run into 2 problems.
1. Most of our DAGs broke due the way Operators are now imported.
https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#deprecated-features

According to the documentation, these deprecations would only cause an
issue in 2.0. However, I needed to fix them now.

So, I needed to change "from airflow.operators import PythonOperator" to
from "from airflow.operators.python_operator import PythonOperator". Am I
missing something?

2. I ran into a migration problem that seems to have cleared itself up. I
did notice that some dags do not have data in their "DAG Runs" column on
the overview page computed. I am looking into that issue presently.
https://www.dropbox.com/s/cn058mtu3vcv8sq/Screenshot%202017-02-06%2018.45.07.png?dl=0

-s

On Mon, Feb 6, 2017 at 4:30 PM, Dan Davydov 
wrote:

> Bolke, attached is the patch for the cgroups fix. Let me know which
> branches you would like me to merge it to. If anyone has complaints about
> the patch let me know (but it does not touch the core of airflow, only the
> new cgroups task runner).
>
> On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand  wrote:
>
>> Actually, I see the error is further down..
>>
>>   File
>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
>> line
>> 469, in do_execute
>>
>> cursor.execute(statement, parameters)
>>
>> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
>> column "dag_id" violates not-null constraint
>>
>> DETAIL:  Failing row contains (null, running, 1, f).
>>
>>  [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
>> %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running',
>> 'dirty': False}]
>>
>> It looks like an autoincrement is missing for this table.
>>
>>
>> I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any version
>> greater than 0.9.8
>>
>> -s
>>
>>
>>
>> On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand 
>> wrote:
>>
>> > I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
>> > https://dist.apache.org/repos/dist/dev/incubator/airflow/
>> > airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
>> > upgradedb didn't quite work. First, I thought it completed successfully,
>> > then saw errors some tables were indeed missing. I ran it again and
>> > encountered the following exception :
>> >
>> > DB: postgresql://app_coust...@db-cousteau.ep.stage.agari.com:543
>> 2/airflow
>> >
>> > [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables
>> >
>> > INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
>> >
>> > INFO  [alembic.runtime.migration] Will assume transactional DDL.
>> >
>> > INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
>> > 211e584da130, add TI state index
>> >
>> > INFO  [alembic.runtime.migration] Running upgrade 211e584da130 ->
>> > 64de9cddf6c9, add task fails journal table
>> >
>> > INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
>> > f2ca10b85618, add dag_stats table
>> >
>> > INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
>> > 4addfa1236f1, Add fractional seconds to mysql tables
>> >
>> > INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
>> > 8504051e801b, xcom dag task indices
>> >
>> > INFO  [alembic.runtime.migration] Running upgrade 8504051e801b ->
>> > 5e7d17757c7a, add pid field to TaskInstance
>> >
>> > INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
>> > 127d2bf2dfa7, Add dag_id/state index on dag_run table
>> >
>> > /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
>> > SAWarning: Column 'dag_stats.dag_id' is marked as a member of the
>> primary
>> > key for table 'dag_stats', but has no Python-side or server-side default
>> > generator indicated, nor does it indicate 'autoincrement=True' or
>> > 'nullable=True', and no explicit value is passed.  Primary key columns
>> > typically may not store NULL. Note that as of SQLAlchemy 1.1,
>> > 'autoincrement=True' must be indicated explicitly for composite (e.g.
>> > multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is
>> > expected for one of the columns in the primary key. CREATE TABLE
>> statements
>> > are impacted by this change as well on most backends.
>> >
>>
>
>

Re: Airflow 1.8.0 Release Candidate 1

[Alex Van Boxel]

I have to give the RC1 a *-1*. I spend hours, or better days to get the RC
running with Celery on our test environment, till I finally found the
commit that killed it:

e7f6212cae82c3a3a0bc17bbcbc70646f67d02eb
[AIRFLOW-813] Fix unterminated unit tests in SchedulerJobTest
Closes #2032 from fenglu-g/master

I was always looking at the wrong this, because the commit only changes a
single default parameter from *None to -1*

I do have the impression I'm the only one running with Celery. Are other
people running with it?

*I propose* *reverting the commit*. Feng, can you elaborate on this change?

Change the default back no *None* in cli.py got it finally working:

'num_runs': Arg(
("-n", "--num_runs"),
default=None, type=int,
help="Set the number of runs to execute before exiting"),

Thanks.

On Tue, Feb 7, 2017 at 3:49 AM siddharth anand  wrote:

I did get 1.8.0 installed and running at Agari.

I did run into 2 problems.
1. Most of our DAGs broke due the way Operators are now imported.
https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#deprecated-features

According to the documentation, these deprecations would only cause an
issue in 2.0. However, I needed to fix them now.

So, I needed to change "from airflow.operators import PythonOperator" to
from "from airflow.operators.python_operator import PythonOperator". Am I
missing something?

2. I ran into a migration problem that seems to have cleared itself up. I
did notice that some dags do not have data in their "DAG Runs" column on
the overview page computed. I am looking into that issue presently.
https://www.dropbox.com/s/cn058mtu3vcv8sq/Screenshot%202017-02-06%2018.45.07.png?dl=0

-s

On Mon, Feb 6, 2017 at 4:30 PM, Dan Davydov 
wrote:

> Bolke, attached is the patch for the cgroups fix. Let me know which
> branches you would like me to merge it to. If anyone has complaints about
> the patch let me know (but it does not touch the core of airflow, only the
> new cgroups task runner).
>
> On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand  wrote:
>
>> Actually, I see the error is further down..
>>
>>   File
>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
>> line
>> 469, in do_execute
>>
>> cursor.execute(statement, parameters)
>>
>> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
>> column "dag_id" violates not-null constraint
>>
>> DETAIL:  Failing row contains (null, running, 1, f).
>>
>>  [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
>> %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running',
>> 'dirty': False}]
>>
>> It looks like an autoincrement is missing for this table.
>>
>>
>> I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any
version
>> greater than 0.9.8
>>
>> -s
>>
>>
>>
>> On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand 
>> wrote:
>>
>> > I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
>> > https://dist.apache.org/repos/dist/dev/incubator/airflow/
>> > airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
>> > upgradedb didn't quite work. First, I thought it completed
successfully,
>> > then saw errors some tables were indeed missing. I ran it again and
>> > encountered the following exception :
>> >
>> > DB: postgresql://app_coust...@db-cousteau.ep.stage.agari.com:543
>> 2/airflow
>> >
>> > [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables
>> >
>> > INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
>> >
>> > INFO  [alembic.runtime.migration] Will assume transactional DDL.
>> >
>> > INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
>> > 211e584da130, add TI state index
>> >
>> > INFO  [alembic.runtime.migration] Running upgrade 211e584da130 ->
>> > 64de9cddf6c9, add task fails journal table
>> >
>> > INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
>> > f2ca10b85618, add dag_stats table
>> >
>> > INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
>> > 4addfa1236f1, Add fractional seconds to mysql tables
>> >
>> > INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
>> > 8504051e801b, xcom dag task indices
>> >
>> > INFO  [alembic.runtime.migration] Running upgrade 8504051e801b ->
>> > 5e7d17757c7a, add pid field to TaskInstance
>> >
>> > INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
>> > 127d2bf2dfa7, Add dag_id/state index on dag_run table
>> >
>> > /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
>> > SAWarning: Column 'dag_stats.dag_id' is marked as a member of the
>> primary
>> > key for table 'dag_stats', but has no Python-side or server-side
default
>> > generator indicated, nor does it indicate 'autoincrement=True' or
>> > 'nullable=True', and no explicit value is passed.  Primary key columns
>> > typically may not store NULL. Note that as of SQLAlchemy 1.1,
>> > 'autoincrement=True' must be indicated explicitly for composite (e.g.
>> > multicolumn) primary key

Re: Airflow 1.8.0 Release Candidate 1

[Bolke de Bruin]

Hey Alex,

Thanks for tracking it down. Can you elaborate want went wrong with celery? The 
lines below do not particularly relate to Celery directly, so I wonder why we 
are not seeing it with LocalExecutor?

Cheers
Bolke
 
> On 7 Feb 2017, at 15:51, Alex Van Boxel  wrote:
> 
> I have to give the RC1 a *-1*. I spend hours, or better days to get the RC
> running with Celery on our test environment, till I finally found the
> commit that killed it:
> 
> e7f6212cae82c3a3a0bc17bbcbc70646f67d02eb
> [AIRFLOW-813] Fix unterminated unit tests in SchedulerJobTest
> Closes #2032 from fenglu-g/master
> 
> I was always looking at the wrong this, because the commit only changes a
> single default parameter from *None to -1*
> 
> I do have the impression I'm the only one running with Celery. Are other
> people running with it?
> 
> *I propose* *reverting the commit*. Feng, can you elaborate on this change?
> 
> Change the default back no *None* in cli.py got it finally working:
> 
> 'num_runs': Arg(
>("-n", "--num_runs"),
>default=None, type=int,
>help="Set the number of runs to execute before exiting"),
> 
> Thanks.
> 
> On Tue, Feb 7, 2017 at 3:49 AM siddharth anand  wrote:
> 
> I did get 1.8.0 installed and running at Agari.
> 
> I did run into 2 problems.
> 1. Most of our DAGs broke due the way Operators are now imported.
> https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#deprecated-features
> 
> According to the documentation, these deprecations would only cause an
> issue in 2.0. However, I needed to fix them now.
> 
> So, I needed to change "from airflow.operators import PythonOperator" to
> from "from airflow.operators.python_operator import PythonOperator". Am I
> missing something?
> 
> 2. I ran into a migration problem that seems to have cleared itself up. I
> did notice that some dags do not have data in their "DAG Runs" column on
> the overview page computed. I am looking into that issue presently.
> https://www.dropbox.com/s/cn058mtu3vcv8sq/Screenshot%202017-02-06%2018.45.07.png?dl=0
> 
> -s
> 
> On Mon, Feb 6, 2017 at 4:30 PM, Dan Davydov 
> wrote:
> 
>> Bolke, attached is the patch for the cgroups fix. Let me know which
>> branches you would like me to merge it to. If anyone has complaints about
>> the patch let me know (but it does not touch the core of airflow, only the
>> new cgroups task runner).
>> 
>> On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand  wrote:
>> 
>>> Actually, I see the error is further down..
>>> 
>>>  File
>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
>>> line
>>> 469, in do_execute
>>> 
>>>cursor.execute(statement, parameters)
>>> 
>>> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
>>> column "dag_id" violates not-null constraint
>>> 
>>> DETAIL:  Failing row contains (null, running, 1, f).
>>> 
>>> [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
>>> %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running',
>>> 'dirty': False}]
>>> 
>>> It looks like an autoincrement is missing for this table.
>>> 
>>> 
>>> I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any
> version
>>> greater than 0.9.8
>>> 
>>> -s
>>> 
>>> 
>>> 
>>> On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand 
>>> wrote:
>>> 
 I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
 https://dist.apache.org/repos/dist/dev/incubator/airflow/
 airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
 upgradedb didn't quite work. First, I thought it completed
> successfully,
 then saw errors some tables were indeed missing. I ran it again and
 encountered the following exception :
 
 DB: postgresql://app_coust...@db-cousteau.ep.stage.agari.com:543
>>> 2/airflow
 
 [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables
 
 INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
 
 INFO  [alembic.runtime.migration] Will assume transactional DDL.
 
 INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
 211e584da130, add TI state index
 
 INFO  [alembic.runtime.migration] Running upgrade 211e584da130 ->
 64de9cddf6c9, add task fails journal table
 
 INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
 f2ca10b85618, add dag_stats table
 
 INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
 4addfa1236f1, Add fractional seconds to mysql tables
 
 INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
 8504051e801b, xcom dag task indices
 
 INFO  [alembic.runtime.migration] Running upgrade 8504051e801b ->
 5e7d17757c7a, add pid field to TaskInstance
 
 INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
 127d2bf2dfa7, Add dag_id/state index on dag_run table
 
 /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
 SAWarning: Column 'dag_stats.da

Re: Airflow 1.8.0 Release Candidate 1

[Alex Van Boxel]

OK, a bit of history... I moved for beta 4 to rc 1 and though I didn't have
problems because it run'ed fine locally for testing. But my production set
is on K8S + Redis (Celery). *What I saw in production was the **import
errors on the DAGS,* first I thought it was due to the fix of showing
errors in the DAG in the UI or my own building. But it wasn't. I had to go
though every commit between beta 4 and rc1 to find the error (build docker
image, deploy in k8s).

So to clarify (I'm sure this is the commit now):

   1. I did a before and after commit test.
   2. I now installed RC1, but with that single line reverted (works after,
   broken before)

I can also only reproduce it in the production, but it's been running on
master till beta 4 quite ok. I don't really know why it does that (a bit
the downside of dynamic typing here...) and actually don't want to dig
further. I've lost day's of valuable time (although I learned a lot about
Python dynamic loading).

*Or we find the problem with the help of Feng (the implementor) or we
revert that single commit*. Or is somebody else running Celery that is not
having this problem?






On Tue, Feb 7, 2017 at 3:57 PM Bolke de Bruin  wrote:

> Hey Alex,
>
> Thanks for tracking it down. Can you elaborate want went wrong with
> celery? The lines below do not particularly relate to Celery directly, so I
> wonder why we are not seeing it with LocalExecutor?
>
> Cheers
> Bolke
>
> > On 7 Feb 2017, at 15:51, Alex Van Boxel  wrote:
> >
> > I have to give the RC1 a *-1*. I spend hours, or better days to get the
> RC
> > running with Celery on our test environment, till I finally found the
> > commit that killed it:
> >
> > e7f6212cae82c3a3a0bc17bbcbc70646f67d02eb
> > [AIRFLOW-813] Fix unterminated unit tests in SchedulerJobTest
> > Closes #2032 from fenglu-g/master
> >
> > I was always looking at the wrong this, because the commit only changes a
> > single default parameter from *None to -1*
> >
> > I do have the impression I'm the only one running with Celery. Are other
> > people running with it?
> >
> > *I propose* *reverting the commit*. Feng, can you elaborate on this
> change?
> >
> > Change the default back no *None* in cli.py got it finally working:
> >
> > 'num_runs': Arg(
> >("-n", "--num_runs"),
> >default=None, type=int,
> >help="Set the number of runs to execute before exiting"),
> >
> > Thanks.
> >
> > On Tue, Feb 7, 2017 at 3:49 AM siddharth anand 
> wrote:
> >
> > I did get 1.8.0 installed and running at Agari.
> >
> > I did run into 2 problems.
> > 1. Most of our DAGs broke due the way Operators are now imported.
> >
> https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#deprecated-features
> >
> > According to the documentation, these deprecations would only cause an
> > issue in 2.0. However, I needed to fix them now.
> >
> > So, I needed to change "from airflow.operators import PythonOperator" to
> > from "from airflow.operators.python_operator import PythonOperator". Am I
> > missing something?
> >
> > 2. I ran into a migration problem that seems to have cleared itself up. I
> > did notice that some dags do not have data in their "DAG Runs" column on
> > the overview page computed. I am looking into that issue presently.
> >
> https://www.dropbox.com/s/cn058mtu3vcv8sq/Screenshot%202017-02-06%2018.45.07.png?dl=0
> >
> > -s
> >
> > On Mon, Feb 6, 2017 at 4:30 PM, Dan Davydov  .invalid>
> > wrote:
> >
> >> Bolke, attached is the patch for the cgroups fix. Let me know which
> >> branches you would like me to merge it to. If anyone has complaints
> about
> >> the patch let me know (but it does not touch the core of airflow, only
> the
> >> new cgroups task runner).
> >>
> >> On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand 
> wrote:
> >>
> >>> Actually, I see the error is further down..
> >>>
> >>>  File
> >>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
> >>> line
> >>> 469, in do_execute
> >>>
> >>>cursor.execute(statement, parameters)
> >>>
> >>> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
> >>> column "dag_id" violates not-null constraint
> >>>
> >>> DETAIL:  Failing row contains (null, running, 1, f).
> >>>
> >>> [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
> >>> %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running',
> >>> 'dirty': False}]
> >>>
> >>> It looks like an autoincrement is missing for this table.
> >>>
> >>>
> >>> I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any
> > version
> >>> greater than 0.9.8
> >>>
> >>> -s
> >>>
> >>>
> >>>
> >>> On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand 
> >>> wrote:
> >>>
>  I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
>  https://dist.apache.org/repos/dist/dev/incubator/airflow/
>  airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
>  upgradedb didn't quite work. First, I thought it completed
> > successfully,
>  then saw 

Re: Airflow 1.8.0 Release Candidate 1

[Feng Lu]

Hi Alex-

Please see the attached screenshots of my local testing using
celeryexecutor (on k8s as well).
All look good and the workflow is successfully completed.

Curious did you also update the worker image?
Sorry for the confusion, happy to debug more if you could share with me
your k8s setup.

Feng

On Tue, Feb 7, 2017 at 8:37 AM, Feng Lu  wrote:

> When num_runs is not explicitly specified, the default is set to -1 to
> match the expectation of SchedulerJob here:
>
> ​
> Doing so also matches the type of num_runs ('int' in this case).
> The scheduler will run non-stop as a result regardless whether dag files
> are present (since the num_runs default is now -1: unlimited).
>
> Based on what Alex described, the import error doesn't look like directly
> related to this change.
> Maybe this one? https://github.com/apache/incubator-airflow/commit/
> 67cbb966410226c1489bb730af3af45330fc51b9
>
> I am still in the middle of running some quick test using celery executor,
> will update the thread once it's done.
>
>
> On Tue, Feb 7, 2017 at 6:56 AM, Bolke de Bruin  wrote:
>
>> Hey Alex,
>>
>> Thanks for tracking it down. Can you elaborate want went wrong with
>> celery? The lines below do not particularly relate to Celery directly, so I
>> wonder why we are not seeing it with LocalExecutor?
>>
>> Cheers
>> Bolke
>>
>> > On 7 Feb 2017, at 15:51, Alex Van Boxel  wrote:
>> >
>> > I have to give the RC1 a *-1*. I spend hours, or better days to get the
>> RC
>> > running with Celery on our test environment, till I finally found the
>> > commit that killed it:
>> >
>> > e7f6212cae82c3a3a0bc17bbcbc70646f67d02eb
>> > [AIRFLOW-813] Fix unterminated unit tests in SchedulerJobTest
>> > Closes #2032 from fenglu-g/master
>> >
>> > I was always looking at the wrong this, because the commit only changes
>> a
>> > single default parameter from *None to -1*
>> >
>> > I do have the impression I'm the only one running with Celery. Are other
>> > people running with it?
>> >
>> > *I propose* *reverting the commit*. Feng, can you elaborate on this
>> change?
>> >
>> > Change the default back no *None* in cli.py got it finally working:
>> >
>> > 'num_runs': Arg(
>> >("-n", "--num_runs"),
>> >default=None, type=int,
>> >help="Set the number of runs to execute before exiting"),
>> >
>> > Thanks.
>> >
>> > On Tue, Feb 7, 2017 at 3:49 AM siddharth anand 
>> wrote:
>> >
>> > I did get 1.8.0 installed and running at Agari.
>> >
>> > I did run into 2 problems.
>> > 1. Most of our DAGs broke due the way Operators are now imported.
>> > https://github.com/apache/incubator-airflow/blob/master/UPDA
>> TING.md#deprecated-features
>> >
>> > According to the documentation, these deprecations would only cause an
>> > issue in 2.0. However, I needed to fix them now.
>> >
>> > So, I needed to change "from airflow.operators import PythonOperator" to
>> > from "from airflow.operators.python_operator import PythonOperator".
>> Am I
>> > missing something?
>> >
>> > 2. I ran into a migration problem that seems to have cleared itself up.
>> I
>> > did notice that some dags do not have data in their "DAG Runs" column on
>> > the overview page computed. I am looking into that issue presently.
>> > https://www.dropbox.com/s/cn058mtu3vcv8sq/Screenshot%202017-
>> 02-06%2018.45.07.png?dl=0
>> >
>> > -s
>> >
>> > On Mon, Feb 6, 2017 at 4:30 PM, Dan Davydov > .invalid>
>> > wrote:
>> >
>> >> Bolke, attached is the patch for the cgroups fix. Let me know which
>> >> branches you would like me to merge it to. If anyone has complaints
>> about
>> >> the patch let me know (but it does not touch the core of airflow, only
>> the
>> >> new cgroups task runner).
>> >>
>> >> On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand 
>> wrote:
>> >>
>> >>> Actually, I see the error is further down..
>> >>>
>> >>>  File
>> >>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/
>> default.py",
>> >>> line
>> >>> 469, in do_execute
>> >>>
>> >>>cursor.execute(statement, parameters)
>> >>>
>> >>> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
>> >>> column "dag_id" violates not-null constraint
>> >>>
>> >>> DETAIL:  Failing row contains (null, running, 1, f).
>> >>>
>> >>> [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
>> >>> %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state':
>> u'running',
>> >>> 'dirty': False}]
>> >>>
>> >>> It looks like an autoincrement is missing for this table.
>> >>>
>> >>>
>> >>> I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any
>> > version
>> >>> greater than 0.9.8
>> >>>
>> >>> -s
>> >>>
>> >>>
>> >>>
>> >>> On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand 
>> >>> wrote:
>> >>>
>>  I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
>>  https://dist.apache.org/repos/dist/dev/incubator/airflow/
>>  airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
>>  upgradedb didn't quite work. First, I thought it completed
>> > successfull

Re: Airflow 1.8.0 Release Candidate 1

[Feng Lu]

When num_runs is not explicitly specified, the default is set to -1 to
match the expectation of SchedulerJob here:

​
Doing so also matches the type of num_runs ('int' in this case).
The scheduler will run non-stop as a result regardless whether dag files
are present (since the num_runs default is now -1: unlimited).

Based on what Alex described, the import error doesn't look like directly
related to this change.
Maybe this one?
https://github.com/apache/incubator-airflow/commit/67cbb966410226c1489bb730af3af45330fc51b9

I am still in the middle of running some quick test using celery executor,
will update the thread once it's done.


On Tue, Feb 7, 2017 at 6:56 AM, Bolke de Bruin  wrote:

> Hey Alex,
>
> Thanks for tracking it down. Can you elaborate want went wrong with
> celery? The lines below do not particularly relate to Celery directly, so I
> wonder why we are not seeing it with LocalExecutor?
>
> Cheers
> Bolke
>
> > On 7 Feb 2017, at 15:51, Alex Van Boxel  wrote:
> >
> > I have to give the RC1 a *-1*. I spend hours, or better days to get the
> RC
> > running with Celery on our test environment, till I finally found the
> > commit that killed it:
> >
> > e7f6212cae82c3a3a0bc17bbcbc70646f67d02eb
> > [AIRFLOW-813] Fix unterminated unit tests in SchedulerJobTest
> > Closes #2032 from fenglu-g/master
> >
> > I was always looking at the wrong this, because the commit only changes a
> > single default parameter from *None to -1*
> >
> > I do have the impression I'm the only one running with Celery. Are other
> > people running with it?
> >
> > *I propose* *reverting the commit*. Feng, can you elaborate on this
> change?
> >
> > Change the default back no *None* in cli.py got it finally working:
> >
> > 'num_runs': Arg(
> >("-n", "--num_runs"),
> >default=None, type=int,
> >help="Set the number of runs to execute before exiting"),
> >
> > Thanks.
> >
> > On Tue, Feb 7, 2017 at 3:49 AM siddharth anand 
> wrote:
> >
> > I did get 1.8.0 installed and running at Agari.
> >
> > I did run into 2 problems.
> > 1. Most of our DAGs broke due the way Operators are now imported.
> > https://github.com/apache/incubator-airflow/blob/master/
> UPDATING.md#deprecated-features
> >
> > According to the documentation, these deprecations would only cause an
> > issue in 2.0. However, I needed to fix them now.
> >
> > So, I needed to change "from airflow.operators import PythonOperator" to
> > from "from airflow.operators.python_operator import PythonOperator". Am
> I
> > missing something?
> >
> > 2. I ran into a migration problem that seems to have cleared itself up. I
> > did notice that some dags do not have data in their "DAG Runs" column on
> > the overview page computed. I am looking into that issue presently.
> > https://www.dropbox.com/s/cn058mtu3vcv8sq/Screenshot%
> 202017-02-06%2018.45.07.png?dl=0
> >
> > -s
> >
> > On Mon, Feb 6, 2017 at 4:30 PM, Dan Davydov  invalid>
> > wrote:
> >
> >> Bolke, attached is the patch for the cgroups fix. Let me know which
> >> branches you would like me to merge it to. If anyone has complaints
> about
> >> the patch let me know (but it does not touch the core of airflow, only
> the
> >> new cgroups task runner).
> >>
> >> On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand 
> wrote:
> >>
> >>> Actually, I see the error is further down..
> >>>
> >>>  File
> >>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
> >>> line
> >>> 469, in do_execute
> >>>
> >>>cursor.execute(statement, parameters)
> >>>
> >>> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
> >>> column "dag_id" violates not-null constraint
> >>>
> >>> DETAIL:  Failing row contains (null, running, 1, f).
> >>>
> >>> [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
> >>> %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running',
> >>> 'dirty': False}]
> >>>
> >>> It looks like an autoincrement is missing for this table.
> >>>
> >>>
> >>> I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any
> > version
> >>> greater than 0.9.8
> >>>
> >>> -s
> >>>
> >>>
> >>>
> >>> On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand 
> >>> wrote:
> >>>
>  I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
>  https://dist.apache.org/repos/dist/dev/incubator/airflow/
>  airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
>  upgradedb didn't quite work. First, I thought it completed
> > successfully,
>  then saw errors some tables were indeed missing. I ran it again and
>  encountered the following exception :
> 
>  DB: postgresql://app_coust...@db-cousteau.ep.stage.agari.com:543
> >>> 2/airflow
> 
>  [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables
> 
>  INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
> 
>  INFO  [alembic.runtime.migration] Will assume transactional DDL.
> 
>  INFO  [alembic.runtime.migration] Running upgrade 2e8

Re: Airflow 1.8.0 Release Candidate 1

[Alex Van Boxel]

Hey Feng,

The upgrades are all automated (including the workers/web/scheduler). And I
tripple checked, I now am test running RC1 just with the your line reverted
(and look ok)

Could you do me a favour and add a test dag where you do a local import.
Example:

bqschema.py

def ranking():
return [
{"name": "bucket_date", "type": "timestamp", "mode": "nullable"},
{"name": "rank", "type": "integer", "mode": "nullable"},
{"name": "audience_preference", "type": "float", "mode": "nullable"},
{"name": "audience_likelihood_share", "type": "float", "mode":
"nullable"}
]


dag.py

import bqschema
*...*

all in the same dag folder. We use it to define out BigQuery schema's into
a seperate file.


On Tue, Feb 7, 2017 at 6:37 PM Feng Lu  wrote:

> Hi Alex-
>
> Please see the attached screenshots of my local testing using
> celeryexecutor (on k8s as well).
> All look good and the workflow is successfully completed.
>
> Curious did you also update the worker image?
> Sorry for the confusion, happy to debug more if you could share with me
> your k8s setup.
>
> Feng
>
> On Tue, Feb 7, 2017 at 8:37 AM, Feng Lu  wrote:
>
> When num_runs is not explicitly specified, the default is set to -1 to
> match the expectation of SchedulerJob here:
> [image: Screen Shot 2017-02-07 at 8.01.26 AM.png]
> ​
> Doing so also matches the type of num_runs ('int' in this case).
> The scheduler will run non-stop as a result regardless whether dag files
> are present (since the num_runs default is now -1: unlimited).
>
> Based on what Alex described, the import error doesn't look like directly
> related to this change.
> Maybe this one?
> https://github.com/apache/incubator-airflow/commit/67cbb966410226c1489bb730af3af45330fc51b9
>
> I am still in the middle of running some quick test using celery executor,
> will update the thread once it's done.
>
>
> On Tue, Feb 7, 2017 at 6:56 AM, Bolke de Bruin  wrote:
>
> Hey Alex,
>
> Thanks for tracking it down. Can you elaborate want went wrong with
> celery? The lines below do not particularly relate to Celery directly, so I
> wonder why we are not seeing it with LocalExecutor?
>
> Cheers
> Bolke
>
> > On 7 Feb 2017, at 15:51, Alex Van Boxel  wrote:
> >
> > I have to give the RC1 a *-1*. I spend hours, or better days to get the
> RC
> > running with Celery on our test environment, till I finally found the
> > commit that killed it:
> >
> > e7f6212cae82c3a3a0bc17bbcbc70646f67d02eb
> > [AIRFLOW-813] Fix unterminated unit tests in SchedulerJobTest
> > Closes #2032 from fenglu-g/master
> >
> > I was always looking at the wrong this, because the commit only changes a
> > single default parameter from *None to -1*
> >
> > I do have the impression I'm the only one running with Celery. Are other
> > people running with it?
> >
> > *I propose* *reverting the commit*. Feng, can you elaborate on this
> change?
> >
> > Change the default back no *None* in cli.py got it finally working:
> >
> > 'num_runs': Arg(
> >("-n", "--num_runs"),
> >default=None, type=int,
> >help="Set the number of runs to execute before exiting"),
> >
> > Thanks.
> >
> > On Tue, Feb 7, 2017 at 3:49 AM siddharth anand 
> wrote:
> >
> > I did get 1.8.0 installed and running at Agari.
> >
> > I did run into 2 problems.
> > 1. Most of our DAGs broke due the way Operators are now imported.
> >
> https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#deprecated-features
> >
> > According to the documentation, these deprecations would only cause an
> > issue in 2.0. However, I needed to fix them now.
> >
> > So, I needed to change "from airflow.operators import PythonOperator" to
> > from "from airflow.operators.python_operator import PythonOperator". Am I
> > missing something?
> >
> > 2. I ran into a migration problem that seems to have cleared itself up. I
> > did notice that some dags do not have data in their "DAG Runs" column on
> > the overview page computed. I am looking into that issue presently.
> >
> https://www.dropbox.com/s/cn058mtu3vcv8sq/Screenshot%202017-02-06%2018.45.07.png?dl=0
> >
> > -s
> >
> > On Mon, Feb 6, 2017 at 4:30 PM, Dan Davydov  .invalid>
> > wrote:
> >
> >> Bolke, attached is the patch for the cgroups fix. Let me know which
> >> branches you would like me to merge it to. If anyone has complaints
> about
> >> the patch let me know (but it does not touch the core of airflow, only
> the
> >> new cgroups task runner).
> >>
> >> On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand 
> wrote:
> >>
> >>> Actually, I see the error is further down..
> >>>
> >>>  File
> >>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
> >>> line
> >>> 469, in do_execute
> >>>
> >>>cursor.execute(statement, parameters)
> >>>
> >>> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
> >>> column "dag_id" violates not-null constraint
> >>>
> >>> DETAIL:  Failing row contains (null, running, 1, f).
> >>>
> >>> [SQL: 'INSERT INTO dag_stats (sta

Re: Airflow 1.8.0 Release Candidate 1

[Bolke de Bruin]

How do you start the scheduler Alex? What are the command line parameters? What 
are the logs when it doesn’t work?

Bolke



> On 7 Feb 2017, at 18:52, Alex Van Boxel  wrote:
> 
> Hey Feng,
> 
> The upgrades are all automated (including the workers/web/scheduler). And I 
> tripple checked, I now am test running RC1 just with the your line reverted 
> (and look ok)
> 
> Could you do me a favour and add a test dag where you do a local import. 
> Example:
> 
> bqschema.py
> def ranking():
> return [
> {"name": "bucket_date", "type": "timestamp", "mode": "nullable"},
> {"name": "rank", "type": "integer", "mode": "nullable"},
> {"name": "audience_preference", "type": "float", "mode": "nullable"},
> {"name": "audience_likelihood_share", "type": "float", "mode": 
> "nullable"}
> ]
> 
> dag.py
> import bqschema
> ...
> all in the same dag folder. We use it to define out BigQuery schema's into a 
> seperate file.
> 
> 
> On Tue, Feb 7, 2017 at 6:37 PM Feng Lu  > wrote:
> Hi Alex-
> 
> Please see the attached screenshots of my local testing using celeryexecutor 
> (on k8s as well). 
> All look good and the workflow is successfully completed.
> 
> Curious did you also update the worker image? 
> Sorry for the confusion, happy to debug more if you could share with me your 
> k8s setup. 
> 
> Feng
> 
> On Tue, Feb 7, 2017 at 8:37 AM, Feng Lu  > wrote:
> When num_runs is not explicitly specified, the default is set to -1 to match 
> the expectation of SchedulerJob here:
> 
> ​
> Doing so also matches the type of num_runs ('int' in this case).
> The scheduler will run non-stop as a result regardless whether dag files are 
> present (since the num_runs default is now -1: unlimited). 
> 
> Based on what Alex described, the import error doesn't look like directly 
> related to this change. 
> Maybe this one? 
> https://github.com/apache/incubator-airflow/commit/67cbb966410226c1489bb730af3af45330fc51b9
>  
> 
> 
> I am still in the middle of running some quick test using celery executor, 
> will update the thread once it's done. 
> 
> 
> On Tue, Feb 7, 2017 at 6:56 AM, Bolke de Bruin  > wrote:
> Hey Alex,
> 
> Thanks for tracking it down. Can you elaborate want went wrong with celery? 
> The lines below do not particularly relate to Celery directly, so I wonder 
> why we are not seeing it with LocalExecutor?
> 
> Cheers
> Bolke
> 
> > On 7 Feb 2017, at 15:51, Alex Van Boxel  > > wrote:
> >
> > I have to give the RC1 a *-1*. I spend hours, or better days to get the RC
> > running with Celery on our test environment, till I finally found the
> > commit that killed it:
> >
> > e7f6212cae82c3a3a0bc17bbcbc70646f67d02eb
> > [AIRFLOW-813] Fix unterminated unit tests in SchedulerJobTest
> > Closes #2032 from fenglu-g/master
> >
> > I was always looking at the wrong this, because the commit only changes a
> > single default parameter from *None to -1*
> >
> > I do have the impression I'm the only one running with Celery. Are other
> > people running with it?
> >
> > *I propose* *reverting the commit*. Feng, can you elaborate on this change?
> >
> > Change the default back no *None* in cli.py got it finally working:
> >
> > 'num_runs': Arg(
> >("-n", "--num_runs"),
> >default=None, type=int,
> >help="Set the number of runs to execute before exiting"),
> >
> > Thanks.
> >
> > On Tue, Feb 7, 2017 at 3:49 AM siddharth anand  > > wrote:
> >
> > I did get 1.8.0 installed and running at Agari.
> >
> > I did run into 2 problems.
> > 1. Most of our DAGs broke due the way Operators are now imported.
> > https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#deprecated-features
> >  
> > 
> >
> > According to the documentation, these deprecations would only cause an
> > issue in 2.0. However, I needed to fix them now.
> >
> > So, I needed to change "from airflow.operators import PythonOperator" to
> > from "from airflow.operators.python_operator import PythonOperator". Am I
> > missing something?
> >
> > 2. I ran into a migration problem that seems to have cleared itself up. I
> > did notice that some dags do not have data in their "DAG Runs" column on
> > the overview page computed. I am looking into that issue presently.
> > https://www.dropbox.com/s/cn058mtu3vcv8sq/Screenshot%202017-02-06%2018.45.07.png?dl=0
> >  
> > 
> >
> > -s
> >
> > On Mon, Feb 6, 2017 at 4:30 PM, Dan Davydov  > .invalid>
> > wrote:
> >
> >> Bolke, attached is the patch for the cgroups fix. Let me know which
> >> branches you would like me to merge it to. If anyone has complai

Re: Airflow 1.8.0 Release Candidate 1

[Jeremiah Lowin]

Sid,

The behavior in your first point is odd -- it should print a deprecation
warning about 2.0 but still work. I just tested it on master and it works
(see output below), so unless something unusual happened in the 1.8 build
it should work there as well:

In [1]: from airflow.operators import PigOperator
[2017-02-07 15:24:17,551] {__init__.py:57} INFO - Using executor
SequentialExecutor
/Users/jlowin/git/airflow/airflow/utils/helpers.py:320: DeprecationWarning:
Importing PigOperator directly from  has been
deprecated. Please import from '.[operator_module]'
instead. Support for direct imports will be dropped entirely in Airflow 2.0.
  DeprecationWarning)

In [2]: PigOperator
Out[2]: pig_operator.PigOperator



On Mon, Feb 6, 2017 at 9:50 PM siddharth anand  wrote:

> I did get 1.8.0 installed and running at Agari.
>
> I did run into 2 problems.
> 1. Most of our DAGs broke due the way Operators are now imported.
>
> https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#deprecated-features
>
> According to the documentation, these deprecations would only cause an
> issue in 2.0. However, I needed to fix them now.
>
> So, I needed to change "from airflow.operators import PythonOperator" to
> from "from airflow.operators.python_operator import PythonOperator". Am I
> missing something?
>
> 2. I ran into a migration problem that seems to have cleared itself up. I
> did notice that some dags do not have data in their "DAG Runs" column on
> the overview page computed. I am looking into that issue presently.
>
> https://www.dropbox.com/s/cn058mtu3vcv8sq/Screenshot%202017-02-06%2018.45.07.png?dl=0
>
> -s
>
> On Mon, Feb 6, 2017 at 4:30 PM, Dan Davydov  .invalid>
> wrote:
>
> > Bolke, attached is the patch for the cgroups fix. Let me know which
> > branches you would like me to merge it to. If anyone has complaints about
> > the patch let me know (but it does not touch the core of airflow, only
> the
> > new cgroups task runner).
> >
> > On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand 
> wrote:
> >
> >> Actually, I see the error is further down..
> >>
> >>   File
> >> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
> >> line
> >> 469, in do_execute
> >>
> >> cursor.execute(statement, parameters)
> >>
> >> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
> >> column "dag_id" violates not-null constraint
> >>
> >> DETAIL:  Failing row contains (null, running, 1, f).
> >>
> >>  [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
> >> %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running',
> >> 'dirty': False}]
> >>
> >> It looks like an autoincrement is missing for this table.
> >>
> >>
> >> I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any
> version
> >> greater than 0.9.8
> >>
> >> -s
> >>
> >>
> >>
> >> On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand 
> >> wrote:
> >>
> >> > I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
> >> > https://dist.apache.org/repos/dist/dev/incubator/airflow/
> >> > airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
> >> > upgradedb didn't quite work. First, I thought it completed
> successfully,
> >> > then saw errors some tables were indeed missing. I ran it again and
> >> > encountered the following exception :
> >> >
> >> > DB: postgresql://app_coust...@db-cousteau.ep.stage.agari.com:543
> >> 2/airflow
> >> >
> >> > [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables
> >> >
> >> > INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
> >> >
> >> > INFO  [alembic.runtime.migration] Will assume transactional DDL.
> >> >
> >> > INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
> >> > 211e584da130, add TI state index
> >> >
> >> > INFO  [alembic.runtime.migration] Running upgrade 211e584da130 ->
> >> > 64de9cddf6c9, add task fails journal table
> >> >
> >> > INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
> >> > f2ca10b85618, add dag_stats table
> >> >
> >> > INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
> >> > 4addfa1236f1, Add fractional seconds to mysql tables
> >> >
> >> > INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
> >> > 8504051e801b, xcom dag task indices
> >> >
> >> > INFO  [alembic.runtime.migration] Running upgrade 8504051e801b ->
> >> > 5e7d17757c7a, add pid field to TaskInstance
> >> >
> >> > INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
> >> > 127d2bf2dfa7, Add dag_id/state index on dag_run table
> >> >
> >> > /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
> >> > SAWarning: Column 'dag_stats.dag_id' is marked as a member of the
> >> primary
> >> > key for table 'dag_stats', but has no Python-side or server-side
> default
> >> > generator indicated, nor does it indicate 'autoincrement=True' or
> >> > 'nullable=True', and no explicit value is passed.  Primary key columns
> >> > typically may not store NULL.

Re: Airflow 1.8.0 Release Candidate 1

[Bolke de Bruin]

Apache scrubs patches, so I will cherry pick it in this case. Will figure out a 
better way to this.

Bolke

> On 7 Feb 2017, at 01:30, Dan Davydov  wrote:
> 
> Bolke, attached is the patch for the cgroups fix. Let me know which branches 
> you would like me to merge it to. If anyone has complaints about the patch 
> let me know (but it does not touch the core of airflow, only the new cgroups 
> task runner).
> 
> On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand  > wrote:
> Actually, I see the error is further down..
> 
>   File
> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line
> 469, in do_execute
> 
> cursor.execute(statement, parameters)
> 
> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
> column "dag_id" violates not-null constraint
> 
> DETAIL:  Failing row contains (null, running, 1, f).
> 
>  [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
> %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running',
> 'dirty': False}]
> 
> It looks like an autoincrement is missing for this table.
> 
> 
> I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any version
> greater than 0.9.8
> 
> -s
> 
> 
> 
> On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand  > wrote:
> 
> > I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/ 
> > 
> > airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
> > upgradedb didn't quite work. First, I thought it completed successfully,
> > then saw errors some tables were indeed missing. I ran it again and
> > encountered the following exception :
> >
> > DB: postgresql://app_coust...@db-cousteau.ep.stage.agari.com:5432/airflow 
> > 
> >
> > [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables
> >
> > INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
> >
> > INFO  [alembic.runtime.migration] Will assume transactional DDL.
> >
> > INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
> > 211e584da130, add TI state index
> >
> > INFO  [alembic.runtime.migration] Running upgrade 211e584da130 ->
> > 64de9cddf6c9, add task fails journal table
> >
> > INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
> > f2ca10b85618, add dag_stats table
> >
> > INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
> > 4addfa1236f1, Add fractional seconds to mysql tables
> >
> > INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
> > 8504051e801b, xcom dag task indices
> >
> > INFO  [alembic.runtime.migration] Running upgrade 8504051e801b ->
> > 5e7d17757c7a, add pid field to TaskInstance
> >
> > INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
> > 127d2bf2dfa7, Add dag_id/state index on dag_run table
> >
> > /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
> > SAWarning: Column 'dag_stats.dag_id' is marked as a member of the primary
> > key for table 'dag_stats', but has no Python-side or server-side default
> > generator indicated, nor does it indicate 'autoincrement=True' or
> > 'nullable=True', and no explicit value is passed.  Primary key columns
> > typically may not store NULL. Note that as of SQLAlchemy 1.1,
> > 'autoincrement=True' must be indicated explicitly for composite (e.g.
> > multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is
> > expected for one of the columns in the primary key. CREATE TABLE statements
> > are impacted by this change as well on most backends.
> >
> 

Re: Airflow 1.8.0 Release Candidate 1

[Ali Naqvi]

Hi Sid,
The deprecation warning is appearing only in the webserver, but is not
preventing the scheduler or worker from completing any tasks.
When you say "Most of our DAGs broke" do you mean they cannot be scheduled
at all?

Re upgradedb
we took a snapshot of our db on AWS and used that as a starting point for
our upgrade. For the migration `airflow upgradedb` worked. We are using MySQL
5.6.27.

Cheers,
Ali

On Tue, Feb 7, 2017 at 3:46 PM, Bolke de Bruin  wrote:

> Apache scrubs patches, so I will cherry pick it in this case. Will figure
> out a better way to this.
>
> Bolke
>
> > On 7 Feb 2017, at 01:30, Dan Davydov 
> wrote:
> >
> > Bolke, attached is the patch for the cgroups fix. Let me know which
> branches you would like me to merge it to. If anyone has complaints about
> the patch let me know (but it does not touch the core of airflow, only the
> new cgroups task runner).
> >
> > On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand  > wrote:
> > Actually, I see the error is further down..
> >
> >   File
> > "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
> line
> > 469, in do_execute
> >
> > cursor.execute(statement, parameters)
> >
> > sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
> > column "dag_id" violates not-null constraint
> >
> > DETAIL:  Failing row contains (null, running, 1, f).
> >
> >  [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
> > %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running',
> > 'dirty': False}]
> >
> > It looks like an autoincrement is missing for this table.
> >
> >
> > I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any
> version
> > greater than 0.9.8
> >
> > -s
> >
> >
> >
> > On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand  > wrote:
> >
> > > I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
> > > https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/>
> > > airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
> > > upgradedb didn't quite work. First, I thought it completed
> successfully,
> > > then saw errors some tables were indeed missing. I ran it again and
> > > encountered the following exception :
> > >
> > > DB: postgresql://app_coust...@db-cousteau.ep.stage.agari.com:
> 5432/airflow  5432/airflow>
> > >
> > > [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables
> > >
> > > INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
> > >
> > > INFO  [alembic.runtime.migration] Will assume transactional DDL.
> > >
> > > INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
> > > 211e584da130, add TI state index
> > >
> > > INFO  [alembic.runtime.migration] Running upgrade 211e584da130 ->
> > > 64de9cddf6c9, add task fails journal table
> > >
> > > INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
> > > f2ca10b85618, add dag_stats table
> > >
> > > INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
> > > 4addfa1236f1, Add fractional seconds to mysql tables
> > >
> > > INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
> > > 8504051e801b, xcom dag task indices
> > >
> > > INFO  [alembic.runtime.migration] Running upgrade 8504051e801b ->
> > > 5e7d17757c7a, add pid field to TaskInstance
> > >
> > > INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
> > > 127d2bf2dfa7, Add dag_id/state index on dag_run table
> > >
> > > /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
> > > SAWarning: Column 'dag_stats.dag_id' is marked as a member of the
> primary
> > > key for table 'dag_stats', but has no Python-side or server-side
> default
> > > generator indicated, nor does it indicate 'autoincrement=True' or
> > > 'nullable=True', and no explicit value is passed.  Primary key columns
> > > typically may not store NULL. Note that as of SQLAlchemy 1.1,
> > > 'autoincrement=True' must be indicated explicitly for composite (e.g.
> > > multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is
> > > expected for one of the columns in the primary key. CREATE TABLE
> statements
> > > are impacted by this change as well on most backends.
> > >
> >
>
>

Re: Airflow 1.8.0 Release Candidate 1

[Bolke de Bruin]

We tracked Sid’s issue down to an issue that can happen if you use pools and do 
an upgrade from an earlier release while these pools are in use. Due to the 
flaky behaviour of pools in 1.7.1.3 and the tightened integrity on pools in 1.8 
it can happen that a pool can have queued tasks, but will never schedule them. 
The workaround is to temporarily increase the size of the queue. This wil be 
noted in UPDATING.md

What is required is to have a garbage collection for pools or a verification 
mechanism between scheduler and executor that pools are really in use. This 
should be targeted for 1.8.1.

- Bolke

> On 7 Feb 2017, at 22:24, Ali Naqvi  wrote:
> 
> Hi Sid,
> The deprecation warning is appearing only in the webserver, but is not
> preventing the scheduler or worker from completing any tasks.
> When you say "Most of our DAGs broke" do you mean they cannot be scheduled
> at all?
> 
> Re upgradedb
> we took a snapshot of our db on AWS and used that as a starting point for
> our upgrade. For the migration `airflow upgradedb` worked. We are using MySQL
> 5.6.27.
> 
> Cheers,
> Ali
> 
> On Tue, Feb 7, 2017 at 3:46 PM, Bolke de Bruin  wrote:
> 
>> Apache scrubs patches, so I will cherry pick it in this case. Will figure
>> out a better way to this.
>> 
>> Bolke
>> 
>>> On 7 Feb 2017, at 01:30, Dan Davydov 
>> wrote:
>>> 
>>> Bolke, attached is the patch for the cgroups fix. Let me know which
>> branches you would like me to merge it to. If anyone has complaints about
>> the patch let me know (but it does not touch the core of airflow, only the
>> new cgroups task runner).
>>> 
>>> On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand > > wrote:
>>> Actually, I see the error is further down..
>>> 
>>>  File
>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
>> line
>>> 469, in do_execute
>>> 
>>>cursor.execute(statement, parameters)
>>> 
>>> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
>>> column "dag_id" violates not-null constraint
>>> 
>>> DETAIL:  Failing row contains (null, running, 1, f).
>>> 
>>> [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
>>> %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running',
>>> 'dirty': False}]
>>> 
>>> It looks like an autoincrement is missing for this table.
>>> 
>>> 
>>> I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any
>> version
>>> greater than 0.9.8
>>> 
>>> -s
>>> 
>>> 
>>> 
>>> On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand > > wrote:
>>> 
 I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
 https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/>
 airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
 upgradedb didn't quite work. First, I thought it completed
>> successfully,
 then saw errors some tables were indeed missing. I ran it again and
 encountered the following exception :
 
 DB: postgresql://app_coust...@db-cousteau.ep.stage.agari.com:
>> 5432/airflow > 5432/airflow>
 
 [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables
 
 INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
 
 INFO  [alembic.runtime.migration] Will assume transactional DDL.
 
 INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
 211e584da130, add TI state index
 
 INFO  [alembic.runtime.migration] Running upgrade 211e584da130 ->
 64de9cddf6c9, add task fails journal table
 
 INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
 f2ca10b85618, add dag_stats table
 
 INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
 4addfa1236f1, Add fractional seconds to mysql tables
 
 INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
 8504051e801b, xcom dag task indices
 
 INFO  [alembic.runtime.migration] Running upgrade 8504051e801b ->
 5e7d17757c7a, add pid field to TaskInstance
 
 INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
 127d2bf2dfa7, Add dag_id/state index on dag_run table
 
 /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
 SAWarning: Column 'dag_stats.dag_id' is marked as a member of the
>> primary
 key for table 'dag_stats', but has no Python-side or server-side
>> default
 generator indicated, nor does it indicate 'autoincrement=True' or
 'nullable=True', and no explicit value is passed.  Primary key columns
 typically may not store NULL. Note that as of SQLAlchemy 1.1,
 'autoincrement=True' must be indicated explicitly for composite (e.g.
 multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is
 expected for one of the columns in the primary key. CREATE TABLE
>> statements
 are impact

Re: Airflow 1.8.0 Release Candidate 1

[Bolke de Bruin]

Alex,

Do you have anything more to go on? I don’t mind reverting the patch, however 
it code part seems unrelated to what you described and the issue wasn’t 
reproducible. I would really like to see more logging and maybe a test in a 
clean environment plus debugging. Preferable I would like to make RC 2 
available today and immediately raise a vote as the *current* changes are 
really small, are confined to contrib and have been tested by the people using 
it.

But I am holding off for now due to your concern.

Cheers
Bolke


> On 7 Feb 2017, at 20:56, Bolke de Bruin  wrote:
> 
> How do you start the scheduler Alex? What are the command line parameters? 
> What are the logs when it doesn’t work?
> 
> Bolke
> 
> 
> 
>> On 7 Feb 2017, at 18:52, Alex Van Boxel > > wrote:
>> 
>> Hey Feng,
>> 
>> The upgrades are all automated (including the workers/web/scheduler). And I 
>> tripple checked, I now am test running RC1 just with the your line reverted 
>> (and look ok)
>> 
>> Could you do me a favour and add a test dag where you do a local import. 
>> Example:
>> 
>> bqschema.py
>> def ranking():
>> return [
>> {"name": "bucket_date", "type": "timestamp", "mode": "nullable"},
>> {"name": "rank", "type": "integer", "mode": "nullable"},
>> {"name": "audience_preference", "type": "float", "mode": "nullable"},
>> {"name": "audience_likelihood_share", "type": "float", "mode": 
>> "nullable"}
>> ]
>> 
>> dag.py
>> import bqschema
>> ...
>> all in the same dag folder. We use it to define out BigQuery schema's into a 
>> seperate file.
>> 
>> 
>> On Tue, Feb 7, 2017 at 6:37 PM Feng Lu > > wrote:
>> Hi Alex-
>> 
>> Please see the attached screenshots of my local testing using celeryexecutor 
>> (on k8s as well). 
>> All look good and the workflow is successfully completed.
>> 
>> Curious did you also update the worker image? 
>> Sorry for the confusion, happy to debug more if you could share with me your 
>> k8s setup. 
>> 
>> Feng
>> 
>> On Tue, Feb 7, 2017 at 8:37 AM, Feng Lu > > wrote:
>> When num_runs is not explicitly specified, the default is set to -1 to match 
>> the expectation of SchedulerJob here:
>> 
>> ​
>> Doing so also matches the type of num_runs ('int' in this case).
>> The scheduler will run non-stop as a result regardless whether dag files are 
>> present (since the num_runs default is now -1: unlimited). 
>> 
>> Based on what Alex described, the import error doesn't look like directly 
>> related to this change. 
>> Maybe this one? 
>> https://github.com/apache/incubator-airflow/commit/67cbb966410226c1489bb730af3af45330fc51b9
>>  
>> 
>> 
>> I am still in the middle of running some quick test using celery executor, 
>> will update the thread once it's done. 
>> 
>> 
>> On Tue, Feb 7, 2017 at 6:56 AM, Bolke de Bruin > > wrote:
>> Hey Alex,
>> 
>> Thanks for tracking it down. Can you elaborate want went wrong with celery? 
>> The lines below do not particularly relate to Celery directly, so I wonder 
>> why we are not seeing it with LocalExecutor?
>> 
>> Cheers
>> Bolke
>> 
>> > On 7 Feb 2017, at 15:51, Alex Van Boxel > > > wrote:
>> >
>> > I have to give the RC1 a *-1*. I spend hours, or better days to get the RC
>> > running with Celery on our test environment, till I finally found the
>> > commit that killed it:
>> >
>> > e7f6212cae82c3a3a0bc17bbcbc70646f67d02eb
>> > [AIRFLOW-813] Fix unterminated unit tests in SchedulerJobTest
>> > Closes #2032 from fenglu-g/master
>> >
>> > I was always looking at the wrong this, because the commit only changes a
>> > single default parameter from *None to -1*
>> >
>> > I do have the impression I'm the only one running with Celery. Are other
>> > people running with it?
>> >
>> > *I propose* *reverting the commit*. Feng, can you elaborate on this change?
>> >
>> > Change the default back no *None* in cli.py got it finally working:
>> >
>> > 'num_runs': Arg(
>> >("-n", "--num_runs"),
>> >default=None, type=int,
>> >help="Set the number of runs to execute before exiting"),
>> >
>> > Thanks.
>> >
>> > On Tue, Feb 7, 2017 at 3:49 AM siddharth anand > > > wrote:
>> >
>> > I did get 1.8.0 installed and running at Agari.
>> >
>> > I did run into 2 problems.
>> > 1. Most of our DAGs broke due the way Operators are now imported.
>> > https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#deprecated-features
>> >  
>> > 
>> >
>> > According to the documentation, these deprecations would only cause an
>> > issue in 2.0. However, I needed to fix them now.
>> >
>> > So, I needed to change "from airflow.operators import PythonOperator" to
>> > from "from airflow.operators.pyth

Re: Airflow 1.8.0 Release Candidate 1

[Alex Van Boxel]

I'm still going over the code how such a small change can have such a huge
effect. Some things that is specific to the setup:

worker/scheduler/webserver all run with no extra parameters
build in docker
Python 2.7.13
Celery with redis
Runs on Kubernetes

When connecting to scheduler pod I see the scheduler forking other
scheduler processes that seem to stop immediately (probably with the Dag
scanning).

It's quite hard debugging in k8s. I'll try to find something more.



On Wed, Feb 8, 2017 at 1:33 PM Bolke de Bruin  wrote:

> Alex,
>
> Do you have anything more to go on? I don’t mind reverting the patch,
> however it code part seems unrelated to what you described and the issue
> wasn’t reproducible. I would really like to see more logging and maybe a
> test in a clean environment plus debugging. Preferable I would like to make
> RC 2 available today and immediately raise a vote as the *current* changes
> are really small, are confined to contrib and have been tested by the
> people using it.
>
> But I am holding off for now due to your concern.
>
> Cheers
> Bolke
>
>
> On 7 Feb 2017, at 20:56, Bolke de Bruin  wrote:
>
> How do you start the scheduler Alex? What are the command line parameters?
> What are the logs when it doesn’t work?
>
> Bolke
>
>
>
> On 7 Feb 2017, at 18:52, Alex Van Boxel  wrote:
>
> Hey Feng,
>
> The upgrades are all automated (including the workers/web/scheduler). And
> I tripple checked, I now am test running RC1 just with the your line
> reverted (and look ok)
>
> Could you do me a favour and add a test dag where you do a local import.
> Example:
>
> bqschema.py
>
> def ranking():
> return [
> {"name": "bucket_date", "type": "timestamp", "mode": "nullable"},
> {"name": "rank", "type": "integer", "mode": "nullable"},
> {"name": "audience_preference", "type": "float", "mode": "nullable"},
> {"name": "audience_likelihood_share", "type": "float", "mode": 
> "nullable"}
> ]
>
>
> dag.py
>
> import bqschema
> *...*
>
> all in the same dag folder. We use it to define out BigQuery schema's into
> a seperate file.
>
>
> On Tue, Feb 7, 2017 at 6:37 PM Feng Lu  wrote:
>
> Hi Alex-
>
> Please see the attached screenshots of my local testing using
> celeryexecutor (on k8s as well).
> All look good and the workflow is successfully completed.
>
> Curious did you also update the worker image?
> Sorry for the confusion, happy to debug more if you could share with me
> your k8s setup.
>
> Feng
>
> On Tue, Feb 7, 2017 at 8:37 AM, Feng Lu  wrote:
>
> When num_runs is not explicitly specified, the default is set to -1 to
> match the expectation of SchedulerJob here:
> 
> ​
> Doing so also matches the type of num_runs ('int' in this case).
> The scheduler will run non-stop as a result regardless whether dag files
> are present (since the num_runs default is now -1: unlimited).
>
> Based on what Alex described, the import error doesn't look like directly
> related to this change.
> Maybe this one?
> https://github.com/apache/incubator-airflow/commit/67cbb966410226c1489bb730af3af45330fc51b9
>
> I am still in the middle of running some quick test using celery executor,
> will update the thread once it's done.
>
>
> On Tue, Feb 7, 2017 at 6:56 AM, Bolke de Bruin  wrote:
>
> Hey Alex,
>
> Thanks for tracking it down. Can you elaborate want went wrong with
> celery? The lines below do not particularly relate to Celery directly, so I
> wonder why we are not seeing it with LocalExecutor?
>
> Cheers
> Bolke
>
> > On 7 Feb 2017, at 15:51, Alex Van Boxel  wrote:
> >
> > I have to give the RC1 a *-1*. I spend hours, or better days to get the
> RC
> > running with Celery on our test environment, till I finally found the
> > commit that killed it:
> >
> > e7f6212cae82c3a3a0bc17bbcbc70646f67d02eb
> > [AIRFLOW-813] Fix unterminated unit tests in SchedulerJobTest
> > Closes #2032 from fenglu-g/master
> >
> > I was always looking at the wrong this, because the commit only changes a
> > single default parameter from *None to -1*
> >
> > I do have the impression I'm the only one running with Celery. Are other
> > people running with it?
> >
> > *I propose* *reverting the commit*. Feng, can you elaborate on this
> change?
> >
> > Change the default back no *None* in cli.py got it finally working:
> >
> > 'num_runs': Arg(
> >("-n", "--num_runs"),
> >default=None, type=int,
> >help="Set the number of runs to execute before exiting"),
> >
> > Thanks.
> >
> > On Tue, Feb 7, 2017 at 3:49 AM siddharth anand 
> wrote:
> >
> > I did get 1.8.0 installed and running at Agari.
> >
> > I did run into 2 problems.
> > 1. Most of our DAGs broke due the way Operators are now imported.
> >
> https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#deprecated-features
> >
> > According to the documentation, these deprecations would only cause an
> > issue in 2.0. However, I needed to fix them now.
> >
> > So, I needed to change "from airflow.operators import PythonOperator" to