Re: best way to handle version upgrades of libraries used by tasks

[Dennis O'Brien]

Thanks for the input!  I'll take a look at using queues for this.

thanks,
Dennis

On Tue, Jan 30, 2018 at 4:17 PM Hbw  wrote:

> Run them on different workers by using queues?
> That way different workers can have different 3rd party libs while sharing
> the same af core.
>
> B
>
> Sent from a device with less than stellar autocorrect
>
> > On Jan 30, 2018, at 9:13 AM, Dennis O'Brien 
> wrote:
> >
> > Hi All,
> >
> > I have a number of jobs that use scikit-learn for scoring players.
> > Occasionally I need to upgrade scikit-learn to take advantage of some new
> > features.  We have a single conda environment that specifies all the
> > dependencies for Airflow as well as for all of our DAGs.  So currently
> > upgrading scikit-learn means upgrading it for all DAGs that use it, and
> > retraining all models for that version.  It becomes a very involved task
> > and I'm hoping to find a better way.
> >
> > One option is to use BashOperator (or something that wraps BashOperator)
> > and have bash use a specific conda environment with that version of
> > scikit-learn.  While simple, I don't like the idea of limiting task input
> > to the command line.  Still, an option.
> >
> > Another option is the DockerOperator.  But when I asked around at a
> > previous Airflow Meetup, I couldn't find anyone actually using it.  It
> also
> > adds some complexity to the build and deploy process in that now I have
> to
> > maintain docker images for all my environments.  Still, not ruling it
> out.
> >
> > And the last option I can think of is just heterogeneous workers.  We are
> > migrating our Airflow infrastructure to AWS ECS (from EC2) and plan on
> > having support for separate worker clusters, so this could include
> workers
> > with different conda environments.  I assume as long as a few key
> packages
> > are identical between scheduler and worker instances (airflow, redis,
> > celery?) the rest can be whatever.
> >
> > Has anyone faced this problem and have some advice?  Am I missing any
> > simpler options?  Any thoughts much appreciated.
> >
> > thanks,
> > Dennis
>

Re: Airflow Documentation - Readthedocs

[Andy Hadjigeorgiou]

Same here - I still have a couple outstanding tickets re: documentation for 
2.0, would be helpful if I had additional access

- Andy

> On Feb 4, 2018, at 12:07 PM, Naik Kaxil  wrote:
> 
> Happy to pick it up if permitted.
> 
> Regards,
> Kaxil
> 
> On 04/02/2018, 16:14, "Bolke de Bruin"  wrote:
> 
>Perfectly fine to me. It would be highly appreciated if this could be 
> picked up so it can be made opart of the release process.
> 
>@Arthur @max: can you share / make available credentials?
> 
>Cheers
>Bolke
> 
> 
>> On 4 Feb 2018, at 15:16, Naik Kaxil  wrote:
>> 
>> Yes, the same happened to me. One of my colleague wanted to use a Google 
>> Cloud Storage operator but he was looking at the old docs so he couldn't 
>> find one.
>> 
>> I suggest we should have documentation at 2 places:
>> 1) PythonHosted - For latest stable airflow version on PyPI 
>> 2) Readthedocs - For up-to-date documentation of Airflow on GitHub
>> 
>> Regards,
>> Kaxil
>> 
>> On 04/02/2018, 13:47, "fo...@driesprongen.nl on behalf of Driesprong, Fokko" 
>>  wrote:
>> 
>>   This is a good question, I see that artwr and Maxime are owners of the
>>   project, maybe they can fix the build? Or add some more Airflow committers
>>   as readthedocs project owners.
>> 
>>   I get quite some questions from colleagues because they are reading old
>>   docs :)
>> 
>>   Cheers, Fokko
>> 
>>   2018-02-04 14:32 GMT+01:00 Naik Kaxil :
>> 
>>> Hi guys,
>>> 
>>> 
>>> 
>>> Are we still using http://airflow.readthedocs.io/ for latest
>>> documentation?
>>> 
>>> 
>>> 
>>> I see that the last build was 2 months ago which failed..
>>> 
>>> 
>>> 
>>> http://readthedocs.org/projects/airflow/builds/
>>> 
>>> 
>>> 
>>> The documentation at http://pythonhosted.org/airflow/ will only be for
>>> the latest airflow version at PyPI (1.9 for now).
>>> 
>>> 
>>> 
>>> It would be good to have documentation of the GitHub version at
>>> readthedocs.
>>> 
>>> 
>>> 
>>> Regards,
>>> 
>>> Kaxil
>>> 
>>> 
>>> Kaxil Naik
>>> 
>>> Data Reply
>>> 38 Grosvenor Gardens
>>> 
>>> London SW1W 0EB - UK
>>> phone: +44 (0)20 7730 6000 <+44%2020%207730%206000>
>>> k.n...@reply.com
>>> www.reply.com
>>> 
>>> [image: Data Reply]
>>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> Kaxil Naik 
>> 
>> Data Reply
>> 38 Grosvenor Gardens
>> London SW1W 0EB - UK 
>> phone: +44 (0)20 7730 6000
>> k.n...@reply.com
>> www.reply.com
> 
> 
> 
> 
> 
> 
> 
> Kaxil Naik 
> 
> Data Reply
> 38 Grosvenor Gardens
> London SW1W 0EB - UK 
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com

Re: Airflow Documentation - Readthedocs

[Naik Kaxil]

Happy to pick it up if permitted.

Regards,
Kaxil

On 04/02/2018, 16:14, "Bolke de Bruin"  wrote:

Perfectly fine to me. It would be highly appreciated if this could be 
picked up so it can be made opart of the release process.

@Arthur @max: can you share / make available credentials?

Cheers
Bolke


> On 4 Feb 2018, at 15:16, Naik Kaxil  wrote:
> 
> Yes, the same happened to me. One of my colleague wanted to use a Google 
Cloud Storage operator but he was looking at the old docs so he couldn't find 
one.
> 
> I suggest we should have documentation at 2 places:
> 1) PythonHosted - For latest stable airflow version on PyPI 
> 2) Readthedocs - For up-to-date documentation of Airflow on GitHub
> 
> Regards,
> Kaxil
> 
> On 04/02/2018, 13:47, "fo...@driesprongen.nl on behalf of Driesprong, 
Fokko"  wrote:
> 
>This is a good question, I see that artwr and Maxime are owners of the
>project, maybe they can fix the build? Or add some more Airflow 
committers
>as readthedocs project owners.
> 
>I get quite some questions from colleagues because they are reading old
>docs :)
> 
>Cheers, Fokko
> 
>2018-02-04 14:32 GMT+01:00 Naik Kaxil :
> 
>> Hi guys,
>> 
>> 
>> 
>> Are we still using http://airflow.readthedocs.io/ for latest
>> documentation?
>> 
>> 
>> 
>> I see that the last build was 2 months ago which failed..
>> 
>> 
>> 
>> http://readthedocs.org/projects/airflow/builds/
>> 
>> 
>> 
>> The documentation at http://pythonhosted.org/airflow/ will only be for
>> the latest airflow version at PyPI (1.9 for now).
>> 
>> 
>> 
>> It would be good to have documentation of the GitHub version at
>> readthedocs.
>> 
>> 
>> 
>> Regards,
>> 
>> Kaxil
>> 
>> 
>> Kaxil Naik
>> 
>> Data Reply
>> 38 Grosvenor Gardens
>> 

>> London SW1W 0EB - UK
>> phone: +44 (0)20 7730 6000 <+44%2020%207730%206000>
>> k.n...@reply.com
>> www.reply.com
>> 
>> [image: Data Reply]
>> 
> 
> 
> 
> 
> 
> 
> Kaxil Naik 
> 
> Data Reply
> 38 Grosvenor Gardens
> London SW1W 0EB - UK 
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com







Kaxil Naik 

Data Reply
38 Grosvenor Gardens
London SW1W 0EB - UK 
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com

Re: Airflow Documentation - Readthedocs

[Bolke de Bruin]

Perfectly fine to me. It would be highly appreciated if this could be picked up 
so it can be made opart of the release process.

@Arthur @max: can you share / make available credentials?

Cheers
Bolke


> On 4 Feb 2018, at 15:16, Naik Kaxil  wrote:
> 
> Yes, the same happened to me. One of my colleague wanted to use a Google 
> Cloud Storage operator but he was looking at the old docs so he couldn't find 
> one.
> 
> I suggest we should have documentation at 2 places:
> 1) PythonHosted - For latest stable airflow version on PyPI 
> 2) Readthedocs - For up-to-date documentation of Airflow on GitHub
> 
> Regards,
> Kaxil
> 
> On 04/02/2018, 13:47, "fo...@driesprongen.nl on behalf of Driesprong, Fokko" 
>  wrote:
> 
>This is a good question, I see that artwr and Maxime are owners of the
>project, maybe they can fix the build? Or add some more Airflow committers
>as readthedocs project owners.
> 
>I get quite some questions from colleagues because they are reading old
>docs :)
> 
>Cheers, Fokko
> 
>2018-02-04 14:32 GMT+01:00 Naik Kaxil :
> 
>> Hi guys,
>> 
>> 
>> 
>> Are we still using http://airflow.readthedocs.io/ for latest
>> documentation?
>> 
>> 
>> 
>> I see that the last build was 2 months ago which failed..
>> 
>> 
>> 
>> http://readthedocs.org/projects/airflow/builds/
>> 
>> 
>> 
>> The documentation at http://pythonhosted.org/airflow/ will only be for
>> the latest airflow version at PyPI (1.9 for now).
>> 
>> 
>> 
>> It would be good to have documentation of the GitHub version at
>> readthedocs.
>> 
>> 
>> 
>> Regards,
>> 
>> Kaxil
>> 
>> 
>> Kaxil Naik
>> 
>> Data Reply
>> 38 Grosvenor Gardens
>> 
>> London SW1W 0EB - UK
>> phone: +44 (0)20 7730 6000 <+44%2020%207730%206000>
>> k.n...@reply.com
>> www.reply.com
>> 
>> [image: Data Reply]
>> 
> 
> 
> 
> 
> 
> 
> Kaxil Naik 
> 
> Data Reply
> 38 Grosvenor Gardens
> London SW1W 0EB - UK 
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com

Re: Airflow Documentation - Readthedocs

[Naik Kaxil]

Yes, the same happened to me. One of my colleague wanted to use a Google Cloud 
Storage operator but he was looking at the old docs so he couldn't find one.

I suggest we should have documentation at 2 places:
1) PythonHosted - For latest stable airflow version on PyPI 
2) Readthedocs - For up-to-date documentation of Airflow on GitHub

Regards,
Kaxil

On 04/02/2018, 13:47, "fo...@driesprongen.nl on behalf of Driesprong, Fokko" 
 wrote:

This is a good question, I see that artwr and Maxime are owners of the
project, maybe they can fix the build? Or add some more Airflow committers
as readthedocs project owners.

I get quite some questions from colleagues because they are reading old
docs :)

Cheers, Fokko

2018-02-04 14:32 GMT+01:00 Naik Kaxil :

> Hi guys,
>
>
>
> Are we still using http://airflow.readthedocs.io/ for latest
> documentation?
>
>
>
> I see that the last build was 2 months ago which failed..
>
>
>
> http://readthedocs.org/projects/airflow/builds/
>
>
>
> The documentation at http://pythonhosted.org/airflow/ will only be for
> the latest airflow version at PyPI (1.9 for now).
>
>
>
> It would be good to have documentation of the GitHub version at
> readthedocs.
>
>
>
> Regards,
>
> Kaxil
>
>
> Kaxil Naik
>
> Data Reply
> 38 Grosvenor Gardens
> 

> London SW1W 0EB - UK
> phone: +44 (0)20 7730 6000 <+44%2020%207730%206000>
> k.n...@reply.com
> www.reply.com
>
> [image: Data Reply]
>






Kaxil Naik 

Data Reply
38 Grosvenor Gardens
London SW1W 0EB - UK 
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com

Re: Airflow Documentation - Readthedocs

[Driesprong, Fokko]

This is a good question, I see that artwr and Maxime are owners of the
project, maybe they can fix the build? Or add some more Airflow committers
as readthedocs project owners.

I get quite some questions from colleagues because they are reading old
docs :)

Cheers, Fokko

2018-02-04 14:32 GMT+01:00 Naik Kaxil :

> Hi guys,
>
>
>
> Are we still using http://airflow.readthedocs.io/ for latest
> documentation?
>
>
>
> I see that the last build was 2 months ago which failed..
>
>
>
> http://readthedocs.org/projects/airflow/builds/
>
>
>
> The documentation at http://pythonhosted.org/airflow/ will only be for
> the latest airflow version at PyPI (1.9 for now).
>
>
>
> It would be good to have documentation of the GitHub version at
> readthedocs.
>
>
>
> Regards,
>
> Kaxil
>
>
> Kaxil Naik
>
> Data Reply
> 38 Grosvenor Gardens
> 
> London SW1W 0EB - UK
> phone: +44 (0)20 7730 6000 <+44%2020%207730%206000>
> k.n...@reply.com
> www.reply.com
>
> [image: Data Reply]
>

Airflow Documentation - Readthedocs

[Naik Kaxil]

Hi guys,

Are we still using http://airflow.readthedocs.io/ for latest documentation?

I see that the last build was 2 months ago which failed..

http://readthedocs.org/projects/airflow/builds/

The documentation at http://pythonhosted.org/airflow/ will only be for the 
latest airflow version at PyPI (1.9 for now).

It would be good to have documentation of the GitHub version at readthedocs.

Regards,
Kaxil


Kaxil Naik

Data Reply
38 Grosvenor Gardens
London SW1W 0EB - UK
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com

[Data Reply]