Re: Python 3 (3.4) compatibility bug?
No, I've been running python 3 in my testing environment for sometime now
(migrating prod soon) which is why I'm surprised I hadn't encountered this
before.
On Wed, Mar 29, 2017 at 7:50 AM Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:
> snakebite isn't py3 compatible.
>
> Did you both upgrade to 1.8.0 and to py3 at the same time?
>
> Max
>
> On Wed, Mar 29, 2017 at 1:27 AM, Nicholas Hodgkinson <
> nik.hodgkin...@collectivehealth.com> wrote:
>
> > So I upgraded my airflow instance to 1.8.0 and found that several of my
> > DAGs wouldn't load due to this exception:
> >
> > [2017-03-29 07:04:25,294] [5] {models.py:266} ERROR - Failed to import:
> > /opt/airflow/dags/automator-sensor.py
> > Traceback (most recent call last):
> > File "/usr/local/lib/python3.4/dist-packages/airflow/models.py", line
> > 263, in process_file
> > m = imp.load_source(mod_name, filepath)
> > File "/usr/lib/python3.4/imp.py", line 171, in load_source
> > module = methods.load()
> > File "", line 1220, in load
> > File "", line 1200, in _load_unlocked
> > File "", line 1129, in _exec
> > File "", line 1471, in exec_module
> > File "", line 321, in
> > _call_with_frames_removed
> > File "/opt/airflow/dags/automator-sensor.py", line 11, in
> > from lib.data_loading import DataLoader
> > File "/opt/airflow/dags/lib/data_loading.py", line 16, in
> > from airflow.operators.sensors import S3KeySensor
> > File
> > "/usr/local/lib/python3.4/dist-packages/airflow/operators/sensors.py",
> > line
> > 33, in
> > from airflow.hooks.hdfs_hook import HDFSHook
> > File "/usr/local/lib/python3.4/dist-packages/airflow/hooks/
> > hdfs_hook.py",
> > line 20, in
> > from snakebite.client import Client, HAClient, Namenode,
> > AutoConfigClient
> > File "/usr/local/lib/python3.4/dist-packages/snakebite/client.py", line
> > 1473
> > baseTime = min(time * (1L << retries), cap);
> > ^
> > SyntaxError: invalid syntax
> >
> > I had to actually uninstall snakebite to get my DAGs to load properly.
> Has
> > anyone else encountered this and is there a workaround other than
> > uninstalling the library? I didn't find an issue on JIRA for it either,
> > happy to submit a ticket if I'm not missing anything.
> >
> > -N
> > nik.hodgkin...@collectivehealth.com
> >
> > --
> >
> >
> > Read our founder's story.
> > <https://collectivehealth.com/blog/started-collective-health/>
> >
> > *This message may contain confidential, proprietary, or protected
> > information. If you are not the intended recipient, you may not review,
> > copy, or distribute this message. If you received this message in error,
> > please notify the sender by reply email and delete this message.*
> >
>
--
-N
nik.hodgkin...@collectivehealth.com
(913) 927-4891
--
Read our founder's story.
<https://collectivehealth.com/blog/started-collective-health/>
*This message may contain confidential, proprietary, or protected
information. If you are not the intended recipient, you may not review,
copy, or distribute this message. If you received this message in error,
please notify the sender by reply email and delete this message.*
Re: Python 3 (3.4) compatibility bug?
snakebite isn't py3 compatible.
Did you both upgrade to 1.8.0 and to py3 at the same time?
Max
On Wed, Mar 29, 2017 at 1:27 AM, Nicholas Hodgkinson <
nik.hodgkin...@collectivehealth.com> wrote:
> So I upgraded my airflow instance to 1.8.0 and found that several of my
> DAGs wouldn't load due to this exception:
>
> [2017-03-29 07:04:25,294] [5] {models.py:266} ERROR - Failed to import:
> /opt/airflow/dags/automator-sensor.py
> Traceback (most recent call last):
> File "/usr/local/lib/python3.4/dist-packages/airflow/models.py", line
> 263, in process_file
> m = imp.load_source(mod_name, filepath)
> File "/usr/lib/python3.4/imp.py", line 171, in load_source
> module = methods.load()
> File "", line 1220, in load
> File "", line 1200, in _load_unlocked
> File "", line 1129, in _exec
> File "", line 1471, in exec_module
> File "", line 321, in
> _call_with_frames_removed
> File "/opt/airflow/dags/automator-sensor.py", line 11, in
> from lib.data_loading import DataLoader
> File "/opt/airflow/dags/lib/data_loading.py", line 16, in
> from airflow.operators.sensors import S3KeySensor
> File
> "/usr/local/lib/python3.4/dist-packages/airflow/operators/sensors.py",
> line
> 33, in
> from airflow.hooks.hdfs_hook import HDFSHook
> File "/usr/local/lib/python3.4/dist-packages/airflow/hooks/
> hdfs_hook.py",
> line 20, in
> from snakebite.client import Client, HAClient, Namenode,
> AutoConfigClient
> File "/usr/local/lib/python3.4/dist-packages/snakebite/client.py", line
> 1473
> baseTime = min(time * (1L << retries), cap);
> ^
> SyntaxError: invalid syntax
>
> I had to actually uninstall snakebite to get my DAGs to load properly. Has
> anyone else encountered this and is there a workaround other than
> uninstalling the library? I didn't find an issue on JIRA for it either,
> happy to submit a ticket if I'm not missing anything.
>
> -N
> nik.hodgkin...@collectivehealth.com
>
> --
>
>
> Read our founder's story.
>
Python 3 (3.4) compatibility bug?
So I upgraded my airflow instance to 1.8.0 and found that several of my
DAGs wouldn't load due to this exception:
[2017-03-29 07:04:25,294] [5] {models.py:266} ERROR - Failed to import:
/opt/airflow/dags/automator-sensor.py
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/airflow/models.py", line
263, in process_file
m = imp.load_source(mod_name, filepath)
File "/usr/lib/python3.4/imp.py", line 171, in load_source
module = methods.load()
File "", line 1220, in load
File "", line 1200, in _load_unlocked
File "", line 1129, in _exec
File "", line 1471, in exec_module
File "", line 321, in
_call_with_frames_removed
File "/opt/airflow/dags/automator-sensor.py", line 11, in
from lib.data_loading import DataLoader
File "/opt/airflow/dags/lib/data_loading.py", line 16, in
from airflow.operators.sensors import S3KeySensor
File
"/usr/local/lib/python3.4/dist-packages/airflow/operators/sensors.py", line
33, in
from airflow.hooks.hdfs_hook import HDFSHook
File "/usr/local/lib/python3.4/dist-packages/airflow/hooks/hdfs_hook.py",
line 20, in
from snakebite.client import Client, HAClient, Namenode,
AutoConfigClient
File "/usr/local/lib/python3.4/dist-packages/snakebite/client.py", line
1473
baseTime = min(time * (1L << retries), cap);
^
SyntaxError: invalid syntax
I had to actually uninstall snakebite to get my DAGs to load properly. Has
anyone else encountered this and is there a workaround other than
uninstalling the library? I didn't find an issue on JIRA for it either,
happy to submit a ticket if I'm not missing anything.
-N
nik.hodgkin...@collectivehealth.com
--
Read our founder's story.
Re: Python 3
Thank you Conor! We have managed to get this working after looking at the configuration you provided. We switched out librabbitmq with pyamqp. Also, thank you to everyone else who responded to this thread! Cheers, Luke Maycock OLIVER WYMAN luke.mayc...@affiliate.oliverwyman.com<mailto:luke.mayc...@affiliate.oliverwyman.com> www.oliverwyman.com<http://www.oliverwyman.com/> From: Conor Nash Sent: 03 January 2017 14:51 To: dev@airflow.incubator.apache.org Subject: Re: Python 3 We are using Airflow with Python 3.5 and RabbitMQ (CeleryExecutor). We're using Celery version 3.1.24. For the BROKER_URL connection string we use "pyamqp://:@host:5671/vhost" and for CELERY_RESULT_BACKEND we use "rpc://". Is Redis the preferred/better supported backend? Cheers, Conor Nash Data Science Consulting www.conornash.com | @conornash <https://twitter.com/conornash> On Sun, Jan 1, 2017 at 9:12 PM, George Leslie-Waksman < geo...@cloverhealth.com.invalid> wrote: > We are using Airflow with Python3, PostgreSQL and Redis; it works just > fine. > > On Thu, Dec 1, 2016 at 10:30 AM Maycock, Luke < > luke.mayc...@affiliate.oliverwyman.com> wrote: > > > Thanks Dennis. We will perhaps try Redis as there has been a ticket to > > make librabbitmq Python 3 compatible since 2012. > > > > > > Cheers, > > Luke Maycock > > OLIVER WYMAN > > luke.mayc...@affiliate.oliverwyman.com > luke.mayc...@affiliate.oliverwyman.com> > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > From: Dennis O'Brien > > Sent: 01 December 2016 04:45 > > To: dev@airflow.incubator.apache.org > > Subject: Re: Python 3 > > > > Hi Luke, > > > > I've been running Airflow on Python 3.5 with Celery. I'm using Redis as > > the message broker (and ElastiCache Redis in production). I haven't > tried > > RabbitMQ so I can't speak to its compatibility with Python 3.5. > > > > On Mon, Nov 28, 2016 at 9:18 AM Maycock, Luke < > > luke.mayc...@affiliate.oliverwyman.com> wrote: > > > > > I thought it would be worth following up on this given our experience > of > > > testing Python 3 compatibility. > > > > > > > > > We are using Celery and RabbitMQ. Celery uses librabbitmq for > interaction > > > with RabbitMQ but librabbitmq is not compatible with Python 3 (see > > > https://github.com/celery/librabbitmq/issues/13). Has anyone else been > > > able to get Airflow/Python 3/Celery working? > > > > > > > > > > > > Cheers, > > > Luke Maycock > > > OLIVER WYMAN > > > luke.mayc...@affiliate.oliverwyman.com > > luke.mayc...@affiliate.oliverwyman.com> > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > > > > From: Maycock, Luke > > > Sent: 28 November 2016 12:33 > > > To: dev@airflow.incubator.apache.org > > > Subject: Re: Python 3 > > > > > > Thank you Dmitriy and Li. > > > > > > Luke Maycock > > > OLIVER WYMAN > > > luke.mayc...@affiliate.oliverwyman.com > > luke.mayc...@affiliate.oliverwyman.com> > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > > > > From: Li Xuan Ji > > > Sent: 28 November 2016 12:20 > > > To: dev@airflow.incubator.apache.org > > > Subject: Re: Python 3 > > > > > > Airflow is supposed to be Python 3 compatible and any > > > incompatibilities are bugs on our side. > > > > > > I haven't heard of anyone running airflow on windows, but I'm not sure > > > what the official stance is. > > > > > > On 28 November 2016 at 07:01, Dmitriy Krasnikov < > dkrasni...@hotmail.com> > > > wrote: > > > > I have been using it and developing plugins for it in Python 3.5 with > > no > > > problems. > > > > > > > > > > > > -Original Message- > > > > From: Maycock, Luke [mailto:luke.mayc...@affiliate.oliverwyman.com] > > > > Sent: Monday, November 28, 2016 6:58 AM > > > > To: dev@airflow.incubator.apache.org > > > > Subject: Python 3 > > > > > > > > Hi All, > > > > > > > > > > > > I have been using Airflow for a while now and have a
Re: Python 3
Per Celery's documentation <http://docs.celeryproject.org/en/latest/getting-started/brokers/>, Redis and RabbitMQ are the best choices. We use Redis at Airbnb and it's been a breeze, I don't think we've ever had any issues coming from that layer. <http://docs.celeryproject.org/en/2.4-archived/getting-started/brokers/sqlalchemy.html> Using SqlAlchemy <http://docs.celeryproject.org/en/2.4-archived/getting-started/brokers/sqlalchemy.html> (backed by mysql or postgres) as the Celery broker may work pretty well at the low volume that Airflow typically needs. The Celery docs say something about "can lead to messages being executed multiple times", but Airflow has safeguards for that so it should not be a problem. Airflow has also safeguards around lost messages. I'd love to hear reports of people using that broker in production, let us know if you do! Max On Tue, Jan 3, 2017 at 6:51 AM, Conor Nash wrote: > We are using Airflow with Python 3.5 and RabbitMQ (CeleryExecutor). We're > using Celery version 3.1.24. For the BROKER_URL connection string we use > "pyamqp://:@host:5671/vhost" and for CELERY_RESULT_BACKEND we > use "rpc://". > > Is Redis the preferred/better supported backend? > > Cheers, > > Conor Nash > Data Science Consulting > www.conornash.com | @conornash <https://twitter.com/conornash> > > On Sun, Jan 1, 2017 at 9:12 PM, George Leslie-Waksman < > geo...@cloverhealth.com.invalid> wrote: > > > We are using Airflow with Python3, PostgreSQL and Redis; it works just > > fine. > > > > On Thu, Dec 1, 2016 at 10:30 AM Maycock, Luke < > > luke.mayc...@affiliate.oliverwyman.com> wrote: > > > > > Thanks Dennis. We will perhaps try Redis as there has been a ticket to > > > make librabbitmq Python 3 compatible since 2012. > > > > > > > > > Cheers, > > > Luke Maycock > > > OLIVER WYMAN > > > luke.mayc...@affiliate.oliverwyman.com > > luke.mayc...@affiliate.oliverwyman.com> > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > > > > From: Dennis O'Brien > > > Sent: 01 December 2016 04:45 > > > To: dev@airflow.incubator.apache.org > > > Subject: Re: Python 3 > > > > > > Hi Luke, > > > > > > I've been running Airflow on Python 3.5 with Celery. I'm using Redis > as > > > the message broker (and ElastiCache Redis in production). I haven't > > tried > > > RabbitMQ so I can't speak to its compatibility with Python 3.5. > > > > > > On Mon, Nov 28, 2016 at 9:18 AM Maycock, Luke < > > > luke.mayc...@affiliate.oliverwyman.com> wrote: > > > > > > > I thought it would be worth following up on this given our experience > > of > > > > testing Python 3 compatibility. > > > > > > > > > > > > We are using Celery and RabbitMQ. Celery uses librabbitmq for > > interaction > > > > with RabbitMQ but librabbitmq is not compatible with Python 3 (see > > > > https://github.com/celery/librabbitmq/issues/13). Has anyone else > been > > > > able to get Airflow/Python 3/Celery working? > > > > > > > > > > > > > > > > Cheers, > > > > Luke Maycock > > > > OLIVER WYMAN > > > > luke.mayc...@affiliate.oliverwyman.com > > > luke.mayc...@affiliate.oliverwyman.com> > > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > > > > > > > > > From: Maycock, Luke > > > > Sent: 28 November 2016 12:33 > > > > To: dev@airflow.incubator.apache.org > > > > Subject: Re: Python 3 > > > > > > > > Thank you Dmitriy and Li. > > > > > > > > Luke Maycock > > > > OLIVER WYMAN > > > > luke.mayc...@affiliate.oliverwyman.com > > > luke.mayc...@affiliate.oliverwyman.com> > > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > > > > > > > > > From: Li Xuan Ji > > > > Sent: 28 November 2016 12:20 > > > > To: dev@airflow.incubator.apache.org > > > > Subject: Re: Python 3 > > > > > > > > Airflow is supposed to be Python 3 compatible and any > > > > incompatibilities are
Re: Python 3
We are using Airflow with Python 3.5 and RabbitMQ (CeleryExecutor). We're using Celery version 3.1.24. For the BROKER_URL connection string we use "pyamqp://:@host:5671/vhost" and for CELERY_RESULT_BACKEND we use "rpc://". Is Redis the preferred/better supported backend? Cheers, Conor Nash Data Science Consulting www.conornash.com | @conornash <https://twitter.com/conornash> On Sun, Jan 1, 2017 at 9:12 PM, George Leslie-Waksman < geo...@cloverhealth.com.invalid> wrote: > We are using Airflow with Python3, PostgreSQL and Redis; it works just > fine. > > On Thu, Dec 1, 2016 at 10:30 AM Maycock, Luke < > luke.mayc...@affiliate.oliverwyman.com> wrote: > > > Thanks Dennis. We will perhaps try Redis as there has been a ticket to > > make librabbitmq Python 3 compatible since 2012. > > > > > > Cheers, > > Luke Maycock > > OLIVER WYMAN > > luke.mayc...@affiliate.oliverwyman.com > luke.mayc...@affiliate.oliverwyman.com> > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > ____ > > From: Dennis O'Brien > > Sent: 01 December 2016 04:45 > > To: dev@airflow.incubator.apache.org > > Subject: Re: Python 3 > > > > Hi Luke, > > > > I've been running Airflow on Python 3.5 with Celery. I'm using Redis as > > the message broker (and ElastiCache Redis in production). I haven't > tried > > RabbitMQ so I can't speak to its compatibility with Python 3.5. > > > > On Mon, Nov 28, 2016 at 9:18 AM Maycock, Luke < > > luke.mayc...@affiliate.oliverwyman.com> wrote: > > > > > I thought it would be worth following up on this given our experience > of > > > testing Python 3 compatibility. > > > > > > > > > We are using Celery and RabbitMQ. Celery uses librabbitmq for > interaction > > > with RabbitMQ but librabbitmq is not compatible with Python 3 (see > > > https://github.com/celery/librabbitmq/issues/13). Has anyone else been > > > able to get Airflow/Python 3/Celery working? > > > > > > > > > > > > Cheers, > > > Luke Maycock > > > OLIVER WYMAN > > > luke.mayc...@affiliate.oliverwyman.com > > luke.mayc...@affiliate.oliverwyman.com> > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > > > > From: Maycock, Luke > > > Sent: 28 November 2016 12:33 > > > To: dev@airflow.incubator.apache.org > > > Subject: Re: Python 3 > > > > > > Thank you Dmitriy and Li. > > > > > > Luke Maycock > > > OLIVER WYMAN > > > luke.mayc...@affiliate.oliverwyman.com > > luke.mayc...@affiliate.oliverwyman.com> > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > > > > From: Li Xuan Ji > > > Sent: 28 November 2016 12:20 > > > To: dev@airflow.incubator.apache.org > > > Subject: Re: Python 3 > > > > > > Airflow is supposed to be Python 3 compatible and any > > > incompatibilities are bugs on our side. > > > > > > I haven't heard of anyone running airflow on windows, but I'm not sure > > > what the official stance is. > > > > > > On 28 November 2016 at 07:01, Dmitriy Krasnikov < > dkrasni...@hotmail.com> > > > wrote: > > > > I have been using it and developing plugins for it in Python 3.5 with > > no > > > problems. > > > > > > > > > > > > -Original Message- > > > > From: Maycock, Luke [mailto:luke.mayc...@affiliate.oliverwyman.com] > > > > Sent: Monday, November 28, 2016 6:58 AM > > > > To: dev@airflow.incubator.apache.org > > > > Subject: Python 3 > > > > > > > > Hi All, > > > > > > > > > > > > I have been using Airflow for a while now and have a couple of > > questions > > > that I am hoping someone knows the answer to: > > > > > > > > 1. Is Airflow now Python 3 compatible? The documentation used to > state > > > that Airflow was only compatible with Python 2.7 (< > > > > > https://web.archive.org/web/20160310111640/http:// > pythonhosted.org/airflow/start.html > > > > > > > > > https://web.archive.org/web/20160502024539/http:// > pythonhosted.org/airflow/start.html > > )
Re: Python 3
We are using Airflow with Python3, PostgreSQL and Redis; it works just fine. On Thu, Dec 1, 2016 at 10:30 AM Maycock, Luke < luke.mayc...@affiliate.oliverwyman.com> wrote: > Thanks Dennis. We will perhaps try Redis as there has been a ticket to > make librabbitmq Python 3 compatible since 2012. > > > Cheers, > Luke Maycock > OLIVER WYMAN > luke.mayc...@affiliate.oliverwyman.com luke.mayc...@affiliate.oliverwyman.com> > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > From: Dennis O'Brien > Sent: 01 December 2016 04:45 > To: dev@airflow.incubator.apache.org > Subject: Re: Python 3 > > Hi Luke, > > I've been running Airflow on Python 3.5 with Celery. I'm using Redis as > the message broker (and ElastiCache Redis in production). I haven't tried > RabbitMQ so I can't speak to its compatibility with Python 3.5. > > On Mon, Nov 28, 2016 at 9:18 AM Maycock, Luke < > luke.mayc...@affiliate.oliverwyman.com> wrote: > > > I thought it would be worth following up on this given our experience of > > testing Python 3 compatibility. > > > > > > We are using Celery and RabbitMQ. Celery uses librabbitmq for interaction > > with RabbitMQ but librabbitmq is not compatible with Python 3 (see > > https://github.com/celery/librabbitmq/issues/13). Has anyone else been > > able to get Airflow/Python 3/Celery working? > > > > > > > > Cheers, > > Luke Maycock > > OLIVER WYMAN > > luke.mayc...@affiliate.oliverwyman.com > luke.mayc...@affiliate.oliverwyman.com> > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > From: Maycock, Luke > > Sent: 28 November 2016 12:33 > > To: dev@airflow.incubator.apache.org > > Subject: Re: Python 3 > > > > Thank you Dmitriy and Li. > > > > Luke Maycock > > OLIVER WYMAN > > luke.mayc...@affiliate.oliverwyman.com > luke.mayc...@affiliate.oliverwyman.com> > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > From: Li Xuan Ji > > Sent: 28 November 2016 12:20 > > To: dev@airflow.incubator.apache.org > > Subject: Re: Python 3 > > > > Airflow is supposed to be Python 3 compatible and any > > incompatibilities are bugs on our side. > > > > I haven't heard of anyone running airflow on windows, but I'm not sure > > what the official stance is. > > > > On 28 November 2016 at 07:01, Dmitriy Krasnikov > > wrote: > > > I have been using it and developing plugins for it in Python 3.5 with > no > > problems. > > > > > > > > > -Original Message- > > > From: Maycock, Luke [mailto:luke.mayc...@affiliate.oliverwyman.com] > > > Sent: Monday, November 28, 2016 6:58 AM > > > To: dev@airflow.incubator.apache.org > > > Subject: Python 3 > > > > > > Hi All, > > > > > > > > > I have been using Airflow for a while now and have a couple of > questions > > that I am hoping someone knows the answer to: > > > > > > 1. Is Airflow now Python 3 compatible? The documentation used to state > > that Airflow was only compatible with Python 2.7 (< > > > https://web.archive.org/web/20160310111640/http://pythonhosted.org/airflow/start.html > > > > > > https://web.archive.org/web/20160502024539/http://pythonhosted.org/airflow/start.html > ) > > but that statement has since been removed. > > > 2. I know Gunicorn does not run on Windows - is this the only > > component of Airflow that does not run on Windows? > > > > > > > > > Cheers, > > > Luke Maycock > > > OLIVER WYMAN > > > luke.mayc...@affiliate.oliverwyman.com > luke.mayc...@affiliate.oliverwyman.com> > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > This e-mail and any attachments may be confidential or legally > > privileged. If you received this message in error or are not the intended > > recipient, you should destroy the e-mail message and any attachments or > > copies, and you are prohibited from retaining, distributing, disclosing > or > > using any information contained herein. Please inform us of the erroneous > > delivery by return e-mail. Thank you for your cooperation. > > > > > > > > -- > > Im Xuan Ji! > > > >
Re: Python 3
Thanks Dennis. We will perhaps try Redis as there has been a ticket to make librabbitmq Python 3 compatible since 2012. Cheers, Luke Maycock OLIVER WYMAN luke.mayc...@affiliate.oliverwyman.com<mailto:luke.mayc...@affiliate.oliverwyman.com> www.oliverwyman.com<http://www.oliverwyman.com/> From: Dennis O'Brien Sent: 01 December 2016 04:45 To: dev@airflow.incubator.apache.org Subject: Re: Python 3 Hi Luke, I've been running Airflow on Python 3.5 with Celery. I'm using Redis as the message broker (and ElastiCache Redis in production). I haven't tried RabbitMQ so I can't speak to its compatibility with Python 3.5. On Mon, Nov 28, 2016 at 9:18 AM Maycock, Luke < luke.mayc...@affiliate.oliverwyman.com> wrote: > I thought it would be worth following up on this given our experience of > testing Python 3 compatibility. > > > We are using Celery and RabbitMQ. Celery uses librabbitmq for interaction > with RabbitMQ but librabbitmq is not compatible with Python 3 (see > https://github.com/celery/librabbitmq/issues/13). Has anyone else been > able to get Airflow/Python 3/Celery working? > > > > Cheers, > Luke Maycock > OLIVER WYMAN > luke.mayc...@affiliate.oliverwyman.com luke.mayc...@affiliate.oliverwyman.com> > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > ____ > From: Maycock, Luke > Sent: 28 November 2016 12:33 > To: dev@airflow.incubator.apache.org > Subject: Re: Python 3 > > Thank you Dmitriy and Li. > > Luke Maycock > OLIVER WYMAN > luke.mayc...@affiliate.oliverwyman.com luke.mayc...@affiliate.oliverwyman.com> > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > ________ > From: Li Xuan Ji > Sent: 28 November 2016 12:20 > To: dev@airflow.incubator.apache.org > Subject: Re: Python 3 > > Airflow is supposed to be Python 3 compatible and any > incompatibilities are bugs on our side. > > I haven't heard of anyone running airflow on windows, but I'm not sure > what the official stance is. > > On 28 November 2016 at 07:01, Dmitriy Krasnikov > wrote: > > I have been using it and developing plugins for it in Python 3.5 with no > problems. > > > > > > -Original Message- > > From: Maycock, Luke [mailto:luke.mayc...@affiliate.oliverwyman.com] > > Sent: Monday, November 28, 2016 6:58 AM > > To: dev@airflow.incubator.apache.org > > Subject: Python 3 > > > > Hi All, > > > > > > I have been using Airflow for a while now and have a couple of questions > that I am hoping someone knows the answer to: > > > > 1. Is Airflow now Python 3 compatible? The documentation used to state > that Airflow was only compatible with Python 2.7 (< > https://web.archive.org/web/20160310111640/http://pythonhosted.org/airflow/start.html > > > https://web.archive.org/web/20160502024539/http://pythonhosted.org/airflow/start.html) > but that statement has since been removed. > > 2. I know Gunicorn does not run on Windows - is this the only > component of Airflow that does not run on Windows? > > > > > > Cheers, > > Luke Maycock > > OLIVER WYMAN > > luke.mayc...@affiliate.oliverwyman.com luke.mayc...@affiliate.oliverwyman.com> > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > This e-mail and any attachments may be confidential or legally > privileged. If you received this message in error or are not the intended > recipient, you should destroy the e-mail message and any attachments or > copies, and you are prohibited from retaining, distributing, disclosing or > using any information contained herein. Please inform us of the erroneous > delivery by return e-mail. Thank you for your cooperation. > > > > -- > Im Xuan Ji! > > > This e-mail and any attachments may be confidential or legally privileged. > If you received this message in error or are not the intended recipient, > you should destroy the e-mail message and any attachments or copies, and > you are prohibited from retaining, distributing, disclosing or using any > information contained herein. Please inform us of the erroneous delivery by > return e-mail. Thank you for your cooperation. > > > This e-mail and any attachments may be confidential or legally privileged. > If you received this message in error or are not the intended recipient, > you should destroy the e-mail message and any attachments or copies, and > you are prohibited from retaining, distributing, disclos
Re: Python 3
Hi Luke, I've been running Airflow on Python 3.5 with Celery. I'm using Redis as the message broker (and ElastiCache Redis in production). I haven't tried RabbitMQ so I can't speak to its compatibility with Python 3.5. On Mon, Nov 28, 2016 at 9:18 AM Maycock, Luke < luke.mayc...@affiliate.oliverwyman.com> wrote: > I thought it would be worth following up on this given our experience of > testing Python 3 compatibility. > > > We are using Celery and RabbitMQ. Celery uses librabbitmq for interaction > with RabbitMQ but librabbitmq is not compatible with Python 3 (see > https://github.com/celery/librabbitmq/issues/13). Has anyone else been > able to get Airflow/Python 3/Celery working? > > > > Cheers, > Luke Maycock > OLIVER WYMAN > luke.mayc...@affiliate.oliverwyman.com luke.mayc...@affiliate.oliverwyman.com> > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > From: Maycock, Luke > Sent: 28 November 2016 12:33 > To: dev@airflow.incubator.apache.org > Subject: Re: Python 3 > > Thank you Dmitriy and Li. > > Luke Maycock > OLIVER WYMAN > luke.mayc...@affiliate.oliverwyman.com luke.mayc...@affiliate.oliverwyman.com> > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > From: Li Xuan Ji > Sent: 28 November 2016 12:20 > To: dev@airflow.incubator.apache.org > Subject: Re: Python 3 > > Airflow is supposed to be Python 3 compatible and any > incompatibilities are bugs on our side. > > I haven't heard of anyone running airflow on windows, but I'm not sure > what the official stance is. > > On 28 November 2016 at 07:01, Dmitriy Krasnikov > wrote: > > I have been using it and developing plugins for it in Python 3.5 with no > problems. > > > > > > -Original Message- > > From: Maycock, Luke [mailto:luke.mayc...@affiliate.oliverwyman.com] > > Sent: Monday, November 28, 2016 6:58 AM > > To: dev@airflow.incubator.apache.org > > Subject: Python 3 > > > > Hi All, > > > > > > I have been using Airflow for a while now and have a couple of questions > that I am hoping someone knows the answer to: > > > > 1. Is Airflow now Python 3 compatible? The documentation used to state > that Airflow was only compatible with Python 2.7 (< > https://web.archive.org/web/20160310111640/http://pythonhosted.org/airflow/start.html > > > https://web.archive.org/web/20160502024539/http://pythonhosted.org/airflow/start.html) > but that statement has since been removed. > > 2. I know Gunicorn does not run on Windows - is this the only > component of Airflow that does not run on Windows? > > > > > > Cheers, > > Luke Maycock > > OLIVER WYMAN > > luke.mayc...@affiliate.oliverwyman.com luke.mayc...@affiliate.oliverwyman.com> > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > This e-mail and any attachments may be confidential or legally > privileged. If you received this message in error or are not the intended > recipient, you should destroy the e-mail message and any attachments or > copies, and you are prohibited from retaining, distributing, disclosing or > using any information contained herein. Please inform us of the erroneous > delivery by return e-mail. Thank you for your cooperation. > > > > -- > Im Xuan Ji! > > > This e-mail and any attachments may be confidential or legally privileged. > If you received this message in error or are not the intended recipient, > you should destroy the e-mail message and any attachments or copies, and > you are prohibited from retaining, distributing, disclosing or using any > information contained herein. Please inform us of the erroneous delivery by > return e-mail. Thank you for your cooperation. > > > This e-mail and any attachments may be confidential or legally privileged. > If you received this message in error or are not the intended recipient, > you should destroy the e-mail message and any attachments or copies, and > you are prohibited from retaining, distributing, disclosing or using any > information contained herein. Please inform us of the erroneous delivery by > return e-mail. Thank you for your cooperation. >
Re: Python 3
I thought it would be worth following up on this given our experience of testing Python 3 compatibility. We are using Celery and RabbitMQ. Celery uses librabbitmq for interaction with RabbitMQ but librabbitmq is not compatible with Python 3 (see https://github.com/celery/librabbitmq/issues/13). Has anyone else been able to get Airflow/Python 3/Celery working? Cheers, Luke Maycock OLIVER WYMAN luke.mayc...@affiliate.oliverwyman.com<mailto:luke.mayc...@affiliate.oliverwyman.com> www.oliverwyman.com<http://www.oliverwyman.com/> From: Maycock, Luke Sent: 28 November 2016 12:33 To: dev@airflow.incubator.apache.org Subject: Re: Python 3 Thank you Dmitriy and Li. Luke Maycock OLIVER WYMAN luke.mayc...@affiliate.oliverwyman.com<mailto:luke.mayc...@affiliate.oliverwyman.com> www.oliverwyman.com<http://www.oliverwyman.com/> From: Li Xuan Ji Sent: 28 November 2016 12:20 To: dev@airflow.incubator.apache.org Subject: Re: Python 3 Airflow is supposed to be Python 3 compatible and any incompatibilities are bugs on our side. I haven't heard of anyone running airflow on windows, but I'm not sure what the official stance is. On 28 November 2016 at 07:01, Dmitriy Krasnikov wrote: > I have been using it and developing plugins for it in Python 3.5 with no > problems. > > > -Original Message- > From: Maycock, Luke [mailto:luke.mayc...@affiliate.oliverwyman.com] > Sent: Monday, November 28, 2016 6:58 AM > To: dev@airflow.incubator.apache.org > Subject: Python 3 > > Hi All, > > > I have been using Airflow for a while now and have a couple of questions that > I am hoping someone knows the answer to: > > 1. Is Airflow now Python 3 compatible? The documentation used to state that > Airflow was only compatible with Python 2.7 > (<https://web.archive.org/web/20160310111640/http://pythonhosted.org/airflow/start.html>https://web.archive.org/web/20160502024539/http://pythonhosted.org/airflow/start.html) > but that statement has since been removed. > 2. I know Gunicorn does not run on Windows - is this the only component of > Airflow that does not run on Windows? > > > Cheers, > Luke Maycock > OLIVER WYMAN > luke.mayc...@affiliate.oliverwyman.com<mailto:luke.mayc...@affiliate.oliverwyman.com> > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > This e-mail and any attachments may be confidential or legally privileged. If > you received this message in error or are not the intended recipient, you > should destroy the e-mail message and any attachments or copies, and you are > prohibited from retaining, distributing, disclosing or using any information > contained herein. Please inform us of the erroneous delivery by return > e-mail. Thank you for your cooperation. -- Im Xuan Ji! This e-mail and any attachments may be confidential or legally privileged. If you received this message in error or are not the intended recipient, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained herein. Please inform us of the erroneous delivery by return e-mail. Thank you for your cooperation. This e-mail and any attachments may be confidential or legally privileged. If you received this message in error or are not the intended recipient, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained herein. Please inform us of the erroneous delivery by return e-mail. Thank you for your cooperation.
Re: Python 3
Thank you Dmitriy and Li. Luke Maycock OLIVER WYMAN luke.mayc...@affiliate.oliverwyman.com<mailto:luke.mayc...@affiliate.oliverwyman.com> www.oliverwyman.com<http://www.oliverwyman.com/> From: Li Xuan Ji Sent: 28 November 2016 12:20 To: dev@airflow.incubator.apache.org Subject: Re: Python 3 Airflow is supposed to be Python 3 compatible and any incompatibilities are bugs on our side. I haven't heard of anyone running airflow on windows, but I'm not sure what the official stance is. On 28 November 2016 at 07:01, Dmitriy Krasnikov wrote: > I have been using it and developing plugins for it in Python 3.5 with no > problems. > > > -Original Message- > From: Maycock, Luke [mailto:luke.mayc...@affiliate.oliverwyman.com] > Sent: Monday, November 28, 2016 6:58 AM > To: dev@airflow.incubator.apache.org > Subject: Python 3 > > Hi All, > > > I have been using Airflow for a while now and have a couple of questions that > I am hoping someone knows the answer to: > > 1. Is Airflow now Python 3 compatible? The documentation used to state that > Airflow was only compatible with Python 2.7 > (<https://web.archive.org/web/20160310111640/http://pythonhosted.org/airflow/start.html>https://web.archive.org/web/20160502024539/http://pythonhosted.org/airflow/start.html) > but that statement has since been removed. > 2. I know Gunicorn does not run on Windows - is this the only component of > Airflow that does not run on Windows? > > > Cheers, > Luke Maycock > OLIVER WYMAN > luke.mayc...@affiliate.oliverwyman.com<mailto:luke.mayc...@affiliate.oliverwyman.com> > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > This e-mail and any attachments may be confidential or legally privileged. If > you received this message in error or are not the intended recipient, you > should destroy the e-mail message and any attachments or copies, and you are > prohibited from retaining, distributing, disclosing or using any information > contained herein. Please inform us of the erroneous delivery by return > e-mail. Thank you for your cooperation. -- Im Xuan Ji! This e-mail and any attachments may be confidential or legally privileged. If you received this message in error or are not the intended recipient, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained herein. Please inform us of the erroneous delivery by return e-mail. Thank you for your cooperation.
Re: Python 3
Airflow is supposed to be Python 3 compatible and any incompatibilities are bugs on our side. I haven't heard of anyone running airflow on windows, but I'm not sure what the official stance is. On 28 November 2016 at 07:01, Dmitriy Krasnikov wrote: > I have been using it and developing plugins for it in Python 3.5 with no > problems. > > > -Original Message- > From: Maycock, Luke [mailto:luke.mayc...@affiliate.oliverwyman.com] > Sent: Monday, November 28, 2016 6:58 AM > To: dev@airflow.incubator.apache.org > Subject: Python 3 > > Hi All, > > > I have been using Airflow for a while now and have a couple of questions that > I am hoping someone knows the answer to: > > 1. Is Airflow now Python 3 compatible? The documentation used to state that > Airflow was only compatible with Python 2.7 > (<https://web.archive.org/web/20160310111640/http://pythonhosted.org/airflow/start.html>https://web.archive.org/web/20160502024539/http://pythonhosted.org/airflow/start.html) > but that statement has since been removed. > 2. I know Gunicorn does not run on Windows - is this the only component of > Airflow that does not run on Windows? > > > Cheers, > Luke Maycock > OLIVER WYMAN > luke.mayc...@affiliate.oliverwyman.com<mailto:luke.mayc...@affiliate.oliverwyman.com> > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > This e-mail and any attachments may be confidential or legally privileged. If > you received this message in error or are not the intended recipient, you > should destroy the e-mail message and any attachments or copies, and you are > prohibited from retaining, distributing, disclosing or using any information > contained herein. Please inform us of the erroneous delivery by return > e-mail. Thank you for your cooperation. -- Im Xuan Ji!
RE: Python 3
I have been using it and developing plugins for it in Python 3.5 with no problems. -Original Message- From: Maycock, Luke [mailto:luke.mayc...@affiliate.oliverwyman.com] Sent: Monday, November 28, 2016 6:58 AM To: dev@airflow.incubator.apache.org Subject: Python 3 Hi All, I have been using Airflow for a while now and have a couple of questions that I am hoping someone knows the answer to: 1. Is Airflow now Python 3 compatible? The documentation used to state that Airflow was only compatible with Python 2.7 (<https://web.archive.org/web/20160310111640/http://pythonhosted.org/airflow/start.html>https://web.archive.org/web/20160502024539/http://pythonhosted.org/airflow/start.html) but that statement has since been removed. 2. I know Gunicorn does not run on Windows - is this the only component of Airflow that does not run on Windows? Cheers, Luke Maycock OLIVER WYMAN luke.mayc...@affiliate.oliverwyman.com<mailto:luke.mayc...@affiliate.oliverwyman.com> www.oliverwyman.com<http://www.oliverwyman.com/> This e-mail and any attachments may be confidential or legally privileged. If you received this message in error or are not the intended recipient, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained herein. Please inform us of the erroneous delivery by return e-mail. Thank you for your cooperation.
Python 3
Hi All, I have been using Airflow for a while now and have a couple of questions that I am hoping someone knows the answer to: 1. Is Airflow now Python 3 compatible? The documentation used to state that Airflow was only compatible with Python 2.7 (<https://web.archive.org/web/20160310111640/http://pythonhosted.org/airflow/start.html>https://web.archive.org/web/20160502024539/http://pythonhosted.org/airflow/start.html) but that statement has since been removed. 2. I know Gunicorn does not run on Windows - is this the only component of Airflow that does not run on Windows? Cheers, Luke Maycock OLIVER WYMAN luke.mayc...@affiliate.oliverwyman.com<mailto:luke.mayc...@affiliate.oliverwyman.com> www.oliverwyman.com<http://www.oliverwyman.com/> This e-mail and any attachments may be confidential or legally privileged. If you received this message in error or are not the intended recipient, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained herein. Please inform us of the erroneous delivery by return e-mail. Thank you for your cooperation.
