We are using Airflow with Python 3.5 and RabbitMQ (CeleryExecutor). We're using Celery version 3.1.24. For the BROKER_URL connection string we use "pyamqp://<user>:<pass>@host:5671/vhost" and for CELERY_RESULT_BACKEND we use "rpc://".
Is Redis the preferred/better supported backend? Cheers, Conor Nash Data Science Consulting www.conornash.com | @conornash <https://twitter.com/conornash> On Sun, Jan 1, 2017 at 9:12 PM, George Leslie-Waksman < geo...@cloverhealth.com.invalid> wrote: > We are using Airflow with Python3, PostgreSQL and Redis; it works just > fine. > > On Thu, Dec 1, 2016 at 10:30 AM Maycock, Luke < > luke.mayc...@affiliate.oliverwyman.com> wrote: > > > Thanks Dennis. We will perhaps try Redis as there has been a ticket to > > make librabbitmq Python 3 compatible since 2012. > > > > > > Cheers, > > Luke Maycock > > OLIVER WYMAN > > luke.mayc...@affiliate.oliverwyman.com<mailto: > > luke.mayc...@affiliate.oliverwyman.com> > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > ________________________________ > > From: Dennis O'Brien <den...@dennisobrien.net> > > Sent: 01 December 2016 04:45 > > To: dev@airflow.incubator.apache.org > > Subject: Re: Python 3 > > > > Hi Luke, > > > > I've been running Airflow on Python 3.5 with Celery. I'm using Redis as > > the message broker (and ElastiCache Redis in production). I haven't > tried > > RabbitMQ so I can't speak to its compatibility with Python 3.5. > > > > On Mon, Nov 28, 2016 at 9:18 AM Maycock, Luke < > > luke.mayc...@affiliate.oliverwyman.com> wrote: > > > > > I thought it would be worth following up on this given our experience > of > > > testing Python 3 compatibility. > > > > > > > > > We are using Celery and RabbitMQ. Celery uses librabbitmq for > interaction > > > with RabbitMQ but librabbitmq is not compatible with Python 3 (see > > > https://github.com/celery/librabbitmq/issues/13). Has anyone else been > > > able to get Airflow/Python 3/Celery working? > > > > > > > > > > > > Cheers, > > > Luke Maycock > > > OLIVER WYMAN > > > luke.mayc...@affiliate.oliverwyman.com<mailto: > > > luke.mayc...@affiliate.oliverwyman.com> > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > ________________________________ > > > From: Maycock, Luke <luke.mayc...@affiliate.oliverwyman.com> > > > Sent: 28 November 2016 12:33 > > > To: dev@airflow.incubator.apache.org > > > Subject: Re: Python 3 > > > > > > Thank you Dmitriy and Li. > > > > > > Luke Maycock > > > OLIVER WYMAN > > > luke.mayc...@affiliate.oliverwyman.com<mailto: > > > luke.mayc...@affiliate.oliverwyman.com> > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > ________________________________ > > > From: Li Xuan Ji <xua...@gmail.com> > > > Sent: 28 November 2016 12:20 > > > To: dev@airflow.incubator.apache.org > > > Subject: Re: Python 3 > > > > > > Airflow is supposed to be Python 3 compatible and any > > > incompatibilities are bugs on our side. > > > > > > I haven't heard of anyone running airflow on windows, but I'm not sure > > > what the official stance is. > > > > > > On 28 November 2016 at 07:01, Dmitriy Krasnikov < > dkrasni...@hotmail.com> > > > wrote: > > > > I have been using it and developing plugins for it in Python 3.5 with > > no > > > problems. > > > > > > > > > > > > -----Original Message----- > > > > From: Maycock, Luke [mailto:luke.mayc...@affiliate.oliverwyman.com] > > > > Sent: Monday, November 28, 2016 6:58 AM > > > > To: dev@airflow.incubator.apache.org > > > > Subject: Python 3 > > > > > > > > Hi All, > > > > > > > > > > > > I have been using Airflow for a while now and have a couple of > > questions > > > that I am hoping someone knows the answer to: > > > > > > > > 1. Is Airflow now Python 3 compatible? The documentation used to > state > > > that Airflow was only compatible with Python 2.7 (< > > > > > https://web.archive.org/web/20160310111640/http:// > pythonhosted.org/airflow/start.html > > > > > > > > > https://web.archive.org/web/20160502024539/http:// > pythonhosted.org/airflow/start.html > > ) > > > but that statement has since been removed. > > > > 2. I know Gunicorn does not run on Windows - is this the only > > > component of Airflow that does not run on Windows? > > > > > > > > > > > > Cheers, > > > > Luke Maycock > > > > OLIVER WYMAN > > > > luke.mayc...@affiliate.oliverwyman.com<mailto: > > > luke.mayc...@affiliate.oliverwyman.com> > > > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > > > > > > > ________________________________ > > > > This e-mail and any attachments may be confidential or legally > > > privileged. If you received this message in error or are not the > intended > > > recipient, you should destroy the e-mail message and any attachments or > > > copies, and you are prohibited from retaining, distributing, disclosing > > or > > > using any information contained herein. Please inform us of the > erroneous > > > delivery by return e-mail. Thank you for your cooperation. > > > > > > > > > > > > -- > > > Im Xuan Ji! > > > > > > ________________________________ > > > This e-mail and any attachments may be confidential or legally > > privileged. > > > If you received this message in error or are not the intended > recipient, > > > you should destroy the e-mail message and any attachments or copies, > and > > > you are prohibited from retaining, distributing, disclosing or using > any > > > information contained herein. Please inform us of the erroneous > delivery > > by > > > return e-mail. Thank you for your cooperation. > > > > > > ________________________________ > > > This e-mail and any attachments may be confidential or legally > > privileged. > > > If you received this message in error or are not the intended > recipient, > > > you should destroy the e-mail message and any attachments or copies, > and > > > you are prohibited from retaining, distributing, disclosing or using > any > > > information contained herein. Please inform us of the erroneous > delivery > > by > > > return e-mail. Thank you for your cooperation. > > > > > > > ________________________________ > > This e-mail and any attachments may be confidential or legally > privileged. > > If you received this message in error or are not the intended recipient, > > you should destroy the e-mail message and any attachments or copies, and > > you are prohibited from retaining, distributing, disclosing or using any > > information contained herein. Please inform us of the erroneous delivery > by > > return e-mail. Thank you for your cooperation. > > >
