I have submitted: https://github.com/apache/incubator-airflow/pull/3388
-- Craig On Mon, May 21, 2018 at 7:00 AM Naik Kaxil <k.n...@reply.com> wrote: > Thanks. Please do that. > > On 21/05/2018, 14:59, "Craig Rodrigues" <crodr...@gmail.com> wrote: > > celery 4.1.1 was just released last night which has all the async > problems fixed: > > https://github.com/celery/celery/commits/v4.1.1 > > I'll test this out, and then submit a PR to bump airflow's celery > version to 4.1.1 > > -- > Craig > > On 2018/05/21 07:20:50, Craig Rodrigues <crodr...@gmail.com> wrote: > > Hi, > > > > I used a requiremens.txt file with these three lines: > > > > git+ > https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3] > > celery>=4.2.0rc3 > > kombu>=4.2.0 > > > > > > I did > > > > pip install -r requirements.txt > > > > When I started my worker, I got: > > > > [2018-05-21 06:39:23,980] {__init__.py:48} INFO - Using executor > CeleryExecutor > > Traceback (most recent call last): > > File "/bin/airflow", line 32, in <module> > > args.func(args) > > File "/usr/lib/python2.7/site-packages/airflow/utils/cli.py", line > 74, in wrapper > > return f(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line > 959, in worker > > worker.run(**options) > > File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line > 257, in run > > **kwargs) > > File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", > line 101, in __init__ > > self.setup_instance(**self.prepare_args(**kwargs)) > > File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", > line 124, in setup_instance > > self.should_use_eventloop() if use_eventloop is None > > File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", > line 243, in should_use_eve > > self._conninfo.transport.implements.async and > > File "/usr/lib/python2.7/site-packages/kombu/transport/base.py", > line 125, in __getattr__ > > raise AttributeError(key) > > AttributeError: async > > > > > > > > I reported this as a bug in celery here: > > > > https://github.com/celery/celery/issues/4747 > > > > It turns out that this is a known problem. It looks like in kombu, > they > > renamed a bunch of uses of "async" to "asynchronous", to not clash > with the new > > 'async' keyword in Python 3.7. > > > > In celery, they are catching up and doing the same rename. > > Almost at the same time that I reported this bug, the issue was > fixed in the master branch of celery: > > > > > https://github.com/celery/celery/commit/c8ef7ad60b72a194654c58beb04a1d65cd0435ad > > > > Hopefully the versions of kombu and celery will stabilize so that it > all works properly with airflow. > > > > -- > > Craig > > > > > > > > > > > Kaxil Naik > > Data Reply > 2nd Floor, Nova South > 160 Victoria Street, Westminster > London SW1E 5LB - UK > phone: +44 (0)20 7730 6000 > k.n...@reply.com > www.reply.com >
