Hi, I used a requiremens.txt file with these three lines:
git+https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3] celery>=4.2.0rc3 kombu>=4.2.0 I did pip install -r requirements.txt When I started my worker, I got: [2018-05-21 06:39:23,980] {__init__.py:48} INFO - Using executor CeleryExecutor Traceback (most recent call last): File "/bin/airflow", line 32, in <module> args.func(args) File "/usr/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in wrapper return f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 959, in worker worker.run(**options) File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line 257, in run **kwargs) File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 101, in __init__ self.setup_instance(**self.prepare_args(**kwargs)) File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 124, in setup_instance self.should_use_eventloop() if use_eventloop is None File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 243, in should_use_eve self._conninfo.transport.implements.async and File "/usr/lib/python2.7/site-packages/kombu/transport/base.py", line 125, in __getattr__ raise AttributeError(key) AttributeError: async I reported this as a bug in celery here: https://github.com/celery/celery/issues/4747 It turns out that this is a known problem. It looks like in kombu, they renamed a bunch of uses of "async" to "asynchronous", to not clash with the new 'async' keyword in Python 3.7. In celery, they are catching up and doing the same rename. Almost at the same time that I reported this bug, the issue was fixed in the master branch of celery: https://github.com/celery/celery/commit/c8ef7ad60b72a194654c58beb04a1d65cd0435ad Hopefully the versions of kombu and celery will stabilize so that it all works properly with airflow. -- Craig
