Valeriy created AIRFLOW-3532:
--------------------------------
Summary: Apache Airflow > 1.8 don't working with Celery 4.x and
don't working Celery 3.x using other than amqp transport
Key: AIRFLOW-3532
URL: https://issues.apache.org/jira/browse/AIRFLOW-3532
Project: Apache Airflow
Issue Type: Bug
Components: celery
Affects Versions: 1.10.1, 1.10.0, 1.9.0
Reporter: Valeriy
I needed Airflow > 1.8 with all the necessary fixes in the cluster
configuration.
I'm used Aiflow 1.10.0/1.10.1 + Celery 4.2.0/4.2.1 and have problem with
working DAG's. After some time, all DAG's gone away in queued. After restarting
the worker, the problem is solved. I tried for a long time to find solutions to
this problem (logs in DEBUG mode showed nothing) and found a number of
discussions, one them:
[https://stackoverflow.com/questions/43524457/airflow-tasks-queued-but-not-running]
*{color:#d04437}As a result, we conclude that Airflow does not work with Celery
4.x!{color}* The code is not adapted to the Celery 4.x.
I decided to try the Celery 3.x and damn I got an WARNING:
{code:java}
[2018-12-17 15:43:11,136: WARNING/MainProcess]
/home/hadoop/youla_airflow/lib/python3.6/site-packages/celery/apps/worker.py:161:
CDeprecationWarning:
Starting from version 3.2 Celery will refuse to accept pickle by default.
The pickle serializer is a security concern as it may give attackers
the ability to execute any command. It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.
If you depend on pickle then you should set a setting to disable this
warning and to be sure that everything will continue working
when you upgrade to Celery 3.2::
CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
You must only enable the serializers that you will actually use.
warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
-------------- celery@youlahdp-driver4.p v3.1.26.post2 (Cipater)
---- **** -----
--- * *** * --
Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-centos-7.5.1804-Core
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: airflow.executors.celery_executor:0x7f0093b86470
- ** ---------- .> transport: amqp://guest:**@localhost:5672//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> default exchange=default(direct) key=default
{code}
Airflow > 1.8 version with Celery 3.x flatly refuses to use transport other
than amqp. About it already wrote here
http://mail-archives.apache.org/mod_mbox/airflow-commits/201801.mbox/%3cjira.13129586.1515519138000.610058.1515519180...@atlassian.jira%3E
My Airflow config:
{code:java}
[celery]
celery_app_name = airflow.executors.celery_executor
worker_concurrency = 16
worker_log_server_port = 8793
broker_url = redis://localhost:6400/0
result_backend = db+postgres://airflow:pass@localhost:5434/airflow
flower_host = 0.0.0.0
flower_url_prefix =
flower_port = 5555
default_queue = default
celery_config_options =
airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
{code}
How do I run Airflow > 1.8 with Celery as a Redis broker? Is that possible?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
