turbaszek opened a new pull request #11336:
URL: https://github.com/apache/airflow/pull/11336
This commit does small refactor of the way we star celery worker.
In this way it will be easier to migrate to Celery 5.0.
Running with currently used Celery version:
```
root@1e7421e00701:/opt/airflow# airflow celery worker
/usr/local/lib/python3.7/site-packages/celery/platforms.py:801:
RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
uid=uid, euid=euid, gid=gid, egid=egid,
Starting flask
* Serving Flask app "airflow.utils.serve_logs" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production
deployment.
Use a production WSGI server instead.
* Debug mode: off
[2020-10-07 20:44:47,384: INFO/Process-1] * Running on http://0.0.0.0:8793/
(Press CTRL+C to quit)
-------------- celery@1e7421e00701 v4.4.7 (cliffs)
--- ***** -----
-- ******* ---- Linux-4.19.76-linuxkit-x86_64-with-debian-10.5 2020-10-07
20:44:47
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:
airflow.executors.celery_executor:0x7f401d0d2a10
- ** ---------- .> transport: redis://redis:6379/0
- ** ---------- .> results: postgresql://postgres:**@postgres/airflow
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this
worker)
--- ***** -----
-------------- [queues]
.> default exchange=default(direct) key=default
[tasks]
. airflow.executors.celery_executor.execute_command
[2020-10-07 20:44:47,988: INFO/MainProcess] Connected to redis://redis:6379/0
[2020-10-07 20:44:47,998: INFO/MainProcess] mingle: searching for neighbors
[2020-10-07 20:44:49,024: INFO/MainProcess] mingle: all alone
[2020-10-07 20:44:49,045: INFO/MainProcess] celery@1e7421e00701 ready.
^C
worker: Hitting Ctrl+C again will terminate all running tasks!
worker: Warm shutdown (MainProcess)
```
Running with Celery 5.0:
```
root@5257c48ec87e:/opt/airflow# airflow celery worker
/usr/local/lib/python3.7/site-packages/celery/platforms.py:798:
RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
uid=uid, euid=euid, gid=gid, egid=egid,
Starting flask
* Serving Flask app "airflow.utils.serve_logs" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production
deployment.
Use a production WSGI server instead.
* Debug mode: off
[2020-10-07 20:40:27,110: INFO/Process-1] * Running on http://0.0.0.0:8793/
(Press CTRL+C to quit)
-------------- celery@5257c48ec87e v5.0.0 (singularity)
--- ***** -----
-- ******* ---- Linux-4.19.76-linuxkit-x86_64-with-debian-10.5 2020-10-07
20:40:27
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:
airflow.executors.celery_executor:0x7fb3e65b5dd0
- ** ---------- .> transport: redis://redis:6379/0
- ** ---------- .> results: postgresql://postgres:**@postgres/airflow
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this
worker)
--- ***** -----
-------------- [queues]
.> default exchange=default(direct) key=default
[tasks]
. airflow.executors.celery_executor.execute_command
[2020-10-07 20:40:27,701: INFO/MainProcess] Connected to redis://redis:6379/0
[2020-10-07 20:40:27,711: INFO/MainProcess] mingle: searching for neighbors
[2020-10-07 20:40:28,737: INFO/MainProcess] mingle: all alone
[2020-10-07 20:40:28,762: INFO/MainProcess] celery@5257c48ec87e ready.
^C
worker: Hitting Ctrl+C again will terminate all running tasks!
worker: Warm shutdown (MainProcess)
```
Related to #11301
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
