turbaszek commented on issue #12519: URL: https://github.com/apache/airflow/issues/12519#issuecomment-731640640
> separate queue and a separate worker, where that worker has its celery.worker_concurrency set to 1 and listens to this separate queue I would say that this is the recommended way. It's a common practice to have separate queue for compute-heavy tasks. If you are using kubernetes - then once Airflow 2.0 is you can consider using `CeleryKubernetesExecutor` https://airflow.readthedocs.io/en/latest/executor/celery_kubernetes.html ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
