Thanks George.
On 2018/06/20 20:06:38, George Leslie-Waksman
wrote:
> "celeryd_concurrency" and "parallelism" serve different purposes
>
> "celeryd_concurrency" determines how many worker subprocesses will be spun
> up on an airflow worker instance (how many concurrent tasks per machine)
>
> "
"celeryd_concurrency" and "parallelism" serve different purposes
"celeryd_concurrency" determines how many worker subprocesses will be spun
up on an airflow worker instance (how many concurrent tasks per machine)
"parallelism" determines how many tasks airflow will schedule at once;
running and q
Hi All,
There seems to be couple of settings in airflow.cfg which controls the number
of tasks that can run in parallel on Airflow( "parallelism" and
"celeryd_concurrency")
In case of celeryExecutor which one is honoured. Do we need to set both or
setting only celeryd_concurrency would work.
