Hey, Had a look at this celery config option, but no luck. Also tried setting executor to Local executor - same result Each task takes no more than 0.1 sec but overall time is huge Thought that it could be due to disabled pickling, enabled it - almost no change :(
-----Original Message----- From: Bolke de Bruin [mailto:bdbr...@gmail.com] Sent: Monday, May 30, 2016 3:09 PM To: dev@airflow.incubator.apache.org Subject: Re: Airflow scheduler/worker inefficient time Have a look at this: https://github.com/apache/incubator-airflow/pull/1509 Sent from my iPhone > On 30 mei 2016, at 14:03, Ryabchuk, Pavlo <ext-pavlo.ryabc...@here.com> wrote: > > Hi all, > > Maybe I am misusing airflow a bit, because I am using it as on demand > (triggered) complex data processing system, but still, the question is, what > are the actual parameters I should play around with in order to speedup > execution? > I have around 250 Dummy tasks (which do nothing) in my DAG and running it > locally with celery executor takes around 1000 sec, which is pretty strange. > I've noticed that a single Dummy task takes some milliseconds. I've tried > playing around with celery concurrency, airflow executor parallelism and > heartbeat, but with almost no result... it's really strange, what am I doing > wrong :) > > Best, > Pavlo > >
