We use max_threads = number of scheduler cores.
On 2018/09/11 09:49:53, Chandu Kavar <ccka...@gmail.com> wrote: > Thanks Raman, > > Understood. > > We have around 500 DAGs. What value do you suggest for max_threads? > > On Tue, Sep 11, 2018, 5:44 PM ramandu...@gmail.com <ramandu...@gmail.com> > wrote: > > > Hi Chandu, > > How many dag files are there on the scheduler. As per my understanding > > scheduler processes each file to trigger any dag/task run. It spawns number > > of processes equivalent to "max_threads" count to parallelize file > > processing. So you can try by increasing airflow config's max_threads count. > > > > Thanks, > > Raman Gupta > > > > On 2018/09/11 07:11:23, Chandu Kavar <ccka...@gmail.com> wrote: > > > Hi All, > > > > > > We are seeing there is a delay between triggering the next task. Please > > > find the attached screenshot. > > > > > > First and the last task are very small. So, you can see the tiny object. > > > > > > [image: image.png] > > > Do you suggest any airflow config can resolve this problem? > > > > > > Thanks, > > > Chandu > > > > > >
