Hi All, We have a use case where there are 100s of DAGs in the scheduler's local dag folder but at a time only ~50 dags are active(Other dags are disabled). New dags keep on adding to the local Dag folder. We are observing that scheduler is taking lot of time(around 20 minutes) in picking those newly created dags. We are using following conf Machine config: 8 core, 16 GB RAM Executor: LocalExecutor Parallelism: 50 max_thread: 4, Dir list interval: 60.
Is there any recommended setting or config to optimise the scheduler's dag listing and parsing. Thanks, Raman
