Hi folks, This config line <https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L414> controls how often the scheduler scan the DAG folder and tries to discover/ forget DAGs.
For doing dag file processing part, scheduler does parse the DAG file everytime before it schedules tasks through DagFileProcessor. Cheers, Kevin Y On Mon, May 28, 2018 at 10:14 PM, Ananth Durai <vanant...@gmail.com> wrote: > It is an interesting question. On a slightly related note, Correct me if > I'm wrong, AFAIK we require restarting airflow scheduler in order pick any > new DAG file changes by the scheduler. In that case, should the scheduler > do the DAGFileProcessing every time before scheduling the tasks? > > Regards, > Ananth.P, > > > > > > > On 28 May 2018 at 21:46, ramandu...@gmail.com <ramandu...@gmail.com> > wrote: > > > Hi All, > > > > We have a use case where there would be 100(s) of DAG files with schedule > > set to "@once". Currently it seems that scheduler processes each and > every > > file and creates a Dag Object. > > Is there a way or config to tell scheduler to stop processing certain > > files. > > > > Thanks, > > Raman Gupta > > >
