With configuration as code, you can't really know whether the DAG definition has changed based on whether the module was altered. This python module could be importing other modules that have been changed, could have read a config file somewhere on the drive that might have changed, or read from a DB that is constantly getting mutated.
There are also issues around the fact that Python caches modules in `sys.modules`, so even though the crawler is re-interpreting modules, imported modules wouldn't get re-interpreted [as our DAG authors expected] For these reasons [and others I won't get into here], we decided that the scheduler would use a subprocess pool and re-interpret the DAGs from scratch at every cycle, insulating the different DAGs and guaranteeing no interpreter caching. Side note: yaml parsing is much more expensive than other markup languages and would recommend working around it to store DAG configuration. Our longest-to-parse DAGs at Airbnb were reading yaml to build build a DAG, and I believe someone wrote custom logic to avoid reparsing the yaml at every cycle. Parsing equivalent json or hocon was an order of magnitude faster. Max On Mon, Apr 24, 2017 at 2:55 PM, Bolke de Bruin <bdbr...@gmail.com> wrote: > Inotify can work without a daemon. Just fire a call to the API when a file > changes. Just a few lines in bash. > > If you bundle you dependencies in a zip you should be fine with the above. > Or if we start using manifests that list the files that are needed in a > dag... > > > Sent from my iPhone > > > On 24 Apr 2017, at 22:46, Dan Davydov <dan.davy...@airbnb.com.INVALID> > wrote: > > > > One idea to solve this is to use a daemon that uses inotify to watch for > > changes in files and then reprocesses just those files. The hard part is > > without any kind of dependency/build system for DAGs it can be hard to > tell > > which DAGs depend on which files. > > > > On Mon, Apr 24, 2017 at 1:21 PM, Gerard Toonstra <gtoons...@gmail.com> > > wrote: > > > >> Hey, > >> > >> I've seen some people complain about DAG file processing times. An issue > >> was raised about this today: > >> > >> https://issues.apache.org/jira/browse/AIRFLOW-1139 > >> > >> I attempted to provide a good explanation what's going on. Feel free to > >> validate and comment. > >> > >> > >> I'm noticing that the file processor is a bit naive in the way it > >> reprocesses DAGs. It doesn't look at the DAG interval for example, so it > >> looks like it reprocesses all files continuously in one big batch, even > if > >> we can determine that the next "schedule" for all its dags are in the > >> future? > >> > >> > >> Wondering if a change in the DagFileProcessingManager could optimize > things > >> a bit here. > >> > >> In the part where it gets the simple_dags from a file it's currently > >> processing: > >> > >> for simple_dag in processor.result: > >> simple_dags.append(simple_dag) > >> > >> the file_path is in the context and the simple_dags should be able to > >> provide the next interval date for each dag in the file. > >> > >> The idea is to add files to a sorted deque by "next_schedule_datetime" > (the > >> minimum next interval date), so that when we build the list > >> "files_paths_to_queue", it can remove files that have dags that we know > >> won't have a new dagrun for a while. > >> > >> One gotcha to resolve after that is to deal with files getting updated > with > >> new dags or changed dag definitions and renames and different interval > >> schedules. > >> > >> Worth a PR to glance over? > >> > >> Rgds, > >> > >> Gerard > >> >
