> On Nov 22, 2018, at 3:37 PM, Alex Guziel <alex.guz...@airbnb.com> wrote: > > I think this is what is going on. The dags are picked by local variables. > I.E. if you do > dag = Dag(...) > dag = Dag(…)
from my_module import create_dag for file in yaml_files: dag = create_dag(file) globals()[dag.dag_id] = dag You notice that create_dag is in a different module. If it is in the same scope (file), it will be fine. > > Only the second dag will be picked up. > > On Thu, Nov 22, 2018 at 2:04 AM Soma S Dhavala <soma.dhav...@gmail.com > <mailto:soma.dhav...@gmail.com>> wrote: > Hey AirFlow Devs: > In our organization, we build a Machine Learning WorkBench with AirFlow as > an orchestrator of the ML Work Flows, and have wrapped AirFlow python > operators to customize the behaviour. These work flows are specified in > YAML. > > We drop a DAG loader (written python) in the default location airflow > expects the DAG files. This DAG loader reads the specified YAML files and > converts them into airflow DAG objects. Essentially, we are > programmatically creating the DAG objects. In order to support muliple > parsers (yaml, json etc), we separated the DAG creation from loading. But > when a DAG is created (in a separate module) and made available to the DAG > loaders, airflow does not pick it up. As an example, consider that I > created a DAG picked it, and will simply unpickle the DAG and give it to > airflow. > > However, in current avatar of airfow, the very creation of DAG has to > happen in the loader itself. As far I am concerned, airflow should not care > where and how the DAG object is created, so long as it is a valid DAG > object. The workaround for us is to mix parser and loader in the same file > and drop it in the airflow default dags folder. During dag_bag creation, > this file is loaded up with import_modules utility and shows up in the UI. > While this is a solution, but it is not clean. > > What do DEVs think about a solution to this problem? Will saving the DAG to > the db and reading it from the db work? Or some core changes need to happen > in the dag_bag creation. Can dag_bag take a bunch of "created" DAGs. > > thanks, > -soma