Hi Nicolas, I don't have a lot of experience with subdags as there's a lot of warnings around them, I would strongly recommend reading this before using them: https://www.astronomer.io/guides/subdags/
I believe there are some big improvements coming in Airflow 2 for subdags where they will act more like regular dags and work in the way you expect. In terms of what you are saying there you have dags depend on each, we use a DummyOperator to be an End Point for the DAG that will always be the same name, and ExternalTaskSensor which will wait for the End Point to be successfully finished. Through a warning, you have to be on exactly the same schedule for this to work or you need to do some datetime conversion logic to provide the correct execution_date. Regards, Damian -----Original Message----- From: Nicolas Paris [mailto:[email protected]] Sent: Sunday, August 11, 2019 7:44 PM To: [email protected] Subject: reuse dags Hi >From the documentation I guess I spotted the way of reusing dags from other python files: this can be done by creating "subdags". I have created several pipelines (dag1.py, dag2.py...) in several python files Right now, I d'like to build a meta-pipeline dag3 = dag1 >> dag2 Do I have to convert dag1 and dag2 to subdag and import them in dag3 ? Or is there a more friendly way Thanks -- nicolas =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================
