Hi Damian > I believe there are some big improvements coming in Airflow 2 for subdags
All right, I won't use them until v2 is available. > In terms of what you are saying there you have dags depend on each, we use a > DummyOperator to be an End Point for the DAG that will always be the same > name, and ExternalTaskSensor which will wait for the End Point to be > successfully finished. Through a warning, you have to be on exactly the same > schedule for this to work or you need to do some datetime conversion logic to > provide the correct execution_date. Interesting but quite complex. I am looking for a built-in simple way to import dags. Reusing and factorizing code is quite common need. So far airflow does not provide yet a solution for such purpose apparently On Mon, Aug 12, 2019 at 01:34:11PM +0000, Shaw, Damian P. wrote: > Hi Nicolas, > > I don't have a lot of experience with subdags as there's a lot of warnings > around them, I would strongly recommend reading this before using them: > https://www.astronomer.io/guides/subdags/ > > I believe there are some big improvements coming in Airflow 2 for subdags > where they will act more like regular dags and work in the way you expect. > > In terms of what you are saying there you have dags depend on each, we use a > DummyOperator to be an End Point for the DAG that will always be the same > name, and ExternalTaskSensor which will wait for the End Point to be > successfully finished. Through a warning, you have to be on exactly the same > schedule for this to work or you need to do some datetime conversion logic to > provide the correct execution_date. > > Regards, > Damian > > -----Original Message----- > From: Nicolas Paris [mailto:[email protected]] > Sent: Sunday, August 11, 2019 7:44 PM > To: [email protected] > Subject: reuse dags > > Hi > > From the documentation I guess I spotted the way of reusing dags from > other python files: this can be done by creating "subdags". > > I have created several pipelines (dag1.py, dag2.py...) in several python files > Right now, I d'like to build a meta-pipeline dag3 = dag1 >> dag2 > > Do I have to convert dag1 and dag2 to subdag and import them in dag3 ? > Or is there a more friendly way > > Thanks > -- > nicolas > > > =============================================================================== > > Please access the attached hyperlink for an important electronic > communications disclaimer: > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > =============================================================================== > > -- nicolas
