> I would say the main reusable objects in airflow are the operator and the
> hook,
About operator, most examples in the documentation use them by
specifying their dag as argument. For this reason I thought they cannot
be reused over python files. Eg:
> run_this = BashOperator( task_id='run_after_loop', bash_command='echo 1',
> dag=dag,)
After digging, I found that note:
> Added in Airflow 1.8
>
> DAGs can be used as context managers to automatically assign new
> operators to that DAG.
>
> with DAG('my_dag', start_date=datetime(2016, 1, 1)) as dag:
> op = DummyOperator('op')
Correct me if I am wrong, this might be a way to define a bunch of
operators in some library files, and import them into dag files ?
Thanks
On Wed, Aug 14, 2019 at 04:13:14PM -0700, Daniel Standish wrote:
> I would say the main reusable objects in airflow are the operator and the
> hook,
> not the dag. These are the primary building blocks that you use to construct
> your pipelines.
>
> That said you may find it useful to look at TriggerDagRunOperator. You can
> have your dag's behavior change based on execution date and dag_run conf,
> which
> are available to all tasks in as a template context variable.
>
> You may also look into generating your dags dynamically.
>
>
>
>
> On Wed, Aug 14, 2019 at 2:58 PM Nicolas Paris <[email protected]>
> wrote:
>
> Hi Damian
>
> > I believe there are some big improvements coming in Airflow 2 for
> subdags
>
> All right, I won't use them until v2 is available.
>
>
> > In terms of what you are saying there you have dags depend on each, we
> use a DummyOperator to be an End Point for the DAG that will always be the
> same name, and ExternalTaskSensor which will wait for the End Point to be
> successfully finished. Through a warning, you have to be on exactly the
> same schedule for this to work or you need to do some datetime conversion
> logic to provide the correct execution_date.
>
> Interesting but quite complex. I am looking for a built-in simple way to
> import dags.
>
> Reusing and factorizing code is quite common need. So far airflow does
> not provide yet a solution for such purpose apparently
>
> On Mon, Aug 12, 2019 at 01:34:11PM +0000, Shaw, Damian P. wrote:
> > Hi Nicolas,
> >
> > I don't have a lot of experience with subdags as there's a lot of
> warnings around them, I would strongly recommend reading this before using
> them: https://www.astronomer.io/guides/subdags/
> >
> > I believe there are some big improvements coming in Airflow 2 for
> subdags
> where they will act more like regular dags and work in the way you expect.
> >
> > In terms of what you are saying there you have dags depend on each, we
> use a DummyOperator to be an End Point for the DAG that will always be the
> same name, and ExternalTaskSensor which will wait for the End Point to be
> successfully finished. Through a warning, you have to be on exactly the
> same schedule for this to work or you need to do some datetime conversion
> logic to provide the correct execution_date.
> >
> > Regards,
> > Damian
> >
> > -----Original Message-----
> > From: Nicolas Paris [mailto:[email protected]]
> > Sent: Sunday, August 11, 2019 7:44 PM
> > To: [email protected]
> > Subject: reuse dags
> >
> > Hi
> >
> > From the documentation I guess I spotted the way of reusing dags from
> > other python files: this can be done by creating "subdags".
> >
> > I have created several pipelines (dag1.py, dag2.py...) in several python
> files
> > Right now, I d'like to build a meta-pipeline dag3 = dag1 >> dag2
> >
> > Do I have to convert dag1 and dag2 to subdag and import them in dag3 ?
> > Or is there a more friendly way
> >
> > Thanks
> > --
> > nicolas
> >
> >
> >
> =========================================================================
> ======
> > Please access the attached hyperlink for an important electronic
> communications disclaimer:
> > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
> >
> =========================================================================
> ======
> >
>
> --
> nicolas
>
--
nicolas
