yuqian90 commented on pull request #10153: URL: https://github.com/apache/airflow/pull/10153#issuecomment-688834630
> > I love this change! This DAG representation is nice and clean. I'm just wondering if this will work with AIP-31 approach > > This would be awesome. > > Actually I started a discussion on https://towardsdatascience.com/managing-dependencies-between-data-pipelines-in-apache-airflow-prefect-f4eba65886df article of Anna (and there is a thread where we discuss it with the author) about some new ways of thinking about building DAGs when we have Task Group concept merged. > > I think this makes it possible to start thinking a bit differently about building complex DAGs and reusing parts of the logic between different DAGS (which Task Grop actually makes very easy to visualize) - it's been possible before but Task Group makes it actually a "first class citizen" to build reusable "DAG fragments" that you will be able to re-use between different DAGs. When paired with DAG trigger/DAG wait operator/sensor I think this is a complete solution that allows to implement arbitrary complex 'dependencies" between pipelines that Anna wrote about in a an elegant way, without introducing unnecessary complexity on the processing side. > > Having functional interface to do that would make it even better and would allow DAG developers to rethink their way of developing DAGs. > > It sounds very exciting to be able to join AIP-31 and AIP-34 results into a coherent and nice approach. Hi @potiuk thanks for the comment and the interesting article link. I totally agree that with `TaskGroup` in mind, we will have a new way of building arbitrarily complex interdependent workflows. In the past we would be trying hard to put these individual workflows onto separate DAGs, and link them up using `ExternalTaskSensor`, `ExternalTaskMarker`, `SubDagOperator`, `TriggerDagRunOperator`, etc With `TaskGroup`, we have one more option: just put the related workflows all in the same DAG and link them up normally like simple tasks. It's also possible to write functions that returns a `TaskGroup` object which can be re-used in different DAGs to re-use the same workflow. That said, I'm not sure if we should really combine the two AIPs. To go along with the idea, I included an example in the previous reply [here](https://github.com/apache/airflow/pull/10153#issuecomment-688778222) showing the `@task` decorator and `TaskGroup` used together. They seem to work nicely with each other just fine even though they were born out of two different AIPs. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
