potiuk commented on pull request #10153: URL: https://github.com/apache/airflow/pull/10153#issuecomment-688622957
> I love this change! This DAG representation is nice and clean. I'm just wondering if this will work with AIP-31 approach This would be awesome. Actually I started a discussion on https://towardsdatascience.com/managing-dependencies-between-data-pipelines-in-apache-airflow-prefect-f4eba65886df article of Anna (and there is a thread where we discuss it with the author) about some new ways of thinking about building DAGs when we have Task Group concept merged. I think this makes it possible to start thinking a bit differently about building complex DAGs and reusing parts of the logic between different DAGS (which Task Grop actually makes very easy to visualize) - it's been possible before but Task Group makes it actually a "first class citizen" to build reusable "DAG fragments" that you will be able to re-use between different DAGs. When paired with DAG trigger/DAG wait operator/sensor I think this is a complete solution that allows to implement arbitrary complex 'dependencies" between pipelines that Anna wrote about in a an elegant way, without introducing unnecessary complexity on the processing side. Having functional interface to do that would make it even better and would allow DAG developers to rethink their way of developing DAGs. It sounds very exciting to be able to join AIP-31 and AIP-34 results into a coherent and nice approach. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
