Kevin, I am not sure I understand your response to Nathan.
I agree that it is also a valid use case, but I don't see how it can be cleanly done while keeping TaskGroup only as a UI concept. Would this require extending the TaskGroup concept to the backend? Best regards, Vikram On Fri, Mar 5, 2021 at 1:31 AM Kevin Yang <yrql...@gmail.com> wrote: > Hi Nathan, > > Thanks a lot for your input and it is indeed a valid use case. This can be > done either keeping TaskGroup as a UI concept or bringing it into the > backend. I'm curious to hear what others think. > > > Cheers, > Kevin Y > > On Thu, Mar 4, 2021 at 12:57 AM Nathan Hadfield <nathan.hadfi...@king.com> > wrote: > >> Hi Kevin, >> >> >> >> A quick piece of input from our recent experiences of working with >> TaskGroup is that we often have dependencies across DAGs that require >> waiting upon the completion of all the tasks in a group. At the moment, >> you basically have two options: >> >> >> >> 1. Create a sensor task in a DAG for every task in the group >> 2. Create a Dummy task after the group that a sensor waits on >> >> >> >> So, I would certainly like TaskGroups to have some notion of run status >> as to better enable downstream decision making. >> >> >> >> I’ve already created a feature ticket to try to add some kind of >> TaskGroup Sensor but perhaps this can also form part of the wider >> discussions here. >> >> >> >> https://github.com/apache/airflow/issues/14563 >> >> >> >> Cheers, >> >> >> >> Nathan >> >> >> >> *From: *Kevin Yang <yrql...@gmail.com> >> *Date: *Thursday, 4 March 2021 at 05:21 >> *To: *dev@airflow.apache.org <dev@airflow.apache.org> >> *Subject: *[DISCUSS] TaskGroup in Tree View >> >> Hi team, >> >> >> >> We are very glad to see the introduction of TaskGroup in Airflow 2.0 and >> really like it. Thanks to Yu Qian and everyone that contributed to it. To >> continue moving towards the goal of replacing SubDagOperator with >> TaskGroup, I'd like to kick off a discussion on bringing TaskGroup into >> Tree View. >> >> >> >> *Why do we need TaskGroup in Tree View?* >> >> For owners of larger DAGs, say a DAG with 500 tasks, Tree View is the >> preferred view for its loading speed and simpler representation. >> SubDagOperator is often used to provide an isolated view into a subset of >> tasks in such large DAGs. To replace such SubDag use cases, TaskGroup will >> need to support Tree View. >> >> >> >> *What should TaskGroup look like in Tree View?* >> >> We didn't have a conclusion during the 1st iteration of TaskGroup. In >> Airbnb, we use SubDag mostly for providing a zoom in view on a small set of >> tasks and the SubDag zoom in feature worked well for us. We'd like to see >> TaskGroup provide a zoom in option for both Graph View and Tree View but >> also like to hear everyone's thoughts. >> >> >> >> *What needs to be in TaskGroup and what doesn't?* >> >> TaskGroup started off as a pure UI concept while SubDag is something >> more, e.g. it has its own DagRun thus isolated scheduling decisions, it can >> serve as a logical isolation layer that holds different sets of DAG level >> params, etc. While we only use SubDag as a UI feature, I think it would be >> a good opportunity for us to discuss what should be TaskGroup and what >> shouldn't. >> >> >> >> Please don't hesitate to share your thoughts. >> >> >> >> >> >> Cheers, >> >> Kevin Y >> >
