Hi Vikram, Good point. What I had in mind was getting the TaskGroup definition in a sensor, e.g. extract the _task_group field from serialized DAG, and query the DB for the TI states within.
You are right that it might not be clean nor does it keep TaskGroup as a UI concept. Cheers, Kevin Y On Fri, Mar 5, 2021 at 8:19 PM Vikram Koka <vik...@astronomer.io.invalid> wrote: > Kevin, > > I am not sure I understand your response to Nathan. > > I agree that it is also a valid use case, but I don't see how it can be > cleanly done while keeping TaskGroup only as a UI concept. > Would this require extending the TaskGroup concept to the backend? > > Best regards, > Vikram > > On Fri, Mar 5, 2021 at 1:31 AM Kevin Yang <yrql...@gmail.com> wrote: > >> Hi Nathan, >> >> Thanks a lot for your input and it is indeed a valid use case. This can >> be done either keeping TaskGroup as a UI concept or bringing it into the >> backend. I'm curious to hear what others think. >> >> >> Cheers, >> Kevin Y >> >> On Thu, Mar 4, 2021 at 12:57 AM Nathan Hadfield <nathan.hadfi...@king.com> >> wrote: >> >>> Hi Kevin, >>> >>> >>> >>> A quick piece of input from our recent experiences of working with >>> TaskGroup is that we often have dependencies across DAGs that require >>> waiting upon the completion of all the tasks in a group. At the moment, >>> you basically have two options: >>> >>> >>> >>> 1. Create a sensor task in a DAG for every task in the group >>> 2. Create a Dummy task after the group that a sensor waits on >>> >>> >>> >>> So, I would certainly like TaskGroups to have some notion of run status >>> as to better enable downstream decision making. >>> >>> >>> >>> I’ve already created a feature ticket to try to add some kind of >>> TaskGroup Sensor but perhaps this can also form part of the wider >>> discussions here. >>> >>> >>> >>> https://github.com/apache/airflow/issues/14563 >>> >>> >>> >>> Cheers, >>> >>> >>> >>> Nathan >>> >>> >>> >>> *From: *Kevin Yang <yrql...@gmail.com> >>> *Date: *Thursday, 4 March 2021 at 05:21 >>> *To: *dev@airflow.apache.org <dev@airflow.apache.org> >>> *Subject: *[DISCUSS] TaskGroup in Tree View >>> >>> Hi team, >>> >>> >>> >>> We are very glad to see the introduction of TaskGroup in Airflow 2.0 and >>> really like it. Thanks to Yu Qian and everyone that contributed to it. To >>> continue moving towards the goal of replacing SubDagOperator with >>> TaskGroup, I'd like to kick off a discussion on bringing TaskGroup into >>> Tree View. >>> >>> >>> >>> *Why do we need TaskGroup in Tree View?* >>> >>> For owners of larger DAGs, say a DAG with 500 tasks, Tree View is the >>> preferred view for its loading speed and simpler representation. >>> SubDagOperator is often used to provide an isolated view into a subset of >>> tasks in such large DAGs. To replace such SubDag use cases, TaskGroup will >>> need to support Tree View. >>> >>> >>> >>> *What should TaskGroup look like in Tree View?* >>> >>> We didn't have a conclusion during the 1st iteration of TaskGroup. In >>> Airbnb, we use SubDag mostly for providing a zoom in view on a small set of >>> tasks and the SubDag zoom in feature worked well for us. We'd like to see >>> TaskGroup provide a zoom in option for both Graph View and Tree View but >>> also like to hear everyone's thoughts. >>> >>> >>> >>> *What needs to be in TaskGroup and what doesn't?* >>> >>> TaskGroup started off as a pure UI concept while SubDag is something >>> more, e.g. it has its own DagRun thus isolated scheduling decisions, it can >>> serve as a logical isolation layer that holds different sets of DAG level >>> params, etc. While we only use SubDag as a UI feature, I think it would be >>> a good opportunity for us to discuss what should be TaskGroup and what >>> shouldn't. >>> >>> >>> >>> Please don't hesitate to share your thoughts. >>> >>> >>> >>> >>> >>> Cheers, >>> >>> Kevin Y >>> >>
