Hi Kevin, Vikram, and Nathan, I think we don't need to restrict too much on keeping TaskGroup only as a UI concept. We are already using TaskGroup to author DAGs and create dependencies, which already lies a bit outside the UI. To fully replace SubDagOperator, I think it's necessary to expand TaskGroup as a *container for tasks* than just UI concept.
As for TaskGroupSensor specifically, I land with the same approach as Kevin, and I have created a draft PR here: https://github.com/apache/airflow/pull/14640 Cheers Bin On Fri, Mar 5, 2021 at 10:00 PM Kevin Yang <yrql...@gmail.com> wrote: > Hi Vikram, > > Good point. What I had in mind was getting the TaskGroup definition in a > sensor, e.g. extract the _task_group field from serialized DAG, and query > the DB for the TI states within. > > You are right that it might not be clean nor does it keep TaskGroup as a > UI concept. > > > Cheers, > Kevin Y > > On Fri, Mar 5, 2021 at 8:19 PM Vikram Koka <vik...@astronomer.io.invalid> > wrote: > >> Kevin, >> >> I am not sure I understand your response to Nathan. >> >> I agree that it is also a valid use case, but I don't see how it can be >> cleanly done while keeping TaskGroup only as a UI concept. >> Would this require extending the TaskGroup concept to the backend? >> >> Best regards, >> Vikram >> >> On Fri, Mar 5, 2021 at 1:31 AM Kevin Yang <yrql...@gmail.com> wrote: >> >>> Hi Nathan, >>> >>> Thanks a lot for your input and it is indeed a valid use case. This can >>> be done either keeping TaskGroup as a UI concept or bringing it into the >>> backend. I'm curious to hear what others think. >>> >>> >>> Cheers, >>> Kevin Y >>> >>> On Thu, Mar 4, 2021 at 12:57 AM Nathan Hadfield < >>> nathan.hadfi...@king.com> wrote: >>> >>>> Hi Kevin, >>>> >>>> >>>> >>>> A quick piece of input from our recent experiences of working with >>>> TaskGroup is that we often have dependencies across DAGs that require >>>> waiting upon the completion of all the tasks in a group. At the moment, >>>> you basically have two options: >>>> >>>> >>>> >>>> 1. Create a sensor task in a DAG for every task in the group >>>> 2. Create a Dummy task after the group that a sensor waits on >>>> >>>> >>>> >>>> So, I would certainly like TaskGroups to have some notion of run status >>>> as to better enable downstream decision making. >>>> >>>> >>>> >>>> I’ve already created a feature ticket to try to add some kind of >>>> TaskGroup Sensor but perhaps this can also form part of the wider >>>> discussions here. >>>> >>>> >>>> >>>> https://github.com/apache/airflow/issues/14563 >>>> >>>> >>>> >>>> Cheers, >>>> >>>> >>>> >>>> Nathan >>>> >>>> >>>> >>>> *From: *Kevin Yang <yrql...@gmail.com> >>>> *Date: *Thursday, 4 March 2021 at 05:21 >>>> *To: *dev@airflow.apache.org <dev@airflow.apache.org> >>>> *Subject: *[DISCUSS] TaskGroup in Tree View >>>> >>>> Hi team, >>>> >>>> >>>> >>>> We are very glad to see the introduction of TaskGroup in Airflow 2.0 >>>> and really like it. Thanks to Yu Qian and everyone that contributed to it. >>>> To continue moving towards the goal of replacing SubDagOperator with >>>> TaskGroup, I'd like to kick off a discussion on bringing TaskGroup into >>>> Tree View. >>>> >>>> >>>> >>>> *Why do we need TaskGroup in Tree View?* >>>> >>>> For owners of larger DAGs, say a DAG with 500 tasks, Tree View is the >>>> preferred view for its loading speed and simpler representation. >>>> SubDagOperator is often used to provide an isolated view into a subset of >>>> tasks in such large DAGs. To replace such SubDag use cases, TaskGroup will >>>> need to support Tree View. >>>> >>>> >>>> >>>> *What should TaskGroup look like in Tree View?* >>>> >>>> We didn't have a conclusion during the 1st iteration of TaskGroup. In >>>> Airbnb, we use SubDag mostly for providing a zoom in view on a small set of >>>> tasks and the SubDag zoom in feature worked well for us. We'd like to see >>>> TaskGroup provide a zoom in option for both Graph View and Tree View but >>>> also like to hear everyone's thoughts. >>>> >>>> >>>> >>>> *What needs to be in TaskGroup and what doesn't?* >>>> >>>> TaskGroup started off as a pure UI concept while SubDag is something >>>> more, e.g. it has its own DagRun thus isolated scheduling decisions, it can >>>> serve as a logical isolation layer that holds different sets of DAG level >>>> params, etc. While we only use SubDag as a UI feature, I think it would be >>>> a good opportunity for us to discuss what should be TaskGroup and what >>>> shouldn't. >>>> >>>> >>>> >>>> Please don't hesitate to share your thoughts. >>>> >>>> >>>> >>>> >>>> >>>> Cheers, >>>> >>>> Kevin Y >>>> >>>
