Hi all, interesting discussion. I would love to hear about some more use-cases where TaskGroup needs to be something more than the UI concept.
All of Kevin's use-cases can be achieved while keeping it as a UI concept.Xinbin can you please expand a bit on your use case. Regards, Kaxil On Sat, Mar 6, 2021, 10:08 Xinbin Huang <bin.huan...@gmail.com> wrote: > Hi Kevin, Vikram, and Nathan, > > I think we don't need to restrict too much on keeping TaskGroup only as a > UI concept. We are already using TaskGroup to author DAGs and create > dependencies, which already lies a bit outside the UI. > To fully replace SubDagOperator, I think it's necessary to expand > TaskGroup as a *container for tasks* than just UI concept. > > As for TaskGroupSensor specifically, I land with the same approach as > Kevin, and I have created a draft PR here: > https://github.com/apache/airflow/pull/14640 > > Cheers > Bin > > On Fri, Mar 5, 2021 at 10:00 PM Kevin Yang <yrql...@gmail.com> wrote: > >> Hi Vikram, >> >> Good point. What I had in mind was getting the TaskGroup definition in a >> sensor, e.g. extract the _task_group field from serialized DAG, and query >> the DB for the TI states within. >> >> You are right that it might not be clean nor does it keep TaskGroup as a >> UI concept. >> >> >> Cheers, >> Kevin Y >> >> On Fri, Mar 5, 2021 at 8:19 PM Vikram Koka <vik...@astronomer.io.invalid> >> wrote: >> >>> Kevin, >>> >>> I am not sure I understand your response to Nathan. >>> >>> I agree that it is also a valid use case, but I don't see how it can be >>> cleanly done while keeping TaskGroup only as a UI concept. >>> Would this require extending the TaskGroup concept to the backend? >>> >>> Best regards, >>> Vikram >>> >>> On Fri, Mar 5, 2021 at 1:31 AM Kevin Yang <yrql...@gmail.com> wrote: >>> >>>> Hi Nathan, >>>> >>>> Thanks a lot for your input and it is indeed a valid use case. This can >>>> be done either keeping TaskGroup as a UI concept or bringing it into the >>>> backend. I'm curious to hear what others think. >>>> >>>> >>>> Cheers, >>>> Kevin Y >>>> >>>> On Thu, Mar 4, 2021 at 12:57 AM Nathan Hadfield < >>>> nathan.hadfi...@king.com> wrote: >>>> >>>>> Hi Kevin, >>>>> >>>>> >>>>> >>>>> A quick piece of input from our recent experiences of working with >>>>> TaskGroup is that we often have dependencies across DAGs that require >>>>> waiting upon the completion of all the tasks in a group. At the moment, >>>>> you basically have two options: >>>>> >>>>> >>>>> >>>>> 1. Create a sensor task in a DAG for every task in the group >>>>> 2. Create a Dummy task after the group that a sensor waits on >>>>> >>>>> >>>>> >>>>> So, I would certainly like TaskGroups to have some notion of run >>>>> status as to better enable downstream decision making. >>>>> >>>>> >>>>> >>>>> I’ve already created a feature ticket to try to add some kind of >>>>> TaskGroup Sensor but perhaps this can also form part of the wider >>>>> discussions here. >>>>> >>>>> >>>>> >>>>> https://github.com/apache/airflow/issues/14563 >>>>> >>>>> >>>>> >>>>> Cheers, >>>>> >>>>> >>>>> >>>>> Nathan >>>>> >>>>> >>>>> >>>>> *From: *Kevin Yang <yrql...@gmail.com> >>>>> *Date: *Thursday, 4 March 2021 at 05:21 >>>>> *To: *dev@airflow.apache.org <dev@airflow.apache.org> >>>>> *Subject: *[DISCUSS] TaskGroup in Tree View >>>>> >>>>> Hi team, >>>>> >>>>> >>>>> >>>>> We are very glad to see the introduction of TaskGroup in Airflow 2.0 >>>>> and really like it. Thanks to Yu Qian and everyone that contributed to it. >>>>> To continue moving towards the goal of replacing SubDagOperator with >>>>> TaskGroup, I'd like to kick off a discussion on bringing TaskGroup into >>>>> Tree View. >>>>> >>>>> >>>>> >>>>> *Why do we need TaskGroup in Tree View?* >>>>> >>>>> For owners of larger DAGs, say a DAG with 500 tasks, Tree View is the >>>>> preferred view for its loading speed and simpler representation. >>>>> SubDagOperator is often used to provide an isolated view into a subset of >>>>> tasks in such large DAGs. To replace such SubDag use cases, TaskGroup will >>>>> need to support Tree View. >>>>> >>>>> >>>>> >>>>> *What should TaskGroup look like in Tree View?* >>>>> >>>>> We didn't have a conclusion during the 1st iteration of TaskGroup. In >>>>> Airbnb, we use SubDag mostly for providing a zoom in view on a small set >>>>> of >>>>> tasks and the SubDag zoom in feature worked well for us. We'd like to see >>>>> TaskGroup provide a zoom in option for both Graph View and Tree View but >>>>> also like to hear everyone's thoughts. >>>>> >>>>> >>>>> >>>>> *What needs to be in TaskGroup and what doesn't?* >>>>> >>>>> TaskGroup started off as a pure UI concept while SubDag is something >>>>> more, e.g. it has its own DagRun thus isolated scheduling decisions, it >>>>> can >>>>> serve as a logical isolation layer that holds different sets of DAG level >>>>> params, etc. While we only use SubDag as a UI feature, I think it would be >>>>> a good opportunity for us to discuss what should be TaskGroup and what >>>>> shouldn't. >>>>> >>>>> >>>>> >>>>> Please don't hesitate to share your thoughts. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Cheers, >>>>> >>>>> Kevin Y >>>>> >>>>
