Hey Kevin, One immediate clarifying question: - For your use case, it seems that you want to continue using TaskGroup only as a "pure UI concept". - But, you want it's representation to also be in the Tree View. - You are not proposing any "execution or scheduling" enhancements [again for your use case] with TaskGroups
Is the above correct? Vikram On Wed, Mar 3, 2021 at 9:21 PM Kevin Yang <yrql...@gmail.com> wrote: > Hi team, > > We are very glad to see the introduction of TaskGroup in Airflow 2.0 and > really like it. Thanks to Yu Qian and everyone that contributed to it. To > continue moving towards the goal of replacing SubDagOperator with > TaskGroup, I'd like to kick off a discussion on bringing TaskGroup into > Tree View. > > *Why do we need TaskGroup in Tree View?* > For owners of larger DAGs, say a DAG with 500 tasks, Tree View is the > preferred view for its loading speed and simpler representation. > SubDagOperator is often used to provide an isolated view into a subset of > tasks in such large DAGs. To replace such SubDag use cases, TaskGroup will > need to support Tree View. > > *What should TaskGroup look like in Tree View**?* > We didn't have a conclusion during the 1st iteration of TaskGroup. In > Airbnb, we use SubDag mostly for providing a zoom in view on a small set of > tasks and the SubDag zoom in feature worked well for us. We'd like to see > TaskGroup provide a zoom in option for both Graph View and Tree View but > also like to hear everyone's thoughts. > > *What needs to be in TaskGroup and what doesn't?* > TaskGroup started off as a pure UI concept while SubDag is something more, > e.g. it has its own DagRun thus isolated scheduling decisions, it can serve > as a logical isolation layer that holds different sets of DAG level params, > etc. While we only use SubDag as a UI feature, I think it would be a good > opportunity for us to discuss what should be TaskGroup and what shouldn't. > > Please don't hesitate to share your thoughts. > > > Cheers, > Kevin Y >
