Hi team,

We are very glad to see the introduction of TaskGroup in Airflow 2.0 and
really like it. Thanks to Yu Qian and everyone that contributed to it. To
continue moving towards the goal of replacing SubDagOperator with
TaskGroup, I'd like to kick off a discussion on bringing TaskGroup into
Tree View.

*Why do we need TaskGroup in Tree View?*
For owners of larger DAGs, say a DAG with 500 tasks, Tree View is the
preferred view for its loading speed and simpler representation.
SubDagOperator is often used to provide an isolated view into a subset of
tasks in such large DAGs. To replace such SubDag use cases, TaskGroup will
need to support Tree View.

*What should TaskGroup look like in Tree View**?*
We didn't have a conclusion during the 1st iteration of TaskGroup. In
Airbnb, we use SubDag mostly for providing a zoom in view on a small set of
tasks and the SubDag zoom in feature worked well for us. We'd like to see
TaskGroup provide a zoom in option for both Graph View and Tree View but
also like to hear everyone's thoughts.

*What needs to be in TaskGroup and what doesn't?*
TaskGroup started off as a pure UI concept while SubDag is something more,
e.g. it has its own DagRun thus isolated scheduling decisions, it can serve
as a logical isolation layer that holds different sets of DAG level params,
etc. While we only use SubDag as a UI feature, I think it would be a good
opportunity for us to discuss what should be TaskGroup and what shouldn't.

Please don't hesitate to share your thoughts.


Cheers,
Kevin Y

Reply via email to