Re: [AIP-34] Rewrite SubDagOperator

Okay. On one hand, we want to automatically prefix task_id so that users don't have to parametrize task_id themselves inside TaskGroup to maintain task_id uniqueness. On the other hand, we don't want people to be surprised when they introduce TaskGroup to an existing DAG and all of a sudden

Re: [AIP-34] Rewrite SubDagOperator

As I mentioned in the issue, I believe prefixing group_id is a nice thing as it makes TaskGroup an equivalent for SubDagOperator. Internally we have a similar concept to TaskGroup called FlattenedSubDagOperator that append the group_id to the task_id. One of the main usages internally for this

Re: [AIP-34] Rewrite SubDagOperator

The vote for this AIP-34 passed. However, there's an interesting discussion going on here

Re: [AIP-34] Rewrite SubDagOperator

Agree on this being non-blocking. Regarding moving to vote, you can take care. Just open a new email thread on dev list and call for a vote. You can see this example from Tomek for AIP-31:

Re: [AIP-34] Rewrite SubDagOperator

Hi, Gerard, yes I agree it's possible to do this at UI level without any fundamental change to the implementation. If expand_group() sees that two groups are fully connected (i.e. every task in one parent group depends on every task in another parent group), it can decide to collapse all those

Re: [AIP-34] Rewrite SubDagOperator

+1 > 2020年8月18日 23:55，Gerard Casas Saez 写道： > > Is it not possible to solve this at the UI level? Aka tell dagre to only > add 1 edge to the group instead of to all nodes in the group? No need to do > SubDag behaviour, but just reduce the edges on the graph. Should reduce > load time if I

Re: [AIP-34] Rewrite SubDagOperator

Is it not possible to solve this at the UI level? Aka tell dagre to only add 1 edge to the group instead of to all nodes in the group? No need to do SubDag behaviour, but just reduce the edges on the graph. Should reduce load time if I understand correctly. I would strongly avoid the Dummy

Re: [AIP-34] Rewrite SubDagOperator

Hi, All, Here's the updated AIP-34 . The PR has been fine-tuned with better UI interactions and added serialization of TaskGroup:

Re: [AIP-34] Rewrite SubDagOperator

Re graph times. That makes sense. Let me know what you find. We may be able to contribute on the lazy loading part. Looking forward to see the updated AIP! Gerard Casas Saez Twitter | Cortex | @casassaez On Fri, Aug 14, 2020 at 6:14 AM Kaxil Naik wrote: >

Re: [AIP-34] Rewrite SubDagOperator

Permissions granted, let me know if you face any issues. On Fri, Aug 14, 2020 at 1:10 PM Yu Qian wrote: > Hi, Kaxil, my ID for cwiki.apache.org is yuqian1990. Thank you! > > On Fri, Aug 14, 2020 at 7:35 PM Kaxil Naik wrote: > > > What's your ID i.e. if you haven't created an account yet,

Re: [AIP-34] Rewrite SubDagOperator

>>>>>>>>> generating 1 > > > >>>>>>>>>>>>>>>> dag, a lot of dynamic dags, or used for SubDag > > > >>> (in > > > >>>>> this > &

Re: [AIP-34] Rewrite SubDagOperator

> > >>>>>> dag). > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> - Then it gets to the idea of replacing subdag > > >>>> with a > > >>>>>>>>>> simpler > > >>>>>>>>>>>>>> concept > > >>&

Re: [AIP-34] Rewrite SubDagOperator

>>>>>>>>>>>>>> AIP >>>>>>>>>>>>>>>>>> or draft PR), we can use the same dag_factory >>>>>>> function >>>>>>>>> for >>>>>>&

Re: [AIP-34] Rewrite SubDagOperator

;>>>>> ExtractSubdagTasksAndAppendToRootdagOperator > >>>>>>>>>>>>>>> (forgive > >>>>>>>>>>>>>>>> me about the crazy name..). In this case, it is > >>>> still > >>>>>>>>>>> necessary > >>>&g

Re: [AIP-34] Rewrite SubDagOperator

;>>>>>>>> helping >>>>>>>>>>>>>>>> conceptualize the functionality of TaskGroup, I >>>> will >>>>>> just >>>>>>>>> paste >>>>>>>>>>> it >>>>>>>>>>>>>> here. >>>>>>>>>>>>>>>> >>>>>>>>>>

Re: [AIP-34] Rewrite SubDagOperator

> > > > > > > > wrote: > > > > > > > > > > > > > > >> I agree that SubDAGs are an overly complex > > > > > abstraction. > > > > > > I > > > > > > > > > think > > > > > > > > > > > what > > > > > > > > > > > > > is > > > > > > > > > > > > > > >> needed/usefu

Re: [AIP-34] Rewrite SubDagOperator

nnot* have dependencies between a Task in a > > > > TaskGroup > > > > > > and > > > > > > > > > > either > > > > > > > > > > > a > > > > > > > > > > > > > >> Task in a different TaskGroup or a Task not in > any > > > > group > > > > > > > > > > > > > >> - You *can* have dependencies between a > TaskGroup > > > and > > > > > > either &g

Re: [AIP-34] Rewrite SubDagOperator

> own > > > > > > > > > > > database > > > > > > > > > > > > >> table and model or just another attribute on > tasks. > > I > > > > > think > > > > > > > you > > > > > &g

Re: [AIP-34] Rewrite SubDagOperator

gt; > > >> to simplify the render of the DAG structure. > > > > > > > > > > > >> > > > > > > > > > > > >> Chris > > > > > > > > > > > >> > > > > >

Re: [AIP-34] Rewrite SubDagOperator

> > > > > > > to > > > > > > > > > > >> fix), > > > > > > > > > > >> > but I am still convinced Ash' idea is the right way > > > > forward > > > > > > > (just > > > > > > > > it > > > > > > &

Re: [AIP-34] Rewrite SubDagOperator

solutions: > > > > > > > > > >> > > > > > > > > https://www.mail-archive.com/dev@airflow.apache.org/msg01202.html > > > > > > > > . A > > > > > > > > > >> > solution I outline there to Jame's problem is e.g. > > > enabling > > > > > the > > > > > > >> > > > > > > > > > >> operator > > > > > > > > > >> > for A

Re: [AIP-34] Rewrite SubDagOperator

t; > > >> > > operator to simplify dependencies too. If you have a > group > > > of > > > > > > tasks > > > > > > > >> that > > > > > > > >> > > need to finish befo

Re: [AIP-34] Rewrite SubDagOperator

> > > > > > > >> > > > On Fri, Jun 12, 2020 at 5:10 AM Ash Berlin-Taylor < > > > > > a...@apache.org > > > > > > > > > > > > > >> > > wrote: > > > > > > >> &

Re: [AIP-34] Rewrite SubDagOperator

> > > >> > > > > > > > > >> > > > I second Ash’s grouping concept. > > > > > > >> > > > > > > > > > >> > > > On Fri, Jun 12, 2020 at 5:10 AM Ash Berlin-Taylor < > > > > > a...@apache.org > > > > > > > > > &g

Re: [AIP-34] Rewrite SubDagOperator

t;> > > > > users actually want to achieve with subdags? > > > > > >> > > > > > > > > > >> > > > > With your proposed change, tasks in subdags could start > > > > running > > > > > in >

Re: [AIP-34] Rewrite SubDagOperator

gt; > > extensively > > > > so > > > > >> > may > > > > >> > > > > be wrong on some of these): > > > > >> > > > > - They need their own dag_id, but it has(?) to be of the > > form > > > &g

Re: [AIP-34] Rewrite SubDagOperator

st a sub dag would mean the sub dag would never > > > >> execute, so > > > >> > > > > the SubDagOperator would fail too. > > > >> > > > > - You had to choose the executor to operator a subdag with > -- > > > >> always > >

Re: [AIP-34] Rewrite SubDagOperator

g parsing*: This rewrites the > > >> > > > > *DagBag.bag_dag* > > >> > > > > >> method to unpack subdag while parsing, and it will give a > > flat > > >> > > > > >> structure at > > >>

Re: [AIP-34] Rewrite SubDagOperator

t; > > > > > > >> > > > > > > >> > > > > > On Jun 12 2020, at 9:51 am, Xinbin Huang < > bin.huan...@gmail.com > >> > > >> > > > wrote: > >> > > > > > > >> > > > >

Re: [AIP-34] Rewrite SubDagOperator

> > on >> > > > > >> rewriting SubDagOperator. This was previously briefly >> mentioned in >> > > the >> > > > > >> discussion about what needs to be done for Airflow 2.0, and >> one of >> > > the >

Re: [AIP-34] Rewrite SubDagOperator

flow 2.0, and >> one of >> > > the >> > > > > >> ideas is to make SubDagOperator attach tasks back to the root >> DAG. >> > > > > >> >> > > > > >> This AIP-34 focuses on solving SubDagOperator

Re: [AIP-34] Rewrite SubDagOperator

> > > > >> ideas is to make SubDagOperator attach tasks back to the root > > DAG. > > > > > > >> > > > > > > >> This AIP-34 focuses on solving SubDagOperator related issues > by > > > > > > reattaching > > &g

Re: [AIP-34] Rewrite SubDagOperator

cuses on solving SubDagOperator related issues > by > > > > > > reattaching > > > > > > >> all tasks back to the root dag while respecting dependencies > > > during > > > > > > >> parsing. The original grouping effect on th

Re: [AIP-34] Rewrite SubDagOperator

> > > > > >> This also makes the dag_factory function more reusable because > you > > > > don't > > > > > >> need to have parent_dag_name and child_dag_name in the function > > > > > signature > > > > > >> anymore. > > > &g

Re: [AIP-34] Rewrite SubDagOperator

gt; rewriting SubDagOperator. This was previously briefly >> mentioned in >> > > the >> > > > > >> discussion about what needs to be done for Airflow 2.0, and >> one of >> > > the >> > > > > >> ideas is to make SubDagOpera

Re: [AIP-34] Rewrite SubDagOperator

grouping related tasks by metadata. > > > > > >> > > > > > >> This also makes the dag_factory function more reusable because > you > > > > don't > > > > > >> need to have parent_dag_name and child_dag_name in the function > > > >

Re: [AIP-34] Rewrite SubDagOperator

ure > > > > >> anymore. > > > > >> > > > > >> Changes proposed: > > > > >> > > > > >> - *Unpack SubDags during dag parsing*: This rewrites the > > > > *DagBag.bag_dag* > > > > >> method

Re: [AIP-34] Rewrite SubDagOperator

tes the > > > > *DagBag.bag_dag* > > > > >> method to unpack subdag while parsing, and it will give a flat > > > > >> structure at > > > > >> the task level > > > > >> - *Simplify SubDagOperator*: The new SubDagOperator acts like a

Re: [AIP-34] Rewrite SubDagOperator

he original methods are removed. The > > > >> signature is > > > >> also changed to *subdag_factory *with *subdag_args *and > > > *subdag_kwargs*. > > > >> This is similar to the PythonOperator signature. > > > >> - *Add a TaskGro

Re: [AIP-34] Rewrite SubDagOperator

thonOperator signature. > > >> - *Add a TaskGroup model and add current_group & parent_group > > attributes > > >> to BaseOperator*: This metadata is used to group tasks for > > >> rendering at > > >> UI level. It may potentially extend

Re: [AIP-34] Rewrite SubDagOperator

gt; >> rendering at > >> UI level. It may potentially extend further to group arbitrary tasks > >> outside the context of subdag to allow group-level operations (i.e. > >> stop/trigger a group of task within the dag) > >> - *Webserver UI for SubDa

Re: [AIP-34] Rewrite SubDagOperator

ions (i.e. >> stop/trigger a group of task within the dag) >> - *Webserver UI for SubDag*: Proposed UI modification to allow >> (un)collapse a group of tasks for a flat structure to pair with the first >> change instead of the original hierarchical structure.

Re: [AIP-34] Rewrite SubDagOperator

in the dag) > - *Webserver UI for SubDag*: Proposed UI modification to allow > (un)collapse a group of tasks for a flat structure to pair with the first > change instead of the original hierarchical structure. > > > Please see related documents and PRs for details: > AIP: >

[AIP-34] Rewrite SubDagOperator

.org/confluence/display/AIRFLOW/AIP-34+Rewrite+SubDagOperator Original Issue: https://github.com/apache/airflow/issues/8078 Draft PR: https://github.com/apache/airflow/pull/9243 Please let me know if there are any aspects that you agree/disagree with or need more clarification (especially the third ch