Re: [AIP-34] Rewrite SubDagOperator

Okay. On one hand, we want to automatically prefix task_id so that users don't have to parametrize task_id themselves inside TaskGroup to maintain task_id uniqueness. On the other hand, we don't want people to be surprised when they introduce TaskGroup to an existing DAG and all of a sudden task_id

Re: [AIP-34] Rewrite SubDagOperator

As I mentioned in the issue, I believe prefixing group_id is a nice thing as it makes TaskGroup an equivalent for SubDagOperator. Internally we have a similar concept to TaskGroup called FlattenedSubDagOperator that append the group_id to the task_id. One of the main usages internally for this ope

Re: [AIP-34] Rewrite SubDagOperator

The vote for this AIP-34 passed. However, there's an interesting discussion going on here reg

Re: [AIP-34] Rewrite SubDagOperator

Agree on this being non-blocking. Regarding moving to vote, you can take care. Just open a new email thread on dev list and call for a vote. You can see this example from Tomek for AIP-31: https://lists.apache.org/thread.html/r16ee2b6263f5849a2e2b1b6c3ba39fb3d643022f052216cb19a0a8da%40%3Cdev.airfl

Re: [AIP-34] Rewrite SubDagOperator

Hi, Gerard, yes I agree it's possible to do this at UI level without any fundamental change to the implementation. If expand_group() sees that two groups are fully connected (i.e. every task in one parent group depends on every task in another parent group), it can decide to collapse all those chil

Re: [AIP-34] Rewrite SubDagOperator

+1 > 2020年8月18日 23:55，Gerard Casas Saez 写道： > > Is it not possible to solve this at the UI level? Aka tell dagre to only > add 1 edge to the group instead of to all nodes in the group? No need to do > SubDag behaviour, but just reduce the edges on the graph. Should reduce > load time if I unders

Re: [AIP-34] Rewrite SubDagOperator

Is it not possible to solve this at the UI level? Aka tell dagre to only add 1 edge to the group instead of to all nodes in the group? No need to do SubDag behaviour, but just reduce the edges on the graph. Should reduce load time if I understand correctly. I would strongly avoid the Dummy operato

Re: [AIP-34] Rewrite SubDagOperator

Hi, All, Here's the updated AIP-34 . The PR has been fine-tuned with better UI interactions and added serialization of TaskGroup: https://github.com/apache/airflow

Re: [AIP-34] Rewrite SubDagOperator

Re graph times. That makes sense. Let me know what you find. We may be able to contribute on the lazy loading part. Looking forward to see the updated AIP! Gerard Casas Saez Twitter | Cortex | @casassaez On Fri, Aug 14, 2020 at 6:14 AM Kaxil Naik wrote: > Permi

Re: [AIP-34] Rewrite SubDagOperator

Permissions granted, let me know if you face any issues. On Fri, Aug 14, 2020 at 1:10 PM Yu Qian wrote: > Hi, Kaxil, my ID for cwiki.apache.org is yuqian1990. Thank you! > > On Fri, Aug 14, 2020 at 7:35 PM Kaxil Naik wrote: > > > What's your ID i.e. if you haven't created an account yet, please

Re: [AIP-34] Rewrite SubDagOperator

t;> or draft PR), we can use the same dag_factory > > > >>>>> function > > > >>>>>>> for > > > >>>>>>>>>>>>> genera

Re: [AIP-34] Rewrite SubDagOperator

t;>>>>> just > > >>>>>>>>>>>>>>>> extract all underlying tasks and append to the > > >>> root > > >>>>>> dag). > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> - Then it gets to the idea of replacing subdag > >

Re: [AIP-34] Rewrite SubDagOperator

with the >>>>> new >>>>>>>>>>>>> SubDagOperator >>>>>>>>>>>>>>> (see >>>>>>>>>>>>>>>>> AIP >>>>>>>>>>>>>>>&

Re: [AIP-34] Rewrite SubDagOperator

t;>>>>>>> SubDag > >>>>>>>>>>>>>>>> and becomes more like > >>>>>>>>>>>> ExtractSubdagTasksAndAppendToRootdagOperator > >>>>>>>>>>>>>>> (forgive > >>>>>>>>>>>>&g

Re: [AIP-34] Rewrite SubDagOperator

idea comes up. Thanks >>>> Chris >>>>>>> Palmer >>>>>>>>> for >>>>>>>>>>>>> helping >>>>>>>>>>>>>>>> conceptualize the functionality of TaskGroup, I >>>> will >>>>>> just >>>>>>>>> paste >>>>>>>>>>> it >&g

Re: [AIP-34] Rewrite SubDagOperator

t 9:45 AM, Chris Palmer > < > > > > > > > > > > ch...@crpalmer.com > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > >> I agree that SubDAGs are an overly complex > > > > > abstraction. > > > > > > I > > > > > > > > > think > > > > > &g

Re: [AIP-34] Rewrite SubDagOperator

> > > > > > > > > TaskGroup, > > > > > > > > > > > but > > > > > > > > > > > > > >> *cannot* have dependencies between a Task in a > > > > TaskGroup > > > > > > and > > > > > > > > > > either > > > > > > > > > > > a > > > > > > > > > > > > > >> Task in a different TaskGroup or a Task not in > any > > >

Re: [AIP-34] Rewrite SubDagOperator

t; > > > > > > > > > > >> Not sure if it would need to be a top level object > > > with > > > > > its > > > > > > > own > > > > > > > > > > > database > > > >

Re: [AIP-34] Rewrite SubDagOperator

t; > > > allows > > > > > > > > > > the > > > > > > > > > > > UI > > > > > > > > > > > >> to simplify the render of the DAG structure. > > > > > > > > > > > >&

Re: [AIP-34] Rewrite SubDagOperator

> >> > Agree with James (and think it's actually the more > > > > important > > > > > > > issue > > > > > > > > > to > > > > > > > > > > >> fix), > > > > > > > > > > >> > but I am still convinced Ash' ide

Re: [AIP-34] Rewrite SubDagOperator

> > why > > > > > > > > > >> subdags > > > > > > > > > >> > are bad and potential solutions: > > > > > > > > > >> > > > > > > > > https://www.mail-archive.com/dev@airflow.apache.org/msg01202.html > > > > > > > > . A > > > > > > > > > >> > solution I outline there to Jame's problem is e.g. > > > enabling > > > >

Re: [AIP-34] Rewrite SubDagOperator

; > > > >> > required to replace all existing subdag functionality. > > > > > > > > >> > > > > > > > > > >> > I've been working with subdags for 3 years and they are > > > > always a > > > > > > > giant > > > > > > > > >> pain > > > > > > > > >> > to us

Re: [AIP-34] Rewrite SubDagOperator

> I'm not sure I totally agree it's just a UI concept. I > use > > > the > > > > > > > subdag > > > > > > > >> > > operator to simplify dependencies too. If you have a > group > > >

Re: [AIP-34] Rewrite SubDagOperator

; > >> > > > > > > > > >> > > > I second Ash’s grouping concept. > > > > > > >> > > > > > > > > > >> > > > On Fri, Jun 12, 2020 at 5:10 AM Ash Berlin-Taylor < > >

Re: [AIP-34] Rewrite SubDagOperator

t; > On Fri, Jun 12, 2020 at 9:53 AM Kyle Hamlin < > > > > hamlin...@gmail.com> > > > > > > >> wrote: > > > > > > >> > > > > > > > > >> > > > I second Ash’s grouping concept. > > > > > > >> > > > > > > > > > >&

Re: [AIP-34] Rewrite SubDagOperator

ept be conceptually simpler, less to get wrong, and > > > closer > > > > > to > > > > > >> > what > > > > > >> > > > > users actually want to achieve with subdags? > > > > > >> > > > > > &

Re: [AIP-34] Rewrite SubDagOperator

omething > > > simpler. > > > > >> > > > > > > > > >> > > > > Problems with subdags (I think. I haven't used them > > > extensively > > > > so > > > > >> > may > > > > >> > > > &

Re: [AIP-34] Rewrite SubDagOperator

an be paused on their own. (Does it make sense > to > > do > > > >> > this? > > > >> > > > > Pausing just a sub dag would mean the sub dag would never > > > >> execute, so > > > >> > > > > the SubDagOperator would fail

Re: [AIP-34] Rewrite SubDagOperator

> progresses. > > >> > > > > > > > >> > > > > > > > >> > > > > >> - *Unpack SubDags during dag parsing*: This rewrites the > > >> > > > > *DagBag.bag_dag* > > >> > > > >

Re: [AIP-34] Rewrite SubDagOperator

e understood your idea here correctly. > >> > > > > > > >> > > > > > -ash > >> > > > > > > >> > > > > > > >> > > > > > On Jun 12 2020, at 9:51 am, Xinbin Huang < > bin.huan...@gmail.com >

Re: [AIP-34] Rewrite SubDagOperator

t; > >> >> > > > > >> Sending a message to everyone and collect feedback on the >> AIP-34 >> > on >> > > > > >> rewriting SubDagOperator. This was previously briefly >> mentioned in >> > > the >> > > > >

Re: [AIP-34] Rewrite SubDagOperator

fly >> mentioned in >> > > the >> > > > > >> discussion about what needs to be done for Airflow 2.0, and >> one of >> > > the >> > > > > >> ideas is to make SubDagOperator attach tasks back to the root >> DAG. >&

Re: [AIP-34] Rewrite SubDagOperator

eds to be done for Airflow 2.0, and > one > > of > > > > the > > > > > > >> ideas is to make SubDagOperator attach tasks back to the root > > DAG. > > > > > > >> > > > > > > >> This AIP-34 focuses on solvi

Re: [AIP-34] Rewrite SubDagOperator

gt; DAG. > > > > > > >> > > > > > > >> This AIP-34 focuses on solving SubDagOperator related issues > by > > > > > > reattaching > > > > > > >> all tasks back to the root dag while respecting dependencies > > >

Re: [AIP-34] Rewrite SubDagOperator

t; grouping related tasks by metadata. > > > > > >> > > > > > >> This also makes the dag_factory function more reusable because > you > > > > don't > > > > > >> need to have parent_dag_name and child_dag_name in the function > >

Re: [AIP-34] Rewrite SubDagOperator

ding a message to everyone and collect feedback on the >> AIP-34 >> > on >> > > > > >> rewriting SubDagOperator. This was previously briefly >> mentioned in >> > > the >> > > > > >> discussion about what needs to be done f

Re: [AIP-34] Rewrite SubDagOperator

iginal grouping effect on the UI will be achieved > > > > through > > > > > >> grouping related tasks by metadata. > > > > > >> > > > > > >> This also makes the dag_factory function more reusable because &g

Re: [AIP-34] Rewrite SubDagOperator

t_dag_name and child_dag_name in the function > > > > signature > > > > >> anymore. > > > > >> > > > > >> Changes proposed: > > > > >> > > > > >> - *Unpack SubDags during dag parsing*: This rewrites

Re: [AIP-34] Rewrite SubDagOperator

> >> - *Unpack SubDags during dag parsing*: This rewrites the > > > > *DagBag.bag_dag* > > > > >> method to unpack subdag while parsing, and it will give a flat > > > > >> structure at > > > > >> the task level > > > > >>

Re: [AIP-34] Rewrite SubDagOperator

ike a > > > >> container and most of the original methods are removed. The > > > >> signature is > > > >> also changed to *subdag_factory *with *subdag_args *and > > > *subdag_kwargs*. > > > >> This is similar to the PythonOperato

Re: [AIP-34] Rewrite SubDagOperator

t; This is similar to the PythonOperator signature. > > >> - *Add a TaskGroup model and add current_group & parent_group > > attributes > > >> to BaseOperator*: This metadata is used to group tasks for > > >> rendering at > > >> UI le

Re: [AIP-34] Rewrite SubDagOperator

to group tasks for > >> rendering at > >> UI level. It may potentially extend further to group arbitrary tasks > >> outside the context of subdag to allow group-level operations (i.e. > >> stop/trigger a group of task within the dag) > >> - *We

Re: [AIP-34] Rewrite SubDagOperator

w group-level operations (i.e. >> stop/trigger a group of task within the dag) >> - *Webserver UI for SubDag*: Proposed UI modification to allow >> (un)collapse a group of tasks for a flat structure to pair with the first >> change instead of the original hie

Re: [AIP-34] Rewrite SubDagOperator

task within the dag) > - *Webserver UI for SubDag*: Proposed UI modification to allow > (un)collapse a group of tasks for a flat structure to pair with the first > change instead of the original hierarchical structure. > > > Please see related documents and PRs for details:

[AIP-34] Rewrite SubDagOperator

pache.org/confluence/display/AIRFLOW/AIP-34+Rewrite+SubDagOperator Original Issue: https://github.com/apache/airflow/issues/8078 Draft PR: https://github.com/apache/airflow/pull/9243 Please let me know if there are any aspects that you agree/disagree with or need more clarification (especially the thi