Thanks Ash for reminding me the problem of image attachments :) Here are the image links for my previous message *Runtime DAG*: https://airflow.apache.org/docs/stable/_images/subdag_before.png *UI Visual*: https://airflow.apache.org/docs/stable/_images/subdag_after.png
On Fri, Jun 12, 2020 at 11:48 AM Xinbin Huang <bin.huan...@gmail.com> wrote: > > > - *Unpack SubDags during dag parsing*: This rewrites the > *DagBag.bag_dag* > > > method to unpack subdag while parsing, and it will give a flat > > > structure at > > > the task level > > > > The serialized_dag representation already does this I think. At least if > > I've understood your idea here correctly. > > I am not sure about serialized_dag representation, but at least it will > still keep the subdag entry in the DAG table? In my proposal as also in the > draft PR, the idea is to *extract the tasks from the subdag and add them > back to the root_dag. *So the runtime DAG graph will look exactly the > same as without subdag but with metadata attached to those sections. These > metadata will be later on used to render in the UI. So after parsing ( > *DagBag.process_file()*), it will just output the *root_dag *instead of > *root_dag + > subdag + subdag + nested subdag* etc. > > - e.g. section-1-* will have metadata current_group=section-1, > parent_group=<the-root-dag-id> (welcome for naming suggestions), the > reason for parent_group is that we can have nested group and still be > able to capture the dependency. > > Runtime DAG: > [image: image.png] > > While at the UI, what we see would be something like this by utilizing the > metadata, and then we can expand or zoom into in some way. > [image: image.png] > > The benefits I can see is that: > 1. We don't need to deal with the extra complexity of SubDag for execution > and scheduling. It will be the same as not using SubDag. > 2. Still have the benefits of modularized and reusable dag code and > declare dependencies between them. And with the new SubDagOperator (see AIP > or draft PR), we can use the same dag_factory function for generating 1 > dag, a lot of dynamic dags, or used for SubDag (in this case, it will just > extract all underlying tasks and append to the root dag). > > - Then it gets to the idea of replacing subdag with a simpler concept > by Ash: the proposed change basically drains out the contents of a SubDag > and becomes more like ExtractSubdagTasksAndAppendToRootdagOperator (forgive > me about the crazy name..). In this case, it is still necessary to keep the > concept of subdag as it is nothing more than a name? > > That's why the TaskGroup idea comes up. Thanks Chris Palmer for helping > conceptualize the functionality of TaskGroup, I will just paste it here. > > > - Tasks can be added to a TaskGroup > > - You *can* have dependencies between Tasks in the same TaskGroup, but > > *cannot* have dependencies between a Task in a TaskGroup and either a > > Task in a different TaskGroup or a Task not in any group > > - You *can* have dependencies between a TaskGroup and either other > > TaskGroups or Tasks not in any group > > - The UI will by default render a TaskGroup as a single "object", but > > which you expand or zoom into in some way > > - You'd need some way to determine what the "status" of a TaskGroup was > > at least for UI display purposes > > I agree with Chris: > - From the backend's view (scheduler & executor), I think TaskGroup should > be ignored during execution. (unless we decide to implement some metadata > operations that allows start/stop a group of tasks etc.) > - From the UI's View, it should be able to pick up the individual tasks' > status and then determine the TaskGroup's status > > Bin > > On Fri, Jun 12, 2020 at 10:28 AM Daniel Imberman < > daniel.imber...@gmail.com> wrote: > >> I hadn’t thought about using the `>>` operator to tie dags together but I >> think that sounds pretty great! I wonder if we could essentially write in >> the ability to set dependencies to all starter-tasks for that DAG. >> >> I’m personally ok with SubDag being a mostly UI concept. It doesn’t need >> to execute separately, you’re just adding more tasks to the queue that will >> be executed when there are resources available. >> >> via Newton Mail [ >> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.14.6&source=email_footer_2 >> ] >> On Fri, Jun 12, 2020 at 9:45 AM, Chris Palmer <ch...@crpalmer.com> wrote: >> I agree that SubDAGs are an overly complex abstraction. I think what is >> needed/useful is a TaskGroup concept. On a high level I think you want >> this >> functionality: >> >> - Tasks can be added to a TaskGroup >> - You *can* have dependencies between Tasks in the same TaskGroup, but >> *cannot* have dependencies between a Task in a TaskGroup and either a >> Task in a different TaskGroup or a Task not in any group >> - You *can* have dependencies between a TaskGroup and either other >> TaskGroups or Tasks not in any group >> - The UI will by default render a TaskGroup as a single "object", but >> which you expand or zoom into in some way >> - You'd need some way to determine what the "status" of a TaskGroup was >> at least for UI display purposes >> >> Not sure if it would need to be a top level object with its own database >> table and model or just another attribute on tasks. I think you could >> build >> it in a way such that from the schedulers point of view a DAG with >> TaskGroups doesn't get treated any differently. So it really just becomes >> a >> shortcut for setting dependencies between sets of Tasks, and allows the UI >> to simplify the render of the DAG structure. >> >> Chris >> >> On Fri, Jun 12, 2020 at 12:12 PM Dan Davydov <ddavy...@twitter.com.invalid >> > >> wrote: >> >> > Agree with James (and think it's actually the more important issue to >> fix), >> > but I am still convinced Ash' idea is the right way forward (just it >> might >> > require a bit more work to deprecate than adding visual grouping in the >> > UI). >> > >> > There was a previous thread about this FYI with more context on why >> subdags >> > are bad and potential solutions: >> > https://www.mail-archive.com/dev@airflow.apache.org/msg01202.html . A >> > solution I outline there to Jame's problem is e.g. enabling the >> >> operator >> > for Airflow operators to work with DAGs as well. I see this being >> separate >> > from Ash' solution for DAG grouping in the UI but one of the two items >> > required to replace all existing subdag functionality. >> > >> > I've been working with subdags for 3 years and they are always a giant >> pain >> > to use. They are a constant source of user confusion and breakages >> during >> > upgrades. Would love to see them gone :). >> > >> > On Fri, Jun 12, 2020 at 11:11 AM James Coder <jcode...@gmail.com> >> wrote: >> > >> > > I'm not sure I totally agree it's just a UI concept. I use the subdag >> > > operator to simplify dependencies too. If you have a group of tasks >> that >> > > need to finish before another group of tasks start, using a subdag is >> a >> > > pretty quick way to set those dependencies and I think also make it >> > easier >> > > to follow the dag code. >> > > >> > > On Fri, Jun 12, 2020 at 9:53 AM Kyle Hamlin <hamlin...@gmail.com> >> wrote: >> > > >> > > > I second Ash’s grouping concept. >> > > > >> > > > On Fri, Jun 12, 2020 at 5:10 AM Ash Berlin-Taylor <a...@apache.org> >> > > wrote: >> > > > >> > > > > Question: >> > > > > >> > > > > Do we even need the SubDagOperator anymore? >> > > > > >> > > > > Would removing it entirely and just replacing it with a UI >> grouping >> > > > > concept be conceptually simpler, less to get wrong, and closer to >> > what >> > > > > users actually want to achieve with subdags? >> > > > > >> > > > > With your proposed change, tasks in subdags could start running in >> > > > > parallel (a good change) -- so should we not also just _enitrely_ >> > > remove >> > > > > the concept of a sub dag and replace it with something simpler. >> > > > > >> > > > > Problems with subdags (I think. I haven't used them extensively so >> > may >> > > > > be wrong on some of these): >> > > > > - They need their own dag_id, but it has(?) to be of the form >> > > > > `parent_dag_id.subdag_id`. >> > > > > - They need their own schedule_interval, but it has to match the >> > parent >> > > > dag >> > > > > - Sub dags can be paused on their own. (Does it make sense to do >> > this? >> > > > > Pausing just a sub dag would mean the sub dag would never >> execute, so >> > > > > the SubDagOperator would fail too. >> > > > > - You had to choose the executor to operator a subdag with -- >> always >> > a >> > > > > bit of a kludge. >> > > > > >> > > > > Thoughts? >> > > > > >> > > > > -ash >> > > > > >> > > > > On Jun 12 2020, at 12:01 pm, Ash Berlin-Taylor <a...@apache.org> >> > wrote: >> > > > > >> > > > > > Workon sub-dags is much needed, I'm excited to see how this >> > > progresses. >> > > > > > >> > > > > > >> > > > > >> - *Unpack SubDags during dag parsing*: This rewrites the >> > > > > *DagBag.bag_dag* >> > > > > >> method to unpack subdag while parsing, and it will give a flat >> > > > > >> structure at >> > > > > >> the task level >> > > > > > >> > > > > > The serialized_dag representation already does this I think. At >> > least >> > > > if >> > > > > > I've understood your idea here correctly. >> > > > > > >> > > > > > -ash >> > > > > > >> > > > > > >> > > > > > On Jun 12 2020, at 9:51 am, Xinbin Huang <bin.huan...@gmail.com >> > >> > > > wrote: >> > > > > > >> > > > > >> Hi everyone, >> > > > > >> >> > > > > >> Sending a message to everyone and collect feedback on the >> AIP-34 >> > on >> > > > > >> rewriting SubDagOperator. This was previously briefly >> mentioned in >> > > the >> > > > > >> discussion about what needs to be done for Airflow 2.0, and >> one of >> > > the >> > > > > >> ideas is to make SubDagOperator attach tasks back to the root >> DAG. >> > > > > >> >> > > > > >> This AIP-34 focuses on solving SubDagOperator related issues by >> > > > > reattaching >> > > > > >> all tasks back to the root dag while respecting dependencies >> > during >> > > > > >> parsing. The original grouping effect on the UI will be >> achieved >> > > > through >> > > > > >> grouping related tasks by metadata. >> > > > > >> >> > > > > >> This also makes the dag_factory function more reusable because >> you >> > > > don't >> > > > > >> need to have parent_dag_name and child_dag_name in the function >> > > > > signature >> > > > > >> anymore. >> > > > > >> >> > > > > >> Changes proposed: >> > > > > >> >> > > > > >> - *Unpack SubDags during dag parsing*: This rewrites the >> > > > > *DagBag.bag_dag* >> > > > > >> method to unpack subdag while parsing, and it will give a flat >> > > > > >> structure at >> > > > > >> the task level >> > > > > >> - *Simplify SubDagOperator*: The new SubDagOperator acts like a >> > > > > >> container and most of the original methods are removed. The >> > > > > >> signature is >> > > > > >> also changed to *subdag_factory *with *subdag_args *and >> > > > > *subdag_kwargs*. >> > > > > >> This is similar to the PythonOperator signature. >> > > > > >> - *Add a TaskGroup model and add current_group & parent_group >> > > > > attributes >> > > > > >> to BaseOperator*: This metadata is used to group tasks for >> > > > > >> rendering at >> > > > > >> UI level. It may potentially extend further to group arbitrary >> > > tasks >> > > > > >> outside the context of subdag to allow group-level operations >> > > (i.e. >> > > > > >> stop/trigger a group of task within the dag) >> > > > > >> - *Webserver UI for SubDag*: Proposed UI modification to allow >> > > > > >> (un)collapse a group of tasks for a flat structure to pair with >> > > the >> > > > > first >> > > > > >> change instead of the original hierarchical structure. >> > > > > >> >> > > > > >> >> > > > > >> Please see related documents and PRs for details: >> > > > > >> AIP: >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+Rewrite+SubDagOperator >> > > > > >> >> > > > > >> Original Issue: https://github.com/apache/airflow/issues/8078 >> > > > > >> Draft PR: https://github.com/apache/airflow/pull/9243 >> > > > > >> >> > > > > >> Please let me know if there are any aspects that you >> > agree/disagree >> > > > > >> with or >> > > > > >> need more clarification (especially the third change regarding >> > > > > TaskGroup). >> > > > > >> Any comments are welcome and I am looking forward to it! >> > > > > >> >> > > > > >> Cheers >> > > > > >> Bin >> > > > > >> >> > > > > >> > > > -- >> > > > Kyle Hamlin >> > > > >> > > >> > > >
