What's your ID i.e. if you haven't created an account yet, please create one at https://cwiki.apache.org/confluence/signup.action and send us your ID and we will add permissions.
Thanks. I'll edit the AIP. May I request permission to edit it? > My wiki user email is yuqian1...@gmail.com. On Fri, Aug 14, 2020 at 9:45 AM Yu Qian <yuqian1...@gmail.com> wrote: > Re, Xinbin. Thanks. I'll edit the AIP. May I request permission to edit it? > My wiki user email is yuqian1...@gmail.com. > > Re Gerard: yes the UI loads all the nodes as json from the web server at > once. However, it only adds the top level nodes and edges to the graph when > the Graph View page is first opened. And then adds the expanded nodes to > the graph as the user expands them. From what I've experienced with DAGs > containing around 400 tasks (not using TaskGroup or SubDagOperator), > opening the whole dag in Graph View usually takes 5 seconds. Less than 60ms > of that is taken by loading the data from webserver. The remaining 4.9s+ is > taken by javascript functions in dagre-d3.min.js such as createNodes, > createEdgeLabels, etc and by rendering the graph. With TaskGroup being used > to group tasks into a smaller number of top-level nodes, the amount of data > loaded from webserver will remain about the same compared to a flat dag of > the same size, but the number of nodes and edges needed to be plot on the > graph can be reduced significantly. So in theory this should speed up the > time it takes to open Graph View even without lazy-loading the data (I'll > experiment to find out). That said, if it comes to a point lazy-loading > helps, we can still implement it as an improvement. > > Re James: the Tree View looks as if all all the groups are fully expanded. > (because under the hood all the tasks are in a single DAG). I'm less > worried about Tree View at the moment because it already has a mechanism > for collapsing tasks by the dependency tree. That said, the Tree View can > definitely be improved too with TaskGroup. (e.g. collapse tasks in the same > TaskGroup when Tree View is first opened). > > For both suggestions, implementing them don't require fundamental changes > to the idea. I think we can have a basic working TaskGroup first, and then > improve it incrementally in several PRs as we get more feedback from the > community. What do you think? > > Qian > > > On Wed, Aug 12, 2020 at 9:15 AM James Coder <jcode...@gmail.com> wrote: > > > I agree this looks great, one question, how does the tree view look? > > > > James Coder > > > > > On Aug 11, 2020, at 6:48 PM, Gerard Casas Saez <gcasass...@twitter.com > .invalid> > > wrote: > > > > > > First of all, this is awesome!! > > > > > > Secondly, checking your UI code, seems you are loading all operators at > > > once. Wondering if we can load them as needed (aka load whenever we > click > > > the TaskGroup). Some of our DAGs are so large that take forever to load > > on > > > the Graph view, so worried about this still being an issue here. It may > > be > > > easily solvable by implementing lazy loading of the graph. Not sure how > > > easy to implement/add to the UI extension (and dont want to push for > > early > > > optimization as its the root of all evil). > > > Gerard Casas Saez > > > Twitter | Cortex | @casassaez <http://twitter.com/casassaez> > > > > > > > > >> On Tue, Aug 11, 2020 at 10:35 AM Xinbin Huang <bin.huan...@gmail.com> > > wrote: > > >> > > >> Hi Yu, > > >> > > >> Thank you so much for taking on this. I was fairly distracted > previously > > >> and I didn't have the time to update the proposal. In fact, after > > >> discussing with Ash, Kaxil and Daniel, the direction of this AIP has > > been > > >> changed to favor the concept of TaskGroup instead of rewriting > > >> SubDagOperator (though it may may sense to deprecate SubDag in a > future > > >> date.). > > >> > > >> Your PR is amazing and it has implemented the desire features. I think > > we > > >> can focus on your new PR instead. Do you mind updating the AIP based > on > > >> what you have done in your PR? > > >> > > >> Best, > > >> Bin > > >> > > >> > > >>> On Tue, Aug 11, 2020 at 7:11 AM Yu Qian <yuqian1...@gmail.com> > wrote: > > >>> > > >>> Hi, all, I've added the basic UI changes to my proposed > implementation > > of > > >>> TaskGroup as UI grouping concept: > > >>> https://github.com/apache/airflow/pull/10153 > > >>> > > >>> I think Chris had a pretty good specification of TaskGroup so i'm > > quoting > > >>> it here. The only thing I don't fully agree with is the restriction > > >>> "... **cannot* > > >>> have dependencies between a Task in a TaskGroup and either a* > > >>> * Task in a different TaskGroup or a Task not in any group*". I > think > > >>> this is over restrictive. Since TaskGroup is a UI concept, tasks can > > have > > >>> dependencies on tasks in other TaskGroup or not in any TaskGroup. In > my > > >> PR, > > >>> this is allowed. The graph edges will update accordingly when > > TaskGroups > > >>> are expanded/collapsed. TaskGroup is only helping to make the UI look > > >> less > > >>> crowded. Under the hood, everything is still a DAG of tasks and edges > > so > > >>> things work normally. Here's a screenshot > > >>> < > > >>> > > >> > > > https://raw.githubusercontent.com/yuqian90/airflow/gif_for_demo/airflow/www/static/screen-shot-short.gif > > >>>> > > >>> of the UI interaction. > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> * - Tasks can be added to a TaskGroup - You *can* have > dependencies > > >>> between Tasks in the same TaskGroup, but *cannot* have dependencies > > >>> between a Task in a TaskGroup and either a Task in a different > > >> TaskGroup > > >>> or a Task not in any group - You *can* have dependencies between a > > >>> TaskGroup and either other TaskGroups or Tasks not in any group - > > The > > >>> UI will by default render a TaskGroup as a single "object", but > which > > >> you > > >>> expand or zoom into in some way - You'd need some way to determine > > what > > >>> the "status" of a TaskGroup was at least for UI display purposes* > > >>> > > >>> > > >>> Regarding Jake's comment, I agree it's possible to implement the > > >> "retrying > > >>> tasks in a group" pattern he mentioned as an optional feature of > > >> TaskGroup > > >>> although that may go against having TaskGroup as a pure UI concept. > For > > >> the > > >>> motivating example Jake provided, I suggest implementing both > > >>> SubmitLongRunningJobTask and PollJobStatusSensor in a single > operator. > > It > > >>> can do something like BaseSensorOperator.execute() does in > "reschedule" > > >>> mode, i.e. it first executes some code to submit the long running job > > to > > >>> the external service, and store the state (e.g. in XCom). Then > > reschedule > > >>> itself. Subsequent runs then pokes for the completion state. > > >>> > > >>> > > >>> On Thu, Aug 6, 2020 at 2:08 AM Jacob Ferriero > > >> <jferri...@google.com.invalid > > >>>> > > >>> wrote: > > >>> > > >>>> I really like this idea of a TaskGroup container as I think this > will > > >> be > > >>>> much easier to use than SubDag. > > >>>> > > >>>> I'd like to propose an optional behavior for special retry mechanics > > >> via > > >>> a > > >>>> TaskGroup.retry_all property. > > >>>> This way I could use TaskGroup to replace my favorite use of SubDag > > for > > >>>> atomically retrying tasks of the pattern "act on external state then > > >>>> reschedule poll until desired state reached". > > >>>> > > >>>> Motivating use case I have for a SubDag is very simple two task > group > > >>>> [SubmitLongRunningJobTask >> PollJobStatusSensor]. > > >>>> I use SubDag is because it gives me an easy way to retry the > > >>> SubmitJobTask > > >>>> if something about the PollJobSensor fails. > > >>>> This pattern would be really nice for jobs that are expected to run > a > > >>> long > > >>>> time (because we can use sensor can use reschedule mode freeing up > > >> slots) > > >>>> but might fail for a retryable reason. > > >>>> However, using SubDag to meet this use case defeats the purpose > > because > > >>>> SubDag infamously > > >>>> < > > >>>> > > >>> > > >> > > > https://medium.com/@team_24989/fixing-subdagoperator-deadlock-in-airflow-6c64312ebb10 > > >>>>> > > >>>> blocks a "controller" slot for the entire duration. > > >>>> This may feel like a cyclic behavior but reality it is very common > for > > >> a > > >>>> single operator to submit job / wait til done. > > >>>> We could use this case refactor many operators (e.g. BQ, Dataproc, > > >>>> Dataflow) to be implemented as TaskGroup[SubmitTask >> PollTask] > with > > >> an > > >>>> optional reschedule mode if user knows that this job may take a long > > >>> time. > > >>>> > > >>>> I'd be happy to the development work on adding this specific retry > > >>> behavior > > >>>> to TaskGroup once the base concept is implemented if others in the > > >>>> community would find this a useful feature. > > >>>> > > >>>> Cheers, > > >>>> Jake > > >>>> > > >>>> On Tue, Aug 4, 2020 at 10:07 AM Jarek Potiuk < > > jarek.pot...@polidea.com > > >>> > > >>>> wrote: > > >>>> > > >>>>> All for it :) . I think we are getting closer to have regular > > >> planning > > >>>> and > > >>>>> making some structured approach to 2.0 and starting task force for > it > > >>>> soon, > > >>>>> so I think this should be perfectly fine to discuss and even start > > >>>>> implementing what's beyond as soon as we make sure that we are > > >>>> prioritizing > > >>>>> 2.0 work. > > >>>>> > > >>>>> J, > > >>>>> > > >>>>> > > >>>>> On Tue, Aug 4, 2020 at 12:09 PM Yu Qian <yuqian1...@gmail.com> > > >> wrote: > > >>>>> > > >>>>>> Hi Jarek, > > >>>>>> > > >>>>>> I agree we should not change the behaviour of the existing > > >>>> SubDagOperator > > >>>>>> till Airflow 2.1. Is it okay to continue the discussion about > > >>> TaskGroup > > >>>>> as > > >>>>>> a brand new concept/feature independent from the existing > > >>>> SubDagOperator? > > >>>>>> In other words, shall we add TaskGroup as a UI grouping concept > > >> like > > >>>> Ash > > >>>>>> suggested, and not touch SubDagOperator atl all. Whenever we are > > >>> ready > > >>>>> with > > >>>>>> TaskGroup, we then deprecate SubDagOperator in Airflow 2.1. > > >>>>>> > > >>>>>> I really like Ash's idea of simplifying the SubDagOperator idea > > >> into > > >>> a > > >>>>>> simple UI grouping concept. I think Xinbin's idea of "reattaching > > >> all > > >>>> the > > >>>>>> tasks to the root DAG" is the way to go. And I see James pointed > > >> out > > >>> we > > >>>>>> need some helper functions to simplify dependencies setting of > > >>>> TaskGroup. > > >>>>>> Xinbin put up a pretty elegant example in his PR > > >>>>>> <https://github.com/apache/airflow/pull/9243>. I think having > > >>>> TaskGroup > > >>>>> as > > >>>>>> a UI concept should be a relatively small change. We can simplify > > >>>>> Xinbin's > > >>>>>> PR further. So I put up this alternative proposal here: > > >>>>>> https://github.com/apache/airflow/pull/10153 > > >>>>>> > > >>>>>> I have not done any UI changes due to lack of experience with web > > >> UI. > > >>>> If > > >>>>>> anyone's interested, please take a look at the PR. > > >>>>>> > > >>>>>> Qian > > >>>>>> > > >>>>>> On Mon, Jun 22, 2020 at 5:15 AM Jarek Potiuk < > > >>> jarek.pot...@polidea.com > > >>>>> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> Similar point here to the other ideas that are popping up. Maybe > > >> we > > >>>>>> should > > >>>>>>> just focus on completing 2.0 and make all discussions about > > >> further > > >>>>>>> improvements to 2.1? While those are important discussions (and > > >> we > > >>>>> should > > >>>>>>> continue them in the near future !) I think at this point > > >> focusing > > >>>> on > > >>>>>>> delivering 2.0 in its current shape should be our focus now ? > > >>>>>>> > > >>>>>>> J. > > >>>>>>> > > >>>>>>> On Thu, Jun 18, 2020 at 6:35 PM Xinbin Huang < > > >>> bin.huan...@gmail.com> > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>>> Hi Daniel > > >>>>>>>> > > >>>>>>>> I agree that the TaskGroup should have the same API as a DAG > > >>> object > > >>>>>>> related > > >>>>>>>> to task dependencies, but it will not have anything related to > > >>>> actual > > >>>>>>>> execution or scheduling. > > >>>>>>>> I will update the AIP according to this over the weekend. > > >>>>>>>> > > >>>>>>>>> We could even make a “DAGTemplate” object s.t. when you > > >> import > > >>>> the > > >>>>>>> object > > >>>>>>>> you can import it with parameters to determine the shape of the > > >>>> DAG. > > >>>>>>>> > > >>>>>>>> Can you elaborate a bit more on this? Does it serve a similar > > >>>> purpose > > >>>>>> as > > >>>>>>> a > > >>>>>>>> DAG factory function? > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Thu, Jun 18, 2020 at 9:13 AM Daniel Imberman < > > >>>>>>> daniel.imber...@gmail.com > > >>>>>>>>> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> Hi Bin, > > >>>>>>>>> > > >>>>>>>>> Why not give the TaskGroup the same API as a DAG object (e.g. > > >>> the > > >>>>>>> bitwise > > >>>>>>>>> operator fro task dependencies). We could even make a > > >>>> “DAGTemplate” > > >>>>>>>> object > > >>>>>>>>> s.t. when you import the object you can import it with > > >>> parameters > > >>>>> to > > >>>>>>>>> determine the shape of the DAG. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> On Wed, Jun 17, 2020 at 8:54 PM, Xinbin Huang < > > >>>>> bin.huan...@gmail.com > > >>>>>>> > > >>>>>>>>> wrote: > > >>>>>>>>> The TaskGroup will not take schedule interval as a parameter > > >>>>> itself, > > >>>>>>> and > > >>>>>>>> it > > >>>>>>>>> depends on the DAG where it attaches to. In my opinion, the > > >>>>> TaskGroup > > >>>>>>>> will > > >>>>>>>>> only contain a group of tasks with interdependencies, and the > > >>>>>> TaskGroup > > >>>>>>>>> behaves like a task. It doesn't contain any > > >>> execution/scheduling > > >>>>>> logic > > >>>>>>>>> (i.e. schedule_interval, concurrency, max_active_runs etc.) > > >>> like > > >>>> a > > >>>>>> DAG > > >>>>>>>>> does. > > >>>>>>>>> > > >>>>>>>>>> For example, there is the scenario that the schedule > > >> interval > > >>>> of > > >>>>>> DAG > > >>>>>>> is > > >>>>>>>>> 1 hour and the schedule interval of TaskGroup is 20 min. > > >>>>>>>>> > > >>>>>>>>> I am curious why you ask this. Is this a use case that you > > >> want > > >>>> to > > >>>>>>>> achieve? > > >>>>>>>>> > > >>>>>>>>> Bin > > >>>>>>>>> > > >>>>>>>>> On Wed, Jun 17, 2020 at 7:59 PM 蒋晓峰 < > > >> thanosxnicho...@gmail.com > > >>>> > > >>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> Hi Bin, > > >>>>>>>>>> Using TaskGroup, Is the schedule interval of TaskGroup the > > >>> same > > >>>>> as > > >>>>>>> the > > >>>>>>>>>> parent DAG? My main concern is whether the schedule > > >> interval > > >>> of > > >>>>>>>> TaskGroup > > >>>>>>>>>> could be different with that of the DAG? For example, there > > >>> is > > >>>>> the > > >>>>>>>>> scenario > > >>>>>>>>>> that the schedule interval of DAG is 1 hour and the > > >> schedule > > >>>>>> interval > > >>>>>>>> of > > >>>>>>>>>> TaskGroup is 20 min. > > >>>>>>>>>> > > >>>>>>>>>> Cheers, > > >>>>>>>>>> Nicholas > > >>>>>>>>>> > > >>>>>>>>>> On Thu, Jun 18, 2020 at 10:30 AM Xinbin Huang < > > >>>>>> bin.huan...@gmail.com > > >>>>>>>> > > >>>>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> Hi Nicholas, > > >>>>>>>>>>> > > >>>>>>>>>>> I am not sure about the old behavior of SubDagOperator, > > >>> maybe > > >>>>> it > > >>>>>>> will > > >>>>>>>>>> throw > > >>>>>>>>>>> an error? But in the original proposal, the subdag's > > >>>>>>>> schedule_interval > > >>>>>>>>>> will > > >>>>>>>>>>> be ignored. Or if we decide to use TaskGroup to replace > > >>>> SubDag, > > >>>>>>> there > > >>>>>>>>>> will > > >>>>>>>>>>> be no subdag schedule_interval. > > >>>>>>>>>>> > > >>>>>>>>>>> Bin > > >>>>>>>>>>> > > >>>>>>>>>>> On Wed, Jun 17, 2020 at 6:21 PM 蒋晓峰 < > > >>>> thanosxnicho...@gmail.com > > >>>>>> > > >>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>>> Hi Bin, > > >>>>>>>>>>>> Thanks for your good proposal. I was confused whether > > >> the > > >>>>>>> schedule > > >>>>>>>>>>>> interval of SubDAG is different from that of the parent > > >>>> DAG? > > >>>>> I > > >>>>>>> have > > >>>>>>>>>>>> discussed with Jiajie Zhong about the schedule interval > > >>> of > > >>>>>>> SubDAG. > > >>>>>>>> If > > >>>>>>>>>> the > > >>>>>>>>>>>> SubDagOperator has a different schedule interval, what > > >>> will > > >>>>>>> happen > > >>>>>>>>> for > > >>>>>>>>>>> the > > >>>>>>>>>>>> scheduler to schedule the parent DAG? > > >>>>>>>>>>>> > > >>>>>>>>>>>> Regards, > > >>>>>>>>>>>> Nicholas Jiang > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Thu, Jun 18, 2020 at 8:04 AM Xinbin Huang < > > >>>>>>>> bin.huan...@gmail.com> > > >>>>>>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>>> Thank you, Max, Kaxil, and everyone's feedback! > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> I have rethought about the concept of subdag and task > > >>>>>> groups. I > > >>>>>>>>> think > > >>>>>>>>>>> the > > >>>>>>>>>>>>> better way to approach this is to entirely remove > > >>> subdag > > >>>>> and > > >>>>>>>>>> introduce > > >>>>>>>>>>>> the > > >>>>>>>>>>>>> concept of TaskGroup, which is a container of tasks > > >>> along > > >>>>>> with > > >>>>>>>>> their > > >>>>>>>>>>>>> dependencies *without execution/scheduling logic as a > > >>>> DAG*. > > >>>>>> The > > >>>>>>>>> only > > >>>>>>>>>>>>> purpose of it is to group a list of tasks, but you > > >>> still > > >>>>> need > > >>>>>>> to > > >>>>>>>>> add > > >>>>>>>>>> it > > >>>>>>>>>>>> to > > >>>>>>>>>>>>> a DAG for execution. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Here is a small code snippet. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> ``` > > >>>>>>>>>>>>> class TaskGroup: > > >>>>>>>>>>>>> """ > > >>>>>>>>>>>>> A TaskGroup contains a group of tasks. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> If default_args is missing, it will take default args > > >>>> from > > >>>>>> the > > >>>>>>>>>> DAG. > > >>>>>>>>>>>>> """ > > >>>>>>>>>>>>> def __init__(self, group_id, default_args): > > >>>>>>>>>>>>> pass > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> """ > > >>>>>>>>>>>>> You can add tasks to a task group similar to adding > > >>> tasks > > >>>>> to > > >>>>>> a > > >>>>>>>> DAG > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> This can be declared in a separate file from the dag > > >>> file > > >>>>>>>>>>>>> """ > > >>>>>>>>>>>>> download_group = TaskGroup(group_id='download', > > >>>>>>>>>>>> default_args=default_args) > > >>>>>>>>>>>>> download_group.add_task(task1) > > >>>>>>>>>>>>> task2.dag = download_group > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> with download_group: > > >>>>>>>>>>>>> task3 = DummyOperator(task_id='task3') > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> [task, task2] >> task3 > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> """Add it to a DAG for execution""" > > >>>>>>>>>>>>> with DAG(dag_id='start_download_dag', > > >>>>>>> default_args=default_args, > > >>>>>>>>>>>>> schedule_interval='@daily', ...) as dag: > > >>>>>>>>>>>>> start = DummyOperator(task_id='start') > > >>>>>>>>>>>>> start >> download_group > > >>>>>>>>>>>>> # this is equivalent to > > >>>>>>>>>>>>> # start >> [task, task2] >> task3 > > >>>>>>>>>>>>> ``` > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> With this, we can still reuse a group of tasks and > > >> set > > >>>>>>>> dependencies > > >>>>>>>>>>>> between > > >>>>>>>>>>>>> them; it avoids the boilerplate code from using > > >>>>>> SubDagOperator, > > >>>>>>>> and > > >>>>>>>>>> we > > >>>>>>>>>>>> can > > >>>>>>>>>>>>> declare dependencies as `task >> task_group >> task`. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> User migration wise, we can introduce it before > > >> Airflow > > >>>> 2.0 > > >>>>>> and > > >>>>>>>>> allow > > >>>>>>>>>>>>> gradual transition. Then we can decide if we still > > >> want > > >>>> to > > >>>>>> keep > > >>>>>>>> the > > >>>>>>>>>>>>> SubDagOperator or simply remove it. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Any thoughts? > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Cheers, > > >>>>>>>>>>>>> Bin > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On Wed, Jun 17, 2020 at 7:37 AM Maxime Beauchemin < > > >>>>>>>>>>>>> maximebeauche...@gmail.com> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> +1, proposal looks good. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> The original intention was really to have tasks > > >>> groups > > >>>>> and > > >>>>>> a > > >>>>>>>>>>>> zoom-in/out > > >>>>>>>>>>>>> in > > >>>>>>>>>>>>>> the UI. The original reasoning was to reuse the DAG > > >>>>> object > > >>>>>>>> since > > >>>>>>>>> it > > >>>>>>>>>>> is > > >>>>>>>>>>>> a > > >>>>>>>>>>>>>> group of tasks, but as highlighted here it does > > >>> create > > >>>>>>>> underlying > > >>>>>>>>>>>>>> confusions since a DAG is much more than just a > > >> group > > >>>> of > > >>>>>>> tasks. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Max > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> On Mon, Jun 15, 2020 at 2:43 AM Poornima Joshi < > > >>>>>>>>>>>>> joshipoornim...@gmail.com> > > >>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Thank you for your email. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On Sat, Jun 13, 2020 at 12:18 AM Xinbin Huang < > > >>>>>>>>>>> bin.huan...@gmail.com > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> - *Unpack SubDags during dag parsing*: This > > >>>>>> rewrites > > >>>>>>>> the > > >>>>>>>>>>>>>>>> *DagBag.bag_dag* > > >>>>>>>>>>>>>>>>>> method to unpack subdag while parsing, and > > >> it > > >>>>> will > > >>>>>>>> give a > > >>>>>>>>>>>> flat > > >>>>>>>>>>>>>>>>>> structure at > > >>>>>>>>>>>>>>>>>> the task level > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> The serialized_dag representation already > > >> does > > >>>>> this I > > >>>>>>>>> think. > > >>>>>>>>>> At > > >>>>>>>>>>>>> least > > >>>>>>>>>>>>>>> if > > >>>>>>>>>>>>>>>>> I've understood your idea here correctly. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> I am not sure about serialized_dag > > >>> representation, > > >>>>> but > > >>>>>> at > > >>>>>>>>> least > > >>>>>>>>>>> it > > >>>>>>>>>>>>> will > > >>>>>>>>>>>>>>>> still keep the subdag entry in the DAG table? > > >> In > > >>> my > > >>>>>>>> proposal > > >>>>>>>>> as > > >>>>>>>>>>>> also > > >>>>>>>>>>>>> in > > >>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>> draft PR, the idea is to *extract the tasks > > >> from > > >>>> the > > >>>>>>> subdag > > >>>>>>>>> and > > >>>>>>>>>>> add > > >>>>>>>>>>>>>> them > > >>>>>>>>>>>>>>>> back to the root_dag. *So the runtime DAG graph > > >>>> will > > >>>>>> look > > >>>>>>>>>> exactly > > >>>>>>>>>>>> the > > >>>>>>>>>>>>>>>> same as without subdag but with metadata > > >> attached > > >>>> to > > >>>>>>> those > > >>>>>>>>>>>> sections. > > >>>>>>>>>>>>>>> These > > >>>>>>>>>>>>>>>> metadata will be later on used to render in the > > >>> UI. > > >>>>> So > > >>>>>>>> after > > >>>>>>>>>>>> parsing > > >>>>>>>>>>>>> ( > > >>>>>>>>>>>>>>>> *DagBag.process_file()*), it will just output > > >> the > > >>>>>>> *root_dag > > >>>>>>>>>>>> *instead > > >>>>>>>>>>>>> of > > >>>>>>>>>>>>>>> *root_dag + > > >>>>>>>>>>>>>>>> subdag + subdag + nested subdag* etc. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> - e.g. section-1-* will have metadata > > >>>>>>>>>> current_group=section-1, > > >>>>>>>>>>>>>>>> parent_group=<the-root-dag-id> (welcome for > > >>> naming > > >>>>>>>>>>> suggestions), > > >>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>> reason for parent_group is that we can have > > >>> nested > > >>>>>> group > > >>>>>>>> and > > >>>>>>>>>>>> still > > >>>>>>>>>>>>>> be > > >>>>>>>>>>>>>>>> able to capture the dependency. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Runtime DAG: > > >>>>>>>>>>>>>>>> [image: image.png] > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> While at the UI, what we see would be something > > >>>> like > > >>>>>> this > > >>>>>>>> by > > >>>>>>>>>>>>> utilizing > > >>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>> metadata, and then we can expand or zoom into > > >> in > > >>>> some > > >>>>>>> way. > > >>>>>>>>>>>>>>>> [image: image.png] > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> The benefits I can see is that: > > >>>>>>>>>>>>>>>> 1. We don't need to deal with the extra > > >>> complexity > > >>>> of > > >>>>>>>> SubDag > > >>>>>>>>>> for > > >>>>>>>>>>>>>>> execution > > >>>>>>>>>>>>>>>> and scheduling. It will be the same as not > > >> using > > >>>>>> SubDag. > > >>>>>>>>>>>>>>>> 2. Still have the benefits of modularized and > > >>>>> reusable > > >>>>>>> dag > > >>>>>>>>> code > > >>>>>>>>>>> and > > >>>>>>>>>>>>>>>> declare dependencies between them. And with the > > >>> new > > >>>>>>>>>>> SubDagOperator > > >>>>>>>>>>>>> (see > > >>>>>>>>>>>>>>> AIP > > >>>>>>>>>>>>>>>> or draft PR), we can use the same dag_factory > > >>>>> function > > >>>>>>> for > > >>>>>>>>>>>>> generating 1 > > >>>>>>>>>>>>>>>> dag, a lot of dynamic dags, or used for SubDag > > >>> (in > > >>>>> this > > >>>>>>>> case, > > >>>>>>>>>> it > > >>>>>>>>>>>> will > > >>>>>>>>>>>>>>> just > > >>>>>>>>>>>>>>>> extract all underlying tasks and append to the > > >>> root > > >>>>>> dag). > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> - Then it gets to the idea of replacing subdag > > >>>> with a > > >>>>>>>>>> simpler > > >>>>>>>>>>>>>> concept > > >>>>>>>>>>>>>>>> by Ash: the proposed change basically drains > > >> out > > >>>> the > > >>>>>>>>>> contents > > >>>>>>>>>>>> of > > >>>>>>>>>>>>> a > > >>>>>>>>>>>>>>> SubDag > > >>>>>>>>>>>>>>>> and becomes more like > > >>>>>>>>>>>> ExtractSubdagTasksAndAppendToRootdagOperator > > >>>>>>>>>>>>>>> (forgive > > >>>>>>>>>>>>>>>> me about the crazy name..). In this case, it is > > >>>> still > > >>>>>>>>>>> necessary > > >>>>>>>>>>>> to > > >>>>>>>>>>>>>>> keep the > > >>>>>>>>>>>>>>>> concept of subdag as it is nothing more than a > > >>>> name? > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> That's why the TaskGroup idea comes up. Thanks > > >>>> Chris > > >>>>>>> Palmer > > >>>>>>>>> for > > >>>>>>>>>>>>> helping > > >>>>>>>>>>>>>>>> conceptualize the functionality of TaskGroup, I > > >>>> will > > >>>>>> just > > >>>>>>>>> paste > > >>>>>>>>>>> it > > >>>>>>>>>>>>>> here. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> - Tasks can be added to a TaskGroup > > >>>>>>>>>>>>>>>>> - You *can* have dependencies between Tasks > > >> in > > >>>> the > > >>>>>> same > > >>>>>>>>>>>> TaskGroup, > > >>>>>>>>>>>>>> but > > >>>>>>>>>>>>>>>>> *cannot* have dependencies between a Task in > > >> a > > >>>>>>> TaskGroup > > >>>>>>>>>> and > > >>>>>>>>>>>>>> either a > > >>>>>>>>>>>>>>>>> Task in a different TaskGroup or a Task not > > >> in > > >>>> any > > >>>>>>> group > > >>>>>>>>>>>>>>>>> - You *can* have dependencies between a > > >>> TaskGroup > > >>>>> and > > >>>>>>>>>> either > > >>>>>>>>>>>>> other > > >>>>>>>>>>>>>>>>> TaskGroups or Tasks not in any group > > >>>>>>>>>>>>>>>>> - The UI will by default render a TaskGroup > > >> as > > >>> a > > >>>>>> single > > >>>>>>>>>>>> "object", > > >>>>>>>>>>>>>> but > > >>>>>>>>>>>>>>>>> which you expand or zoom into in some way > > >>>>>>>>>>>>>>>>> - You'd need some way to determine what the > > >>>>> "status" > > >>>>>>> of a > > >>>>>>>>>>>>> TaskGroup > > >>>>>>>>>>>>>>> was > > >>>>>>>>>>>>>>>>> at least for UI display purposes > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> I agree with Chris: > > >>>>>>>>>>>>>>>> - From the backend's view (scheduler & > > >>> executor), I > > >>>>>> think > > >>>>>>>>>>> TaskGroup > > >>>>>>>>>>>>>>> should > > >>>>>>>>>>>>>>>> be ignored during execution. (unless we decide > > >> to > > >>>>>>> implement > > >>>>>>>>>> some > > >>>>>>>>>>>>>> metadata > > >>>>>>>>>>>>>>>> operations that allows start/stop a group of > > >>> tasks > > >>>>>> etc.) > > >>>>>>>>>>>>>>>> - From the UI's View, it should be able to pick > > >>> up > > >>>>> the > > >>>>>>>>>> individual > > >>>>>>>>>>>>>> tasks' > > >>>>>>>>>>>>>>>> status and then determine the TaskGroup's > > >> status > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Bin > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> On Fri, Jun 12, 2020 at 10:28 AM Daniel > > >> Imberman > > >>> < > > >>>>>>>>>>>>>>>> daniel.imber...@gmail.com> wrote: > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> I hadn’t thought about using the `>>` operator > > >>> to > > >>>>> tie > > >>>>>>> dags > > >>>>>>>>>>>> together > > >>>>>>>>>>>>>> but > > >>>>>>>>>>>>>>> I > > >>>>>>>>>>>>>>>>> think that sounds pretty great! I wonder if we > > >>>> could > > >>>>>>>>>> essentially > > >>>>>>>>>>>>> write > > >>>>>>>>>>>>>>> in > > >>>>>>>>>>>>>>>>> the ability to set dependencies to all > > >>>> starter-tasks > > >>>>>> for > > >>>>>>>>> that > > >>>>>>>>>>> DAG. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> I’m personally ok with SubDag being a mostly > > >> UI > > >>>>>> concept. > > >>>>>>>> It > > >>>>>>>>>>>> doesn’t > > >>>>>>>>>>>>>> need > > >>>>>>>>>>>>>>>>> to execute separately, you’re just adding more > > >>>> tasks > > >>>>>> to > > >>>>>>>> the > > >>>>>>>>>>> queue > > >>>>>>>>>>>>> that > > >>>>>>>>>>>>>>> will > > >>>>>>>>>>>>>>>>> be executed when there are resources > > >> available. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> via Newton Mail [ > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.14.6&source=email_footer_2 > > >>>>>>>>>>>>>>>>> ] > > >>>>>>>>>>>>>>>>> On Fri, Jun 12, 2020 at 9:45 AM, Chris Palmer > > >> < > > >>>>>>>>>>> ch...@crpalmer.com > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>> I agree that SubDAGs are an overly complex > > >>>>>> abstraction. > > >>>>>>> I > > >>>>>>>>>> think > > >>>>>>>>>>>> what > > >>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>>>> needed/useful is a TaskGroup concept. On a > > >> high > > >>>>> level > > >>>>>> I > > >>>>>>>>> think > > >>>>>>>>>>> you > > >>>>>>>>>>>>> want > > >>>>>>>>>>>>>>>>> this > > >>>>>>>>>>>>>>>>> functionality: > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> - Tasks can be added to a TaskGroup > > >>>>>>>>>>>>>>>>> - You *can* have dependencies between Tasks in > > >>> the > > >>>>>> same > > >>>>>>>>>>> TaskGroup, > > >>>>>>>>>>>>> but > > >>>>>>>>>>>>>>>>> *cannot* have dependencies between a Task in a > > >>>>>> TaskGroup > > >>>>>>>> and > > >>>>>>>>>>>> either > > >>>>>>>>>>>>> a > > >>>>>>>>>>>>>>>>> Task in a different TaskGroup or a Task not in > > >>> any > > >>>>>> group > > >>>>>>>>>>>>>>>>> - You *can* have dependencies between a > > >>> TaskGroup > > >>>>> and > > >>>>>>>> either > > >>>>>>>>>>> other > > >>>>>>>>>>>>>>>>> TaskGroups or Tasks not in any group > > >>>>>>>>>>>>>>>>> - The UI will by default render a TaskGroup > > >> as a > > >>>>>> single > > >>>>>>>>>>> "object", > > >>>>>>>>>>>>> but > > >>>>>>>>>>>>>>>>> which you expand or zoom into in some way > > >>>>>>>>>>>>>>>>> - You'd need some way to determine what the > > >>>> "status" > > >>>>>> of > > >>>>>>> a > > >>>>>>>>>>>> TaskGroup > > >>>>>>>>>>>>>> was > > >>>>>>>>>>>>>>>>> at least for UI display purposes > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Not sure if it would need to be a top level > > >>> object > > >>>>>> with > > >>>>>>>> its > > >>>>>>>>>> own > > >>>>>>>>>>>>>> database > > >>>>>>>>>>>>>>>>> table and model or just another attribute on > > >>>> tasks. > > >>>>> I > > >>>>>>>> think > > >>>>>>>>>> you > > >>>>>>>>>>>>> could > > >>>>>>>>>>>>>>>>> build > > >>>>>>>>>>>>>>>>> it in a way such that from the schedulers > > >> point > > >>> of > > >>>>>> view > > >>>>>>> a > > >>>>>>>>> DAG > > >>>>>>>>>>> with > > >>>>>>>>>>>>>>>>> TaskGroups doesn't get treated any > > >> differently. > > >>> So > > >>>>> it > > >>>>>>>> really > > >>>>>>>>>>> just > > >>>>>>>>>>>>>>> becomes > > >>>>>>>>>>>>>>>>> a > > >>>>>>>>>>>>>>>>> shortcut for setting dependencies between sets > > >>> of > > >>>>>> Tasks, > > >>>>>>>> and > > >>>>>>>>>>>> allows > > >>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>> UI > > >>>>>>>>>>>>>>>>> to simplify the render of the DAG structure. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Chris > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> On Fri, Jun 12, 2020 at 12:12 PM Dan Davydov > > >>>>>>>>>>>>>>> <ddavy...@twitter.com.invalid > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Agree with James (and think it's actually > > >> the > > >>>> more > > >>>>>>>>> important > > >>>>>>>>>>>> issue > > >>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>> fix), > > >>>>>>>>>>>>>>>>>> but I am still convinced Ash' idea is the > > >>> right > > >>>>> way > > >>>>>>>>> forward > > >>>>>>>>>>>> (just > > >>>>>>>>>>>>> it > > >>>>>>>>>>>>>>>>> might > > >>>>>>>>>>>>>>>>>> require a bit more work to deprecate than > > >>> adding > > >>>>>>> visual > > >>>>>>>>>>> grouping > > >>>>>>>>>>>>> in > > >>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>> UI). > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> There was a previous thread about this FYI > > >>> with > > >>>>> more > > >>>>>>>>> context > > >>>>>>>>>>> on > > >>>>>>>>>>>>> why > > >>>>>>>>>>>>>>>>> subdags > > >>>>>>>>>>>>>>>>>> are bad and potential solutions: > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>> > > >> https://www.mail-archive.com/dev@airflow.apache.org/msg01202.html > > >>>>>>>>>>>>>> . A > > >>>>>>>>>>>>>>>>>> solution I outline there to Jame's problem > > >> is > > >>>> e.g. > > >>>>>>>>> enabling > > >>>>>>>>>>> the > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> operator > > >>>>>>>>>>>>>>>>>> for Airflow operators to work with DAGs as > > >>>> well. I > > >>>>>> see > > >>>>>>>>> this > > >>>>>>>>>>>> being > > >>>>>>>>>>>>>>>>> separate > > >>>>>>>>>>>>>>>>>> from Ash' solution for DAG grouping in the > > >> UI > > >>>> but > > >>>>>> one > > >>>>>>> of > > >>>>>>>>> the > > >>>>>>>>>>> two > > >>>>>>>>>>>>>> items > > >>>>>>>>>>>>>>>>>> required to replace all existing subdag > > >>>>>> functionality. > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> I've been working with subdags for 3 years > > >> and > > >>>>> they > > >>>>>>> are > > >>>>>>>>>>> always a > > >>>>>>>>>>>>>> giant > > >>>>>>>>>>>>>>>>> pain > > >>>>>>>>>>>>>>>>>> to use. They are a constant source of user > > >>>>> confusion > > >>>>>>> and > > >>>>>>>>>>>> breakages > > >>>>>>>>>>>>>>>>> during > > >>>>>>>>>>>>>>>>>> upgrades. Would love to see them gone :). > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> On Fri, Jun 12, 2020 at 11:11 AM James > > >> Coder < > > >>>>>>>>>>>> jcode...@gmail.com> > > >>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> I'm not sure I totally agree it's just a > > >> UI > > >>>>>>> concept. I > > >>>>>>>>> use > > >>>>>>>>>>> the > > >>>>>>>>>>>>>>> subdag > > >>>>>>>>>>>>>>>>>>> operator to simplify dependencies too. If > > >>> you > > >>>>>> have a > > >>>>>>>>> group > > >>>>>>>>>>> of > > >>>>>>>>>>>>>> tasks > > >>>>>>>>>>>>>>>>> that > > >>>>>>>>>>>>>>>>>>> need to finish before another group of > > >> tasks > > >>>>>> start, > > >>>>>>>>> using > > >>>>>>>>>> a > > >>>>>>>>>>>>> subdag > > >>>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>>>> a > > >>>>>>>>>>>>>>>>>>> pretty quick way to set those dependencies > > >>>> and I > > >>>>>>> think > > >>>>>>>>>> also > > >>>>>>>>>>>> make > > >>>>>>>>>>>>>> it > > >>>>>>>>>>>>>>>>>> easier > > >>>>>>>>>>>>>>>>>>> to follow the dag code. > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> On Fri, Jun 12, 2020 at 9:53 AM Kyle > > >> Hamlin > > >>> < > > >>>>>>>>>>>>> hamlin...@gmail.com> > > >>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> I second Ash’s grouping concept. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> On Fri, Jun 12, 2020 at 5:10 AM Ash > > >>>>>> Berlin-Taylor > > >>>>>>> < > > >>>>>>>>>>>>>> a...@apache.org > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Question: > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Do we even need the SubDagOperator > > >>>> anymore? > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Would removing it entirely and just > > >>>>> replacing > > >>>>>> it > > >>>>>>>>> with > > >>>>>>>>>> a > > >>>>>>>>>>> UI > > >>>>>>>>>>>>>>>>> grouping > > >>>>>>>>>>>>>>>>>>>>> concept be conceptually simpler, less > > >> to > > >>>> get > > >>>>>>>> wrong, > > >>>>>>>>>> and > > >>>>>>>>>>>>> closer > > >>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>>> what > > >>>>>>>>>>>>>>>>>>>>> users actually want to achieve with > > >>>> subdags? > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> With your proposed change, tasks in > > >>>> subdags > > >>>>>>> could > > >>>>>>>>>> start > > >>>>>>>>>>>>>> running > > >>>>>>>>>>>>>>> in > > >>>>>>>>>>>>>>>>>>>>> parallel (a good change) -- so should > > >> we > > >>>> not > > >>>>>>> also > > >>>>>>>>> just > > >>>>>>>>>>>>>>> _enitrely_ > > >>>>>>>>>>>>>>>>>>> remove > > >>>>>>>>>>>>>>>>>>>>> the concept of a sub dag and replace > > >> it > > >>>> with > > >>>>>>>>> something > > >>>>>>>>>>>>>> simpler. > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Problems with subdags (I think. I > > >>> haven't > > >>>>> used > > >>>>>>>> them > > >>>>>>>>>>>>>> extensively > > >>>>>>>>>>>>>>> so > > >>>>>>>>>>>>>>>>>> may > > >>>>>>>>>>>>>>>>>>>>> be wrong on some of these): > > >>>>>>>>>>>>>>>>>>>>> - They need their own dag_id, but it > > >>>> has(?) > > >>>>> to > > >>>>>>> be > > >>>>>>>> of > > >>>>>>>>>> the > > >>>>>>>>>>>>> form > > >>>>>>>>>>>>>>>>>>>>> `parent_dag_id.subdag_id`. > > >>>>>>>>>>>>>>>>>>>>> - They need their own > > >> schedule_interval, > > >>>> but > > >>>>>> it > > >>>>>>>> has > > >>>>>>>>> to > > >>>>>>>>>>>> match > > >>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>> parent > > >>>>>>>>>>>>>>>>>>>> dag > > >>>>>>>>>>>>>>>>>>>>> - Sub dags can be paused on their own. > > >>>> (Does > > >>>>>> it > > >>>>>>>> make > > >>>>>>>>>>> sense > > >>>>>>>>>>>>> to > > >>>>>>>>>>>>>> do > > >>>>>>>>>>>>>>>>>> this? > > >>>>>>>>>>>>>>>>>>>>> Pausing just a sub dag would mean the > > >>> sub > > >>>>> dag > > >>>>>>>> would > > >>>>>>>>>>> never > > >>>>>>>>>>>>>>>>> execute, so > > >>>>>>>>>>>>>>>>>>>>> the SubDagOperator would fail too. > > >>>>>>>>>>>>>>>>>>>>> - You had to choose the executor to > > >>>>> operator a > > >>>>>>>>> subdag > > >>>>>>>>>>> with > > >>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>>>> always > > >>>>>>>>>>>>>>>>>> a > > >>>>>>>>>>>>>>>>>>>>> bit of a kludge. > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Thoughts? > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> -ash > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> On Jun 12 2020, at 12:01 pm, Ash > > >>>>>> Berlin-Taylor < > > >>>>>>>>>>>>>> a...@apache.org> > > >>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> Workon sub-dags is much needed, I'm > > >>>>> excited > > >>>>>> to > > >>>>>>>> see > > >>>>>>>>>> how > > >>>>>>>>>>>>> this > > >>>>>>>>>>>>>>>>>>> progresses. > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> - *Unpack SubDags during dag > > >>> parsing*: > > >>>>> This > > >>>>>>>>>> rewrites > > >>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>> *DagBag.bag_dag* > > >>>>>>>>>>>>>>>>>>>>>>> method to unpack subdag while > > >>> parsing, > > >>>>> and > > >>>>>> it > > >>>>>>>>> will > > >>>>>>>>>>>> give a > > >>>>>>>>>>>>>>> flat > > >>>>>>>>>>>>>>>>>>>>>>> structure at > > >>>>>>>>>>>>>>>>>>>>>>> the task level > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> The serialized_dag representation > > >>>> already > > >>>>>> does > > >>>>>>>>> this > > >>>>>>>>>> I > > >>>>>>>>>>>>> think. > > >>>>>>>>>>>>>>> At > > >>>>>>>>>>>>>>>>>> least > > >>>>>>>>>>>>>>>>>>>> if > > >>>>>>>>>>>>>>>>>>>>>> I've understood your idea here > > >>>> correctly. > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> -ash > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> On Jun 12 2020, at 9:51 am, Xinbin > > >>>> Huang < > > >>>>>>>>>>>>>>> bin.huan...@gmail.com > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> Hi everyone, > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> Sending a message to everyone and > > >>>> collect > > >>>>>>>>> feedback > > >>>>>>>>>> on > > >>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>> AIP-34 > > >>>>>>>>>>>>>>>>>> on > > >>>>>>>>>>>>>>>>>>>>>>> rewriting SubDagOperator. This was > > >>>>>> previously > > >>>>>>>>>> briefly > > >>>>>>>>>>>>>>>>> mentioned in > > >>>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>>>> discussion about what needs to be > > >>> done > > >>>>> for > > >>>>>>>>> Airflow > > >>>>>>>>>>> 2.0, > > >>>>>>>>>>>>> and > > >>>>>>>>>>>>>>>>> one of > > >>>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>>>> ideas is to make SubDagOperator > > >>> attach > > >>>>>> tasks > > >>>>>>>> back > > >>>>>>>>>> to > > >>>>>>>>>>>> the > > >>>>>>>>>>>>>> root > > >>>>>>>>>>>>>>>>> DAG. > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> This AIP-34 focuses on solving > > >>>>>> SubDagOperator > > >>>>>>>>>> related > > >>>>>>>>>>>>>> issues > > >>>>>>>>>>>>>>> by > > >>>>>>>>>>>>>>>>>>>>> reattaching > > >>>>>>>>>>>>>>>>>>>>>>> all tasks back to the root dag > > >> while > > >>>>>>> respecting > > >>>>>>>>>>>>>> dependencies > > >>>>>>>>>>>>>>>>>> during > > >>>>>>>>>>>>>>>>>>>>>>> parsing. The original grouping > > >> effect > > >>>> on > > >>>>>> the > > >>>>>>> UI > > >>>>>>>>>> will > > >>>>>>>>>>> be > > >>>>>>>>>>>>>>>>> achieved > > >>>>>>>>>>>>>>>>>>>> through > > >>>>>>>>>>>>>>>>>>>>>>> grouping related tasks by metadata. > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> This also makes the dag_factory > > >>>> function > > >>>>>> more > > >>>>>>>>>>> reusable > > >>>>>>>>>>>>>>> because > > >>>>>>>>>>>>>>>>> you > > >>>>>>>>>>>>>>>>>>>> don't > > >>>>>>>>>>>>>>>>>>>>>>> need to have parent_dag_name and > > >>>>>>> child_dag_name > > >>>>>>>>> in > > >>>>>>>>>>> the > > >>>>>>>>>>>>>>> function > > >>>>>>>>>>>>>>>>>>>>> signature > > >>>>>>>>>>>>>>>>>>>>>>> anymore. > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> Changes proposed: > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> - *Unpack SubDags during dag > > >>> parsing*: > > >>>>> This > > >>>>>>>>>> rewrites > > >>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>> *DagBag.bag_dag* > > >>>>>>>>>>>>>>>>>>>>>>> method to unpack subdag while > > >>> parsing, > > >>>>> and > > >>>>>> it > > >>>>>>>>> will > > >>>>>>>>>>>> give a > > >>>>>>>>>>>>>>> flat > > >>>>>>>>>>>>>>>>>>>>>>> structure at > > >>>>>>>>>>>>>>>>>>>>>>> the task level > > >>>>>>>>>>>>>>>>>>>>>>> - *Simplify SubDagOperator*: The > > >> new > > >>>>>>>>> SubDagOperator > > >>>>>>>>>>>> acts > > >>>>>>>>>>>>>>> like a > > >>>>>>>>>>>>>>>>>>>>>>> container and most of the original > > >>>>> methods > > >>>>>>> are > > >>>>>>>>>>> removed. > > >>>>>>>>>>>>> The > > >>>>>>>>>>>>>>>>>>>>>>> signature is > > >>>>>>>>>>>>>>>>>>>>>>> also changed to *subdag_factory > > >> *with > > >>>>>>>>> *subdag_args > > >>>>>>>>>>> *and > > >>>>>>>>>>>>>>>>>>>>> *subdag_kwargs*. > > >>>>>>>>>>>>>>>>>>>>>>> This is similar to the > > >> PythonOperator > > >>>>>>>> signature. > > >>>>>>>>>>>>>>>>>>>>>>> - *Add a TaskGroup model and add > > >>>>>>> current_group > > >>>>>>>> & > > >>>>>>>>>>>>>> parent_group > > >>>>>>>>>>>>>>>>>>>>> attributes > > >>>>>>>>>>>>>>>>>>>>>>> to BaseOperator*: This metadata is > > >>> used > > >>>>> to > > >>>>>>>> group > > >>>>>>>>>>> tasks > > >>>>>>>>>>>>> for > > >>>>>>>>>>>>>>>>>>>>>>> rendering at > > >>>>>>>>>>>>>>>>>>>>>>> UI level. It may potentially extend > > >>>>> further > > >>>>>>> to > > >>>>>>>>>> group > > >>>>>>>>>>>>>>> arbitrary > > >>>>>>>>>>>>>>>>>>> tasks > > >>>>>>>>>>>>>>>>>>>>>>> outside the context of subdag to > > >>> allow > > >>>>>>>>> group-level > > >>>>>>>>>>>>>> operations > > >>>>>>>>>>>>>>>>>>> (i.e. > > >>>>>>>>>>>>>>>>>>>>>>> stop/trigger a group of task within > > >>> the > > >>>>>> dag) > > >>>>>>>>>>>>>>>>>>>>>>> - *Webserver UI for SubDag*: > > >> Proposed > > >>>> UI > > >>>>>>>>>> modification > > >>>>>>>>>>>> to > > >>>>>>>>>>>>>>> allow > > >>>>>>>>>>>>>>>>>>>>>>> (un)collapse a group of tasks for a > > >>>> flat > > >>>>>>>>> structure > > >>>>>>>>>> to > > >>>>>>>>>>>>> pair > > >>>>>>>>>>>>>>> with > > >>>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>> first > > >>>>>>>>>>>>>>>>>>>>>>> change instead of the original > > >>>>> hierarchical > > >>>>>>>>>>> structure. > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> Please see related documents and > > >> PRs > > >>>> for > > >>>>>>>> details: > > >>>>>>>>>>>>>>>>>>>>>>> AIP: > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+Rewrite+SubDagOperator > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> Original Issue: > > >>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/8078 > > >>>>>>>>>>>>>>>>>>>>>>> Draft PR: > > >>>>>>>>>>> https://github.com/apache/airflow/pull/9243 > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> Please let me know if there are any > > >>>>> aspects > > >>>>>>>> that > > >>>>>>>>>> you > > >>>>>>>>>>>>>>>>>> agree/disagree > > >>>>>>>>>>>>>>>>>>>>>>> with or > > >>>>>>>>>>>>>>>>>>>>>>> need more clarification (especially > > >>> the > > >>>>>> third > > >>>>>>>>>> change > > >>>>>>>>>>>>>>> regarding > > >>>>>>>>>>>>>>>>>>>>> TaskGroup). > > >>>>>>>>>>>>>>>>>>>>>>> Any comments are welcome and I am > > >>>> looking > > >>>>>>>> forward > > >>>>>>>>>> to > > >>>>>>>>>>>> it! > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>>> Cheers > > >>>>>>>>>>>>>>>>>>>>>>> Bin > > >>>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>>>>>>> Kyle Hamlin > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>> Thanks & Regards > > >>>>>>>>>>>>>>> Poornima > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> -- > > >>>>>>> > > >>>>>>> Jarek Potiuk > > >>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > > >>>>>>> > > >>>>>>> M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129 > > >>>>> <+48%20660%20796%20129>> > > >>>>>>> [image: Polidea] <https://www.polidea.com/> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> > > >>>>> Jarek Potiuk > > >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > > >>>>> > > >>>>> M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129 > > >>>>> <+48%20660%20796%20129>> > > >>>>> [image: Polidea] <https://www.polidea.com/> > > >>>>> > > >>>> > > >>>> > > >>>> -- > > >>>> > > >>>> *Jacob Ferriero* > > >>>> > > >>>> Strategic Cloud Engineer: Data Engineering > > >>>> > > >>>> jferri...@google.com > > >>>> > > >>>> 617-714-2509 > > >>>> > > >>> > > >> > > > > >
