Re graph times. That makes sense. Let me know what you find. We may be able to contribute on the lazy loading part.
Looking forward to see the updated AIP! Gerard Casas Saez Twitter | Cortex | @casassaez <http://twitter.com/casassaez> On Fri, Aug 14, 2020 at 6:14 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > Permissions granted, let me know if you face any issues. > > On Fri, Aug 14, 2020 at 1:10 PM Yu Qian <yuqian1...@gmail.com> wrote: > > > Hi, Kaxil, my ID for cwiki.apache.org is yuqian1990. Thank you! > > > > On Fri, Aug 14, 2020 at 7:35 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > > > What's your ID i.e. if you haven't created an account yet, please > create > > > one at https://cwiki.apache.org/confluence/signup.action and send us > > your > > > ID and we will add permissions. > > > > > > Thanks. I'll edit the AIP. May I request permission to edit it? > > > > My wiki user email is yuqian1...@gmail.com. > > > > > > > > > On Fri, Aug 14, 2020 at 9:45 AM Yu Qian <yuqian1...@gmail.com> wrote: > > > > > > > Re, Xinbin. Thanks. I'll edit the AIP. May I request permission to > edit > > > it? > > > > My wiki user email is yuqian1...@gmail.com. > > > > > > > > Re Gerard: yes the UI loads all the nodes as json from the web server > > at > > > > once. However, it only adds the top level nodes and edges to the > graph > > > when > > > > the Graph View page is first opened. And then adds the expanded nodes > > to > > > > the graph as the user expands them. From what I've experienced with > > DAGs > > > > containing around 400 tasks (not using TaskGroup or SubDagOperator), > > > > opening the whole dag in Graph View usually takes 5 seconds. Less > than > > > 60ms > > > > of that is taken by loading the data from webserver. The remaining > > 4.9s+ > > > is > > > > taken by javascript functions in dagre-d3.min.js such as createNodes, > > > > createEdgeLabels, etc and by rendering the graph. With TaskGroup > being > > > used > > > > to group tasks into a smaller number of top-level nodes, the amount > of > > > data > > > > loaded from webserver will remain about the same compared to a flat > dag > > > of > > > > the same size, but the number of nodes and edges needed to be plot on > > the > > > > graph can be reduced significantly. So in theory this should speed up > > the > > > > time it takes to open Graph View even without lazy-loading the data > > (I'll > > > > experiment to find out). That said, if it comes to a point > lazy-loading > > > > helps, we can still implement it as an improvement. > > > > > > > > Re James: the Tree View looks as if all all the groups are fully > > > expanded. > > > > (because under the hood all the tasks are in a single DAG). I'm less > > > > worried about Tree View at the moment because it already has a > > mechanism > > > > for collapsing tasks by the dependency tree. That said, the Tree View > > can > > > > definitely be improved too with TaskGroup. (e.g. collapse tasks in > the > > > same > > > > TaskGroup when Tree View is first opened). > > > > > > > > For both suggestions, implementing them don't require fundamental > > changes > > > > to the idea. I think we can have a basic working TaskGroup first, and > > > then > > > > improve it incrementally in several PRs as we get more feedback from > > the > > > > community. What do you think? > > > > > > > > Qian > > > > > > > > > > > > On Wed, Aug 12, 2020 at 9:15 AM James Coder <jcode...@gmail.com> > > wrote: > > > > > > > > > I agree this looks great, one question, how does the tree view > look? > > > > > > > > > > James Coder > > > > > > > > > > > On Aug 11, 2020, at 6:48 PM, Gerard Casas Saez < > > > gcasass...@twitter.com > > > > .invalid> > > > > > wrote: > > > > > > > > > > > > First of all, this is awesome!! > > > > > > > > > > > > Secondly, checking your UI code, seems you are loading all > > operators > > > at > > > > > > once. Wondering if we can load them as needed (aka load whenever > we > > > > click > > > > > > the TaskGroup). Some of our DAGs are so large that take forever > to > > > load > > > > > on > > > > > > the Graph view, so worried about this still being an issue here. > It > > > may > > > > > be > > > > > > easily solvable by implementing lazy loading of the graph. Not > sure > > > how > > > > > > easy to implement/add to the UI extension (and dont want to push > > for > > > > > early > > > > > > optimization as its the root of all evil). > > > > > > Gerard Casas Saez > > > > > > Twitter | Cortex | @casassaez <http://twitter.com/casassaez> > > > > > > > > > > > > > > > > > >> On Tue, Aug 11, 2020 at 10:35 AM Xinbin Huang < > > > bin.huan...@gmail.com> > > > > > wrote: > > > > > >> > > > > > >> Hi Yu, > > > > > >> > > > > > >> Thank you so much for taking on this. I was fairly distracted > > > > previously > > > > > >> and I didn't have the time to update the proposal. In fact, > after > > > > > >> discussing with Ash, Kaxil and Daniel, the direction of this AIP > > has > > > > > been > > > > > >> changed to favor the concept of TaskGroup instead of rewriting > > > > > >> SubDagOperator (though it may may sense to deprecate SubDag in a > > > > future > > > > > >> date.). > > > > > >> > > > > > >> Your PR is amazing and it has implemented the desire features. I > > > think > > > > > we > > > > > >> can focus on your new PR instead. Do you mind updating the AIP > > based > > > > on > > > > > >> what you have done in your PR? > > > > > >> > > > > > >> Best, > > > > > >> Bin > > > > > >> > > > > > >> > > > > > >>> On Tue, Aug 11, 2020 at 7:11 AM Yu Qian <yuqian1...@gmail.com> > > > > wrote: > > > > > >>> > > > > > >>> Hi, all, I've added the basic UI changes to my proposed > > > > implementation > > > > > of > > > > > >>> TaskGroup as UI grouping concept: > > > > > >>> https://github.com/apache/airflow/pull/10153 > > > > > >>> > > > > > >>> I think Chris had a pretty good specification of TaskGroup so > i'm > > > > > quoting > > > > > >>> it here. The only thing I don't fully agree with is the > > restriction > > > > > >>> "... **cannot* > > > > > >>> have dependencies between a Task in a TaskGroup and either a* > > > > > >>> * Task in a different TaskGroup or a Task not in any > group*". I > > > > think > > > > > >>> this is over restrictive. Since TaskGroup is a UI concept, > tasks > > > can > > > > > have > > > > > >>> dependencies on tasks in other TaskGroup or not in any > TaskGroup. > > > In > > > > my > > > > > >> PR, > > > > > >>> this is allowed. The graph edges will update accordingly when > > > > > TaskGroups > > > > > >>> are expanded/collapsed. TaskGroup is only helping to make the > UI > > > look > > > > > >> less > > > > > >>> crowded. Under the hood, everything is still a DAG of tasks and > > > edges > > > > > so > > > > > >>> things work normally. Here's a screenshot > > > > > >>> < > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > https://raw.githubusercontent.com/yuqian90/airflow/gif_for_demo/airflow/www/static/screen-shot-short.gif > > > > > >>>> > > > > > >>> of the UI interaction. > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> * - Tasks can be added to a TaskGroup - You *can* have > > > > dependencies > > > > > >>> between Tasks in the same TaskGroup, but *cannot* have > > > dependencies > > > > > >>> between a Task in a TaskGroup and either a Task in a > different > > > > > >> TaskGroup > > > > > >>> or a Task not in any group - You *can* have dependencies > > between > > > a > > > > > >>> TaskGroup and either other TaskGroups or Tasks not in any > group > > > - > > > > > The > > > > > >>> UI will by default render a TaskGroup as a single "object", but > > > > which > > > > > >> you > > > > > >>> expand or zoom into in some way - You'd need some way to > > > determine > > > > > what > > > > > >>> the "status" of a TaskGroup was at least for UI display > > purposes* > > > > > >>> > > > > > >>> > > > > > >>> Regarding Jake's comment, I agree it's possible to implement > the > > > > > >> "retrying > > > > > >>> tasks in a group" pattern he mentioned as an optional feature > of > > > > > >> TaskGroup > > > > > >>> although that may go against having TaskGroup as a pure UI > > concept. > > > > For > > > > > >> the > > > > > >>> motivating example Jake provided, I suggest implementing both > > > > > >>> SubmitLongRunningJobTask and PollJobStatusSensor in a single > > > > operator. > > > > > It > > > > > >>> can do something like BaseSensorOperator.execute() does in > > > > "reschedule" > > > > > >>> mode, i.e. it first executes some code to submit the long > running > > > job > > > > > to > > > > > >>> the external service, and store the state (e.g. in XCom). Then > > > > > reschedule > > > > > >>> itself. Subsequent runs then pokes for the completion state. > > > > > >>> > > > > > >>> > > > > > >>> On Thu, Aug 6, 2020 at 2:08 AM Jacob Ferriero > > > > > >> <jferri...@google.com.invalid > > > > > >>>> > > > > > >>> wrote: > > > > > >>> > > > > > >>>> I really like this idea of a TaskGroup container as I think > this > > > > will > > > > > >> be > > > > > >>>> much easier to use than SubDag. > > > > > >>>> > > > > > >>>> I'd like to propose an optional behavior for special retry > > > mechanics > > > > > >> via > > > > > >>> a > > > > > >>>> TaskGroup.retry_all property. > > > > > >>>> This way I could use TaskGroup to replace my favorite use of > > > SubDag > > > > > for > > > > > >>>> atomically retrying tasks of the pattern "act on external > state > > > then > > > > > >>>> reschedule poll until desired state reached". > > > > > >>>> > > > > > >>>> Motivating use case I have for a SubDag is very simple two > task > > > > group > > > > > >>>> [SubmitLongRunningJobTask >> PollJobStatusSensor]. > > > > > >>>> I use SubDag is because it gives me an easy way to retry the > > > > > >>> SubmitJobTask > > > > > >>>> if something about the PollJobSensor fails. > > > > > >>>> This pattern would be really nice for jobs that are expected > to > > > run > > > > a > > > > > >>> long > > > > > >>>> time (because we can use sensor can use reschedule mode > freeing > > up > > > > > >> slots) > > > > > >>>> but might fail for a retryable reason. > > > > > >>>> However, using SubDag to meet this use case defeats the > purpose > > > > > because > > > > > >>>> SubDag infamously > > > > > >>>> < > > > > > >>>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > https://medium.com/@team_24989/fixing-subdagoperator-deadlock-in-airflow-6c64312ebb10 > > > > > >>>>> > > > > > >>>> blocks a "controller" slot for the entire duration. > > > > > >>>> This may feel like a cyclic behavior but reality it is very > > common > > > > for > > > > > >> a > > > > > >>>> single operator to submit job / wait til done. > > > > > >>>> We could use this case refactor many operators (e.g. BQ, > > Dataproc, > > > > > >>>> Dataflow) to be implemented as TaskGroup[SubmitTask >> > PollTask] > > > > with > > > > > >> an > > > > > >>>> optional reschedule mode if user knows that this job may take > a > > > long > > > > > >>> time. > > > > > >>>> > > > > > >>>> I'd be happy to the development work on adding this specific > > retry > > > > > >>> behavior > > > > > >>>> to TaskGroup once the base concept is implemented if others in > > the > > > > > >>>> community would find this a useful feature. > > > > > >>>> > > > > > >>>> Cheers, > > > > > >>>> Jake > > > > > >>>> > > > > > >>>> On Tue, Aug 4, 2020 at 10:07 AM Jarek Potiuk < > > > > > jarek.pot...@polidea.com > > > > > >>> > > > > > >>>> wrote: > > > > > >>>> > > > > > >>>>> All for it :) . I think we are getting closer to have regular > > > > > >> planning > > > > > >>>> and > > > > > >>>>> making some structured approach to 2.0 and starting task > force > > > for > > > > it > > > > > >>>> soon, > > > > > >>>>> so I think this should be perfectly fine to discuss and even > > > start > > > > > >>>>> implementing what's beyond as soon as we make sure that we > are > > > > > >>>> prioritizing > > > > > >>>>> 2.0 work. > > > > > >>>>> > > > > > >>>>> J, > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> On Tue, Aug 4, 2020 at 12:09 PM Yu Qian < > yuqian1...@gmail.com> > > > > > >> wrote: > > > > > >>>>> > > > > > >>>>>> Hi Jarek, > > > > > >>>>>> > > > > > >>>>>> I agree we should not change the behaviour of the existing > > > > > >>>> SubDagOperator > > > > > >>>>>> till Airflow 2.1. Is it okay to continue the discussion > about > > > > > >>> TaskGroup > > > > > >>>>> as > > > > > >>>>>> a brand new concept/feature independent from the existing > > > > > >>>> SubDagOperator? > > > > > >>>>>> In other words, shall we add TaskGroup as a UI grouping > > concept > > > > > >> like > > > > > >>>> Ash > > > > > >>>>>> suggested, and not touch SubDagOperator atl all. Whenever we > > are > > > > > >>> ready > > > > > >>>>> with > > > > > >>>>>> TaskGroup, we then deprecate SubDagOperator in Airflow 2.1. > > > > > >>>>>> > > > > > >>>>>> I really like Ash's idea of simplifying the SubDagOperator > > idea > > > > > >> into > > > > > >>> a > > > > > >>>>>> simple UI grouping concept. I think Xinbin's idea of > > > "reattaching > > > > > >> all > > > > > >>>> the > > > > > >>>>>> tasks to the root DAG" is the way to go. And I see James > > pointed > > > > > >> out > > > > > >>> we > > > > > >>>>>> need some helper functions to simplify dependencies setting > of > > > > > >>>> TaskGroup. > > > > > >>>>>> Xinbin put up a pretty elegant example in his PR > > > > > >>>>>> <https://github.com/apache/airflow/pull/9243>. I think > having > > > > > >>>> TaskGroup > > > > > >>>>> as > > > > > >>>>>> a UI concept should be a relatively small change. We can > > > simplify > > > > > >>>>> Xinbin's > > > > > >>>>>> PR further. So I put up this alternative proposal here: > > > > > >>>>>> https://github.com/apache/airflow/pull/10153 > > > > > >>>>>> > > > > > >>>>>> I have not done any UI changes due to lack of experience > with > > > web > > > > > >> UI. > > > > > >>>> If > > > > > >>>>>> anyone's interested, please take a look at the PR. > > > > > >>>>>> > > > > > >>>>>> Qian > > > > > >>>>>> > > > > > >>>>>> On Mon, Jun 22, 2020 at 5:15 AM Jarek Potiuk < > > > > > >>> jarek.pot...@polidea.com > > > > > >>>>> > > > > > >>>>>> wrote: > > > > > >>>>>> > > > > > >>>>>>> Similar point here to the other ideas that are popping up. > > > Maybe > > > > > >> we > > > > > >>>>>> should > > > > > >>>>>>> just focus on completing 2.0 and make all discussions about > > > > > >> further > > > > > >>>>>>> improvements to 2.1? While those are important discussions > > (and > > > > > >> we > > > > > >>>>> should > > > > > >>>>>>> continue them in the near future !) I think at this point > > > > > >> focusing > > > > > >>>> on > > > > > >>>>>>> delivering 2.0 in its current shape should be our focus > now ? > > > > > >>>>>>> > > > > > >>>>>>> J. > > > > > >>>>>>> > > > > > >>>>>>> On Thu, Jun 18, 2020 at 6:35 PM Xinbin Huang < > > > > > >>> bin.huan...@gmail.com> > > > > > >>>>>>> wrote: > > > > > >>>>>>> > > > > > >>>>>>>> Hi Daniel > > > > > >>>>>>>> > > > > > >>>>>>>> I agree that the TaskGroup should have the same API as a > DAG > > > > > >>> object > > > > > >>>>>>> related > > > > > >>>>>>>> to task dependencies, but it will not have anything > related > > to > > > > > >>>> actual > > > > > >>>>>>>> execution or scheduling. > > > > > >>>>>>>> I will update the AIP according to this over the weekend. > > > > > >>>>>>>> > > > > > >>>>>>>>> We could even make a “DAGTemplate” object s.t. when you > > > > > >> import > > > > > >>>> the > > > > > >>>>>>> object > > > > > >>>>>>>> you can import it with parameters to determine the shape > of > > > the > > > > > >>>> DAG. > > > > > >>>>>>>> > > > > > >>>>>>>> Can you elaborate a bit more on this? Does it serve a > > similar > > > > > >>>> purpose > > > > > >>>>>> as > > > > > >>>>>>> a > > > > > >>>>>>>> DAG factory function? > > > > > >>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> On Thu, Jun 18, 2020 at 9:13 AM Daniel Imberman < > > > > > >>>>>>> daniel.imber...@gmail.com > > > > > >>>>>>>>> > > > > > >>>>>>>> wrote: > > > > > >>>>>>>> > > > > > >>>>>>>>> Hi Bin, > > > > > >>>>>>>>> > > > > > >>>>>>>>> Why not give the TaskGroup the same API as a DAG object > > (e.g. > > > > > >>> the > > > > > >>>>>>> bitwise > > > > > >>>>>>>>> operator fro task dependencies). We could even make a > > > > > >>>> “DAGTemplate” > > > > > >>>>>>>> object > > > > > >>>>>>>>> s.t. when you import the object you can import it with > > > > > >>> parameters > > > > > >>>>> to > > > > > >>>>>>>>> determine the shape of the DAG. > > > > > >>>>>>>>> > > > > > >>>>>>>>> > > > > > >>>>>>>>> On Wed, Jun 17, 2020 at 8:54 PM, Xinbin Huang < > > > > > >>>>> bin.huan...@gmail.com > > > > > >>>>>>> > > > > > >>>>>>>>> wrote: > > > > > >>>>>>>>> The TaskGroup will not take schedule interval as a > > parameter > > > > > >>>>> itself, > > > > > >>>>>>> and > > > > > >>>>>>>> it > > > > > >>>>>>>>> depends on the DAG where it attaches to. In my opinion, > the > > > > > >>>>> TaskGroup > > > > > >>>>>>>> will > > > > > >>>>>>>>> only contain a group of tasks with interdependencies, and > > the > > > > > >>>>>> TaskGroup > > > > > >>>>>>>>> behaves like a task. It doesn't contain any > > > > > >>> execution/scheduling > > > > > >>>>>> logic > > > > > >>>>>>>>> (i.e. schedule_interval, concurrency, max_active_runs > etc.) > > > > > >>> like > > > > > >>>> a > > > > > >>>>>> DAG > > > > > >>>>>>>>> does. > > > > > >>>>>>>>> > > > > > >>>>>>>>>> For example, there is the scenario that the schedule > > > > > >> interval > > > > > >>>> of > > > > > >>>>>> DAG > > > > > >>>>>>> is > > > > > >>>>>>>>> 1 hour and the schedule interval of TaskGroup is 20 min. > > > > > >>>>>>>>> > > > > > >>>>>>>>> I am curious why you ask this. Is this a use case that > you > > > > > >> want > > > > > >>>> to > > > > > >>>>>>>> achieve? > > > > > >>>>>>>>> > > > > > >>>>>>>>> Bin > > > > > >>>>>>>>> > > > > > >>>>>>>>> On Wed, Jun 17, 2020 at 7:59 PM 蒋晓峰 < > > > > > >> thanosxnicho...@gmail.com > > > > > >>>> > > > > > >>>>>> wrote: > > > > > >>>>>>>>> > > > > > >>>>>>>>>> Hi Bin, > > > > > >>>>>>>>>> Using TaskGroup, Is the schedule interval of TaskGroup > the > > > > > >>> same > > > > > >>>>> as > > > > > >>>>>>> the > > > > > >>>>>>>>>> parent DAG? My main concern is whether the schedule > > > > > >> interval > > > > > >>> of > > > > > >>>>>>>> TaskGroup > > > > > >>>>>>>>>> could be different with that of the DAG? For example, > > there > > > > > >>> is > > > > > >>>>> the > > > > > >>>>>>>>> scenario > > > > > >>>>>>>>>> that the schedule interval of DAG is 1 hour and the > > > > > >> schedule > > > > > >>>>>> interval > > > > > >>>>>>>> of > > > > > >>>>>>>>>> TaskGroup is 20 min. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Cheers, > > > > > >>>>>>>>>> Nicholas > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> On Thu, Jun 18, 2020 at 10:30 AM Xinbin Huang < > > > > > >>>>>> bin.huan...@gmail.com > > > > > >>>>>>>> > > > > > >>>>>>>>>> wrote: > > > > > >>>>>>>>>> > > > > > >>>>>>>>>>> Hi Nicholas, > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> I am not sure about the old behavior of SubDagOperator, > > > > > >>> maybe > > > > > >>>>> it > > > > > >>>>>>> will > > > > > >>>>>>>>>> throw > > > > > >>>>>>>>>>> an error? But in the original proposal, the subdag's > > > > > >>>>>>>> schedule_interval > > > > > >>>>>>>>>> will > > > > > >>>>>>>>>>> be ignored. Or if we decide to use TaskGroup to replace > > > > > >>>> SubDag, > > > > > >>>>>>> there > > > > > >>>>>>>>>> will > > > > > >>>>>>>>>>> be no subdag schedule_interval. > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> Bin > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> On Wed, Jun 17, 2020 at 6:21 PM 蒋晓峰 < > > > > > >>>> thanosxnicho...@gmail.com > > > > > >>>>>> > > > > > >>>>>>>> wrote: > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>>> Hi Bin, > > > > > >>>>>>>>>>>> Thanks for your good proposal. I was confused whether > > > > > >> the > > > > > >>>>>>> schedule > > > > > >>>>>>>>>>>> interval of SubDAG is different from that of the > parent > > > > > >>>> DAG? > > > > > >>>>> I > > > > > >>>>>>> have > > > > > >>>>>>>>>>>> discussed with Jiajie Zhong about the schedule > interval > > > > > >>> of > > > > > >>>>>>> SubDAG. > > > > > >>>>>>>> If > > > > > >>>>>>>>>> the > > > > > >>>>>>>>>>>> SubDagOperator has a different schedule interval, what > > > > > >>> will > > > > > >>>>>>> happen > > > > > >>>>>>>>> for > > > > > >>>>>>>>>>> the > > > > > >>>>>>>>>>>> scheduler to schedule the parent DAG? > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> Regards, > > > > > >>>>>>>>>>>> Nicholas Jiang > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> On Thu, Jun 18, 2020 at 8:04 AM Xinbin Huang < > > > > > >>>>>>>> bin.huan...@gmail.com> > > > > > >>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Thank you, Max, Kaxil, and everyone's feedback! > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> I have rethought about the concept of subdag and task > > > > > >>>>>> groups. I > > > > > >>>>>>>>> think > > > > > >>>>>>>>>>> the > > > > > >>>>>>>>>>>>> better way to approach this is to entirely remove > > > > > >>> subdag > > > > > >>>>> and > > > > > >>>>>>>>>> introduce > > > > > >>>>>>>>>>>> the > > > > > >>>>>>>>>>>>> concept of TaskGroup, which is a container of tasks > > > > > >>> along > > > > > >>>>>> with > > > > > >>>>>>>>> their > > > > > >>>>>>>>>>>>> dependencies *without execution/scheduling logic as a > > > > > >>>> DAG*. > > > > > >>>>>> The > > > > > >>>>>>>>> only > > > > > >>>>>>>>>>>>> purpose of it is to group a list of tasks, but you > > > > > >>> still > > > > > >>>>> need > > > > > >>>>>>> to > > > > > >>>>>>>>> add > > > > > >>>>>>>>>> it > > > > > >>>>>>>>>>>> to > > > > > >>>>>>>>>>>>> a DAG for execution. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Here is a small code snippet. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> ``` > > > > > >>>>>>>>>>>>> class TaskGroup: > > > > > >>>>>>>>>>>>> """ > > > > > >>>>>>>>>>>>> A TaskGroup contains a group of tasks. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> If default_args is missing, it will take default args > > > > > >>>> from > > > > > >>>>>> the > > > > > >>>>>>>>>> DAG. > > > > > >>>>>>>>>>>>> """ > > > > > >>>>>>>>>>>>> def __init__(self, group_id, default_args): > > > > > >>>>>>>>>>>>> pass > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> """ > > > > > >>>>>>>>>>>>> You can add tasks to a task group similar to adding > > > > > >>> tasks > > > > > >>>>> to > > > > > >>>>>> a > > > > > >>>>>>>> DAG > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> This can be declared in a separate file from the dag > > > > > >>> file > > > > > >>>>>>>>>>>>> """ > > > > > >>>>>>>>>>>>> download_group = TaskGroup(group_id='download', > > > > > >>>>>>>>>>>> default_args=default_args) > > > > > >>>>>>>>>>>>> download_group.add_task(task1) > > > > > >>>>>>>>>>>>> task2.dag = download_group > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> with download_group: > > > > > >>>>>>>>>>>>> task3 = DummyOperator(task_id='task3') > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> [task, task2] >> task3 > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> """Add it to a DAG for execution""" > > > > > >>>>>>>>>>>>> with DAG(dag_id='start_download_dag', > > > > > >>>>>>> default_args=default_args, > > > > > >>>>>>>>>>>>> schedule_interval='@daily', ...) as dag: > > > > > >>>>>>>>>>>>> start = DummyOperator(task_id='start') > > > > > >>>>>>>>>>>>> start >> download_group > > > > > >>>>>>>>>>>>> # this is equivalent to > > > > > >>>>>>>>>>>>> # start >> [task, task2] >> task3 > > > > > >>>>>>>>>>>>> ``` > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> With this, we can still reuse a group of tasks and > > > > > >> set > > > > > >>>>>>>> dependencies > > > > > >>>>>>>>>>>> between > > > > > >>>>>>>>>>>>> them; it avoids the boilerplate code from using > > > > > >>>>>> SubDagOperator, > > > > > >>>>>>>> and > > > > > >>>>>>>>>> we > > > > > >>>>>>>>>>>> can > > > > > >>>>>>>>>>>>> declare dependencies as `task >> task_group >> task`. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> User migration wise, we can introduce it before > > > > > >> Airflow > > > > > >>>> 2.0 > > > > > >>>>>> and > > > > > >>>>>>>>> allow > > > > > >>>>>>>>>>>>> gradual transition. Then we can decide if we still > > > > > >> want > > > > > >>>> to > > > > > >>>>>> keep > > > > > >>>>>>>> the > > > > > >>>>>>>>>>>>> SubDagOperator or simply remove it. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Any thoughts? > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Cheers, > > > > > >>>>>>>>>>>>> Bin > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> On Wed, Jun 17, 2020 at 7:37 AM Maxime Beauchemin < > > > > > >>>>>>>>>>>>> maximebeauche...@gmail.com> wrote: > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> +1, proposal looks good. > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> The original intention was really to have tasks > > > > > >>> groups > > > > > >>>>> and > > > > > >>>>>> a > > > > > >>>>>>>>>>>> zoom-in/out > > > > > >>>>>>>>>>>>> in > > > > > >>>>>>>>>>>>>> the UI. The original reasoning was to reuse the DAG > > > > > >>>>> object > > > > > >>>>>>>> since > > > > > >>>>>>>>> it > > > > > >>>>>>>>>>> is > > > > > >>>>>>>>>>>> a > > > > > >>>>>>>>>>>>>> group of tasks, but as highlighted here it does > > > > > >>> create > > > > > >>>>>>>> underlying > > > > > >>>>>>>>>>>>>> confusions since a DAG is much more than just a > > > > > >> group > > > > > >>>> of > > > > > >>>>>>> tasks. > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> Max > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> On Mon, Jun 15, 2020 at 2:43 AM Poornima Joshi < > > > > > >>>>>>>>>>>>> joshipoornim...@gmail.com> > > > > > >>>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> Thank you for your email. > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> On Sat, Jun 13, 2020 at 12:18 AM Xinbin Huang < > > > > > >>>>>>>>>>> bin.huan...@gmail.com > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> - *Unpack SubDags during dag parsing*: This > > > > > >>>>>> rewrites > > > > > >>>>>>>> the > > > > > >>>>>>>>>>>>>>>> *DagBag.bag_dag* > > > > > >>>>>>>>>>>>>>>>>> method to unpack subdag while parsing, and > > > > > >> it > > > > > >>>>> will > > > > > >>>>>>>> give a > > > > > >>>>>>>>>>>> flat > > > > > >>>>>>>>>>>>>>>>>> structure at > > > > > >>>>>>>>>>>>>>>>>> the task level > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> The serialized_dag representation already > > > > > >> does > > > > > >>>>> this I > > > > > >>>>>>>>> think. > > > > > >>>>>>>>>> At > > > > > >>>>>>>>>>>>> least > > > > > >>>>>>>>>>>>>>> if > > > > > >>>>>>>>>>>>>>>>> I've understood your idea here correctly. > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> I am not sure about serialized_dag > > > > > >>> representation, > > > > > >>>>> but > > > > > >>>>>> at > > > > > >>>>>>>>> least > > > > > >>>>>>>>>>> it > > > > > >>>>>>>>>>>>> will > > > > > >>>>>>>>>>>>>>>> still keep the subdag entry in the DAG table? > > > > > >> In > > > > > >>> my > > > > > >>>>>>>> proposal > > > > > >>>>>>>>> as > > > > > >>>>>>>>>>>> also > > > > > >>>>>>>>>>>>> in > > > > > >>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>> draft PR, the idea is to *extract the tasks > > > > > >> from > > > > > >>>> the > > > > > >>>>>>> subdag > > > > > >>>>>>>>> and > > > > > >>>>>>>>>>> add > > > > > >>>>>>>>>>>>>> them > > > > > >>>>>>>>>>>>>>>> back to the root_dag. *So the runtime DAG graph > > > > > >>>> will > > > > > >>>>>> look > > > > > >>>>>>>>>> exactly > > > > > >>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>> same as without subdag but with metadata > > > > > >> attached > > > > > >>>> to > > > > > >>>>>>> those > > > > > >>>>>>>>>>>> sections. > > > > > >>>>>>>>>>>>>>> These > > > > > >>>>>>>>>>>>>>>> metadata will be later on used to render in the > > > > > >>> UI. > > > > > >>>>> So > > > > > >>>>>>>> after > > > > > >>>>>>>>>>>> parsing > > > > > >>>>>>>>>>>>> ( > > > > > >>>>>>>>>>>>>>>> *DagBag.process_file()*), it will just output > > > > > >> the > > > > > >>>>>>> *root_dag > > > > > >>>>>>>>>>>> *instead > > > > > >>>>>>>>>>>>> of > > > > > >>>>>>>>>>>>>>> *root_dag + > > > > > >>>>>>>>>>>>>>>> subdag + subdag + nested subdag* etc. > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> - e.g. section-1-* will have metadata > > > > > >>>>>>>>>> current_group=section-1, > > > > > >>>>>>>>>>>>>>>> parent_group=<the-root-dag-id> (welcome for > > > > > >>> naming > > > > > >>>>>>>>>>> suggestions), > > > > > >>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>> reason for parent_group is that we can have > > > > > >>> nested > > > > > >>>>>> group > > > > > >>>>>>>> and > > > > > >>>>>>>>>>>> still > > > > > >>>>>>>>>>>>>> be > > > > > >>>>>>>>>>>>>>>> able to capture the dependency. > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> Runtime DAG: > > > > > >>>>>>>>>>>>>>>> [image: image.png] > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> While at the UI, what we see would be something > > > > > >>>> like > > > > > >>>>>> this > > > > > >>>>>>>> by > > > > > >>>>>>>>>>>>> utilizing > > > > > >>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>> metadata, and then we can expand or zoom into > > > > > >> in > > > > > >>>> some > > > > > >>>>>>> way. > > > > > >>>>>>>>>>>>>>>> [image: image.png] > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> The benefits I can see is that: > > > > > >>>>>>>>>>>>>>>> 1. We don't need to deal with the extra > > > > > >>> complexity > > > > > >>>> of > > > > > >>>>>>>> SubDag > > > > > >>>>>>>>>> for > > > > > >>>>>>>>>>>>>>> execution > > > > > >>>>>>>>>>>>>>>> and scheduling. It will be the same as not > > > > > >> using > > > > > >>>>>> SubDag. > > > > > >>>>>>>>>>>>>>>> 2. Still have the benefits of modularized and > > > > > >>>>> reusable > > > > > >>>>>>> dag > > > > > >>>>>>>>> code > > > > > >>>>>>>>>>> and > > > > > >>>>>>>>>>>>>>>> declare dependencies between them. And with the > > > > > >>> new > > > > > >>>>>>>>>>> SubDagOperator > > > > > >>>>>>>>>>>>> (see > > > > > >>>>>>>>>>>>>>> AIP > > > > > >>>>>>>>>>>>>>>> or draft PR), we can use the same dag_factory > > > > > >>>>> function > > > > > >>>>>>> for > > > > > >>>>>>>>>>>>> generating 1 > > > > > >>>>>>>>>>>>>>>> dag, a lot of dynamic dags, or used for SubDag > > > > > >>> (in > > > > > >>>>> this > > > > > >>>>>>>> case, > > > > > >>>>>>>>>> it > > > > > >>>>>>>>>>>> will > > > > > >>>>>>>>>>>>>>> just > > > > > >>>>>>>>>>>>>>>> extract all underlying tasks and append to the > > > > > >>> root > > > > > >>>>>> dag). > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> - Then it gets to the idea of replacing subdag > > > > > >>>> with a > > > > > >>>>>>>>>> simpler > > > > > >>>>>>>>>>>>>> concept > > > > > >>>>>>>>>>>>>>>> by Ash: the proposed change basically drains > > > > > >> out > > > > > >>>> the > > > > > >>>>>>>>>> contents > > > > > >>>>>>>>>>>> of > > > > > >>>>>>>>>>>>> a > > > > > >>>>>>>>>>>>>>> SubDag > > > > > >>>>>>>>>>>>>>>> and becomes more like > > > > > >>>>>>>>>>>> ExtractSubdagTasksAndAppendToRootdagOperator > > > > > >>>>>>>>>>>>>>> (forgive > > > > > >>>>>>>>>>>>>>>> me about the crazy name..). In this case, it is > > > > > >>>> still > > > > > >>>>>>>>>>> necessary > > > > > >>>>>>>>>>>> to > > > > > >>>>>>>>>>>>>>> keep the > > > > > >>>>>>>>>>>>>>>> concept of subdag as it is nothing more than a > > > > > >>>> name? > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> That's why the TaskGroup idea comes up. Thanks > > > > > >>>> Chris > > > > > >>>>>>> Palmer > > > > > >>>>>>>>> for > > > > > >>>>>>>>>>>>> helping > > > > > >>>>>>>>>>>>>>>> conceptualize the functionality of TaskGroup, I > > > > > >>>> will > > > > > >>>>>> just > > > > > >>>>>>>>> paste > > > > > >>>>>>>>>>> it > > > > > >>>>>>>>>>>>>> here. > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> - Tasks can be added to a TaskGroup > > > > > >>>>>>>>>>>>>>>>> - You *can* have dependencies between Tasks > > > > > >> in > > > > > >>>> the > > > > > >>>>>> same > > > > > >>>>>>>>>>>> TaskGroup, > > > > > >>>>>>>>>>>>>> but > > > > > >>>>>>>>>>>>>>>>> *cannot* have dependencies between a Task in > > > > > >> a > > > > > >>>>>>> TaskGroup > > > > > >>>>>>>>>> and > > > > > >>>>>>>>>>>>>> either a > > > > > >>>>>>>>>>>>>>>>> Task in a different TaskGroup or a Task not > > > > > >> in > > > > > >>>> any > > > > > >>>>>>> group > > > > > >>>>>>>>>>>>>>>>> - You *can* have dependencies between a > > > > > >>> TaskGroup > > > > > >>>>> and > > > > > >>>>>>>>>> either > > > > > >>>>>>>>>>>>> other > > > > > >>>>>>>>>>>>>>>>> TaskGroups or Tasks not in any group > > > > > >>>>>>>>>>>>>>>>> - The UI will by default render a TaskGroup > > > > > >> as > > > > > >>> a > > > > > >>>>>> single > > > > > >>>>>>>>>>>> "object", > > > > > >>>>>>>>>>>>>> but > > > > > >>>>>>>>>>>>>>>>> which you expand or zoom into in some way > > > > > >>>>>>>>>>>>>>>>> - You'd need some way to determine what the > > > > > >>>>> "status" > > > > > >>>>>>> of a > > > > > >>>>>>>>>>>>> TaskGroup > > > > > >>>>>>>>>>>>>>> was > > > > > >>>>>>>>>>>>>>>>> at least for UI display purposes > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> I agree with Chris: > > > > > >>>>>>>>>>>>>>>> - From the backend's view (scheduler & > > > > > >>> executor), I > > > > > >>>>>> think > > > > > >>>>>>>>>>> TaskGroup > > > > > >>>>>>>>>>>>>>> should > > > > > >>>>>>>>>>>>>>>> be ignored during execution. (unless we decide > > > > > >> to > > > > > >>>>>>> implement > > > > > >>>>>>>>>> some > > > > > >>>>>>>>>>>>>> metadata > > > > > >>>>>>>>>>>>>>>> operations that allows start/stop a group of > > > > > >>> tasks > > > > > >>>>>> etc.) > > > > > >>>>>>>>>>>>>>>> - From the UI's View, it should be able to pick > > > > > >>> up > > > > > >>>>> the > > > > > >>>>>>>>>> individual > > > > > >>>>>>>>>>>>>> tasks' > > > > > >>>>>>>>>>>>>>>> status and then determine the TaskGroup's > > > > > >> status > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> Bin > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> On Fri, Jun 12, 2020 at 10:28 AM Daniel > > > > > >> Imberman > > > > > >>> < > > > > > >>>>>>>>>>>>>>>> daniel.imber...@gmail.com> wrote: > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> I hadn’t thought about using the `>>` operator > > > > > >>> to > > > > > >>>>> tie > > > > > >>>>>>> dags > > > > > >>>>>>>>>>>> together > > > > > >>>>>>>>>>>>>> but > > > > > >>>>>>>>>>>>>>> I > > > > > >>>>>>>>>>>>>>>>> think that sounds pretty great! I wonder if we > > > > > >>>> could > > > > > >>>>>>>>>> essentially > > > > > >>>>>>>>>>>>> write > > > > > >>>>>>>>>>>>>>> in > > > > > >>>>>>>>>>>>>>>>> the ability to set dependencies to all > > > > > >>>> starter-tasks > > > > > >>>>>> for > > > > > >>>>>>>>> that > > > > > >>>>>>>>>>> DAG. > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> I’m personally ok with SubDag being a mostly > > > > > >> UI > > > > > >>>>>> concept. > > > > > >>>>>>>> It > > > > > >>>>>>>>>>>> doesn’t > > > > > >>>>>>>>>>>>>> need > > > > > >>>>>>>>>>>>>>>>> to execute separately, you’re just adding more > > > > > >>>> tasks > > > > > >>>>>> to > > > > > >>>>>>>> the > > > > > >>>>>>>>>>> queue > > > > > >>>>>>>>>>>>> that > > > > > >>>>>>>>>>>>>>> will > > > > > >>>>>>>>>>>>>>>>> be executed when there are resources > > > > > >> available. > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> via Newton Mail [ > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>> > > > > > >>>>>> > > > > > >>>>> > > > > > >>>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.14.6&source=email_footer_2 > > > > > >>>>>>>>>>>>>>>>> ] > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 12, 2020 at 9:45 AM, Chris Palmer > > > > > >> < > > > > > >>>>>>>>>>> ch...@crpalmer.com > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>>>>>> I agree that SubDAGs are an overly complex > > > > > >>>>>> abstraction. > > > > > >>>>>>> I > > > > > >>>>>>>>>> think > > > > > >>>>>>>>>>>> what > > > > > >>>>>>>>>>>>>> is > > > > > >>>>>>>>>>>>>>>>> needed/useful is a TaskGroup concept. On a > > > > > >> high > > > > > >>>>> level > > > > > >>>>>> I > > > > > >>>>>>>>> think > > > > > >>>>>>>>>>> you > > > > > >>>>>>>>>>>>> want > > > > > >>>>>>>>>>>>>>>>> this > > > > > >>>>>>>>>>>>>>>>> functionality: > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> - Tasks can be added to a TaskGroup > > > > > >>>>>>>>>>>>>>>>> - You *can* have dependencies between Tasks in > > > > > >>> the > > > > > >>>>>> same > > > > > >>>>>>>>>>> TaskGroup, > > > > > >>>>>>>>>>>>> but > > > > > >>>>>>>>>>>>>>>>> *cannot* have dependencies between a Task in a > > > > > >>>>>> TaskGroup > > > > > >>>>>>>> and > > > > > >>>>>>>>>>>> either > > > > > >>>>>>>>>>>>> a > > > > > >>>>>>>>>>>>>>>>> Task in a different TaskGroup or a Task not in > > > > > >>> any > > > > > >>>>>> group > > > > > >>>>>>>>>>>>>>>>> - You *can* have dependencies between a > > > > > >>> TaskGroup > > > > > >>>>> and > > > > > >>>>>>>> either > > > > > >>>>>>>>>>> other > > > > > >>>>>>>>>>>>>>>>> TaskGroups or Tasks not in any group > > > > > >>>>>>>>>>>>>>>>> - The UI will by default render a TaskGroup > > > > > >> as a > > > > > >>>>>> single > > > > > >>>>>>>>>>> "object", > > > > > >>>>>>>>>>>>> but > > > > > >>>>>>>>>>>>>>>>> which you expand or zoom into in some way > > > > > >>>>>>>>>>>>>>>>> - You'd need some way to determine what the > > > > > >>>> "status" > > > > > >>>>>> of > > > > > >>>>>>> a > > > > > >>>>>>>>>>>> TaskGroup > > > > > >>>>>>>>>>>>>> was > > > > > >>>>>>>>>>>>>>>>> at least for UI display purposes > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> Not sure if it would need to be a top level > > > > > >>> object > > > > > >>>>>> with > > > > > >>>>>>>> its > > > > > >>>>>>>>>> own > > > > > >>>>>>>>>>>>>> database > > > > > >>>>>>>>>>>>>>>>> table and model or just another attribute on > > > > > >>>> tasks. > > > > > >>>>> I > > > > > >>>>>>>> think > > > > > >>>>>>>>>> you > > > > > >>>>>>>>>>>>> could > > > > > >>>>>>>>>>>>>>>>> build > > > > > >>>>>>>>>>>>>>>>> it in a way such that from the schedulers > > > > > >> point > > > > > >>> of > > > > > >>>>>> view > > > > > >>>>>>> a > > > > > >>>>>>>>> DAG > > > > > >>>>>>>>>>> with > > > > > >>>>>>>>>>>>>>>>> TaskGroups doesn't get treated any > > > > > >> differently. > > > > > >>> So > > > > > >>>>> it > > > > > >>>>>>>> really > > > > > >>>>>>>>>>> just > > > > > >>>>>>>>>>>>>>> becomes > > > > > >>>>>>>>>>>>>>>>> a > > > > > >>>>>>>>>>>>>>>>> shortcut for setting dependencies between sets > > > > > >>> of > > > > > >>>>>> Tasks, > > > > > >>>>>>>> and > > > > > >>>>>>>>>>>> allows > > > > > >>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>> UI > > > > > >>>>>>>>>>>>>>>>> to simplify the render of the DAG structure. > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> Chris > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 12, 2020 at 12:12 PM Dan Davydov > > > > > >>>>>>>>>>>>>>> <ddavy...@twitter.com.invalid > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> Agree with James (and think it's actually > > > > > >> the > > > > > >>>> more > > > > > >>>>>>>>> important > > > > > >>>>>>>>>>>> issue > > > > > >>>>>>>>>>>>>> to > > > > > >>>>>>>>>>>>>>>>> fix), > > > > > >>>>>>>>>>>>>>>>>> but I am still convinced Ash' idea is the > > > > > >>> right > > > > > >>>>> way > > > > > >>>>>>>>> forward > > > > > >>>>>>>>>>>> (just > > > > > >>>>>>>>>>>>> it > > > > > >>>>>>>>>>>>>>>>> might > > > > > >>>>>>>>>>>>>>>>>> require a bit more work to deprecate than > > > > > >>> adding > > > > > >>>>>>> visual > > > > > >>>>>>>>>>> grouping > > > > > >>>>>>>>>>>>> in > > > > > >>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>> UI). > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> There was a previous thread about this FYI > > > > > >>> with > > > > > >>>>> more > > > > > >>>>>>>>> context > > > > > >>>>>>>>>>> on > > > > > >>>>>>>>>>>>> why > > > > > >>>>>>>>>>>>>>>>> subdags > > > > > >>>>>>>>>>>>>>>>>> are bad and potential solutions: > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>> > > > > > >>>>>>> > > > > > >> > https://www.mail-archive.com/dev@airflow.apache.org/msg01202.html > > > > > >>>>>>>>>>>>>> . A > > > > > >>>>>>>>>>>>>>>>>> solution I outline there to Jame's problem > > > > > >> is > > > > > >>>> e.g. > > > > > >>>>>>>>> enabling > > > > > >>>>>>>>>>> the > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> operator > > > > > >>>>>>>>>>>>>>>>>> for Airflow operators to work with DAGs as > > > > > >>>> well. I > > > > > >>>>>> see > > > > > >>>>>>>>> this > > > > > >>>>>>>>>>>> being > > > > > >>>>>>>>>>>>>>>>> separate > > > > > >>>>>>>>>>>>>>>>>> from Ash' solution for DAG grouping in the > > > > > >> UI > > > > > >>>> but > > > > > >>>>>> one > > > > > >>>>>>> of > > > > > >>>>>>>>> the > > > > > >>>>>>>>>>> two > > > > > >>>>>>>>>>>>>> items > > > > > >>>>>>>>>>>>>>>>>> required to replace all existing subdag > > > > > >>>>>> functionality. > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> I've been working with subdags for 3 years > > > > > >> and > > > > > >>>>> they > > > > > >>>>>>> are > > > > > >>>>>>>>>>> always a > > > > > >>>>>>>>>>>>>> giant > > > > > >>>>>>>>>>>>>>>>> pain > > > > > >>>>>>>>>>>>>>>>>> to use. They are a constant source of user > > > > > >>>>> confusion > > > > > >>>>>>> and > > > > > >>>>>>>>>>>> breakages > > > > > >>>>>>>>>>>>>>>>> during > > > > > >>>>>>>>>>>>>>>>>> upgrades. Would love to see them gone :). > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> On Fri, Jun 12, 2020 at 11:11 AM James > > > > > >> Coder < > > > > > >>>>>>>>>>>> jcode...@gmail.com> > > > > > >>>>>>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> I'm not sure I totally agree it's just a > > > > > >> UI > > > > > >>>>>>> concept. I > > > > > >>>>>>>>> use > > > > > >>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>> subdag > > > > > >>>>>>>>>>>>>>>>>>> operator to simplify dependencies too. If > > > > > >>> you > > > > > >>>>>> have a > > > > > >>>>>>>>> group > > > > > >>>>>>>>>>> of > > > > > >>>>>>>>>>>>>> tasks > > > > > >>>>>>>>>>>>>>>>> that > > > > > >>>>>>>>>>>>>>>>>>> need to finish before another group of > > > > > >> tasks > > > > > >>>>>> start, > > > > > >>>>>>>>> using > > > > > >>>>>>>>>> a > > > > > >>>>>>>>>>>>> subdag > > > > > >>>>>>>>>>>>>>> is > > > > > >>>>>>>>>>>>>>>>> a > > > > > >>>>>>>>>>>>>>>>>>> pretty quick way to set those dependencies > > > > > >>>> and I > > > > > >>>>>>> think > > > > > >>>>>>>>>> also > > > > > >>>>>>>>>>>> make > > > > > >>>>>>>>>>>>>> it > > > > > >>>>>>>>>>>>>>>>>> easier > > > > > >>>>>>>>>>>>>>>>>>> to follow the dag code. > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> On Fri, Jun 12, 2020 at 9:53 AM Kyle > > > > > >> Hamlin > > > > > >>> < > > > > > >>>>>>>>>>>>> hamlin...@gmail.com> > > > > > >>>>>>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>> I second Ash’s grouping concept. > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>> On Fri, Jun 12, 2020 at 5:10 AM Ash > > > > > >>>>>> Berlin-Taylor > > > > > >>>>>>> < > > > > > >>>>>>>>>>>>>> a...@apache.org > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> Question: > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> Do we even need the SubDagOperator > > > > > >>>> anymore? > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> Would removing it entirely and just > > > > > >>>>> replacing > > > > > >>>>>> it > > > > > >>>>>>>>> with > > > > > >>>>>>>>>> a > > > > > >>>>>>>>>>> UI > > > > > >>>>>>>>>>>>>>>>> grouping > > > > > >>>>>>>>>>>>>>>>>>>>> concept be conceptually simpler, less > > > > > >> to > > > > > >>>> get > > > > > >>>>>>>> wrong, > > > > > >>>>>>>>>> and > > > > > >>>>>>>>>>>>> closer > > > > > >>>>>>>>>>>>>>> to > > > > > >>>>>>>>>>>>>>>>>> what > > > > > >>>>>>>>>>>>>>>>>>>>> users actually want to achieve with > > > > > >>>> subdags? > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> With your proposed change, tasks in > > > > > >>>> subdags > > > > > >>>>>>> could > > > > > >>>>>>>>>> start > > > > > >>>>>>>>>>>>>> running > > > > > >>>>>>>>>>>>>>> in > > > > > >>>>>>>>>>>>>>>>>>>>> parallel (a good change) -- so should > > > > > >> we > > > > > >>>> not > > > > > >>>>>>> also > > > > > >>>>>>>>> just > > > > > >>>>>>>>>>>>>>> _enitrely_ > > > > > >>>>>>>>>>>>>>>>>>> remove > > > > > >>>>>>>>>>>>>>>>>>>>> the concept of a sub dag and replace > > > > > >> it > > > > > >>>> with > > > > > >>>>>>>>> something > > > > > >>>>>>>>>>>>>> simpler. > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> Problems with subdags (I think. I > > > > > >>> haven't > > > > > >>>>> used > > > > > >>>>>>>> them > > > > > >>>>>>>>>>>>>> extensively > > > > > >>>>>>>>>>>>>>> so > > > > > >>>>>>>>>>>>>>>>>> may > > > > > >>>>>>>>>>>>>>>>>>>>> be wrong on some of these): > > > > > >>>>>>>>>>>>>>>>>>>>> - They need their own dag_id, but it > > > > > >>>> has(?) > > > > > >>>>> to > > > > > >>>>>>> be > > > > > >>>>>>>> of > > > > > >>>>>>>>>> the > > > > > >>>>>>>>>>>>> form > > > > > >>>>>>>>>>>>>>>>>>>>> `parent_dag_id.subdag_id`. > > > > > >>>>>>>>>>>>>>>>>>>>> - They need their own > > > > > >> schedule_interval, > > > > > >>>> but > > > > > >>>>>> it > > > > > >>>>>>>> has > > > > > >>>>>>>>> to > > > > > >>>>>>>>>>>> match > > > > > >>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>> parent > > > > > >>>>>>>>>>>>>>>>>>>> dag > > > > > >>>>>>>>>>>>>>>>>>>>> - Sub dags can be paused on their own. > > > > > >>>> (Does > > > > > >>>>>> it > > > > > >>>>>>>> make > > > > > >>>>>>>>>>> sense > > > > > >>>>>>>>>>>>> to > > > > > >>>>>>>>>>>>>> do > > > > > >>>>>>>>>>>>>>>>>> this? > > > > > >>>>>>>>>>>>>>>>>>>>> Pausing just a sub dag would mean the > > > > > >>> sub > > > > > >>>>> dag > > > > > >>>>>>>> would > > > > > >>>>>>>>>>> never > > > > > >>>>>>>>>>>>>>>>> execute, so > > > > > >>>>>>>>>>>>>>>>>>>>> the SubDagOperator would fail too. > > > > > >>>>>>>>>>>>>>>>>>>>> - You had to choose the executor to > > > > > >>>>> operator a > > > > > >>>>>>>>> subdag > > > > > >>>>>>>>>>> with > > > > > >>>>>>>>>>>>> -- > > > > > >>>>>>>>>>>>>>>>> always > > > > > >>>>>>>>>>>>>>>>>> a > > > > > >>>>>>>>>>>>>>>>>>>>> bit of a kludge. > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> Thoughts? > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> -ash > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> On Jun 12 2020, at 12:01 pm, Ash > > > > > >>>>>> Berlin-Taylor < > > > > > >>>>>>>>>>>>>> a...@apache.org> > > > > > >>>>>>>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Workon sub-dags is much needed, I'm > > > > > >>>>> excited > > > > > >>>>>> to > > > > > >>>>>>>> see > > > > > >>>>>>>>>> how > > > > > >>>>>>>>>>>>> this > > > > > >>>>>>>>>>>>>>>>>>> progresses. > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> - *Unpack SubDags during dag > > > > > >>> parsing*: > > > > > >>>>> This > > > > > >>>>>>>>>> rewrites > > > > > >>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>> *DagBag.bag_dag* > > > > > >>>>>>>>>>>>>>>>>>>>>>> method to unpack subdag while > > > > > >>> parsing, > > > > > >>>>> and > > > > > >>>>>> it > > > > > >>>>>>>>> will > > > > > >>>>>>>>>>>> give a > > > > > >>>>>>>>>>>>>>> flat > > > > > >>>>>>>>>>>>>>>>>>>>>>> structure at > > > > > >>>>>>>>>>>>>>>>>>>>>>> the task level > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> The serialized_dag representation > > > > > >>>> already > > > > > >>>>>> does > > > > > >>>>>>>>> this > > > > > >>>>>>>>>> I > > > > > >>>>>>>>>>>>> think. > > > > > >>>>>>>>>>>>>>> At > > > > > >>>>>>>>>>>>>>>>>> least > > > > > >>>>>>>>>>>>>>>>>>>> if > > > > > >>>>>>>>>>>>>>>>>>>>>> I've understood your idea here > > > > > >>>> correctly. > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> -ash > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> On Jun 12 2020, at 9:51 am, Xinbin > > > > > >>>> Huang < > > > > > >>>>>>>>>>>>>>> bin.huan...@gmail.com > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Hi everyone, > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Sending a message to everyone and > > > > > >>>> collect > > > > > >>>>>>>>> feedback > > > > > >>>>>>>>>> on > > > > > >>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>> AIP-34 > > > > > >>>>>>>>>>>>>>>>>> on > > > > > >>>>>>>>>>>>>>>>>>>>>>> rewriting SubDagOperator. This was > > > > > >>>>>> previously > > > > > >>>>>>>>>> briefly > > > > > >>>>>>>>>>>>>>>>> mentioned in > > > > > >>>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>> discussion about what needs to be > > > > > >>> done > > > > > >>>>> for > > > > > >>>>>>>>> Airflow > > > > > >>>>>>>>>>> 2.0, > > > > > >>>>>>>>>>>>> and > > > > > >>>>>>>>>>>>>>>>> one of > > > > > >>>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>> ideas is to make SubDagOperator > > > > > >>> attach > > > > > >>>>>> tasks > > > > > >>>>>>>> back > > > > > >>>>>>>>>> to > > > > > >>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>> root > > > > > >>>>>>>>>>>>>>>>> DAG. > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> This AIP-34 focuses on solving > > > > > >>>>>> SubDagOperator > > > > > >>>>>>>>>> related > > > > > >>>>>>>>>>>>>> issues > > > > > >>>>>>>>>>>>>>> by > > > > > >>>>>>>>>>>>>>>>>>>>> reattaching > > > > > >>>>>>>>>>>>>>>>>>>>>>> all tasks back to the root dag > > > > > >> while > > > > > >>>>>>> respecting > > > > > >>>>>>>>>>>>>> dependencies > > > > > >>>>>>>>>>>>>>>>>> during > > > > > >>>>>>>>>>>>>>>>>>>>>>> parsing. The original grouping > > > > > >> effect > > > > > >>>> on > > > > > >>>>>> the > > > > > >>>>>>> UI > > > > > >>>>>>>>>> will > > > > > >>>>>>>>>>> be > > > > > >>>>>>>>>>>>>>>>> achieved > > > > > >>>>>>>>>>>>>>>>>>>> through > > > > > >>>>>>>>>>>>>>>>>>>>>>> grouping related tasks by metadata. > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> This also makes the dag_factory > > > > > >>>> function > > > > > >>>>>> more > > > > > >>>>>>>>>>> reusable > > > > > >>>>>>>>>>>>>>> because > > > > > >>>>>>>>>>>>>>>>> you > > > > > >>>>>>>>>>>>>>>>>>>> don't > > > > > >>>>>>>>>>>>>>>>>>>>>>> need to have parent_dag_name and > > > > > >>>>>>> child_dag_name > > > > > >>>>>>>>> in > > > > > >>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>> function > > > > > >>>>>>>>>>>>>>>>>>>>> signature > > > > > >>>>>>>>>>>>>>>>>>>>>>> anymore. > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Changes proposed: > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> - *Unpack SubDags during dag > > > > > >>> parsing*: > > > > > >>>>> This > > > > > >>>>>>>>>> rewrites > > > > > >>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>> *DagBag.bag_dag* > > > > > >>>>>>>>>>>>>>>>>>>>>>> method to unpack subdag while > > > > > >>> parsing, > > > > > >>>>> and > > > > > >>>>>> it > > > > > >>>>>>>>> will > > > > > >>>>>>>>>>>> give a > > > > > >>>>>>>>>>>>>>> flat > > > > > >>>>>>>>>>>>>>>>>>>>>>> structure at > > > > > >>>>>>>>>>>>>>>>>>>>>>> the task level > > > > > >>>>>>>>>>>>>>>>>>>>>>> - *Simplify SubDagOperator*: The > > > > > >> new > > > > > >>>>>>>>> SubDagOperator > > > > > >>>>>>>>>>>> acts > > > > > >>>>>>>>>>>>>>> like a > > > > > >>>>>>>>>>>>>>>>>>>>>>> container and most of the original > > > > > >>>>> methods > > > > > >>>>>>> are > > > > > >>>>>>>>>>> removed. > > > > > >>>>>>>>>>>>> The > > > > > >>>>>>>>>>>>>>>>>>>>>>> signature is > > > > > >>>>>>>>>>>>>>>>>>>>>>> also changed to *subdag_factory > > > > > >> *with > > > > > >>>>>>>>> *subdag_args > > > > > >>>>>>>>>>> *and > > > > > >>>>>>>>>>>>>>>>>>>>> *subdag_kwargs*. > > > > > >>>>>>>>>>>>>>>>>>>>>>> This is similar to the > > > > > >> PythonOperator > > > > > >>>>>>>> signature. > > > > > >>>>>>>>>>>>>>>>>>>>>>> - *Add a TaskGroup model and add > > > > > >>>>>>> current_group > > > > > >>>>>>>> & > > > > > >>>>>>>>>>>>>> parent_group > > > > > >>>>>>>>>>>>>>>>>>>>> attributes > > > > > >>>>>>>>>>>>>>>>>>>>>>> to BaseOperator*: This metadata is > > > > > >>> used > > > > > >>>>> to > > > > > >>>>>>>> group > > > > > >>>>>>>>>>> tasks > > > > > >>>>>>>>>>>>> for > > > > > >>>>>>>>>>>>>>>>>>>>>>> rendering at > > > > > >>>>>>>>>>>>>>>>>>>>>>> UI level. It may potentially extend > > > > > >>>>> further > > > > > >>>>>>> to > > > > > >>>>>>>>>> group > > > > > >>>>>>>>>>>>>>> arbitrary > > > > > >>>>>>>>>>>>>>>>>>> tasks > > > > > >>>>>>>>>>>>>>>>>>>>>>> outside the context of subdag to > > > > > >>> allow > > > > > >>>>>>>>> group-level > > > > > >>>>>>>>>>>>>> operations > > > > > >>>>>>>>>>>>>>>>>>> (i.e. > > > > > >>>>>>>>>>>>>>>>>>>>>>> stop/trigger a group of task within > > > > > >>> the > > > > > >>>>>> dag) > > > > > >>>>>>>>>>>>>>>>>>>>>>> - *Webserver UI for SubDag*: > > > > > >> Proposed > > > > > >>>> UI > > > > > >>>>>>>>>> modification > > > > > >>>>>>>>>>>> to > > > > > >>>>>>>>>>>>>>> allow > > > > > >>>>>>>>>>>>>>>>>>>>>>> (un)collapse a group of tasks for a > > > > > >>>> flat > > > > > >>>>>>>>> structure > > > > > >>>>>>>>>> to > > > > > >>>>>>>>>>>>> pair > > > > > >>>>>>>>>>>>>>> with > > > > > >>>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>> first > > > > > >>>>>>>>>>>>>>>>>>>>>>> change instead of the original > > > > > >>>>> hierarchical > > > > > >>>>>>>>>>> structure. > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Please see related documents and > > > > > >> PRs > > > > > >>>> for > > > > > >>>>>>>> details: > > > > > >>>>>>>>>>>>>>>>>>>>>>> AIP: > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>> > > > > > >>>>>> > > > > > >>>>> > > > > > >>>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+Rewrite+SubDagOperator > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Original Issue: > > > > > >>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/8078 > > > > > >>>>>>>>>>>>>>>>>>>>>>> Draft PR: > > > > > >>>>>>>>>>> https://github.com/apache/airflow/pull/9243 > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Please let me know if there are any > > > > > >>>>> aspects > > > > > >>>>>>>> that > > > > > >>>>>>>>>> you > > > > > >>>>>>>>>>>>>>>>>> agree/disagree > > > > > >>>>>>>>>>>>>>>>>>>>>>> with or > > > > > >>>>>>>>>>>>>>>>>>>>>>> need more clarification (especially > > > > > >>> the > > > > > >>>>>> third > > > > > >>>>>>>>>> change > > > > > >>>>>>>>>>>>>>> regarding > > > > > >>>>>>>>>>>>>>>>>>>>> TaskGroup). > > > > > >>>>>>>>>>>>>>>>>>>>>>> Any comments are welcome and I am > > > > > >>>> looking > > > > > >>>>>>>> forward > > > > > >>>>>>>>>> to > > > > > >>>>>>>>>>>> it! > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Cheers > > > > > >>>>>>>>>>>>>>>>>>>>>>> Bin > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>> -- > > > > > >>>>>>>>>>>>>>>>>>>> Kyle Hamlin > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> -- > > > > > >>>>>>>>>>>>>>> Thanks & Regards > > > > > >>>>>>>>>>>>>>> Poornima > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> -- > > > > > >>>>>>> > > > > > >>>>>>> Jarek Potiuk > > > > > >>>>>>> Polidea <https://www.polidea.com/> | Principal Software > > > Engineer > > > > > >>>>>>> > > > > > >>>>>>> M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129 > > > > > >>>>> <+48%20660%20796%20129>> > > > > > >>>>>>> [image: Polidea] <https://www.polidea.com/> > > > > > >>>>>>> > > > > > >>>>>> > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> -- > > > > > >>>>> > > > > > >>>>> Jarek Potiuk > > > > > >>>>> Polidea <https://www.polidea.com/> | Principal Software > > Engineer > > > > > >>>>> > > > > > >>>>> M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129 > > > > > >>>>> <+48%20660%20796%20129>> > > > > > >>>>> [image: Polidea] <https://www.polidea.com/> > > > > > >>>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> -- > > > > > >>>> > > > > > >>>> *Jacob Ferriero* > > > > > >>>> > > > > > >>>> Strategic Cloud Engineer: Data Engineering > > > > > >>>> > > > > > >>>> jferri...@google.com > > > > > >>>> > > > > > >>>> 617-714-2509 > > > > > >>>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > >
