yes， no impact on master
Best Regards --------------- DolphinScheduler(Incubator) PPMC Lidong Dai 代立冬 [email protected] --------------- gabrywu <[email protected]> 于2020年5月21日周四 下午4:21写道： > should have any impact on maste ---> should not have any impact on maste > > gabrywu <[email protected]> 于2020年5月21日周四 下午4:19写道： > > > 100 sql tasks belonging to a business package. So heavyweight business > > logic ! It will be difficult to maintain. > > Maybe we can create a new JOB type called DAG, which split the 100 SQLs > > depend on their data lineage. > > Meanwhile the DAG job has some limitations ,for example ,25 subtask at > > most. > > > > Too many jobs under one label should have any impact on master, jobs > > reached same schedule time will. > > After all ,Jobs under one label will not be triggered at the same time. > > > > Hemin Wen <[email protected]> 于2020年5月21日周四 上午10:34写道： > > > >> First, There are 100 (for example) sql tasks belonging to a business > >> package, Does not need to be split, > >> scheduling still has to serve the actual business. > >> > >> I agree with the concept of JOB, I understand what you mean is that the > >> master scheduling granularity is changed to job level, > >> But, If there are too many jobs under one label, it will affect the > >> execution of other label jobs, > >> So I think the current master is correct at the workflow level, parallel > >> between workflows, queued within workflows. > >> > >> I understand that this need to solve is the dependency between batch > tasks > >> and batch creation tasks. > >> The list is for easier viewing of tasks. User manual maintenance of > batch > >> tasks is a time-consuming operation. > >> So, I suggest to start with how to solve creating batch tasks and how to > >> solve dependencies between tasks. > >> > >> > >> -------------------- > >> DolphinScheduler(Incubator) Commtter > >> Hemin Wen 温合民 > >> [email protected] > >> -------------------- > >> > >> > >> GabryWu <[email protected]> 于2020年5月20日周三 下午9:58写道： > >> > >> > If DAG removed ,master only need to pick up JOBs which reached > schedule > >> > time, and dispatches them to one worker .However the JOBs will be in > the > >> > queue before dispatched . > >> > The master will be lightweight , and complement work will be easy. The > >> > JOBs which have same label will be dispatched to workers and they will > >> > execute parallelly or in a sequence relying on their DEPENDENCY > >> > > >> > ---Original--- > >> > *From:* "GabryWu"<[email protected]> > >> > *Date:* Wed, May 20, 2020 18:41 PM > >> > *To:* "dev"<[email protected]>; > >> > *Cc:* "wenhemin"<[email protected]>; > >> > *Subject:* Re: How do you think Task DAG dependency and List > dependency？ > >> > > >> > In that case ,I recommend spliting the 100 SQLs into different JOB > which > >> > has same label, each JOB has it's own business logic .Finally Add > >> > DEPENDENCY to each JOB. JOBs having same label can be shown in front > >> page > >> > through DAG > >> > > >> > > >> > ---Original--- > >> > *From:* "Hemin Wen"<[email protected]> > >> > *Date:* Wed, May 20, 2020 17:38 PM > >> > *To:* "GabryWu"<[email protected]>; > >> > *Subject:* Re: How do you think Task DAG dependency and List > dependency？ > >> > > >> > For example, A workflow has 100 sql scripts task, tasks depend on each > >> > other through data lineage. > >> > This requirement is not suitable for DAG configuration. > >> > > >> > > >> > -------------------- > >> > DolphinScheduler(Incubator) Commtter > >> > Hemin Wen 温合民 > >> > [email protected] > >> > -------------------- > >> > > >> > > >> > GabryWu <[email protected]> 于2020年5月20日周三 下午5:31写道： > >> > > >> >> what do you mean batch tasks > >> >> > >> >> ------------------ 原始邮件 ------------------ > >> >> *发件人:* "wenhemin"<[email protected]>; > >> >> *发送时间:* 2020年5月20日(星期三) 下午5:04 > >> >> *收件人:* "dev"<[email protected]>; > >> >> *主题:* Re: How do you think Task DAG dependency and List dependency？ > >> >> > >> >> I do not recommend remove DAG. > >> >> I think, DS lacks support for batch tasks. > >> >> DAG solves the expression of different types of tasks, there is no > way > >> to > >> >> express batch tasks of the same type. > >> >> > >> >> I don't know if what I understand is accurate, It is recommended to > >> >> understand in depth the problem that users > >> >> want to solve by using "List dependency”. > >> >> > >> >> -------------------- > >> >> DolphinScheduler(Incubator) Commtter > >> >> Hemin Wen 温合民 > >> >> [email protected] > >> >> -------------------- > >> >> > >> >> > >> >> lidong dai <[email protected]> 于2020年5月20日周三 下午3:40写道： > >> >> > >> >> > I know what you said, there maybe need more people to discuss this > >> >> topic, > >> >> > I want to know other's opinion, how do they think this question > >> >> > > >> >> > > >> >> > > >> >> > Best Regards > >> >> > --------------- > >> >> > DolphinScheduler(Incubator) PPMC > >> >> > Lidong Dai 代立冬 > >> >> > [email protected] > >> >> > --------------- > >> >> > > >> >> > > >> >> > GabryWu <[email protected]> 于2020年5月19日周二 上午7:19写道： > >> >> > > >> >> > > not actually. I mean refactor backend code to remove DAG, where > >> only > >> >> JOB > >> >> > > and DEPENDENCY remained > >> >> > > > >> >> > > ---Original--- > >> >> > > *From:* "lidong dai"<[email protected]> > >> >> > > *Date:* Mon, May 18, 2020 22:47 PM > >> >> > > *To:* "GabryWu"<[email protected]>; > >> >> > > *Cc:* "dev"<[email protected]>; > >> >> > > *Subject:* Re: How do you think Task DAG dependency and List > >> >> dependency？ > >> >> > > > >> >> > > I think your said is "List dependency", List dependency is also > >> DAG, > >> >> it's > >> >> > > only different in use. when use List dependency, you can add > your > >> >> > upstream > >> >> > > dependency for the task, this is convenient for hugely tasks in > one > >> >> > > workflow > >> >> > > > >> >> > > > >> >> > > Best Regards > >> >> > > --------------- > >> >> > > DolphinScheduler(Incubator) PPMC > >> >> > > Lidong Dai 代立冬 > >> >> > > [email protected] > >> >> > > --------------- > >> >> > > > >> >> > > > >> >> > > GabryWu <[email protected]> 于2020年5月13日周三 上午11:08写道： > >> >> > > > >> >> > >> Until now ,DAG is one physical concept， which means that DAG is > >> >> heavy > >> >> > >> class , and introduces other classes and stored in a big json > >> field. > >> >> > >> On the bigdata platform , DAG is not a good concept for > >> Schedulers, > >> >> > which > >> >> > >> can be abandoned. > >> >> > >> If DAG was abandoned, JOB and DEPENDENCY will simply the > >> >> architecture > >> >> > >> and make DolphinScheduler stabilize and extended easily and > >> easier to > >> >> > >> search jobs > >> >> > >> However abandoning DAG doesn't mean remove DAG graph in the > front > >> >> end, > >> >> > >> which is also an important visualization way for JOBs > >> >> > >> We can add one Job and Dependency manually ,and visualize them > in > >> DAG > >> >> > >> graph automatically > >> >> > >> > >> >> > >> > >> >> > >> > >> >> > >> ------------------ 原始邮件 ------------------ > >> >> > >> *发件人:* "lidong dai"<[email protected]>; > >> >> > >> *发送时间:* 2020年5月12日(星期二) 下午5:26 > >> >> > >> *收件人:* "dev"<[email protected]>; > >> >> > >> *主题:* Re: How do you think Task DAG dependency and List > >> dependency？ > >> >> > >> > >> >> > >> yes, your description is proper, thanks > >> >> > >> > >> >> > >> > >> >> > >> Best Regards > >> >> > >> --------------- > >> >> > >> DolphinScheduler(Incubator) PPMC > >> >> > >> Lidong Dai 代立冬 > >> >> > >> [email protected] > >> >> > >> --------------- > >> >> > >> > >> >> > >> > >> >> > >> leon bao <[email protected]> 于2020年5月12日周二 下午5:06写道： > >> >> > >> > >> >> > >> > i think you want to show the DAG using list mode. > >> >> > >> > DS show DAG using graphic mode, that will have some problems: > >> >> > >> > > >> >> > >> > 1. once the number of tasks is large, graphic mode will be > >> >> confusing. > >> >> > >> > 2. it is difficult to find the specified task in a complex > DAG. > >> >> > >> > > >> >> > >> > lidong dai <[email protected]> 于2020年5月12日周二 下午4:31写道： > >> >> > >> > > >> >> > >> > > I want to say do we need to implement List dependency？ > >> >> > >> > > > >> >> > >> > > > >> >> > >> > > > >> >> > >> > > Best Regards > >> >> > >> > > --------------- > >> >> > >> > > DolphinScheduler(Incubator) PPMC > >> >> > >> > > Lidong Dai 代立冬 > >> >> > >> > > [email protected] > >> >> > >> > > --------------- > >> >> > >> > > > >> >> > >> > > > >> >> > >> > > JUN GAO <[email protected]> 于2020年5月12日周二 下午12:26写道： > >> >> > >> > > > >> >> > >> > > > Hi , @lidong dai > >> >> > >> > > > Sorry , I don`t know your mean about this . > >> >> > >> > > > You want to discuss how to implement Task DAG dependency > and > >> >> List > >> >> > >> > > > dependency ? Or discuss what is Task DAG dependency and > List > >> >> > >> > dependency ? > >> >> > >> > > > you can show us what`s your need and ideas , So that we > can > >> >> know > >> >> > the > >> >> > >> > > topic > >> >> > >> > > > of discussion. > >> >> > >> > > > > >> >> > >> > > > Thank you ! > >> >> > >> > > > > >> >> > >> > > > lidong dai <[email protected]> 于2020年5月11日周一 > 下午10:34写道： > >> >> > >> > > > > >> >> > >> > > > > hi ， > >> >> > >> > > > > > >> >> > >> > > > > Task dependency between upstream and downstream > called > >> >> List > >> >> > >> > > > dependency > >> >> > >> > > > > do you have ideas about this topic? > >> >> > >> > > > > > >> >> > >> > > > > > >> >> > >> > > > > > >> >> > >> > > > > Best Regards > >> >> > >> > > > > --------------- > >> >> > >> > > > > DolphinScheduler(Incubator) PPMC > >> >> > >> > > > > Lidong Dai 代立冬 > >> >> > >> > > > > [email protected] > >> >> > >> > > > > --------------- > >> >> > >> > > > > > >> >> > >> > > > > >> >> > >> > > > > >> >> > >> > > > -- > >> >> > >> > > > > >> >> > >> > > > DolphinScheduler(Incubator) PPMC > >> >> > >> > > > Jun Gao 高俊 > >> >> > >> > > > [email protected] > >> >> > >> > > > > >> >> > >> > > > >> >> > >> > > >> >> > >> > > >> >> > >> > -- > >> >> > >> > DolphinScheduler(Incubator) PPMC > >> >> > >> > BaoLiang 鲍亮 > >> >> > >> > [email protected] > >> >> > >> > > >> >> > >> > >> >> > >> > >> >> > > >> >> > >> >> > >> > > >
