should have any impact on maste ---> should not have any impact on maste
gabrywu <[email protected]> 于2020年5月21日周四 下午4:19写道： > 100 sql tasks belonging to a business package. So heavyweight business > logic ! It will be difficult to maintain. > Maybe we can create a new JOB type called DAG, which split the 100 SQLs > depend on their data lineage. > Meanwhile the DAG job has some limitations ,for example ,25 subtask at > most. > > Too many jobs under one label should have any impact on master, jobs > reached same schedule time will. > After all ,Jobs under one label will not be triggered at the same time. > > Hemin Wen <[email protected]> 于2020年5月21日周四 上午10:34写道： > >> First, There are 100 (for example) sql tasks belonging to a business >> package, Does not need to be split, >> scheduling still has to serve the actual business. >> >> I agree with the concept of JOB, I understand what you mean is that the >> master scheduling granularity is changed to job level, >> But, If there are too many jobs under one label, it will affect the >> execution of other label jobs, >> So I think the current master is correct at the workflow level, parallel >> between workflows, queued within workflows. >> >> I understand that this need to solve is the dependency between batch tasks >> and batch creation tasks. >> The list is for easier viewing of tasks. User manual maintenance of batch >> tasks is a time-consuming operation. >> So, I suggest to start with how to solve creating batch tasks and how to >> solve dependencies between tasks. >> >> >> -------------------- >> DolphinScheduler(Incubator) Commtter >> Hemin Wen 温合民 >> [email protected] >> -------------------- >> >> >> GabryWu <[email protected]> 于2020年5月20日周三 下午9:58写道： >> >> > If DAG removed ,master only need to pick up JOBs which reached schedule >> > time, and dispatches them to one worker .However the JOBs will be in the >> > queue before dispatched . >> > The master will be lightweight , and complement work will be easy. The >> > JOBs which have same label will be dispatched to workers and they will >> > execute parallelly or in a sequence relying on their DEPENDENCY >> > >> > ---Original--- >> > *From:* "GabryWu"<[email protected]> >> > *Date:* Wed, May 20, 2020 18:41 PM >> > *To:* "dev"<[email protected]>; >> > *Cc:* "wenhemin"<[email protected]>; >> > *Subject:* Re: How do you think Task DAG dependency and List dependency？ >> > >> > In that case ,I recommend spliting the 100 SQLs into different JOB which >> > has same label, each JOB has it's own business logic .Finally Add >> > DEPENDENCY to each JOB. JOBs having same label can be shown in front >> page >> > through DAG >> > >> > >> > ---Original--- >> > *From:* "Hemin Wen"<[email protected]> >> > *Date:* Wed, May 20, 2020 17:38 PM >> > *To:* "GabryWu"<[email protected]>; >> > *Subject:* Re: How do you think Task DAG dependency and List dependency？ >> > >> > For example, A workflow has 100 sql scripts task, tasks depend on each >> > other through data lineage. >> > This requirement is not suitable for DAG configuration. >> > >> > >> > -------------------- >> > DolphinScheduler(Incubator) Commtter >> > Hemin Wen 温合民 >> > [email protected] >> > -------------------- >> > >> > >> > GabryWu <[email protected]> 于2020年5月20日周三 下午5:31写道： >> > >> >> what do you mean batch tasks >> >> >> >> ------------------ 原始邮件 ------------------ >> >> *发件人:* "wenhemin"<[email protected]>; >> >> *发送时间:* 2020年5月20日(星期三) 下午5:04 >> >> *收件人:* "dev"<[email protected]>; >> >> *主题:* Re: How do you think Task DAG dependency and List dependency？ >> >> >> >> I do not recommend remove DAG. >> >> I think, DS lacks support for batch tasks. >> >> DAG solves the expression of different types of tasks, there is no way >> to >> >> express batch tasks of the same type. >> >> >> >> I don't know if what I understand is accurate, It is recommended to >> >> understand in depth the problem that users >> >> want to solve by using "List dependency”. >> >> >> >> -------------------- >> >> DolphinScheduler(Incubator) Commtter >> >> Hemin Wen 温合民 >> >> [email protected] >> >> -------------------- >> >> >> >> >> >> lidong dai <[email protected]> 于2020年5月20日周三 下午3:40写道： >> >> >> >> > I know what you said, there maybe need more people to discuss this >> >> topic, >> >> > I want to know other's opinion, how do they think this question >> >> > >> >> > >> >> > >> >> > Best Regards >> >> > --------------- >> >> > DolphinScheduler(Incubator) PPMC >> >> > Lidong Dai 代立冬 >> >> > [email protected] >> >> > --------------- >> >> > >> >> > >> >> > GabryWu <[email protected]> 于2020年5月19日周二 上午7:19写道： >> >> > >> >> > > not actually. I mean refactor backend code to remove DAG, where >> only >> >> JOB >> >> > > and DEPENDENCY remained >> >> > > >> >> > > ---Original--- >> >> > > *From:* "lidong dai"<[email protected]> >> >> > > *Date:* Mon, May 18, 2020 22:47 PM >> >> > > *To:* "GabryWu"<[email protected]>; >> >> > > *Cc:* "dev"<[email protected]>; >> >> > > *Subject:* Re: How do you think Task DAG dependency and List >> >> dependency？ >> >> > > >> >> > > I think your said is "List dependency", List dependency is also >> DAG, >> >> it's >> >> > > only different in use. when use List dependency, you can add your >> >> > upstream >> >> > > dependency for the task, this is convenient for hugely tasks in one >> >> > > workflow >> >> > > >> >> > > >> >> > > Best Regards >> >> > > --------------- >> >> > > DolphinScheduler(Incubator) PPMC >> >> > > Lidong Dai 代立冬 >> >> > > [email protected] >> >> > > --------------- >> >> > > >> >> > > >> >> > > GabryWu <[email protected]> 于2020年5月13日周三 上午11:08写道： >> >> > > >> >> > >> Until now ,DAG is one physical concept， which means that DAG is >> >> heavy >> >> > >> class , and introduces other classes and stored in a big json >> field. >> >> > >> On the bigdata platform , DAG is not a good concept for >> Schedulers, >> >> > which >> >> > >> can be abandoned. >> >> > >> If DAG was abandoned, JOB and DEPENDENCY will simply the >> >> architecture >> >> > >> and make DolphinScheduler stabilize and extended easily and >> easier to >> >> > >> search jobs >> >> > >> However abandoning DAG doesn't mean remove DAG graph in the front >> >> end, >> >> > >> which is also an important visualization way for JOBs >> >> > >> We can add one Job and Dependency manually ,and visualize them in >> DAG >> >> > >> graph automatically >> >> > >> >> >> > >> >> >> > >> >> >> > >> ------------------ 原始邮件 ------------------ >> >> > >> *发件人:* "lidong dai"<[email protected]>; >> >> > >> *发送时间:* 2020年5月12日(星期二) 下午5:26 >> >> > >> *收件人:* "dev"<[email protected]>; >> >> > >> *主题:* Re: How do you think Task DAG dependency and List >> dependency？ >> >> > >> >> >> > >> yes, your description is proper, thanks >> >> > >> >> >> > >> >> >> > >> Best Regards >> >> > >> --------------- >> >> > >> DolphinScheduler(Incubator) PPMC >> >> > >> Lidong Dai 代立冬 >> >> > >> [email protected] >> >> > >> --------------- >> >> > >> >> >> > >> >> >> > >> leon bao <[email protected]> 于2020年5月12日周二 下午5:06写道： >> >> > >> >> >> > >> > i think you want to show the DAG using list mode. >> >> > >> > DS show DAG using graphic mode, that will have some problems: >> >> > >> > >> >> > >> > 1. once the number of tasks is large, graphic mode will be >> >> confusing. >> >> > >> > 2. it is difficult to find the specified task in a complex DAG. >> >> > >> > >> >> > >> > lidong dai <[email protected]> 于2020年5月12日周二 下午4:31写道： >> >> > >> > >> >> > >> > > I want to say do we need to implement List dependency？ >> >> > >> > > >> >> > >> > > >> >> > >> > > >> >> > >> > > Best Regards >> >> > >> > > --------------- >> >> > >> > > DolphinScheduler(Incubator) PPMC >> >> > >> > > Lidong Dai 代立冬 >> >> > >> > > [email protected] >> >> > >> > > --------------- >> >> > >> > > >> >> > >> > > >> >> > >> > > JUN GAO <[email protected]> 于2020年5月12日周二 下午12:26写道： >> >> > >> > > >> >> > >> > > > Hi , @lidong dai >> >> > >> > > > Sorry , I don`t know your mean about this . >> >> > >> > > > You want to discuss how to implement Task DAG dependency and >> >> List >> >> > >> > > > dependency ? Or discuss what is Task DAG dependency and List >> >> > >> > dependency ? >> >> > >> > > > you can show us what`s your need and ideas , So that we can >> >> know >> >> > the >> >> > >> > > topic >> >> > >> > > > of discussion. >> >> > >> > > > >> >> > >> > > > Thank you ! >> >> > >> > > > >> >> > >> > > > lidong dai <[email protected]> 于2020年5月11日周一 下午10:34写道： >> >> > >> > > > >> >> > >> > > > > hi ， >> >> > >> > > > > >> >> > >> > > > > Task dependency between upstream and downstream called >> >> List >> >> > >> > > > dependency >> >> > >> > > > > do you have ideas about this topic? >> >> > >> > > > > >> >> > >> > > > > >> >> > >> > > > > >> >> > >> > > > > Best Regards >> >> > >> > > > > --------------- >> >> > >> > > > > DolphinScheduler(Incubator) PPMC >> >> > >> > > > > Lidong Dai 代立冬 >> >> > >> > > > > [email protected] >> >> > >> > > > > --------------- >> >> > >> > > > > >> >> > >> > > > >> >> > >> > > > >> >> > >> > > > -- >> >> > >> > > > >> >> > >> > > > DolphinScheduler(Incubator) PPMC >> >> > >> > > > Jun Gao 高俊 >> >> > >> > > > [email protected] >> >> > >> > > > >> >> > >> > > >> >> > >> > >> >> > >> > >> >> > >> > -- >> >> > >> > DolphinScheduler(Incubator) PPMC >> >> > >> > BaoLiang 鲍亮 >> >> > >> > [email protected] >> >> > >> > >> >> > >> >> >> > >> >> >> > >> >> >> >> >> >
