Re: [SUGGESTION]Release branch management

2020-05-12 Thread Xiaochun Liu
@ > kezhenxu94 I mean the new bug fix PR can be merged into the release branch, more people can participate in release branch. Best Regards --- DolphinScheduler(Incubator) Committer Xiaochun Liu 刘小春 liuxiaoc...@apache.org --- > 2020年5月13日 下午2:06,kezhenxu94@apache 写道

Re: [SUGGESTION]Release branch management

2020-05-12 Thread liwenhe
Yeah, I agree with you. We had better reduce PR during release, shouldn't integrate the new feature into the dev branch, and we must ensure that release and dev branch should be consistent. If so, it will not be difficult to merge in the future. Xiaochun Liu 于2020年5月13日周三 下午1:43写道: > > Hi, all >

Re: [SUGGESTION]Release branch management

2020-05-12 Thread kezhenxu94@apache
I’m totally confused why do you create a release branch if you’re going to continuously merge new codes into THIS release branch? IMO, “release branch” is for releasing and it aims to freeze the codes in some degree, new codes can be still merged into the “dev” branch, which doesn’t block the

[DISCUSS] the data source creation permission

2020-05-12 Thread wenhemin
Hi! In frontend, everyone has creation permission of the data source. This can easily lead to repeated creation of data sources, and there are different naming rules. I think, creating a data source is a low-frequency operation. So, permission should be revoked, only admin has create permission.

[SUGGESTION]Release branch management

2020-05-12 Thread Xiaochun Liu
Hi, all We encountered the following problems in the release branch management: 1. The release participation rate is low, it is difficult for community developers, or it is not easy to participate in the release process. It is mainly because everyone still fixes the bug by default, and the new

回复: How do you think Task DAG dependency and List dependency?

2020-05-12 Thread GabryWu
Until now ,DAG is one physical  concept, which means that DAG is heavy class , and introduces other classes and stored in a big json field. On the bigdata platform , DAG is not a good concept for Schedulers, which can be abandoned. If DAG was abandoned,   JOB and DEPENDENCY will simply the archi

bind to random port for master & worker

2020-05-12 Thread GabryWu
hi ,all ,for now the workers and masters will bing to a fixed port . I think we need to replace it with random port, and store the current port number  to zookeeper through heartbeat report. Do you have any suggestions?

Re: [PROPOSAL][Feature] Support plugin in Alert service

2020-05-12 Thread JUN GAO
In addition to the job can be plugged in, is there any other content that can be plugged in? If so, we need to consider the structural design in advance, because this is a very important design transformation, we can spend more time to think more JUN GAO 于2020年5月13日周三 上午10:11写道: > @hgaol There

Re: [PROPOSAL][Feature] Support plugin in Alert service

2020-05-12 Thread JUN GAO
@hgaol There are some discuss about DS Plugin . But this is an early design and can only be used as a reference . I hope it can help your design. https://github.com/apache/incubator-dolphinscheduler/issues/201 I see the model `dolphinscheduler-plugin-api` . When I want to add a ClickHouse Job

Re: [PROPOSAL][Feature] Support plugin in Alert service

2020-05-12 Thread hgaol
Hi @Gang Li, I agree with you that the plug-in module should be a common module for all plug-in design. Currently I just create the SPI and plugin interface for alert plug-in. If need more type for future, it’s very easy to define other plugins. Currently the alert service can detect the plug-i

Re: Re: [PROPOSAL][Feature] Support plugin in Alert service

2020-05-12 Thread JUN GAO
we can refer to presto plug-in, etc. 1. Design the SPI and define the interface that the plugin needs to implement 2. All plugins create new models to implement the interface in SPI. The benefits are as follows: 1. For developers, they only need to care about the interface in SPI, and do not need t

Re: [PROPOSAL][Feature] Support plugin in Alert service

2020-05-12 Thread leon bao
*> A better way is that user do not need to change the source code to support new alert method. Instead, they just need to implement some interface, and compliance with the convention, then package all(include dependencies) in a single jar file, and put the jar file in some specific place, such as

Re: Re: [PROPOSAL][Feature] Support plugin in Alert service

2020-05-12 Thread lgcareer2...@outlook.com
Hi Gao, May be I didn't express clearly,I think the dolphinscheduler-plugin module is a common module for all plugin design, It's not neccessarily to put the implement of a plugin in the dolphinscheduler-plugin module. I also agree with the following way your said,I see the pr you submited didn

回复: Re:introduce retryer for async function

2020-05-12 Thread GabryWu
I create a new PR, https://github.com/apache/incubator-dolphinscheduler/pull/2679 Because I'm not sure which methods will use the Retryer, so I just create RetryerUtils which main function is retryCall  that will retry 3 times if exceptions throw or return result is false and wait 1 second betwe

Re: How do you think Task DAG dependency and List dependency?

2020-05-12 Thread lidong dai
yes, your description is proper, thanks Best Regards --- DolphinScheduler(Incubator) PPMC Lidong Dai 代立冬 dailidon...@gmail.com --- leon bao 于2020年5月12日周二 下午5:06写道: > i think you want to show the DAG using list mode. > DS show DAG using graphic mode, that will have some

[DISCUSS] Dependent task check rules

2020-05-12 Thread wenhemin
Hello! Abort dependent task check rules, I think the current implementation is problematic. E.g. check logic: this week The current implementation: 1.get this Monday 2.the date list of this Monday to current date(asc order) 3.Iterate through the list of dates in sequence to check the status of a

Re: How do you think Task DAG dependency and List dependency?

2020-05-12 Thread leon bao
i think you want to show the DAG using list mode. DS show DAG using graphic mode, that will have some problems: 1. once the number of tasks is large, graphic mode will be confusing. 2. it is difficult to find the specified task in a complex DAG. lidong dai 于2020年5月12日周二 下午4:31写道: > I want to s

?????? ????: ??DISCUSS??Deleting the process instance does not delete the log file of the task instance

2020-05-12 Thread xingchun.chen
OK??I will test it  tanks --  -- ??: "qiaozhan...@outlook.com"https://github.com/apache/incubator-dolphinscheduler/pull/2693??commited dev-1.3.0 ??please verify ?? DolphinScheduler(Incubator)  PPMC Zhanwei Qiao ?? qia

回复: 回复: 【DISCUSS】Deleting the process instance does not delete the log file of the task instance

2020-05-12 Thread qiaozhan...@outlook.com
This PR https://github.com/apache/incubator-dolphinscheduler/pull/2693,commited dev-1.3.0 ,please verify ― DolphinScheduler(Incubator) PPMC Zhanwei Qiao 乔占卫 qiaozhan...@outlook.com 发件人: qiaozhan...@outlook.com 发送时间: 2020-05-12 16:55 收件人: dev 主题: 回复: 回复: 【DISCUSS】Deleting the pro

回复: 回复: 【DISCUSS】Deleting the process instance does not delete the log file of the task instance

2020-05-12 Thread qiaozhan...@outlook.com
Hi xinchun I have commited,please test Thx ― DolphinScheduler(Incubator) PPMC Zhanwei Qiao 乔占卫 qiaozhan...@outlook.com 发件人: qiaozhan...@outlook.com 发送时间: 2020-05-11 17:39 收件人: dev 主题: 回复: 【DISCUSS】Deleting the process instance does not delete the log file of the task instance

Re: How do you think Task DAG dependency and List dependency?

2020-05-12 Thread lidong dai
I want to say do we need to implement List dependency? Best Regards --- DolphinScheduler(Incubator) PPMC Lidong Dai 代立冬 dailidon...@gmail.com --- JUN GAO 于2020年5月12日周二 下午12:26写道: > Hi , @lidong dai > Sorry , I don`t know your mean about this . > You want to discuss ho