Cool. Agree+1 Sent from my iPhone
> On May 14, 2020, at 22:27, zhangchunyang <[email protected]> wrote: > > 赞 > > > > > > > > > > > > > > > > > > At 2020-05-14 19:51:26, "lidong dai" <[email protected]> wrote: >> I agree with you, I think a dolphinscheduler-plugin module and sub module >> under dolphinscheduler-plugin will be better >> >> >> Best Regards >> --------------- >> DolphinScheduler(Incubator) PPMC >> Lidong Dai 代立冬 >> [email protected] >> --------------- >> >> >> leon bao <[email protected]> 于2020年5月14日周四 下午7:18写道： >> >>> I think a common plugin module is necessary, but I don't recommend putting >>> all the plugins in the root directory, which looks complicated. >>> We can have ds-plugin module in root with many sub modules, such as >>> alert-plugin / task-plugin /... >>> >>> hgaol <[email protected]> 于2020年5月13日周三 下午8:24写道： >>> >>>> Hi @Jun Gao, >>>> >>>> I went through the link you sent, and I agree that if the job was plugged >>>> in will be better. But I’m not quite familiar with DS job currently, so I >>>> may not able to answer your question about how to add a ClickHouse >>> plugin. >>>> >>>> I’ve written interfaces about Alert plugin to support different alert >>>> type. If you want to implement alert plugin, you can refer to the PR >>>> mentioned before. >>>> >>>> For job plugin, I think you can leverage the plugin design and enhance it >>>> to support job plugin. You may need to define a JobProvider, a JobPlugin >>>> interface, option with a AbstractJobPlugin. >>>> >>>> Currently, this PR is about Alert plugin. For job plugin or other plugin >>>> design, I think it may be many topics and details to discuss. >>>> >>>> >>>> Thanks, >>>> Han Gao >>>> >>>> >>>>> On May 13, 2020, at 10:19, JUN GAO <[email protected]> wrote: >>>>> >>>>> In addition to the job can be plugged in, is there any other content that >>>>> can be plugged in? If so, we need to consider the structural design in >>>>> advance, because this is a very important design transformation, we can >>>>> spend more time to think more >>>>> >>>>> JUN GAO <[email protected]> 于2020年5月13日周三 上午10:11写道： >>>>> >>>>>> @hgaol <[email protected]> There are some discuss about DS Plugin >>>>>> . But this is an early design and can only be used as a reference . I >>> hope >>>>> it can help your design. >>>>> https://github.com/apache/incubator-dolphinscheduler/issues/201 >>>>> >>>>> I see the model `dolphinscheduler-plugin-api` . When I want to add a >>>>> ClickHouse Job Plugin, should I create a new module called >>>>> plugin-clickhouse-job directly in the root directory of the project >>>>> (because I don’t know if only the job can do plugin)? Or we create a >>> moudle >>>>> called ds-job-plugin in the root directory, and then create a moudle >>> called >>>>> clickhouse-job under this moudle? Can you make a standardized design in >>>>> this regard? Everyone can follow this design afterwards. >>>>> >>>>> @leon bao @Gang li Do you guys have any ideas ？ >>>>> >>>>> >>>>> >>>>> hgaol <[email protected]> 于2020年5月12日周二 下午9:33写道： >>>>> >>>>>> Hi @Gang Li, I agree with you that the plug-in module should be a >>> common >>>>>> module for all plug-in design. Currently I just create the SPI and >>> plugin >>>>>> interface for alert plug-in. If need more type for future, it’s very >>> easy >>>>>> to define other plugins. >>>>>> >>>>>> Currently the alert service can detect the plug-in jars and load it. >>> But >>>>>> I didn’t implement how to collaborate with front-end about the alert >>> types >>>>>> API and how many properties a plugin should have yet. For example, if >>> we >>>>>> add a webhook plugin, we should have the http method(form select), >>> url(form >>>>>> text), etc… So the plugin interface may change to achieve this goal. >>>>>> >>>>>> Hi @leon bao, as mentioned above, I agree to make the plugin module as >>> a >>>>>> common module for all plugin design, is that what you mean? >>>>>> >>>>>>> why don't we put the alert plug-in as an alert module? >>>>>> I’m not clear about it, do you mean we could just put alert plugin in >>>>>> alert module? I decide to put frequently used plugins in alert module, >>> as >>>>>> for others like SMS(which may has different api), make it as a plugin >>> jar >>>>>> file. >>>>>> >>>>>> Hi @Jun Gao, yes! I also use the SPI feature, it is quite convenient. >>>>>> Developers just need to implement the interfaces in SPI, which are in >>>>>> `dolphinscheduler-plugin-api` module. >>>>>> >>>>>> Hi all, thanks for your careful review! Here is some link. >>>>>> >>>>>> PR about plugin management. >>>>>> <https://github.com/apache/incubator-dolphinscheduler/pull/2572> >>>>>> PR about how to develop a simple plugin. >>>>>> <https://github.com/apache/incubator-dolphinscheduler/pull/2593> >>>>>> README about how to develop a simple plugin. >>>>>> < >>> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07 >>>> >>>>>> >>>>>> Thanks in advance! >>>>>> >>>>>> Thanks, >>>>>> hgaol >>>>>> >>>>>> On May 12, 2020, at 18:57, JUN GAO <[email protected]> wrote: >>>>>> >>>>>> we can refer to presto plug-in, etc. >>>>>> 1. Design the SPI and define the interface that the plugin needs to >>>>>> implement >>>>>> 2. All plugins create new models to implement the interface in SPI. >>>>>> The benefits are as follows: >>>>>> 1. For developers, they only need to care about the interface in SPI, >>>>>> and do not need to care about other people's code, so it is easier to >>> get >>>>>> started developing plug-ins. >>>>>> 2. For code reviewers, to improve the efficiency of review, independent >>>>>> models will be more easily reviewed. If there is a problem with the >>> code, >>>>>> it is easy to roll back. >>>>>> 3. For users, there are more flexible options. For example, some people >>>>>> only want to use the A plug-in but not the B plug-in. >>>>>> >>>>>> Thank you ! >>>>>> >>>>>> [email protected] <[email protected]> 于2020年5月12日周二 >>>>>> 下午6:31写道： >>>>>> >>>>>>> Hi Gao, >>>>>>> >>>>>>> May be I didn't express clearly,I think the dolphinscheduler-plugin >>>>>>> module is a common module for all plugin design, >>>>>>> It's not neccessarily to put the implement of a plugin in the >>>>>>> dolphinscheduler-plugin module. >>>>>>> >>>>>>> I also agree with the following way your said,I see the pr you >>> submited >>>>>>> didn't match it now,Will you give a deep design about the following >>> way? >>>>>>> >>>>>>>> A better way is that user do not need to change the source code to >>>>>>> support new alert method. Instead, they just need to implement some >>>>>>> interface, and compliance with the convention, then package >>> all(include >>>>>>> dependencies) in a single jar file, and put the jar file in some >>> specific >>>>>>> place, such as $DS_HOME/plugins. After that, when start Alert >>> service, the >>>>>>> plugins can be recognized and loaded. So user can choose the alert >>> method >>>>>>> in the portal. >>>>>>> >>>>>>> >>>>>>> >>>>>>> DolphinScheduler(Incubator) PPMC >>>>>>> Gang Li 李岗 >>>>>>> >>>>>>> [email protected]<mailto:[email protected]> >>>>>>> >>>>>>> From: hgaol >>>>>>> Date: 2020-05-11 19:00 >>>>>>> To: [email protected] >>>>>>> CC: dev >>>>>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >>>>>>> Hi Gang, >>>>>>> >>>>>>> I agree with that making examples as a pure folder, thanks for your >>>>>>> suggestion. >>>>>>> >>>>>>> For dolphinscheduler-plugin-api, do you mean it’s better to rename >>> this >>>>>>> module? I’m not clear about ‘all plugin function need to be >>> implemented >>>>>>> here’, do you mean we should put implemented plugin here? >>>>>>> >>>>>>> BTW, I’ve write a simple doc about how to implement an alert plugin. >>>>>>> >>> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07 >>>>>>> >>>>>>> Also, I think for some frequently used plugins(email/webhook/...), we >>>>>>> could just implement in alert service. Here is the former PR about it. >>>>>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 >>>>>>> >>>>>>> Thanks for your careful review! >>>>>>> >>>>>>> Thanks, >>>>>>> hgaol >>>>>>> >>>>>>> On May 11, 2020, at 18:09, [email protected] wrote: >>>>>>> >>>>>>> Hi Gao, >>>>>>> >>>>>>> I agree with the following you said,If this is a plugin module,all the >>>>>>> plugin function need to be implemented here. >>>>>>> Now the name is dolphinscheduler-plugin-api,I think we can change the >>>>>>> module name, Such as `dolphinscheduler-plugin`. >>>>>>>> I think dolphinscheduler-plugin-api could not only support alert >>>>>>> plugin, but also support java job task for future. That’s another >>> topic. >>>>>>> >>>>>>> Yes,I think it just a folder,not a sub-module. >>>>>>> We can also look at other developers' comments on this. >>>>>>>> Do you mean the examples module should not be a sub-module in >>>>>>> dolphinscheduler project pom file, just a pure folder? >>>>>>> >>>>>>> >>>>>>> >>>>>>> DolphinScheduler(Incubator) PPMC >>>>>>> Gang Li 李岗 >>>>>>> >>>>>>> [email protected]<mailto:[email protected]> >>>>>>> >>>>>>> From: hgaol >>>>>>> Date: 2020-05-11 17:45 >>>>>>> To: dev; lgcareer2019 >>>>>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >>>>>>> Hi Gang, >>>>>>> >>>>>>> Thanks for your suggestion. >>>>>>> 1.Whether the function of module dolphinscheduler-plugin-api can be >>>>>>> puted in the alert module. >>>>>>> [hgaol] The customer should depend on this module to implement their >>>>>>> own plugin. If put it into alert service, customer should put >>>>>>> `dolphinscheduler-alert` as dependence, which is too heavy for a >>> plugin. >>>>>>> >>>>>>> Also, I think dolphinscheduler-plugin-api could not only support alert >>>>>>> plugin, but also support java job task for future. That’s another >>> topic. >>>>>>> >>>>>>> 2. I think the examples module can be a folder. >>>>>>> [hgaol] Do you mean the examples module should not be a sub-module in >>>>>>> dolphinscheduler project pom file, just a pure folder? >>>>>>> >>>>>>> Thanks, >>>>>>> hgaol >>>>>>> >>>>>>> On May 11, 2020, at 11:51, [email protected] wrote: >>>>>>> >>>>>>> Hi hgaol, >>>>>>> Sorry for not replying in time. >>>>>>> I saw you added the module dolphinscheduler-plugin-api and examples >>>>>>> module,here are my two points want to discuss. >>>>>>> >>>>>>> 1、Whether the function of module dolphinscheduler-plugin-api can be >>>>>>> puted in the alert module. >>>>>>> 2、I think the examples module can be a folder. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> DolphinScheduler(Incubator) PPMC >>>>>>> Gang Li 李岗 >>>>>>> >>>>>>> [email protected]<mailto:[email protected]> >>>>>>> >>>>>>> From: hgaol >>>>>>> Date: 2020-04-29 23:37 >>>>>>> To: dev >>>>>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >>>>>>> Hi folks, >>>>>>> >>>>>>> I’ve created a PR for this feature, here is the link >>>>>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 < >>>>>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572> >>>>>>> >>>>>>> Feel free to let me know if there is any suggestion. >>>>>>> >>>>>>> Thanks, >>>>>>> Han Gao >>>>>>> >>>>>>> On Apr 28, 2020, at 15:25, xingchun.chen <[email protected]> >>> wrote: >>>>>>> >>>>>>> It sounds very good, looking forward to your contribution >>>>>>> >>>>>>> >>>>>>> best wish! >>>>>>> >>>>>>> >>>>>>> ------------------ 原始邮件 ------------------ >>>>>>> 发件人: "hgaol"<[email protected]>; >>>>>>> 发送时间: 2020年4月28日(星期二) 下午3:11 >>>>>>> 收件人: "dev"<[email protected]>; >>>>>>> >>>>>>> 主题: [PROPOSAL][Feature] Support plugin in Alert service >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi folks, >>>>>>> >>>>>>> I think it will be useful for Alert service supporting the plugin >>>>>>> management. I’ve create an issue in Github and described about it, >>> here < >>>>>>> https://github.com/apache/incubator-dolphinscheduler/issues/2553> >>>>>>> is the link. Copy here for convenience. >>>>>>> >>>>>>> > I think the plugin feature is needed by the customers, because >>>>>>> different user may have different environment. For example, I want to >>> use >>>>>>> SMS to send alert information, but the SMS API is provided by some >>> system >>>>>>> we don't know before, and our DS has already deployed. In this case, >>> I just >>>>>>> want to implement the plugin code, and don't change the source code. >>>>>>> > >>>>>>> > Another benefits are developers can implement different plugin, >>>>>>> like slack, wechat, telegram, github issue, and etc... >>>>>>> > >>>>>>> > Describe the solution you'd like >>>>>>> > A clear and concise description of what you want to happen. >>>>>>> > >>>>>>> > A better way is that user do not need to change the source code >>> to >>>>>>> support new alert method. Instead, they just need to implement some >>>>>>> interface, and compliance with the convention, then package >>> all(include >>>>>>> dependencies) in a single jar file, and put the jar file in some >>> specific >>>>>>> place, such as $DS_HOME/plugins. After that, when start Alert >>> service, the >>>>>>> plugins can be recognized and loaded. So user can choose the alert >>> method >>>>>>> in the portal. >>>>>>> > >>>>>>> > Currently, I've done some investigations, and if this feature is >>>>>>> accepted, I think maybe I can implement it. >>>>>>> > >>>>>>> > Firstly, I'd better to adapt the current function, which is send >>>>>>> mail and something like that. >>>>>>> > >>>>>>> > Then, we can add some more features, such as Alert handler >>>>>>> instance, which is created to describe the alert >>>>>>> information(email/slack/wechat/..., to whom, the title, content, and >>> etc…). >>>>>>> > >>>>>>> > BTW, email and webhook are used frequently. I think they can be >>>>>>> the default plugin, and written in the source code, instead of a jar >>> file. >>>>>>> >>>>>>> Thanks, >>>>>>> hgaol >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> DolphinScheduler(Incubator) PPMC >>>>>> Jun Gao 高俊 >>>>>> [email protected] >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> >>>>> DolphinScheduler(Incubator) PPMC >>>>> Jun Gao 高俊 >>>>> [email protected] >>>>> >>>>> >>>> >>>> -- >>>> >>>> DolphinScheduler(Incubator) PPMC >>>> Jun Gao 高俊 >>>> [email protected] >>>> >>>> >>>> >>> >>> -- >>> DolphinScheduler(Incubator) PPMC >>> BaoLiang 鲍亮 >>> [email protected] >>>
