赞
At 2020-05-14 19:51:26, "lidong dai" <[email protected]> wrote: >I agree with you, I think a dolphinscheduler-plugin module and sub module >under dolphinscheduler-plugin will be better > > >Best Regards >--------------- >DolphinScheduler(Incubator) PPMC >Lidong Dai 代立冬 >[email protected] >--------------- > > >leon bao <[email protected]> 于2020年5月14日周四 下午7:18写道： > >> I think a common plugin module is necessary, but I don't recommend putting >> all the plugins in the root directory, which looks complicated. >> We can have ds-plugin module in root with many sub modules, such as >> alert-plugin / task-plugin /... >> >> hgaol <[email protected]> 于2020年5月13日周三 下午8:24写道： >> >> > Hi @Jun Gao, >> > >> > I went through the link you sent, and I agree that if the job was plugged >> > in will be better. But I’m not quite familiar with DS job currently, so I >> > may not able to answer your question about how to add a ClickHouse >> plugin. >> > >> > I’ve written interfaces about Alert plugin to support different alert >> > type. If you want to implement alert plugin, you can refer to the PR >> > mentioned before. >> > >> > For job plugin, I think you can leverage the plugin design and enhance it >> > to support job plugin. You may need to define a JobProvider, a JobPlugin >> > interface, option with a AbstractJobPlugin. >> > >> > Currently, this PR is about Alert plugin. For job plugin or other plugin >> > design, I think it may be many topics and details to discuss. >> > >> > >> > Thanks, >> > Han Gao >> > >> > >> > On May 13, 2020, at 10:19, JUN GAO <[email protected]> wrote: >> > >> > In addition to the job can be plugged in, is there any other content that >> > can be plugged in? If so, we need to consider the structural design in >> > advance, because this is a very important design transformation, we can >> > spend more time to think more >> > >> > JUN GAO <[email protected]> 于2020年5月13日周三 上午10:11写道： >> > >> >> @hgaol <[email protected]> There are some discuss about DS Plugin >> >> . But this is an early design and can only be used as a reference . I >> hope >> >> it can help your design. >> >> https://github.com/apache/incubator-dolphinscheduler/issues/201 >> >> >> >> I see the model `dolphinscheduler-plugin-api` . When I want to add a >> >> ClickHouse Job Plugin, should I create a new module called >> >> plugin-clickhouse-job directly in the root directory of the project >> >> (because I don’t know if only the job can do plugin)? Or we create a >> moudle >> >> called ds-job-plugin in the root directory, and then create a moudle >> called >> >> clickhouse-job under this moudle? Can you make a standardized design in >> >> this regard? Everyone can follow this design afterwards. >> >> >> >> @leon bao @Gang li Do you guys have any ideas ？ >> >> >> >> >> >> >> >> hgaol <[email protected]> 于2020年5月12日周二 下午9:33写道： >> >> >> >>> Hi @Gang Li, I agree with you that the plug-in module should be a >> common >> >>> module for all plug-in design. Currently I just create the SPI and >> plugin >> >>> interface for alert plug-in. If need more type for future, it’s very >> easy >> >>> to define other plugins. >> >>> >> >>> Currently the alert service can detect the plug-in jars and load it. >> But >> >>> I didn’t implement how to collaborate with front-end about the alert >> types >> >>> API and how many properties a plugin should have yet. For example, if >> we >> >>> add a webhook plugin, we should have the http method(form select), >> url(form >> >>> text), etc… So the plugin interface may change to achieve this goal. >> >>> >> >>> Hi @leon bao, as mentioned above, I agree to make the plugin module as >> a >> >>> common module for all plugin design, is that what you mean? >> >>> >> >>> > why don't we put the alert plug-in as an alert module? >> >>> I’m not clear about it, do you mean we could just put alert plugin in >> >>> alert module? I decide to put frequently used plugins in alert module, >> as >> >>> for others like SMS(which may has different api), make it as a plugin >> jar >> >>> file. >> >>> >> >>> Hi @Jun Gao, yes! I also use the SPI feature, it is quite convenient. >> >>> Developers just need to implement the interfaces in SPI, which are in >> >>> `dolphinscheduler-plugin-api` module. >> >>> >> >>> Hi all, thanks for your careful review! Here is some link. >> >>> >> >>> PR about plugin management. >> >>> <https://github.com/apache/incubator-dolphinscheduler/pull/2572> >> >>> PR about how to develop a simple plugin. >> >>> <https://github.com/apache/incubator-dolphinscheduler/pull/2593> >> >>> README about how to develop a simple plugin. >> >>> < >> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07 >> > >> >>> >> >>> Thanks in advance! >> >>> >> >>> Thanks, >> >>> hgaol >> >>> >> >>> On May 12, 2020, at 18:57, JUN GAO <[email protected]> wrote: >> >>> >> >>> we can refer to presto plug-in, etc. >> >>> 1. Design the SPI and define the interface that the plugin needs to >> >>> implement >> >>> 2. All plugins create new models to implement the interface in SPI. >> >>> The benefits are as follows: >> >>> 1. For developers, they only need to care about the interface in SPI, >> >>> and do not need to care about other people's code, so it is easier to >> get >> >>> started developing plug-ins. >> >>> 2. For code reviewers, to improve the efficiency of review, independent >> >>> models will be more easily reviewed. If there is a problem with the >> code, >> >>> it is easy to roll back. >> >>> 3. For users, there are more flexible options. For example, some people >> >>> only want to use the A plug-in but not the B plug-in. >> >>> >> >>> Thank you ! >> >>> >> >>> [email protected] <[email protected]> 于2020年5月12日周二 >> >>> 下午6:31写道： >> >>> >> >>>> Hi Gao, >> >>>> >> >>>> May be I didn't express clearly,I think the dolphinscheduler-plugin >> >>>> module is a common module for all plugin design, >> >>>> It's not neccessarily to put the implement of a plugin in the >> >>>> dolphinscheduler-plugin module. >> >>>> >> >>>> I also agree with the following way your said,I see the pr you >> submited >> >>>> didn't match it now,Will you give a deep design about the following >> way? >> >>>> >> >>>> > A better way is that user do not need to change the source code to >> >>>> support new alert method. Instead, they just need to implement some >> >>>> interface, and compliance with the convention, then package >> all(include >> >>>> dependencies) in a single jar file, and put the jar file in some >> specific >> >>>> place, such as $DS_HOME/plugins. After that, when start Alert >> service, the >> >>>> plugins can be recognized and loaded. So user can choose the alert >> method >> >>>> in the portal. >> >>>> >> >>>> >> >>>> >> >>>> DolphinScheduler(Incubator) PPMC >> >>>> Gang Li 李岗 >> >>>> >> >>>> [email protected]<mailto:[email protected]> >> >>>> >> >>>> From: hgaol >> >>>> Date: 2020-05-11 19:00 >> >>>> To: [email protected] >> >>>> CC: dev >> >>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >> >>>> Hi Gang, >> >>>> >> >>>> I agree with that making examples as a pure folder, thanks for your >> >>>> suggestion. >> >>>> >> >>>> For dolphinscheduler-plugin-api, do you mean it’s better to rename >> this >> >>>> module? I’m not clear about ‘all plugin function need to be >> implemented >> >>>> here’, do you mean we should put implemented plugin here? >> >>>> >> >>>> BTW, I’ve write a simple doc about how to implement an alert plugin. >> >>>> >> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07 >> >>>> >> >>>> Also, I think for some frequently used plugins(email/webhook/...), we >> >>>> could just implement in alert service. Here is the former PR about it. >> >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 >> >>>> >> >>>> Thanks for your careful review! >> >>>> >> >>>> Thanks, >> >>>> hgaol >> >>>> >> >>>> On May 11, 2020, at 18:09, [email protected] wrote: >> >>>> >> >>>> Hi Gao, >> >>>> >> >>>> I agree with the following you said,If this is a plugin module,all the >> >>>> plugin function need to be implemented here. >> >>>> Now the name is dolphinscheduler-plugin-api,I think we can change the >> >>>> module name, Such as `dolphinscheduler-plugin`. >> >>>> > I think dolphinscheduler-plugin-api could not only support alert >> >>>> plugin, but also support java job task for future. That’s another >> topic. >> >>>> >> >>>> Yes,I think it just a folder,not a sub-module. >> >>>> We can also look at other developers' comments on this. >> >>>> >Do you mean the examples module should not be a sub-module in >> >>>> dolphinscheduler project pom file, just a pure folder? >> >>>> >> >>>> >> >>>> >> >>>> DolphinScheduler(Incubator) PPMC >> >>>> Gang Li 李岗 >> >>>> >> >>>> [email protected]<mailto:[email protected]> >> >>>> >> >>>> From: hgaol >> >>>> Date: 2020-05-11 17:45 >> >>>> To: dev; lgcareer2019 >> >>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >> >>>> Hi Gang, >> >>>> >> >>>> Thanks for your suggestion. >> >>>> 1.Whether the function of module dolphinscheduler-plugin-api can be >> >>>> puted in the alert module. >> >>>> [hgaol] The customer should depend on this module to implement their >> >>>> own plugin. If put it into alert service, customer should put >> >>>> `dolphinscheduler-alert` as dependence, which is too heavy for a >> plugin. >> >>>> >> >>>> Also, I think dolphinscheduler-plugin-api could not only support alert >> >>>> plugin, but also support java job task for future. That’s another >> topic. >> >>>> >> >>>> 2. I think the examples module can be a folder. >> >>>> [hgaol] Do you mean the examples module should not be a sub-module in >> >>>> dolphinscheduler project pom file, just a pure folder? >> >>>> >> >>>> Thanks, >> >>>> hgaol >> >>>> >> >>>> On May 11, 2020, at 11:51, [email protected] wrote: >> >>>> >> >>>> Hi hgaol, >> >>>> Sorry for not replying in time. >> >>>> I saw you added the module dolphinscheduler-plugin-api and examples >> >>>> module,here are my two points want to discuss. >> >>>> >> >>>> 1、Whether the function of module dolphinscheduler-plugin-api can be >> >>>> puted in the alert module. >> >>>> 2、I think the examples module can be a folder. >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> DolphinScheduler(Incubator) PPMC >> >>>> Gang Li 李岗 >> >>>> >> >>>> [email protected]<mailto:[email protected]> >> >>>> >> >>>> From: hgaol >> >>>> Date: 2020-04-29 23:37 >> >>>> To: dev >> >>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >> >>>> Hi folks, >> >>>> >> >>>> I’ve created a PR for this feature, here is the link >> >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 < >> >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572> >> >>>> >> >>>> Feel free to let me know if there is any suggestion. >> >>>> >> >>>> Thanks, >> >>>> Han Gao >> >>>> >> >>>> On Apr 28, 2020, at 15:25, xingchun.chen <[email protected]> >> wrote: >> >>>> >> >>>> It sounds very good, looking forward to your contribution >> >>>> >> >>>> >> >>>> best wish! >> >>>> >> >>>> >> >>>> ------------------ 原始邮件 ------------------ >> >>>> 发件人: "hgaol"<[email protected]>; >> >>>> 发送时间: 2020年4月28日(星期二) 下午3:11 >> >>>> 收件人: "dev"<[email protected]>; >> >>>> >> >>>> 主题: [PROPOSAL][Feature] Support plugin in Alert service >> >>>> >> >>>> >> >>>> >> >>>> Hi folks, >> >>>> >> >>>> I think it will be useful for Alert service supporting the plugin >> >>>> management. I’ve create an issue in Github and described about it, >> here < >> >>>> https://github.com/apache/incubator-dolphinscheduler/issues/2553> >> >>>> is the link. Copy here for convenience. >> >>>> >> >>>> > I think the plugin feature is needed by the customers, because >> >>>> different user may have different environment. For example, I want to >> use >> >>>> SMS to send alert information, but the SMS API is provided by some >> system >> >>>> we don't know before, and our DS has already deployed. In this case, >> I just >> >>>> want to implement the plugin code, and don't change the source code. >> >>>> > >> >>>> > Another benefits are developers can implement different plugin, >> >>>> like slack, wechat, telegram, github issue, and etc... >> >>>> > >> >>>> > Describe the solution you'd like >> >>>> > A clear and concise description of what you want to happen. >> >>>> > >> >>>> > A better way is that user do not need to change the source code >> to >> >>>> support new alert method. Instead, they just need to implement some >> >>>> interface, and compliance with the convention, then package >> all(include >> >>>> dependencies) in a single jar file, and put the jar file in some >> specific >> >>>> place, such as $DS_HOME/plugins. After that, when start Alert >> service, the >> >>>> plugins can be recognized and loaded. So user can choose the alert >> method >> >>>> in the portal. >> >>>> > >> >>>> > Currently, I've done some investigations, and if this feature is >> >>>> accepted, I think maybe I can implement it. >> >>>> > >> >>>> > Firstly, I'd better to adapt the current function, which is send >> >>>> mail and something like that. >> >>>> > >> >>>> > Then, we can add some more features, such as Alert handler >> >>>> instance, which is created to describe the alert >> >>>> information(email/slack/wechat/..., to whom, the title, content, and >> etc…). >> >>>> > >> >>>> > BTW, email and webhook are used frequently. I think they can be >> >>>> the default plugin, and written in the source code, instead of a jar >> file. >> >>>> >> >>>> Thanks, >> >>>> hgaol >> >>>> >> >>>> >> >>> >> >>> -- >> >>> >> >>> DolphinScheduler(Incubator) PPMC >> >>> Jun Gao 高俊 >> >>> [email protected] >> >>> >> >>> >> >>> >> >> >> >> -- >> >> >> >> DolphinScheduler(Incubator) PPMC >> >> Jun Gao 高俊 >> >> [email protected] >> >> >> >> >> > >> > -- >> > >> > DolphinScheduler(Incubator) PPMC >> > Jun Gao 高俊 >> > [email protected] >> > >> > >> > >> >> -- >> DolphinScheduler(Incubator) PPMC >> BaoLiang 鲍亮 >> [email protected] >>
