@hgaol <[email protected]> There are some discuss about DS Plugin . But this is an early design and can only be used as a reference . I hope it can help your design. https://github.com/apache/incubator-dolphinscheduler/issues/201
I see the model `dolphinscheduler-plugin-api` . When I want to add a ClickHouse Job Plugin, should I create a new module called plugin-clickhouse-job directly in the root directory of the project (because I don’t know if only the job can do plugin)? Or we create a moudle called ds-job-plugin in the root directory, and then create a moudle called clickhouse-job under this moudle? Can you make a standardized design in this regard? Everyone can follow this design afterwards. @leon bao @Gang li Do you guys have any ideas ？ hgaol <[email protected]> 于2020年5月12日周二 下午9:33写道： > Hi @Gang Li, I agree with you that the plug-in module should be a common > module for all plug-in design. Currently I just create the SPI and plugin > interface for alert plug-in. If need more type for future, it’s very easy > to define other plugins. > > Currently the alert service can detect the plug-in jars and load it. But I > didn’t implement how to collaborate with front-end about the alert types > API and how many properties a plugin should have yet. For example, if we > add a webhook plugin, we should have the http method(form select), url(form > text), etc… So the plugin interface may change to achieve this goal. > > Hi @leon bao, as mentioned above, I agree to make the plugin module as a > common module for all plugin design, is that what you mean? > > > why don't we put the alert plug-in as an alert module? > I’m not clear about it, do you mean we could just put alert plugin in > alert module? I decide to put frequently used plugins in alert module, as > for others like SMS(which may has different api), make it as a plugin jar > file. > > Hi @Jun Gao, yes! I also use the SPI feature, it is quite convenient. > Developers just need to implement the interfaces in SPI, which are in > `dolphinscheduler-plugin-api` module. > > Hi all, thanks for your careful review! Here is some link. > > PR about plugin management. > <https://github.com/apache/incubator-dolphinscheduler/pull/2572> > PR about how to develop a simple plugin. > <https://github.com/apache/incubator-dolphinscheduler/pull/2593> > README about how to develop a simple plugin. > <https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07> > > Thanks in advance! > > Thanks, > hgaol > > On May 12, 2020, at 18:57, JUN GAO <[email protected]> wrote: > > we can refer to presto plug-in, etc. > 1. Design the SPI and define the interface that the plugin needs to > implement > 2. All plugins create new models to implement the interface in SPI. > The benefits are as follows: > 1. For developers, they only need to care about the interface in SPI, and > do not need to care about other people's code, so it is easier to get > started developing plug-ins. > 2. For code reviewers, to improve the efficiency of review, independent > models will be more easily reviewed. If there is a problem with the code, > it is easy to roll back. > 3. For users, there are more flexible options. For example, some people > only want to use the A plug-in but not the B plug-in. > > Thank you ! > > [email protected] <[email protected]> 于2020年5月12日周二 > 下午6:31写道： > >> Hi Gao, >> >> May be I didn't express clearly,I think the dolphinscheduler-plugin >> module is a common module for all plugin design, >> It's not neccessarily to put the implement of a plugin in the >> dolphinscheduler-plugin module. >> >> I also agree with the following way your said,I see the pr you submited >> didn't match it now,Will you give a deep design about the following way? >> >> > A better way is that user do not need to change the source code to >> support new alert method. Instead, they just need to implement some >> interface, and compliance with the convention, then package all(include >> dependencies) in a single jar file, and put the jar file in some specific >> place, such as $DS_HOME/plugins. After that, when start Alert service, the >> plugins can be recognized and loaded. So user can choose the alert method >> in the portal. >> >> >> >> DolphinScheduler(Incubator) PPMC >> Gang Li 李岗 >> >> [email protected]<mailto:[email protected]> >> >> From: hgaol >> Date: 2020-05-11 19:00 >> To: [email protected] >> CC: dev >> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >> Hi Gang, >> >> I agree with that making examples as a pure folder, thanks for your >> suggestion. >> >> For dolphinscheduler-plugin-api, do you mean it’s better to rename this >> module? I’m not clear about ‘all plugin function need to be implemented >> here’, do you mean we should put implemented plugin here? >> >> BTW, I’ve write a simple doc about how to implement an alert plugin. >> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07 >> >> Also, I think for some frequently used plugins(email/webhook/...), we >> could just implement in alert service. Here is the former PR about it. >> https://github.com/apache/incubator-dolphinscheduler/pull/2572 >> >> Thanks for your careful review! >> >> Thanks, >> hgaol >> >> On May 11, 2020, at 18:09, [email protected] wrote: >> >> Hi Gao, >> >> I agree with the following you said,If this is a plugin module,all the >> plugin function need to be implemented here. >> Now the name is dolphinscheduler-plugin-api,I think we can change the >> module name, Such as `dolphinscheduler-plugin`. >> > I think dolphinscheduler-plugin-api could not only support alert >> plugin, but also support java job task for future. That’s another topic. >> >> Yes,I think it just a folder,not a sub-module. >> We can also look at other developers' comments on this. >> >Do you mean the examples module should not be a sub-module in >> dolphinscheduler project pom file, just a pure folder? >> >> >> >> DolphinScheduler(Incubator) PPMC >> Gang Li 李岗 >> >> [email protected]<mailto:[email protected]> >> >> From: hgaol >> Date: 2020-05-11 17:45 >> To: dev; lgcareer2019 >> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >> Hi Gang, >> >> Thanks for your suggestion. >> 1.Whether the function of module dolphinscheduler-plugin-api can be puted >> in the alert module. >> [hgaol] The customer should depend on this module to implement their own >> plugin. If put it into alert service, customer should put >> `dolphinscheduler-alert` as dependence, which is too heavy for a plugin. >> >> Also, I think dolphinscheduler-plugin-api could not only support alert >> plugin, but also support java job task for future. That’s another topic. >> >> 2. I think the examples module can be a folder. >> [hgaol] Do you mean the examples module should not be a sub-module in >> dolphinscheduler project pom file, just a pure folder? >> >> Thanks, >> hgaol >> >> On May 11, 2020, at 11:51, [email protected] wrote: >> >> Hi hgaol, >> Sorry for not replying in time. >> I saw you added the module dolphinscheduler-plugin-api and examples >> module,here are my two points want to discuss. >> >> 1、Whether the function of module dolphinscheduler-plugin-api can be puted >> in the alert module. >> 2、I think the examples module can be a folder. >> >> >> >> >> >> >> DolphinScheduler(Incubator) PPMC >> Gang Li 李岗 >> >> [email protected]<mailto:[email protected]> >> >> From: hgaol >> Date: 2020-04-29 23:37 >> To: dev >> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >> Hi folks, >> >> I’ve created a PR for this feature, here is the link >> https://github.com/apache/incubator-dolphinscheduler/pull/2572 < >> https://github.com/apache/incubator-dolphinscheduler/pull/2572> >> >> Feel free to let me know if there is any suggestion. >> >> Thanks, >> Han Gao >> >> On Apr 28, 2020, at 15:25, xingchun.chen <[email protected]> wrote: >> >> It sounds very good, looking forward to your contribution >> >> >> best wish! >> >> >> ------------------ 原始邮件 ------------------ >> 发件人: "hgaol"<[email protected]>; >> 发送时间: 2020年4月28日(星期二) 下午3:11 >> 收件人: "dev"<[email protected]>; >> >> 主题: [PROPOSAL][Feature] Support plugin in Alert service >> >> >> >> Hi folks, >> >> I think it will be useful for Alert service supporting the plugin >> management. I’ve create an issue in Github and described about it, here < >> https://github.com/apache/incubator-dolphinscheduler/issues/2553> is >> the link. Copy here for convenience. >> >> > I think the plugin feature is needed by the customers, because >> different user may have different environment. For example, I want to use >> SMS to send alert information, but the SMS API is provided by some system >> we don't know before, and our DS has already deployed. In this case, I just >> want to implement the plugin code, and don't change the source code. >> > >> > Another benefits are developers can implement different plugin, like >> slack, wechat, telegram, github issue, and etc... >> > >> > Describe the solution you'd like >> > A clear and concise description of what you want to happen. >> > >> > A better way is that user do not need to change the source code to >> support new alert method. Instead, they just need to implement some >> interface, and compliance with the convention, then package all(include >> dependencies) in a single jar file, and put the jar file in some specific >> place, such as $DS_HOME/plugins. After that, when start Alert service, the >> plugins can be recognized and loaded. So user can choose the alert method >> in the portal. >> > >> > Currently, I've done some investigations, and if this feature is >> accepted, I think maybe I can implement it. >> > >> > Firstly, I'd better to adapt the current function, which is send >> mail and something like that. >> > >> > Then, we can add some more features, such as Alert handler instance, >> which is created to describe the alert information(email/slack/wechat/..., >> to whom, the title, content, and etc…). >> > >> > BTW, email and webhook are used frequently. I think they can be the >> default plugin, and written in the source code, instead of a jar file. >> >> Thanks, >> hgaol >> >> > > -- > > DolphinScheduler(Incubator) PPMC > Jun Gao 高俊 > [email protected] > > > -- DolphinScheduler(Incubator) PPMC Jun Gao 高俊 [email protected]
