Hi @Jun Gao, I went through the link you sent, and I agree that if the job was plugged in will be better. But I’m not quite familiar with DS job currently, so I may not able to answer your question about how to add a ClickHouse plugin.
I’ve written interfaces about Alert plugin to support different alert type. If you want to implement alert plugin, you can refer to the PR mentioned before. For job plugin, I think you can leverage the plugin design and enhance it to support job plugin. You may need to define a JobProvider, a JobPlugin interface, option with a AbstractJobPlugin. Currently, this PR is about Alert plugin. For job plugin or other plugin design, I think it may be many topics and details to discuss. Thanks, Han Gao > On May 13, 2020, at 10:19, JUN GAO <[email protected]> wrote: > > In addition to the job can be plugged in, is there any other content that can > be plugged in? If so, we need to consider the structural design in advance, > because this is a very important design transformation, we can spend more > time to think more > > JUN GAO <[email protected] <mailto:[email protected]>> 于2020年5月13日周三 > 上午10:11写道： > @hgaol <mailto:[email protected]> There are some discuss about DS Plugin . > But this is an early design and can only be used as a reference . I hope it > can help your design. > https://github.com/apache/incubator-dolphinscheduler/issues/201 > <https://github.com/apache/incubator-dolphinscheduler/issues/201> > > I see the model `dolphinscheduler-plugin-api` . When I want to add a > ClickHouse Job Plugin, should I create a new module called > plugin-clickhouse-job directly in the root directory of the project (because > I don’t know if only the job can do plugin)? Or we create a moudle called > ds-job-plugin in the root directory, and then create a moudle called > clickhouse-job under this moudle? Can you make a standardized design in this > regard? Everyone can follow this design afterwards. > > @leon bao @Gang li Do you guys have any ideas ？ > > > > hgaol <[email protected] <mailto:[email protected]>> 于2020年5月12日周二 > 下午9:33写道： > Hi @Gang Li, I agree with you that the plug-in module should be a common > module for all plug-in design. Currently I just create the SPI and plugin > interface for alert plug-in. If need more type for future, it’s very easy to > define other plugins. > > Currently the alert service can detect the plug-in jars and load it. But I > didn’t implement how to collaborate with front-end about the alert types API > and how many properties a plugin should have yet. For example, if we add a > webhook plugin, we should have the http method(form select), url(form text), > etc… So the plugin interface may change to achieve this goal. > > Hi @leon bao, as mentioned above, I agree to make the plugin module as a > common module for all plugin design, is that what you mean? > > > why don't we put the alert plug-in as an alert module? > I’m not clear about it, do you mean we could just put alert plugin in alert > module? I decide to put frequently used plugins in alert module, as for > others like SMS(which may has different api), make it as a plugin jar file. > > Hi @Jun Gao, yes! I also use the SPI feature, it is quite convenient. > Developers just need to implement the interfaces in SPI, which are in > `dolphinscheduler-plugin-api` module. > > Hi all, thanks for your careful review! Here is some link. > > PR about plugin management. > <https://github.com/apache/incubator-dolphinscheduler/pull/2572> > PR about how to develop a simple plugin. > <https://github.com/apache/incubator-dolphinscheduler/pull/2593> > README about how to develop a simple plugin. > <https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07> > > Thanks in advance! > > Thanks, > hgaol > >> On May 12, 2020, at 18:57, JUN GAO <[email protected] >> <mailto:[email protected]>> wrote: >> >> we can refer to presto plug-in, etc. >> 1. Design the SPI and define the interface that the plugin needs to implement >> 2. All plugins create new models to implement the interface in SPI. >> The benefits are as follows: >> 1. For developers, they only need to care about the interface in SPI, and do >> not need to care about other people's code, so it is easier to get started >> developing plug-ins. >> 2. For code reviewers, to improve the efficiency of review, independent >> models will be more easily reviewed. If there is a problem with the code, it >> is easy to roll back. >> 3. For users, there are more flexible options. For example, some people only >> want to use the A plug-in but not the B plug-in. >> >> Thank you ! >> >> [email protected] <mailto:[email protected]> >> <[email protected] <mailto:[email protected]>> 于2020年5月12日周二 >> 下午6:31写道： >> Hi Gao, >> >> May be I didn't express clearly,I think the dolphinscheduler-plugin module >> is a common module for all plugin design, >> It's not neccessarily to put the implement of a plugin in the >> dolphinscheduler-plugin module. >> >> I also agree with the following way your said,I see the pr you submited >> didn't match it now,Will you give a deep design about the following way? >> >> > A better way is that user do not need to change the source code to support >> > new alert method. Instead, they just need to implement some interface, and >> > compliance with the convention, then package all(include dependencies) in >> > a single jar file, and put the jar file in some specific place, such as >> > $DS_HOME/plugins. After that, when start Alert service, the plugins can be >> > recognized and loaded. So user can choose the alert method in the portal. >> >> >> >> DolphinScheduler(Incubator) PPMC >> Gang Li 李岗 >> >> [email protected] >> <mailto:[email protected]><mailto:[email protected] >> <mailto:[email protected]>> >> >> From: hgaol >> Date: 2020-05-11 19:00 >> To: [email protected] <mailto:[email protected]> >> CC: dev >> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >> Hi Gang, >> >> I agree with that making examples as a pure folder, thanks for your >> suggestion. >> >> For dolphinscheduler-plugin-api, do you mean it’s better to rename this >> module? I’m not clear about ‘all plugin function need to be implemented >> here’, do you mean we should put implemented plugin here? >> >> BTW, I’ve write a simple doc about how to implement an alert >> plugin.https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07 >> >> <https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07> >> >> Also, I think for some frequently used plugins(email/webhook/...), we could >> just implement in alert service. Here is the former PR about it. >> https://github.com/apache/incubator-dolphinscheduler/pull/2572 >> <https://github.com/apache/incubator-dolphinscheduler/pull/2572> >> >> Thanks for your careful review! >> >> Thanks, >> hgaol >> >> On May 11, 2020, at 18:09, [email protected] >> <mailto:[email protected]> wrote: >> >> Hi Gao, >> >> I agree with the following you said,If this is a plugin module,all the >> plugin function need to be implemented here. >> Now the name is dolphinscheduler-plugin-api,I think we can change the module >> name, Such as `dolphinscheduler-plugin`. >> > I think dolphinscheduler-plugin-api could not only support alert plugin, >> > but also support java job task for future. That’s another topic. >> >> Yes,I think it just a folder,not a sub-module. >> We can also look at other developers' comments on this. >> >Do you mean the examples module should not be a sub-module in >> >dolphinscheduler project pom file, just a pure folder? >> >> >> >> DolphinScheduler(Incubator) PPMC >> Gang Li 李岗 >> >> [email protected] >> <mailto:[email protected]><mailto:[email protected] >> <mailto:[email protected]>> >> >> From: hgaol >> Date: 2020-05-11 17:45 >> To: dev; lgcareer2019 >> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >> Hi Gang, >> >> Thanks for your suggestion. >> 1.Whether the function of module dolphinscheduler-plugin-api can be puted in >> the alert module. >> [hgaol] The customer should depend on this module to implement their own >> plugin. If put it into alert service, customer should put >> `dolphinscheduler-alert` as dependence, which is too heavy for a plugin. >> >> Also, I think dolphinscheduler-plugin-api could not only support alert >> plugin, but also support java job task for future. That’s another topic. >> >> 2. I think the examples module can be a folder. >> [hgaol] Do you mean the examples module should not be a sub-module in >> dolphinscheduler project pom file, just a pure folder? >> >> Thanks, >> hgaol >> >> On May 11, 2020, at 11:51, [email protected] >> <mailto:[email protected]> wrote: >> >> Hi hgaol, >> Sorry for not replying in time. >> I saw you added the module dolphinscheduler-plugin-api and examples >> module,here are my two points want to discuss. >> >> 1、Whether the function of module dolphinscheduler-plugin-api can be puted in >> the alert module. >> 2、I think the examples module can be a folder. >> >> >> >> >> >> >> DolphinScheduler(Incubator) PPMC >> Gang Li 李岗 >> >> [email protected] >> <mailto:[email protected]><mailto:[email protected] >> <mailto:[email protected]>> >> >> From: hgaol >> Date: 2020-04-29 23:37 >> To: dev >> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >> Hi folks, >> >> I’ve created a PR for this feature, here is the link >> https://github.com/apache/incubator-dolphinscheduler/pull/2572 >> <https://github.com/apache/incubator-dolphinscheduler/pull/2572> >> <https://github.com/apache/incubator-dolphinscheduler/pull/2572 >> <https://github.com/apache/incubator-dolphinscheduler/pull/2572>> >> >> Feel free to let me know if there is any suggestion. >> >> Thanks, >> Han Gao >> >> On Apr 28, 2020, at 15:25, xingchun.chen <[email protected] >> <mailto:[email protected]>> wrote: >> >> It sounds very good, looking forward to your contribution >> >> >> best wish! >> >> >> ------------------ 原始邮件 ------------------ >> 发件人: "hgaol"<[email protected] <mailto:[email protected]>>; >> 发送时间: 2020年4月28日(星期二) 下午3:11 >> 收件人: "dev"<[email protected] >> <mailto:[email protected]>>; >> >> 主题: [PROPOSAL][Feature] Support plugin in Alert service >> >> >> >> Hi folks, >> >> I think it will be useful for Alert service supporting the plugin >> management. I’ve create an issue in Github and described about it, here >> <https://github.com/apache/incubator-dolphinscheduler/issues/2553> >> <https://github.com/apache/incubator-dolphinscheduler/issues/2553>>; is >> the link. Copy here for convenience. >> >> > I think the plugin feature is needed by the customers, because >> different user may have different environment. For example, I want to use >> SMS to send alert information, but the SMS API is provided by some system we >> don't know before, and our DS has already deployed. In this case, I just >> want to implement the plugin code, and don't change the source code. >> > >> > Another benefits are developers can implement different plugin, like >> slack, wechat, telegram, github issue, and etc... >> > >> > Describe the solution you'd like >> > A clear and concise description of what you want to happen. >> > >> > A better way is that user do not need to change the source code to >> support new alert method. Instead, they just need to implement some >> interface, and compliance with the convention, then package all(include >> dependencies) in a single jar file, and put the jar file in some specific >> place, such as $DS_HOME/plugins. After that, when start Alert service, the >> plugins can be recognized and loaded. So user can choose the alert method in >> the portal. >> > >> > Currently, I've done some investigations, and if this feature is >> accepted, I think maybe I can implement it. >> > >> > Firstly, I'd better to adapt the current function, which is send mail >> and something like that. >> > >> > Then, we can add some more features, such as Alert handler instance, >> which is created to describe the alert information(email/slack/wechat/..., >> to whom, the title, content, and etc…). >> > >> > BTW, email and webhook are used frequently. I think they can be the >> default plugin, and written in the source code, instead of a jar file. >> >> Thanks, >> hgaol >> >> >> >> -- >> DolphinScheduler(Incubator) PPMC >> Jun Gao 高俊 >> >> [email protected] <mailto:[email protected]> >> > > > > -- > DolphinScheduler(Incubator) PPMC > Jun Gao 高俊 > > [email protected] <mailto:[email protected]> > > > > -- > DolphinScheduler(Incubator) PPMC > Jun Gao 高俊 > > [email protected] <mailto:[email protected]> >
