In addition to the job can be plugged in, is there any other content that can be plugged in? If so, we need to consider the structural design in advance, because this is a very important design transformation, we can spend more time to think more
JUN GAO <[email protected]> 于2020年5月13日周三 上午10:11写道： > @hgaol <[email protected]> There are some discuss about DS Plugin > . But this is an early design and can only be used as a reference . I hope > it can help your design. > https://github.com/apache/incubator-dolphinscheduler/issues/201 > > I see the model `dolphinscheduler-plugin-api` . When I want to add a > ClickHouse Job Plugin, should I create a new module called > plugin-clickhouse-job directly in the root directory of the project > (because I don’t know if only the job can do plugin)? Or we create a moudle > called ds-job-plugin in the root directory, and then create a moudle called > clickhouse-job under this moudle? Can you make a standardized design in > this regard? Everyone can follow this design afterwards. > > @leon bao @Gang li Do you guys have any ideas ？ > > > > hgaol <[email protected]> 于2020年5月12日周二 下午9:33写道： > >> Hi @Gang Li, I agree with you that the plug-in module should be a common >> module for all plug-in design. Currently I just create the SPI and plugin >> interface for alert plug-in. If need more type for future, it’s very easy >> to define other plugins. >> >> Currently the alert service can detect the plug-in jars and load it. But >> I didn’t implement how to collaborate with front-end about the alert types >> API and how many properties a plugin should have yet. For example, if we >> add a webhook plugin, we should have the http method(form select), url(form >> text), etc… So the plugin interface may change to achieve this goal. >> >> Hi @leon bao, as mentioned above, I agree to make the plugin module as a >> common module for all plugin design, is that what you mean? >> >> > why don't we put the alert plug-in as an alert module? >> I’m not clear about it, do you mean we could just put alert plugin in >> alert module? I decide to put frequently used plugins in alert module, as >> for others like SMS(which may has different api), make it as a plugin jar >> file. >> >> Hi @Jun Gao, yes! I also use the SPI feature, it is quite convenient. >> Developers just need to implement the interfaces in SPI, which are in >> `dolphinscheduler-plugin-api` module. >> >> Hi all, thanks for your careful review! Here is some link. >> >> PR about plugin management. >> <https://github.com/apache/incubator-dolphinscheduler/pull/2572> >> PR about how to develop a simple plugin. >> <https://github.com/apache/incubator-dolphinscheduler/pull/2593> >> README about how to develop a simple plugin. >> <https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07> >> >> Thanks in advance! >> >> Thanks, >> hgaol >> >> On May 12, 2020, at 18:57, JUN GAO <[email protected]> wrote: >> >> we can refer to presto plug-in, etc. >> 1. Design the SPI and define the interface that the plugin needs to >> implement >> 2. All plugins create new models to implement the interface in SPI. >> The benefits are as follows: >> 1. For developers, they only need to care about the interface in SPI, and >> do not need to care about other people's code, so it is easier to get >> started developing plug-ins. >> 2. For code reviewers, to improve the efficiency of review, independent >> models will be more easily reviewed. If there is a problem with the code, >> it is easy to roll back. >> 3. For users, there are more flexible options. For example, some people >> only want to use the A plug-in but not the B plug-in. >> >> Thank you ! >> >> [email protected] <[email protected]> 于2020年5月12日周二 >> 下午6:31写道： >> >>> Hi Gao, >>> >>> May be I didn't express clearly,I think the dolphinscheduler-plugin >>> module is a common module for all plugin design, >>> It's not neccessarily to put the implement of a plugin in the >>> dolphinscheduler-plugin module. >>> >>> I also agree with the following way your said,I see the pr you submited >>> didn't match it now,Will you give a deep design about the following way? >>> >>> > A better way is that user do not need to change the source code to >>> support new alert method. Instead, they just need to implement some >>> interface, and compliance with the convention, then package all(include >>> dependencies) in a single jar file, and put the jar file in some specific >>> place, such as $DS_HOME/plugins. After that, when start Alert service, the >>> plugins can be recognized and loaded. So user can choose the alert method >>> in the portal. >>> >>> >>> >>> DolphinScheduler(Incubator) PPMC >>> Gang Li 李岗 >>> >>> [email protected]<mailto:[email protected]> >>> >>> From: hgaol >>> Date: 2020-05-11 19:00 >>> To: [email protected] >>> CC: dev >>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >>> Hi Gang, >>> >>> I agree with that making examples as a pure folder, thanks for your >>> suggestion. >>> >>> For dolphinscheduler-plugin-api, do you mean it’s better to rename this >>> module? I’m not clear about ‘all plugin function need to be implemented >>> here’, do you mean we should put implemented plugin here? >>> >>> BTW, I’ve write a simple doc about how to implement an alert plugin. >>> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07 >>> >>> Also, I think for some frequently used plugins(email/webhook/...), we >>> could just implement in alert service. Here is the former PR about it. >>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 >>> >>> Thanks for your careful review! >>> >>> Thanks, >>> hgaol >>> >>> On May 11, 2020, at 18:09, [email protected] wrote: >>> >>> Hi Gao, >>> >>> I agree with the following you said,If this is a plugin module,all the >>> plugin function need to be implemented here. >>> Now the name is dolphinscheduler-plugin-api,I think we can change the >>> module name, Such as `dolphinscheduler-plugin`. >>> > I think dolphinscheduler-plugin-api could not only support alert >>> plugin, but also support java job task for future. That’s another topic. >>> >>> Yes,I think it just a folder,not a sub-module. >>> We can also look at other developers' comments on this. >>> >Do you mean the examples module should not be a sub-module in >>> dolphinscheduler project pom file, just a pure folder? >>> >>> >>> >>> DolphinScheduler(Incubator) PPMC >>> Gang Li 李岗 >>> >>> [email protected]<mailto:[email protected]> >>> >>> From: hgaol >>> Date: 2020-05-11 17:45 >>> To: dev; lgcareer2019 >>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >>> Hi Gang, >>> >>> Thanks for your suggestion. >>> 1.Whether the function of module dolphinscheduler-plugin-api can be >>> puted in the alert module. >>> [hgaol] The customer should depend on this module to implement their own >>> plugin. If put it into alert service, customer should put >>> `dolphinscheduler-alert` as dependence, which is too heavy for a plugin. >>> >>> Also, I think dolphinscheduler-plugin-api could not only support alert >>> plugin, but also support java job task for future. That’s another topic. >>> >>> 2. I think the examples module can be a folder. >>> [hgaol] Do you mean the examples module should not be a sub-module in >>> dolphinscheduler project pom file, just a pure folder? >>> >>> Thanks, >>> hgaol >>> >>> On May 11, 2020, at 11:51, [email protected] wrote: >>> >>> Hi hgaol, >>> Sorry for not replying in time. >>> I saw you added the module dolphinscheduler-plugin-api and examples >>> module,here are my two points want to discuss. >>> >>> 1、Whether the function of module dolphinscheduler-plugin-api can be >>> puted in the alert module. >>> 2、I think the examples module can be a folder. >>> >>> >>> >>> >>> >>> >>> DolphinScheduler(Incubator) PPMC >>> Gang Li 李岗 >>> >>> [email protected]<mailto:[email protected]> >>> >>> From: hgaol >>> Date: 2020-04-29 23:37 >>> To: dev >>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >>> Hi folks, >>> >>> I’ve created a PR for this feature, here is the link >>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 < >>> https://github.com/apache/incubator-dolphinscheduler/pull/2572> >>> >>> Feel free to let me know if there is any suggestion. >>> >>> Thanks, >>> Han Gao >>> >>> On Apr 28, 2020, at 15:25, xingchun.chen <[email protected]> wrote: >>> >>> It sounds very good, looking forward to your contribution >>> >>> >>> best wish! >>> >>> >>> ------------------ 原始邮件 ------------------ >>> 发件人: "hgaol"<[email protected]>; >>> 发送时间: 2020年4月28日(星期二) 下午3:11 >>> 收件人: "dev"<[email protected]>; >>> >>> 主题: [PROPOSAL][Feature] Support plugin in Alert service >>> >>> >>> >>> Hi folks, >>> >>> I think it will be useful for Alert service supporting the plugin >>> management. I’ve create an issue in Github and described about it, here < >>> https://github.com/apache/incubator-dolphinscheduler/issues/2553> is >>> the link. Copy here for convenience. >>> >>> > I think the plugin feature is needed by the customers, because >>> different user may have different environment. For example, I want to use >>> SMS to send alert information, but the SMS API is provided by some system >>> we don't know before, and our DS has already deployed. In this case, I just >>> want to implement the plugin code, and don't change the source code. >>> > >>> > Another benefits are developers can implement different plugin, >>> like slack, wechat, telegram, github issue, and etc... >>> > >>> > Describe the solution you'd like >>> > A clear and concise description of what you want to happen. >>> > >>> > A better way is that user do not need to change the source code to >>> support new alert method. Instead, they just need to implement some >>> interface, and compliance with the convention, then package all(include >>> dependencies) in a single jar file, and put the jar file in some specific >>> place, such as $DS_HOME/plugins. After that, when start Alert service, the >>> plugins can be recognized and loaded. So user can choose the alert method >>> in the portal. >>> > >>> > Currently, I've done some investigations, and if this feature is >>> accepted, I think maybe I can implement it. >>> > >>> > Firstly, I'd better to adapt the current function, which is send >>> mail and something like that. >>> > >>> > Then, we can add some more features, such as Alert handler >>> instance, which is created to describe the alert >>> information(email/slack/wechat/..., to whom, the title, content, and etc…). >>> > >>> > BTW, email and webhook are used frequently. I think they can be the >>> default plugin, and written in the source code, instead of a jar file. >>> >>> Thanks, >>> hgaol >>> >>> >> >> -- >> >> DolphinScheduler(Incubator) PPMC >> Jun Gao 高俊 >> [email protected] >> >> >> > > -- > > DolphinScheduler(Incubator) PPMC > Jun Gao 高俊 > [email protected] > > -- DolphinScheduler(Incubator) PPMC Jun Gao 高俊 [email protected]
