I have some idea: 1. Remove the alert service and change it to a tool jar package. Whether it is the master or the worker, who needs to send a message calls the alert interface. 2. The alert service no longer scans the db and provides an interface. What is sent is determined by the caller. The alert need to saving the alert message and sending the message via email or WeChat. 3. If the master, worker, or other services can call the alert toolkit to send messages, the alert can exist as an independent module, otherwise it is not suitable for the common layer, the dao layer, or the service layer. And if want to extend the function of alert, only need to modify the alert module.
By doing this, we can reduce one service(alert) and simplify deployment. What do you guys think？ guo jiwei <[email protected]> 于2019年12月12日周四 上午10:14写道： > To baoliang： > Is alert server distributed ? it's a standalone module with only a > function scanning db and triggering alert . > Trigger event will delay if in an individual module even not scan db. > do you have some good idea to let alert trigger in time ? do you get my > point of in time ? > > > On Wed, Dec 11, 2019 at 8:26 PM leon bao <[email protected]> wrote: > > > As an open source project, I think it's important to stay open and > > extensible. > > Our current alert does not have many shortcomings except for scanning the > > database with one thread, at the same time, the alert service as a > separate > > module provides better scalability. > > So I don't think it is necessary to merge alert into other modules > without > > much benefit. > > > > guo jiwei <[email protected]> 于2019年12月11日周三 下午7:33写道： > > > > > To xiaochun. > > > it's not a good way. > > > Alert must be trigger by who is scheduling the task, in DS, it's > > > MasterServer. Because, only by who is scheduling the task, it can know > > the > > > task status in time. If the task is timeout, trigger timeout alert in > > time > > > is very important for users. > > > this is why we have to move alert into server module. > > > Alert implementation will be refactor in the future. not scan db > > anymore. > > > > > > On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <[email protected]> > > > wrote: > > > > > > > To guo jiwei: > > > > > > > > Why not put together with the api server, the alert server function > is > > > > very small, > > > > and the load will not be very high. If logs are stored together in > the > > > > future, > > > > we can combine alert server, log server, api server together, these > can > > > be > > > > called management server. > > > > > > > > Best Regards > > > > --------------- > > > > DolphinScheduler(Incubator) Committer > > > > Xiaochun Liu 刘小春 > > > > [email protected] > > > > --------------- > > > > > > > > > > > > > > > > > 在 2019年12月11日，下午7:15，guo jiwei <[email protected]> 写道： > > > > > > > > > > To ligang. > > > > > it's right. > > > > > But alert server is only a small function. we define it as an > > > > > individual module and as a server. do you thing alert is > expensive > > or > > > > > taking more resource ? if not , why a single module ? > > > > > And alert server trigger task event by scanning db, do you think > it > > > is a > > > > > nice way ? > > > > > Moving into server module is only our first step for simplifying > > user > > > > > deployment. Extension of alert can be updated via redeploy server > and > > > > it's > > > > > not a frequent operation. > > > > > As the architecture changes, alert implementation will change. > > > > > > > > > > > > > > > On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <[email protected]> > > wrote: > > > > > > > > > >> I think from another angle,Master and Worker as key services,I > think > > > not > > > > >> to redeployment during normal execution. > > > > >> If tasks are still running,redeploy master and worker may be lead > > to > > > > >> missed the timed task. > > > > >> > > > > >> ________________________________ > > > > >> DolphinScheduler(Incubator) PPMC > > > > >> Gang Li 李岗 > > > > >> > > > > >> [email protected]<mailto:[email protected]> > > > > >> > > > > >> From: guo jiwei<mailto:[email protected]> > > > > >> Date: 2019-12-11 18:11 > > > > >> To: dev<mailto:[email protected]> > > > > >> Subject: Re: Aproposal for DolphinScheduler Simplified Deployment > > > > >> To ligang. > > > > >> redeploy is simple, but what about the latency of alert ? > > > > >> it's easy to redeploy master server to update alert > > > > >> > > > > >> > > > > >> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <[email protected]> > > wrote: > > > > >> > > > > >>> I think the alert module can be retained. Currently, it only > > supports > > > > >>> email and webchat, > > > > >>> but more alarm modes can be added in the future. > > > > >>> > > > > >>> At present，alert is a independent service. the alert service only > > > > >> consumes > > > > >>> alarm information in the database, other services produce these > > alarm > > > > >>> information. > > > > >>> If a new alarm mode is added, It is only necessary to redeploy > the > > > > alert > > > > >>> service. > > > > >>> > > > > >>> > > > > >>> > > > > >>> ________________________________ > > > > >>> DolphinScheduler(Incubator) PPMC > > > > >>> Gang Li 李岗 > > > > >>> > > > > >>> [email protected]<mailto:[email protected]> > > > > >>> > > > > >>> 发件人： qiao zhanwei<mailto:[email protected]> > > > > >>> 发送时间： 2019-12-10 14:24 > > > > >>> 收件人： dev<mailto:[email protected]> > > > > >>> 主题： Aproposal for DolphinScheduler Simplified Deployment > > > > >>> > > > > >>> Hello All , > > > > >>> > > > > >>> Now DolphinScheduler has so many Configuration files > > > > >>> > > > > >>> for example : > > > > >>> > > > > >>> dolphinscheduler-alert : > > > > >>> alert.properties > > > > >>> > > > > >>> dolphinscheduler-api : > > > > >>> application-api.properties > > > > >>> application-combined.properties > > > > >>> > > > > >>> > > > > >>> dolphinscheduler-common : > > > > >>> hadoop.properties > > > > >>> common.properties > > > > >>> quartz.properties > > > > >>> zookeeper.properties > > > > >>> > > > > >>> dolphinscheduler-dao : > > > > >>> application-dao.properties > > > > >>> > > > > >>> dolphinscheduler-server : > > > > >>> application-master.properties > > > > >>> application-master.properties > > > > >>> master.properties > > > > >>> worker.properties > > > > >>> > > > > >>> .dolphinscheduler_env.sh > > > > >>> > > > > >>> Can we simplify deployment ? > > > > >>> > > > > >>> Main point : > > > > >>> > > > > >>> 1 configuration file simplification and merged configuration > file > > > > >>> 2 master server remove port > > > > >>> 3 support offline installation，remove kazoo dependencies in > install > > > and > > > > >>> monitor > > > > >>> 4 instll.sh script simplification > > > > >>> > > > > >>> > > > > >>> ————————————— > > > > >>> DolphinScheduler(Incubator) PPMC > > > > >>> Zhanwei Qiao 乔占卫 > > > > >>> > > > > >>> [email protected] > > > > >>> > > > > >> > > > > > > > > > > > > > > > > > -- > > DolphinScheduler(Incubator) PPMC > > BaoLiang 鲍亮 > > [email protected] > > > > guo jiwei <[email protected]> 于2019年12月11日周三 下午7:33写道： > > > > > To xiaochun. > > > it's not a good way. > > > Alert must be trigger by who is scheduling the task, in DS, it's > > > MasterServer. Because, only by who is scheduling the task, it can know > > the > > > task status in time. If the task is timeout, trigger timeout alert in > > time > > > is very important for users. > > > this is why we have to move alert into server module. > > > Alert implementation will be refactor in the future. not scan db > > anymore. > > > > > > On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <[email protected]> > > > wrote: > > > > > > > To guo jiwei: > > > > > > > > Why not put together with the api server, the alert server function > is > > > > very small, > > > > and the load will not be very high. If logs are stored together in > the > > > > future, > > > > we can combine alert server, log server, api server together, these > can > > > be > > > > called management server. > > > > > > > > Best Regards > > > > --------------- > > > > DolphinScheduler(Incubator) Committer > > > > Xiaochun Liu 刘小春 > > > > [email protected] > > > > --------------- > > > > > > > > > > > > > > > > > 在 2019年12月11日，下午7:15，guo jiwei <[email protected]> 写道： > > > > > > > > > > To ligang. > > > > > it's right. > > > > > But alert server is only a small function. we define it as an > > > > > individual module and as a server. do you thing alert is > expensive > > or > > > > > taking more resource ? if not , why a single module ? > > > > > And alert server trigger task event by scanning db, do you think > it > > > is a > > > > > nice way ? > > > > > Moving into server module is only our first step for simplifying > > user > > > > > deployment. Extension of alert can be updated via redeploy server > and > > > > it's > > > > > not a frequent operation. > > > > > As the architecture changes, alert implementation will change. > > > > > > > > > > > > > > > On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <[email protected]> > > wrote: > > > > > > > > > >> I think from another angle,Master and Worker as key services,I > think > > > not > > > > >> to redeployment during normal execution. > > > > >> If tasks are still running,redeploy master and worker may be lead > > to > > > > >> missed the timed task. > > > > >> > > > > >> ________________________________ > > > > >> DolphinScheduler(Incubator) PPMC > > > > >> Gang Li 李岗 > > > > >> > > > > >> [email protected]<mailto:[email protected]> > > > > >> > > > > >> From: guo jiwei<mailto:[email protected]> > > > > >> Date: 2019-12-11 18:11 > > > > >> To: dev<mailto:[email protected]> > > > > >> Subject: Re: Aproposal for DolphinScheduler Simplified Deployment > > > > >> To ligang. > > > > >> redeploy is simple, but what about the latency of alert ? > > > > >> it's easy to redeploy master server to update alert > > > > >> > > > > >> > > > > >> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <[email protected]> > > wrote: > > > > >> > > > > >>> I think the alert module can be retained. Currently, it only > > supports > > > > >>> email and webchat, > > > > >>> but more alarm modes can be added in the future. > > > > >>> > > > > >>> At present，alert is a independent service. the alert service only > > > > >> consumes > > > > >>> alarm information in the database, other services produce these > > alarm > > > > >>> information. > > > > >>> If a new alarm mode is added, It is only necessary to redeploy > the > > > > alert > > > > >>> service. > > > > >>> > > > > >>> > > > > >>> > > > > >>> ________________________________ > > > > >>> DolphinScheduler(Incubator) PPMC > > > > >>> Gang Li 李岗 > > > > >>> > > > > >>> [email protected]<mailto:[email protected]> > > > > >>> > > > > >>> 发件人： qiao zhanwei<mailto:[email protected]> > > > > >>> 发送时间： 2019-12-10 14:24 > > > > >>> 收件人： dev<mailto:[email protected]> > > > > >>> 主题： Aproposal for DolphinScheduler Simplified Deployment > > > > >>> > > > > >>> Hello All , > > > > >>> > > > > >>> Now DolphinScheduler has so many Configuration files > > > > >>> > > > > >>> for example : > > > > >>> > > > > >>> dolphinscheduler-alert : > > > > >>> alert.properties > > > > >>> > > > > >>> dolphinscheduler-api : > > > > >>> application-api.properties > > > > >>> application-combined.properties > > > > >>> > > > > >>> > > > > >>> dolphinscheduler-common : > > > > >>> hadoop.properties > > > > >>> common.properties > > > > >>> quartz.properties > > > > >>> zookeeper.properties > > > > >>> > > > > >>> dolphinscheduler-dao : > > > > >>> application-dao.properties > > > > >>> > > > > >>> dolphinscheduler-server : > > > > >>> application-master.properties > > > > >>> application-master.properties > > > > >>> master.properties > > > > >>> worker.properties > > > > >>> > > > > >>> .dolphinscheduler_env.sh > > > > >>> > > > > >>> Can we simplify deployment ? > > > > >>> > > > > >>> Main point : > > > > >>> > > > > >>> 1 configuration file simplification and merged configuration > file > > > > >>> 2 master server remove port > > > > >>> 3 support offline installation，remove kazoo dependencies in > install > > > and > > > > >>> monitor > > > > >>> 4 instll.sh script simplification > > > > >>> > > > > >>> > > > > >>> ————————————— > > > > >>> DolphinScheduler(Incubator) PPMC > > > > >>> Zhanwei Qiao 乔占卫 > > > > >>> > > > > >>> [email protected] > > > > >>> > > > > >> > > > > > > > > > > > > > > > > > -- > > DolphinScheduler(Incubator) PPMC > > BaoLiang 鲍亮 > > [email protected] > > >
