To baoliang： Is alert server distributed ? it's a standalone module with only a function scanning db and triggering alert . Trigger event will delay if in an individual module even not scan db. do you have some good idea to let alert trigger in time ? do you get my point of in time ?
On Wed, Dec 11, 2019 at 8:26 PM leon bao <[email protected]> wrote: > As an open source project, I think it's important to stay open and > extensible. > Our current alert does not have many shortcomings except for scanning the > database with one thread, at the same time, the alert service as a separate > module provides better scalability. > So I don't think it is necessary to merge alert into other modules without > much benefit. > > guo jiwei <[email protected]> 于2019年12月11日周三 下午7:33写道： > > > To xiaochun. > > it's not a good way. > > Alert must be trigger by who is scheduling the task, in DS, it's > > MasterServer. Because, only by who is scheduling the task, it can know > the > > task status in time. If the task is timeout, trigger timeout alert in > time > > is very important for users. > > this is why we have to move alert into server module. > > Alert implementation will be refactor in the future. not scan db > anymore. > > > > On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <[email protected]> > > wrote: > > > > > To guo jiwei: > > > > > > Why not put together with the api server, the alert server function is > > > very small, > > > and the load will not be very high. If logs are stored together in the > > > future, > > > we can combine alert server, log server, api server together, these can > > be > > > called management server. > > > > > > Best Regards > > > --------------- > > > DolphinScheduler(Incubator) Committer > > > Xiaochun Liu 刘小春 > > > [email protected] > > > --------------- > > > > > > > > > > > > > 在 2019年12月11日，下午7:15，guo jiwei <[email protected]> 写道： > > > > > > > > To ligang. > > > > it's right. > > > > But alert server is only a small function. we define it as an > > > > individual module and as a server. do you thing alert is expensive > or > > > > taking more resource ? if not , why a single module ? > > > > And alert server trigger task event by scanning db, do you think it > > is a > > > > nice way ? > > > > Moving into server module is only our first step for simplifying > user > > > > deployment. Extension of alert can be updated via redeploy server and > > > it's > > > > not a frequent operation. > > > > As the architecture changes, alert implementation will change. > > > > > > > > > > > > On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <[email protected]> > wrote: > > > > > > > >> I think from another angle,Master and Worker as key services,I think > > not > > > >> to redeployment during normal execution. > > > >> If tasks are still running,redeploy master and worker may be lead > to > > > >> missed the timed task. > > > >> > > > >> ________________________________ > > > >> DolphinScheduler(Incubator) PPMC > > > >> Gang Li 李岗 > > > >> > > > >> [email protected]<mailto:[email protected]> > > > >> > > > >> From: guo jiwei<mailto:[email protected]> > > > >> Date: 2019-12-11 18:11 > > > >> To: dev<mailto:[email protected]> > > > >> Subject: Re: Aproposal for DolphinScheduler Simplified Deployment > > > >> To ligang. > > > >> redeploy is simple, but what about the latency of alert ? > > > >> it's easy to redeploy master server to update alert > > > >> > > > >> > > > >> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <[email protected]> > wrote: > > > >> > > > >>> I think the alert module can be retained. Currently, it only > supports > > > >>> email and webchat, > > > >>> but more alarm modes can be added in the future. > > > >>> > > > >>> At present，alert is a independent service. the alert service only > > > >> consumes > > > >>> alarm information in the database, other services produce these > alarm > > > >>> information. > > > >>> If a new alarm mode is added, It is only necessary to redeploy the > > > alert > > > >>> service. > > > >>> > > > >>> > > > >>> > > > >>> ________________________________ > > > >>> DolphinScheduler(Incubator) PPMC > > > >>> Gang Li 李岗 > > > >>> > > > >>> [email protected]<mailto:[email protected]> > > > >>> > > > >>> 发件人： qiao zhanwei<mailto:[email protected]> > > > >>> 发送时间： 2019-12-10 14:24 > > > >>> 收件人： dev<mailto:[email protected]> > > > >>> 主题： Aproposal for DolphinScheduler Simplified Deployment > > > >>> > > > >>> Hello All , > > > >>> > > > >>> Now DolphinScheduler has so many Configuration files > > > >>> > > > >>> for example : > > > >>> > > > >>> dolphinscheduler-alert : > > > >>> alert.properties > > > >>> > > > >>> dolphinscheduler-api : > > > >>> application-api.properties > > > >>> application-combined.properties > > > >>> > > > >>> > > > >>> dolphinscheduler-common : > > > >>> hadoop.properties > > > >>> common.properties > > > >>> quartz.properties > > > >>> zookeeper.properties > > > >>> > > > >>> dolphinscheduler-dao : > > > >>> application-dao.properties > > > >>> > > > >>> dolphinscheduler-server : > > > >>> application-master.properties > > > >>> application-master.properties > > > >>> master.properties > > > >>> worker.properties > > > >>> > > > >>> .dolphinscheduler_env.sh > > > >>> > > > >>> Can we simplify deployment ? > > > >>> > > > >>> Main point : > > > >>> > > > >>> 1 configuration file simplification and merged configuration file > > > >>> 2 master server remove port > > > >>> 3 support offline installation，remove kazoo dependencies in install > > and > > > >>> monitor > > > >>> 4 instll.sh script simplification > > > >>> > > > >>> > > > >>> ————————————— > > > >>> DolphinScheduler(Incubator) PPMC > > > >>> Zhanwei Qiao 乔占卫 > > > >>> > > > >>> [email protected] > > > >>> > > > >> > > > > > > > > > > > -- > DolphinScheduler(Incubator) PPMC > BaoLiang 鲍亮 > [email protected] > > guo jiwei <[email protected]> 于2019年12月11日周三 下午7:33写道： > > > To xiaochun. > > it's not a good way. > > Alert must be trigger by who is scheduling the task, in DS, it's > > MasterServer. Because, only by who is scheduling the task, it can know > the > > task status in time. If the task is timeout, trigger timeout alert in > time > > is very important for users. > > this is why we have to move alert into server module. > > Alert implementation will be refactor in the future. not scan db > anymore. > > > > On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <[email protected]> > > wrote: > > > > > To guo jiwei: > > > > > > Why not put together with the api server, the alert server function is > > > very small, > > > and the load will not be very high. If logs are stored together in the > > > future, > > > we can combine alert server, log server, api server together, these can > > be > > > called management server. > > > > > > Best Regards > > > --------------- > > > DolphinScheduler(Incubator) Committer > > > Xiaochun Liu 刘小春 > > > [email protected] > > > --------------- > > > > > > > > > > > > > 在 2019年12月11日，下午7:15，guo jiwei <[email protected]> 写道： > > > > > > > > To ligang. > > > > it's right. > > > > But alert server is only a small function. we define it as an > > > > individual module and as a server. do you thing alert is expensive > or > > > > taking more resource ? if not , why a single module ? > > > > And alert server trigger task event by scanning db, do you think it > > is a > > > > nice way ? > > > > Moving into server module is only our first step for simplifying > user > > > > deployment. Extension of alert can be updated via redeploy server and > > > it's > > > > not a frequent operation. > > > > As the architecture changes, alert implementation will change. > > > > > > > > > > > > On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <[email protected]> > wrote: > > > > > > > >> I think from another angle,Master and Worker as key services,I think > > not > > > >> to redeployment during normal execution. > > > >> If tasks are still running,redeploy master and worker may be lead > to > > > >> missed the timed task. > > > >> > > > >> ________________________________ > > > >> DolphinScheduler(Incubator) PPMC > > > >> Gang Li 李岗 > > > >> > > > >> [email protected]<mailto:[email protected]> > > > >> > > > >> From: guo jiwei<mailto:[email protected]> > > > >> Date: 2019-12-11 18:11 > > > >> To: dev<mailto:[email protected]> > > > >> Subject: Re: Aproposal for DolphinScheduler Simplified Deployment > > > >> To ligang. > > > >> redeploy is simple, but what about the latency of alert ? > > > >> it's easy to redeploy master server to update alert > > > >> > > > >> > > > >> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <[email protected]> > wrote: > > > >> > > > >>> I think the alert module can be retained. Currently, it only > supports > > > >>> email and webchat, > > > >>> but more alarm modes can be added in the future. > > > >>> > > > >>> At present，alert is a independent service. the alert service only > > > >> consumes > > > >>> alarm information in the database, other services produce these > alarm > > > >>> information. > > > >>> If a new alarm mode is added, It is only necessary to redeploy the > > > alert > > > >>> service. > > > >>> > > > >>> > > > >>> > > > >>> ________________________________ > > > >>> DolphinScheduler(Incubator) PPMC > > > >>> Gang Li 李岗 > > > >>> > > > >>> [email protected]<mailto:[email protected]> > > > >>> > > > >>> 发件人： qiao zhanwei<mailto:[email protected]> > > > >>> 发送时间： 2019-12-10 14:24 > > > >>> 收件人： dev<mailto:[email protected]> > > > >>> 主题： Aproposal for DolphinScheduler Simplified Deployment > > > >>> > > > >>> Hello All , > > > >>> > > > >>> Now DolphinScheduler has so many Configuration files > > > >>> > > > >>> for example : > > > >>> > > > >>> dolphinscheduler-alert : > > > >>> alert.properties > > > >>> > > > >>> dolphinscheduler-api : > > > >>> application-api.properties > > > >>> application-combined.properties > > > >>> > > > >>> > > > >>> dolphinscheduler-common : > > > >>> hadoop.properties > > > >>> common.properties > > > >>> quartz.properties > > > >>> zookeeper.properties > > > >>> > > > >>> dolphinscheduler-dao : > > > >>> application-dao.properties > > > >>> > > > >>> dolphinscheduler-server : > > > >>> application-master.properties > > > >>> application-master.properties > > > >>> master.properties > > > >>> worker.properties > > > >>> > > > >>> .dolphinscheduler_env.sh > > > >>> > > > >>> Can we simplify deployment ? > > > >>> > > > >>> Main point : > > > >>> > > > >>> 1 configuration file simplification and merged configuration file > > > >>> 2 master server remove port > > > >>> 3 support offline installation，remove kazoo dependencies in install > > and > > > >>> monitor > > > >>> 4 instll.sh script simplification > > > >>> > > > >>> > > > >>> ————————————— > > > >>> DolphinScheduler(Incubator) PPMC > > > >>> Zhanwei Qiao 乔占卫 > > > >>> > > > >>> [email protected] > > > >>> > > > >> > > > > > > > > > > > -- > DolphinScheduler(Incubator) PPMC > BaoLiang 鲍亮 > [email protected] >
