Thank you! On Wed, Apr 8, 2020 at 20:47 Ethan Li <[email protected]> wrote:
> In summary, on 2.x, you shouldn't need to worry about PaceMaker. And I > don't suggest to support running older version of topologies running on 2.x > storm cluster. It is painful in practice. > > On Wed, Apr 8, 2020 at 10:44 PM Ethan Li <[email protected]> > wrote: > >> Hi Andrew, >> >> There are two places Pacemaker can be used. >> >> 1. Storm 2.x actually supports running older version of topologies (i.e. >> topology compiled with storm 0.x or 1.x). We have been doing it inside my >> company for a long time. It's kind of painful to support it but it works. >> When older version of topologies running on 2.x Storm cluster, it runs >> older worker code and it can use PaceMaker to mitigate the performance >> issue on zookeeper. Hence the code here still check the heartbeat from >> Zookeeper if any. >> https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L2219-L2222 >> >> >> 2. In 2.x, workers write heartbeats to disk and supervisor pick them up >> and send to nimbus directly. It has another timer (run every 60s by >> default) to send executor status through PaceMaker/Zookeeper. >> https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java#L217-L220 >> And these are executor metrics (emitted, executed, etc.) used on UI. >> This timer only happens every 60s. So it shouldn't overload zookeeper. But >> Pacemaker can still be used here. The graph at >> https://github.com/apache/storm/pull/2389 might help to understand this. >> >> The code is merged at https://github.com/apache/storm/pull/2433 >> >> >> On Wed, Mar 18, 2020 at 6:38 PM Andrew Neilson <[email protected]> >> wrote: >> >>> Hi Ethan, >>> >>> Pacemaker is not required but still can be used. >>>> >>> >>> Under what circumstances would Pacemaker be used on v2.0+? It's not >>> totally clear to me from how that ticket is written but it looks like that >>> replaced all of the heartbeat logic that was managed by Pacemaker and ZK in >>> older versions. >>> >>> Thanks, >>> Andrew >>> >>> On Thu, Mar 5, 2020 at 7:08 PM Ethan Li <[email protected]> >>> wrote: >>> >>>> Hi Surajeet, >>>> >>>> In 2.0, with the change in STORM-2693 >>>> <https://issues.apache.org/jira/browse/STORM-2693>, supervisors will >>>> send worker heartbeat to nimbus directly. Pacemaker is not required but >>>> still can be used. >>>> >>>> >>>> Links: >>>> [1] https://issues.apache.org/jira/browse/STORM-2693 >>>> >>>> Best, >>>> Ethan >>>> >>>> On Feb 18, 2020, at 1:03 PM, Surajeet Dev <[email protected]> >>>> wrote: >>>> >>>> I am aware that Pacemaker was introduced to avoid Zookeeper becoming a >>>> bottleneck when there is high volume of worker heartbeat. >>>> >>>> Will Pacemaker be still required if we upgrade to Storm 2.0? Or is it >>>> with 2.0 , the whole thing has been re-architected? >>>> >>>> Regards >>>> Surajeet >>>> >>>> >>>>
