Hi Andrew, There are two places Pacemaker can be used.
1. Storm 2.x actually supports running older version of topologies (i.e. topology compiled with storm 0.x or 1.x). We have been doing it inside my company for a long time. It's kind of painful to support it but it works. When older version of topologies running on 2.x Storm cluster, it runs older worker code and it can use PaceMaker to mitigate the performance issue on zookeeper. Hence the code here still check the heartbeat from Zookeeper if any. https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L2219-L2222 2. In 2.x, workers write heartbeats to disk and supervisor pick them up and send to nimbus directly. It has another timer (run every 60s by default) to send executor status through PaceMaker/Zookeeper. https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java#L217-L220 And these are executor metrics (emitted, executed, etc.) used on UI. This timer only happens every 60s. So it shouldn't overload zookeeper. But Pacemaker can still be used here. The graph at https://github.com/apache/storm/pull/2389 might help to understand this. The code is merged at https://github.com/apache/storm/pull/2433 On Wed, Mar 18, 2020 at 6:38 PM Andrew Neilson <[email protected]> wrote: > Hi Ethan, > > Pacemaker is not required but still can be used. >> > > Under what circumstances would Pacemaker be used on v2.0+? It's not > totally clear to me from how that ticket is written but it looks like that > replaced all of the heartbeat logic that was managed by Pacemaker and ZK in > older versions. > > Thanks, > Andrew > > On Thu, Mar 5, 2020 at 7:08 PM Ethan Li <[email protected]> wrote: > >> Hi Surajeet, >> >> In 2.0, with the change in STORM-2693 >> <https://issues.apache.org/jira/browse/STORM-2693>, supervisors will >> send worker heartbeat to nimbus directly. Pacemaker is not required but >> still can be used. >> >> >> Links: >> [1] https://issues.apache.org/jira/browse/STORM-2693 >> >> Best, >> Ethan >> >> On Feb 18, 2020, at 1:03 PM, Surajeet Dev <[email protected]> >> wrote: >> >> I am aware that Pacemaker was introduced to avoid Zookeeper becoming a >> bottleneck when there is high volume of worker heartbeat. >> >> Will Pacemaker be still required if we upgrade to Storm 2.0? Or is it >> with 2.0 , the whole thing has been re-architected? >> >> Regards >> Surajeet >> >> >>
