In summary, on 2.x, you shouldn't need to worry about PaceMaker. And I don't suggest to support running older version of topologies running on 2.x storm cluster. It is painful in practice.
On Wed, Apr 8, 2020 at 10:44 PM Ethan Li <ethanopensou...@gmail.com> wrote: > Hi Andrew, > > There are two places Pacemaker can be used. > > 1. Storm 2.x actually supports running older version of topologies (i.e. > topology compiled with storm 0.x or 1.x). We have been doing it inside my > company for a long time. It's kind of painful to support it but it works. > When older version of topologies running on 2.x Storm cluster, it runs > older worker code and it can use PaceMaker to mitigate the performance > issue on zookeeper. Hence the code here still check the heartbeat from > Zookeeper if any. > https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L2219-L2222 > > > 2. In 2.x, workers write heartbeats to disk and supervisor pick them up > and send to nimbus directly. It has another timer (run every 60s by > default) to send executor status through PaceMaker/Zookeeper. > https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java#L217-L220 > And these are executor metrics (emitted, executed, etc.) used on UI. > This timer only happens every 60s. So it shouldn't overload zookeeper. But > Pacemaker can still be used here. The graph at > https://github.com/apache/storm/pull/2389 might help to understand this. > > The code is merged at https://github.com/apache/storm/pull/2433 > > > On Wed, Mar 18, 2020 at 6:38 PM Andrew Neilson <arsneil...@gmail.com> > wrote: > >> Hi Ethan, >> >> Pacemaker is not required but still can be used. >>> >> >> Under what circumstances would Pacemaker be used on v2.0+? It's not >> totally clear to me from how that ticket is written but it looks like that >> replaced all of the heartbeat logic that was managed by Pacemaker and ZK in >> older versions. >> >> Thanks, >> Andrew >> >> On Thu, Mar 5, 2020 at 7:08 PM Ethan Li <ethanopensou...@gmail.com> >> wrote: >> >>> Hi Surajeet, >>> >>> In 2.0, with the change in STORM-2693 >>> <https://issues.apache.org/jira/browse/STORM-2693>, supervisors will >>> send worker heartbeat to nimbus directly. Pacemaker is not required but >>> still can be used. >>> >>> >>> Links: >>> [1] https://issues.apache.org/jira/browse/STORM-2693 >>> >>> Best, >>> Ethan >>> >>> On Feb 18, 2020, at 1:03 PM, Surajeet Dev <surajeetapa...@gmail.com> >>> wrote: >>> >>> I am aware that Pacemaker was introduced to avoid Zookeeper becoming a >>> bottleneck when there is high volume of worker heartbeat. >>> >>> Will Pacemaker be still required if we upgrade to Storm 2.0? Or is it >>> with 2.0 , the whole thing has been re-architected? >>> >>> Regards >>> Surajeet >>> >>> >>>
