Yes that message means that ZK does not always inform you of a change. This
mostly happens when reconnecting after losing a connection.
- Bobby
On Wednesday, March 25, 2015 3:44 AM, Erik Weathers
<eweath...@groupon.com> wrote:
Thanks for the response Bobby, on both fronts.
Can you please confirm my new understanding of this potentially misleading
comment above *mk-storm-cluster-state?*
https://github.com/apache/storm/blob/v0.9.3/storm-core/src/clj/backtype/storm/cluster.clj#L233
;; Watches should be used for optimization. When ZK is reconnecting,
they're not guaranteed to be called.
This comment sounds like a TODO, but based on your response I assume the
code is already using ZooKeeper watches. So now I believe that this
comment is instead warning potential callers that these watch-based
callbacks aren't reliable. Is that correct?
Thanks again!
- Erik
On Tue, Mar 24, 2015 at 6:49 AM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:
> Two things. First storm uses zookeeper to store state and offers a watch
> that can inform a client when a particular znode in zookeeper changes. The
> callbacks provide a way for a client to be informed that the particular
> part of the state they care about has changed, or nil if you don't want
> it. The ZK functionality is not always perfect in getting the callbacks,
> so it is always a good idea to back up the callback with polling
> periodically. Also at least for people I know working on DRUID the order
> of the callbacks on certain versions of ZK can be out of order, so if there
> are a lot of changes happening to a single ZNode you may need to be
> careful, luckily this is not an issue for how storm currently uses ZK.
>
> Now for the second thing. JStorm is a fork of Apache Storm where the
> clojure code was translated into java code, so if you are having trouble
> reading the clojure code you can look at what JStorm is doing for a hint.
> Be aware that the two projects have diverged somewhat so they are not going
> to be identical in their functionality.
> That being said both projects have been talking with one another about
> combining. https://issues.apache.org/jira/browse/STORM-717 is a JIRA to
> work through making that happen. If you or anyone else has an opinion on
> this please feel free to discuss it on that JIRA on here on the dev list,
> I'll send out another e-mail to provide a better place than piggybacking it
> here. The goal would be to maintain binary compatibility, except possibly
> in the case of dependencies. It would be great to have both groups working
> together instead of duplication of effort. We also felt that having more
> of the code base in java would possibly make it more accessible to a wider
> range of developers. That being said this project is a community effort so
> anyone who wants to help, or has opinions on this please help out/let us
> know what you think. - Bobby
>
>
>
> On Tuesday, March 24, 2015 12:51 AM, Erik Weathers <
> eweath...@groupon.com> wrote:
>
>
> hi Longda, thanks for the response. Interesting project there. I would
> appreciate seeing the architecture/work-flow file if you could please share
> it.
> However, we are using standard Storm, and I need to continue supporting our
> existing system -- which requires me to dig into the Clojure-based storm
> code.
>
> Any other kind passersby that can help me?
>
> - Erik
>
> On Monday, March 23, 2015, 封仲淹(纪君祥) <zhongyan.f...@alibaba-inc.com> wrote:
>
> >
> > You can read the source code of Jstorm, It is java Storm. It is easy to
> > read
> > https://github.com/alibaba/jstorm
> > I can share one freemind file to you to demonstrate the architecture and
> > work flow.
> >
> >
> >
> > Best Regards
> > Longda
> >
> >
> > -----邮件原件-----
> > 发件人: Erik Weathers [mailto:eweath...@groupon.com]
> > 发送时间: 2015年3月24日 8:58
> > 收件人: dev@storm.apache.org
> > 主题: begging for a bit of help reading storm-core code
> >
> > hi Storm Devs!
> >
> > I've been trying to discern what the purpose is for the ubiquitous
> > "callback" variables in cluster.clj and supervisor.clj code. It occurred
> > to me that some of you fine folks on the dev list must have already
> braved
> > this test and could help dispel my ignorance. Specifically, I'm trying
> to
> > determine what the mk-storm-cluster-state function is actually doing, and
> > understanding the callbacks seems critical to that endeavor.
> >
> > I had hoped that commit log messages might help me infer the point of
> > these callbacks, but this particular pattern hasn't changed since the
> > initial commit by Nathan Marz.
> >
> > Thanks for whatever help you can provide!
> >
> > - Erik
> >
> > P.S., I have many other similar questions about the storm-core Clojure
> > code, so I wonder if someone might have a suggestion of a different route
> > for to requesting such code-reading help?
> >
> >
>
>
>
