Quick update for future readers :) The shared collection, using static instances works.
>From this task I learned: - Instantiate in the 'prepare' method and not in constructor - Don't pass shared objects from the main to the bolts - Good (?) old fashioned singleton pattern with synchronized Eyal *Eyal Golan* Connect [image: Gmail] <egola...@gmail.com> [image: LinkedIn] <http://www.linkedin.com/in/egolan74> [image: My Blog] <http://eyalgo.com/> [image: Twitter] <http://twitter.com/eyalgo_egolan> [image: Skype] [image: GitHub] <https://github.com/eyalgo> [image: Facebook] <http://www.facebook.com/eyal.golan.14> P Save a tree. Please don't print this e-mail unless it's really necessary On Thu, Dec 4, 2014 at 11:20 PM, Dima Dragan <dima.dra...@belleron.net> wrote: > Hi Eyal, > > I used the same approach as Michael. > > As far as I understand, your shared collection should be changeable. So > do not forget use concurrent collection. > > Solution for copies by worker are: > > 1) Represent one collection as group of collections (collection per > worker), using storm grouping make sure that input bolts emit to the > appropriate cache-bolts. > > 2) use external storage for state persistence. > > It depends from your needs. > > Best regards, > Dmytro Dragan > On Dec 4, 2014 10:54 PM, "Michael Rose" <mich...@fullcontact.com> wrote: > >> Yes, as long as they're in the same JVM. If you have 4 workers, that's 4 >> JVMs, and 4 copies of the data. >> >> Storm will ensure that if you have 4 bolts of type A, and 8 of type B, >> that 1 of type A and 2 of type B exist in each JVM. Since the singleton's >> field is static, it'll be shared. >> >> Michael Rose (@Xorlev <https://twitter.com/xorlev>) >> Senior Platform Engineer, FullContact <http://www.fullcontact.com/> >> mich...@fullcontact.com >> >> On Thu, Dec 4, 2014 at 1:34 PM, Eyal Golan <egola...@gmail.com> wrote: >> >>> Michael, >>> I understand your solution, however, if bolt of type A "wins" the >>> initialization, will bolts of type B see the same instance? >>> >>> Sent from my Android phone >>> On Dec 4, 2014 10:26 PM, "Eyal Golan" <egola...@gmail.com> wrote: >>> >>>> Thanks Michael and Ben ! >>>> >>>> Sent from my Android phone >>>> On Dec 4, 2014 9:53 PM, "Flint, Ben" <bfl...@popcap.com> wrote: >>>> >>>>> I’ve used Hazelcast for use cases like this with great success. It >>>>> makes it relatively straightforward to implement a distributed data >>>>> structure. >>>>> >>>>> From: Eyal Golan <egola...@gmail.com> >>>>> Reply-To: "user@storm.apache.org" <user@storm.apache.org> >>>>> Date: Thursday, December 4, 2014 at 11:40 AM >>>>> To: "user@storm.apache.org" <user@storm.apache.org> >>>>> Subject: Re: Shared static data between bolts >>>>> >>>>> Thanks! >>>>> All the answers give me good idea. >>>>> Here's why I need this list be sharable. >>>>> One bolt needs the list for computation. >>>>> Another bolt will periodically update the list. >>>>> In other words, the list is changeable. >>>>> >>>>> I tried passing it in conf but there are some 3rd party libs in the >>>>> wrapper. >>>>> >>>>> Anyway, I also need to improve the topology. Separate bolt to two etc. >>>>> That way, it would be easier to handle that cache. >>>>> >>>>> One more question, I thought about giving hazelcast a try. >>>>> Create local node and then the bolts will have clients. >>>>> >>>>> What do you think? >>>>> >>>>> Sent from my Android phone >>>>> On Dec 4, 2014 9:14 PM, "Kushan Maskey" < >>>>> kushan.mas...@mmillerassociates.com> wrote: >>>>> >>>>>> Hi Eyal, >>>>>> >>>>>> I had the similar issue earlier. Static objects are not a good way >>>>>> of passing the variables around in Storm due to clustered nature of the >>>>>> Storm. You will see null for all the static variables. >>>>>> >>>>>> This is how I implemented mine, for instance I wanted to pass >>>>>> around the server properties such and host name and username/password. I >>>>>> read the properties file and then created a map of these properties then >>>>>> set it into the Config object. Like below. >>>>>> >>>>>> *In topology:* >>>>>> >>>>>> Map<Stirng, Stirng> map = new HashMap(); >>>>>> map.put("hostname", "localhost"); >>>>>> map.put("username", "testuser"); >>>>>> map.put("password", "testpassord"); >>>>>> map.put("port", "1234"); >>>>>> >>>>>> Config config = new Config(); >>>>>> >>>>>> config.put("MY_MAP", map) >>>>>> *In Bolts:* >>>>>> >>>>>> public void prepare(Map stormConf, TopologyContext context) { >>>>>> >>>>>> HashMap<Stirng, Stirng> map = stormConfig.get("MY_MAP"); >>>>>> >>>>>> } >>>>>> This how i resolved my problem.Try that if that helps. >>>>>> -- >>>>>> Kushan Maskey >>>>>> 817.403.7500 >>>>>> M. Miller & Associates <http://mmillerassociates.com/> >>>>>> kushan.mas...@mmillerassociates.com >>>>>> >>>>>> On Thu, Dec 4, 2014 at 1:01 PM, Eyal Golan <egola...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> I am newbie with Storm and having some issues with static objects in >>>>>>> the topology. >>>>>>> In the main class, where I set the topology, I also set a static >>>>>>> object. It's actually a wrapper for a collection of strings. >>>>>>> This object needs to he shared between different bolts. >>>>>>> It's big so I want it to be only one instance. >>>>>>> >>>>>>> The problem is that when a bolt tries to get this static instance: >>>>>>> example - MyMain.Shared... It's null. Although I instantiated it in the >>>>>>> main method. >>>>>>> >>>>>>> I guess I'm doing something wrong. >>>>>>> >>>>>>> Thanks for any assistance. >>>>>>> >>>>>>> Eyal >>>>>>> >>>>>>> Sent from my Android phone >>>>>>> >>>>>> >>>>>> >>
