Hi Michael,
If a bolt of type A on a JVM modifies the shared data, will another bolt of
type B on another JVM see the changes? To be clear, I have a use-case where I
need multiple bolts of multiple types to be able to read/write on the shared
collection. You can think of it as a shared database. In this case, can I still
use this solution or do I need to use Hazelcast/Redis/Memcached?
Best regards,
Huy, Le Van
On Thursday, Dec 4, 2014 at 8:53 p.m., Michael Rose <mich...@fullcontact.com>,
wrote:
Yes, as long as they're in the same JVM. If you have 4 workers, that's 4 JVMs,
and 4 copies of the data.
Storm will ensure that if you have 4 bolts of type A, and 8 of type B, that 1
of type A and 2 of type B exist in each JVM. Since the singleton's field is
static, it'll be shared.
Michael Rose (@Xorlev)
Senior Platform Engineer, FullContact
mich...@fullcontact.com
On Thu, Dec 4, 2014 at 1:34 PM, Eyal Golan <egola...@gmail.com> wrote:
Michael,
I understand your solution, however, if bolt of type A "wins" the
initialization, will bolts of type B see the same instance?
Sent from my Android phone
On Dec 4, 2014 10:26 PM, "Eyal Golan" <egola...@gmail.com> wrote:
Thanks Michael and Ben !
Sent from my Android phone
On Dec 4, 2014 9:53 PM, "Flint, Ben" <bfl...@popcap.com> wrote:
I’ve used Hazelcast for use cases like this with great success. It makes it
relatively straightforward to implement a distributed data structure.
From: Eyal Golan <egola...@gmail.com>
Reply-To: "user@storm.apache.org" <user@storm.apache.org>
Date: Thursday, December 4, 2014 at 11:40 AM
To: "user@storm.apache.org" <user@storm.apache.org>
Subject: Re: Shared static data between bolts
Thanks!
All the answers give me good idea.
Here's why I need this list be sharable.
One bolt needs the list for computation.
Another bolt will periodically update the list.
In other words, the list is changeable.
I tried passing it in conf but there are some 3rd party libs in the wrapper.
Anyway, I also need to improve the topology. Separate bolt to two etc. That
way, it would be easier to handle that cache.
One more question, I thought about giving hazelcast a try.
Create local node and then the bolts will have clients.
What do you think?
Sent from my Android phone
On Dec 4, 2014 9:14 PM, "Kushan Maskey" <kushan.mas...@mmillerassociates.com>
wrote:
Hi Eyal,
I had the similar issue earlier. Static objects are not a good way of passing
the variables around in Storm due to clustered nature of the Storm. You will
see null for all the static variables.
This is how I implemented mine, for instance I wanted to pass around the server
properties such and host name and username/password. I read the properties file
and then created a map of these properties then set it into the Config object.
Like below.
In topology:
Map<Stirng, Stirng> map = new HashMap();
map.put("hostname", "localhost");
map.put("username", "testuser");
map.put("password", "testpassord");
map.put("port", "1234");
Config config = new Config();
config.put("MY_MAP", map)
In Bolts:
public void prepare(Map stormConf, TopologyContext context) {
HashMap<Stirng, Stirng> map = stormConfig.get("MY_MAP");
}
This how i resolved my problem.Try that if that helps.
--
Kushan Maskey
817.403.7500
M. Miller & Associates
kushan.mas...@mmillerassociates.com
On Thu, Dec 4, 2014 at 1:01 PM, Eyal Golan
<egola...@gmail.com> wrote:
Hi all,
I am newbie with Storm and having some issues with static objects in the
topology.
In the main class, where I set the topology, I also set a static object. It's
actually a wrapper for a collection of strings.
This object needs to he shared between different bolts.
It's big so I want it to be only one instance.
The problem is that when a bolt tries to get this static instance: example -
MyMain.Shared... It's null. Although I instantiated it in the main method.
I guess I'm doing something wrong.
Thanks for any assistance.
Eyal
Sent from my Android phone
