how can i increase size of heap

i want to increase size of heap for worker to -Xms4G how can i write it in storm.yaml ? and how can i set the number of CPU consuming bolts to the amount of cores ? Thanks for any help

Question on SolrUpdateBolt

Hi, I have a question on SolrUpdateBolt.execute() method. It seems that SolrUpdateBolt is sending every tuple to Solr in the execute() method but sending a commit()

Is Storm 1.0.0 compatible with Kafka 0.8.2.x?

Hi Everyone, I know that Storm 1.0.0 is designed to work with Kafka 0.9.0.1, but is it possible for Storm 1.0.0 to work with 0.8.2.1? I did some integration testing with 0.9.0.1 client with 0.8.2.1 broker, which is yucky, and it appears this won't work. I checked out the v1.0.0 tag, changed the

Re: Storm Metrics Consumer Not Receiving Tuples

I took thread dumps of the worker where the graphite consumer bolt executor was running but I didn't see any BLOCKED threads or anything out of the ordinary. This is the thread dump for the graphite metrics consumer bolt: "Thread-23-__metricscom.verisign.storm.metrics.GraphiteMetricsConsumer"

Re: Storm Metrics Consumer Not Receiving Tuples

You might want to check the thread dump and verify if some bolt is stuck somewhere Excuse typos On Apr 15, 2016 11:08 PM, "Kevin Conaway" wrote: > Was the bolt really "stuck" though given that the failure was at the spout > level (because the spout couldn't connect to

Re: Storm Metrics Consumer Not Receiving Tuples

Was the bolt really "stuck" though given that the failure was at the spout level (because the spout couldn't connect to the Kafka broker)? Additionally, we restarted the Kafka broker and it seemed like the spout was able to reconnect but we never saw messages from through on the metric consumer

Re: Storm Metrics Consumer Not Receiving Tuples

Kevin, That would explain it. A stuck bolt will stall the whole topology. MetricConsumer runs as a bolt so it will be blocked as well Excuse typos On Apr 15, 2016 10:29 PM, "Kevin Conaway" wrote: > Two more data points on this: > > 1.) We are registering the graphite

Re: Losing Tuples

I've placed two logs in the bolts to verify that tuples are missing. One log right before the tuple is emitted and another at the beginning of the execute method for the downstream bolt. These logs should contain the same statements, however, the downstream bolt is lacking close to 1000 of the

Re: Storm Metrics Consumer Not Receiving Tuples

Two more data points on this: 1.) We are registering the graphite MetricsConsumer on our Topology Config, not globally in storm.yaml. I don't know if this makes a difference. 2.) We re-ran another test last night and it ran fine for about 6 hours until the Kafka brokers ran out of disk space

Losing Tuples

Hi all, I've recently run into a problem where my topology seems to be losing tuples after some continuous processing. That is the number of tuples emitted from one bolt doesn't equal the number of tuples ack'ed for the downstream bolt. It's also not reporting any tuples as having failed, I

Re: initState method not invoked in Storm 1.0

Yes the static dispatch of overloads can cause issues here. Plan to address in https://issues.apache.org/jira/browse/STORM-1714. Thanks for bringing this up. From: Alexander T Reply-To: "user@storm.apache.org" Date: Friday, April 15, 2016 at 3:02 PM To: "user@storm.apache.org" Subject: Re:

Re: storm debug messages in worker logs

Add -XX:+PrintFlagsFinal and -XX:+PrintFlagsInitial to worker.childopts, and show all provided arguments. 2016-04-15 2:05 GMT+03:00 : > I don't have topology.debug defined anywhere. > > > To be clear, my log4j config is: > > > protocol="UDP"

Re: initState method not invoked in Storm 1.0

Hi Arun, Yes, that is one way of getting it wrong. Another way is assigning your bolt to a variable of a stateless type before doing setBolt. Since Java lacks dynamic dispatch, the stateless overload will be chosen in that case. Regards, Alex On Apr 15, 2016 11:16 AM, "Arun Mahadevan"

Re: initState method not invoked in Storm 1.0

Yes, In the stateful bolts you need to anchor the tuples while emitting and ack them. If you have any non-stateful bolts in the same topology they should also do the anchoring/acking (or extend BaseBasicBolt). From: Spico Florin Reply-To: "user@storm.apache.org" Date: Friday, April 15,

Re: initState method not invoked in Storm 1.0

Hi! Thank you for your answers. Yes I will provide the hint parallelism. Regarding, the constraint that I'm facing should I emit from both spout and stateful bolt anchored tuples (that way the the exception was complaining)? I look forward for your answers, Florin On Fri, Apr 15, 2016 at 12:16

Re: initState method not invoked in Storm 1.0

Ah, I see what you mean. The “setBolt” method without parallelism hint is not overloaded for stateful bolts so if parallelism hint is not specified it ends up as being normal bolt. Will raise a JIRA for fixing this. Spico, For now, can you provide parallelism hint while you add stateful bolt

Re: initState method not invoked in Storm 1.0

Hi, Alex! There is no special method addBolt in the API TopologyBuilder for . I have used the topologyBuilder.setBolt("myId", new TimeSeriesStatefulBolt()) without parallelism_hint declared and the API dismiss the statefulness. I did NOT get any errors in the Logs. What I have found is the

Re: initState method not invoked in Storm 1.0

Hi Arun, I meant that it's very easy to use the wrong setBolt method overload by mistake, since stateful bolts are supertypes of stateless ones. Regards Alex On Apr 15, 2016 10:54 AM, "Arun Mahadevan" wrote: Its the same method (builder.setBolt) that adds stateful bolts to a

Re: initState method not invoked in Storm 1.0

Hi Spico, Are you adding your bolt to the topology with the special methods for stateful bolts? It's quite easy to use the regular addBolt method and it will in that case be treated as a stateless one. Cheers Alex On Apr 15, 2016 10:33 AM, "Spico Florin" wrote: > Hello!

Re: where are my jobs ?

I ran into a very similar scenario. Disk filling up --> zookeeper crash --> topologies disappear. The lesson learnt was to isolate zookeeper from other processes which should have been done from the start. Here is the code which cleans up topologies -

Re: Storm transaction boundary

The I/O with redis would concern me, admittedly I don't have much exp with it. You can't write into your destination store directly as they come in? Do you need some kind of transactional support on the destination too ? What are you writing into ? In cases where you need transactional support

Re: Storm transaction boundary

Thanks John for the details on storm design. I am planning below approach for our design. Keep the messages in redis and increment the message count. Have a key for total count from the last message. This message can come at any time since ordering is not guaranteed. Each time you receive a