How does one distribute database iteration across workers?

I've seen this: http://storm.apache.org/releases/0.10.0/Understanding-the-parallelism-of-a-Storm-topology.html but it doesn't explain how workers coordinate with each other, so requesting a bit of clarity. I'm considering a situation where I have 2 million rows in MySQL or MongoDB. 1. I want to

Re: hdfs-bolt write/sync problems

Problems may be in the hadoop, like wrong permission. 2015-04-29 20:51 GMT+08:00 Volker Janz : > Hi, > > we are using the storm-hdfs bolt (0.9.4) to write data from Kafka to > Hadoop (Hadoop 2.5.0-cdh5.2.0). > > This works fine for us but we discovered some unexpected

Re: DRPCExecutionException(msg:Request failed)

i searched and found it may be cause of Grabage collector problem On Mon, Apr 18, 2016 at 9:54 PM, sam mohel wrote: > i got this error in nimbus log file > [ERROR] Connection timed out for connection string (localhost:2181/storm) > and timeout (15000) / elapsed (16002) >

Re: GC overhead limit exceeded

I want to take your opinion , Should i increase RAM of my laptop , is that a good solution for GC OOM ? Thanks On Sat, Apr 16, 2016 at 11:36 PM, sam mohel wrote: > i'm really sorry for my lt questions > problem still but i noticed that when i changed size of worker to be >

Re: DRPCExecutionException(msg:Request failed)

i got this error in nimbus log file [ERROR] Connection timed out for connection string (localhost:2181/storm) and timeout (15000) / elapsed (16002) org.apache.storm.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at

Storm 1.0.0 DRPC connection refused

Hi Good People! I’m trying to update my cluster running Storm 0.10.0 with DRPC to Storm 1.0.0. I’ve updated all machines with latest version of Storm, changed storm.yml configurations (e.g. nimbus.host -> nimbus.seeds), changed dependencies in topology. When I starting nimbus, supervisors, ui,

Re: Losing Tuples

The spout is a KafkaSpout and I only have one spout task. The reason I set the MaxSpoutPendingValue so high was that in the topology, each tuple processed in a bolt tends to create more tuples. So, although the KafkaSpout only receives one message, it results in thousands of tuples downstream.

Re: DRPCExecutionException(msg:Request failed)

Yes I started it . the strange thing that I ran this code before without any problem ! On Monday, April 18, 2016, Spico Florin wrote: > Hi! > Have you started the DRPC server? > Please have a look at: >

Re: Storm 0.10.0 Benchmark issue

Hello again, I am not using kafka-spouts and I did not run any benchmarks on my setup. Can I ask what is your parallelism hint on the kafka-spouts? Because, it might be the case that 7000 t/s might be the ceiling for your parallelism hint and you may have to increase that in order to achieve

Re: DRPCExecutionException(msg:Request failed)

Hi! Have you started the DRPC server? Please have a look at: http://stackoverflow.com/questions/23693871/storm-basicdrpc-client-execute I hope that this help. Florin On Mon, Apr 18, 2016 at 2:18 AM, sam mohel wrote: > i got this error when i submitted topology with