Re: Worker dies (bolt)

Hei I have made a new test and discovered that in my environment a very simple bolt will die too after around 2500 cycle. Bolt's code: 1 package storm; 2 3 import backtype.storm.task.TopologyContext; 4 import backtype.storm.topology.BasicOutputCollector; 5 import

Re: Worker dies (bolt)

Some new information. Set debug true and from active worker log I can see: if worker is ok: 2014-06-03 11:04:55 b.s.d.task [INFO] Emitting: hbasewriter __ack_ack [7197822474056634252 -608920652033678418] 2014-06-03 11:04:55 b.s.d.executor [INFO] Processing received message source:

RE: Storm with RDBMS

Thanks Alex and Ncleung for your inputs. Both the options looks to be valid based on the data size. I am thinking now, even RDBMS might scale well for this scenario if bolt is using only “Read” operations on database. However, will update later on the approach taken. Regards Balakrishna From:

Fwd: Can storm use the jar from Eclipse?

Hi, Right now, I have my storm cluster setting up (three machines respectively for zookeeper, nimbus, supervisor). Before submitting the topology to the storm cluster, I usually test storm program in local model with Eclipse. And then I use Eclipse to package the storm project and send to nimbus

MultiLag (Python) bolt gives error

Hi, I'm using python bolt which is in the resource directory but storm giving me error *7226 [Thread-19-exclaim1] INFO backtype.storm.daemon.executor - Preparing bolt

Kafka-storm - trident transactional topology - bad perf

I have a simple trident transactional topology that does something like the following: kafka transactional spout (~3000 rec/sec, 6 partitions thus paraHint=6) -- aggregate with reducerAggregator (paraHint=20) -- transactional state (I tried MemoryMapState, MemcachedMapState and CassandraMapState)

Re: Worker dies (bolt)

Ok got more info. Looks like the problem is related with spout. I changed spout: 32 public void open(Map conf, TopologyContext context,SpoutOutputCollector collector) 33 { 34 this.collector = collector; 35 36 Properties props = new

RE: Workers constantly restarted due to session timeout

Thank you Derek for the explanation between :disallowed and :timed-out. That was extremely helpful in understanding what decisions Storm is making. I increased the timeouts for both messages to 5 minutes and returned the zookeeper session timeouts to their default values. This made it plain

Re: Workers constantly restarted due to session timeout

1) Is it appropriate to run Zookeeper in parallel on the same node with the storm services? I recommend separate, and even then to ZK storage to a path on its own disk device if possible. ZK is a bottleneck for storm, and when it is too slow lots of bad things can happen. Some folks use

java.lang.IllegalArgumentException: timeout value is negative

I'm getting this exception 2014-06-03 19:59:13 STDIO [ERROR][id:] Jun 03, 2014 7:59:13 PM org.jboss.netty.channel.DefaultChannelPipeline WARNING: An exception was thrown by a user handler while handling an exception event ([id: 0xdcf3be42] EXCEPTION: java.net.ConnectException: Connection refused)

how does PersistentAggregate distribute the DB Calls ?

How does PersistentAggregate distribute the database calls across all the worked nodes ? Does it do the global aggregation then choose a single host to do a multiget/multiput to the external db ? Thanks -- Raphael Hsieh

Re: how does PersistentAggregate distribute the DB Calls ?

Thanks for your quick reply nathan. So I'm doing some debugging of my topology, and I've removed all the logic from my MultiPut function, replacing it with a single System.out.println() Then i am monitoring my logs to check when this gets printed out. It looks like every single one of my hosts

If I register a metrics object with a bolt/spout, will it run in the same thread as the bolt/spout?

Or do I need to worry about synchronization issue? Thanks in advance!

If I register a metrics object with a bolt/spout task, will it run in the same thread as the task?

Sorry, changed the title to make it more accurate. On Tue, Jun 3, 2014 at 11:12 PM, Xueming Li james.xueming...@gmail.com wrote: Or do I need to worry about synchronization issue? Thanks in advance!