State of Inter-worker communication during the course of a running topology after Connection exceptions

Hello All, I had a question regarding the state of the inter-worker communication while a Topology running, after a worker restart. We have observed in our Topology that a worker gets restarted due to a zookeeper heartbeat timing out. The supervisor daemon on that node promptly restarts the

Mixing java/python with trident

hello We have a Trident topology, and some bolts need to execute python code. currently, we call them by WebService but it's harder to scale... Need to modify the parallellism of this bolt Need to add VM of those webservices, Need to modify haproxy to include those new servers... I know

Question on Parallelsim

I was reading on How many Workers should I use? (link https://storm.apache.org/documentation/FAQ.html#how-many-workers-should-i-use?) and it suggested us to use parallelism hint that is same as the total number of cores in the cluster. I just want to clarify that this parallelism is solely for

Re: Question on Parallelsim

What would be best to do if you have more than the number of cores? For example, we have 4 spout and 13 bolts and our machine has 32 CPUs with 8 cores each.. *Seungtack Baek | Precocity, LLC* Tel/Direct: (972) 378-1030 | Mobile: (214) 477-5715 *seungtackb...@precocityllc.com

Re: Question on Parallelsim

I would say, configure so that your total parallelism matches the number of cores available (i.e. if you have a topology with X spouts, Y boltAs and Z boltBs, make it so that X+Y+Z = cores available). And one worker per machine, inter-JVM communications are expensive. When you have more bolts and

Re: Storm Message Flow Question

It surely did! Thanks for such a precint answer! Thanks, Baek On Jun 8, 2015, at 12:43 AM, Vineet Mishra clearmido...@gmail.com wrote: Any Storm Streaming job runs in its own space and doesn't interact with other topology. Your tuple distribution will be across the topology within the

Re: Best Spout implementation for Reading input Data From File

Hello! You can also have a look at this post: http://stackoverflow.com/questions/24413088/storm-max-spout-pending. It might be helpful. Regards, Florin On Sun, Jun 7, 2015 at 4:17 PM, Nathan Leung ncle...@gmail.com wrote: You should emit with a message id, which will prevent too many

RE: unsubscribe

Hi Mark This doesn't work. I had already sent a tons of unsubscribe requests to user-unsubscr...@storm.apache.org, and didn't got any mail back to confirm I'm unsubscribing.I also had contacted the admin, but got no response. It's really frustrating. Can you please help? Thanks Date: Sat, 6

Re: unsubscribe

Ditto.. From: Cynthia Huynh huynh_cynt...@hotmail.commailto:huynh_cynt...@hotmail.com Reply-To: user@storm.apache.orgmailto:user@storm.apache.org user@storm.apache.orgmailto:user@storm.apache.org Date: Monday, June 8, 2015 at 8:22 AM To: user@storm.apache.orgmailto:user@storm.apache.org

Re: Does Storm keep sending tick tuples to bolts when a topology is deactivated?

Hi! AFAIK, setup-tick! is called from start of executor (which schedules tick timer for each executor), and tick tuples will be emitted unless worker is going to be shutdown. In short, your fellow is correct. Thanks! Jungtaek Lim (HeartSaVioR) 2015-06-08 21:52 GMT+09:00 Dan Blanchard

Storm Trident for kafka not showing up proper emitted/transfered values in UI

Hello, I am trying to use trident topology for Kafka i.e. *TransactionalTridentKafkaSpout*. All works fine except the Storm UI. Even though I've not produced any data to my topic, the Storm UI keeps showing invalid emitted/transferred values. Meaning the count keeps on increasing even when there

Re: Does Storm keep sending tick tuples to bolts when a topology is deactivated?

Hi, I have checked the source of Storm executor.clj, ticks is an independent timer thread and it does not care whether the topology is alive or dead. It just send information to `receive-queue` with stream id `_tick`. But I think there is question that only the `spout` executor can receive

RE: unsubscribe

Same here please . From: bhn...@hotmail.com To: user@storm.apache.org Subject: RE: unsubscribe Date: Mon, 8 Jun 2015 09:00:23 + Hi Mark This doesn't work. I had already sent a tons of unsubscribe requests to user-unsubscr...@storm.apache.org, and didn't got any mail back to confirm

Does Storm keep sending tick tuples to bolts when a topology is deactivated?

As part of the development of streamparse, we have a BatchingBolt that processes tuples in batches. It's intended for use with things like databases that are more performant when you send things in batches. I've recently proposed switching our BatchingBolt implementation over from using a