Darsh, I am in a bit of a crush for a deployment and perf testing but I will ask my experts to take a look tomorrow. I will set up a single nimbus and 3 zookeepers in one data center, then I will distribute the supervisors and see what happens. I think we will always be limited in how we back up nimbus but I think we should be able to share processing for supervisors across data centers.
Keep in touch and let me know if this is really what you are looking to test. I have a lot of toys at the moment and this seems like a worth while test for everyone. Patrick Patrick Brinton Sr. Engineer SWLM | ¤Target | 7000 Target Parkway North | Brooklyn Park, MN 55445 | 612.599.6523 (ph) | patrick.brin...@target.com<mailto:c...@target.com> From: Darsh <darsh.pa...@gmail.com<mailto:darsh.pa...@gmail.com>> Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>> Date: Thursday, April 28, 2016 at 5:29 PM To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>> Subject: Re: Distribute load across multiple storm clusters. Patrick, Thank you for replying. I did try but load isn't distributed. Both clusters are processing all the events individually on the topic. Darsh On Thu, Apr 28, 2016 at 2:56 PM, Patrick.Brinton <patrick.brin...@target.com<mailto:patrick.brin...@target.com>> wrote: Darsh, I have never tried but I have a setup where I could try. As long as you point to the same zooKeeper I think it would work. Give it a try and let me know if you hit issues. If you do I will mimic your setup and we should be able to figure it out. Patrick Patrick Brinton Sr. Engineer SWLM | ¤Target | 7000 Target Parkway North | Brooklyn Park, MN 55445 | 612.599.6523<tel:612.599.6523> (ph) | patrick.brin...@target.com<mailto:c...@target.com> From: Darsh <darsh.pa...@gmail.com<mailto:darsh.pa...@gmail.com>> Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>> Date: Thursday, April 28, 2016 at 12:39 PM To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>> Subject: Distribute load across multiple storm clusters. Hi, Is it possible to to distribute load across 2 clusters if I deploy same topology to 2 storm clusters with same spout id? We have 2 separate storm clusters with 0.10.0 version of storm running in 2 different availability zones. We are using storm-kafka spout (with simple consumer) to process data from kafka and external common zookeeper to store the kafka offsets.Kafka Topic has 32 partitions. 8 executors(parallelism_hint) for spout in each cluster. Thanks Darsh -- Thanks Darsh
