RE: Internal shard communication - performance?
Hi Tim, Torsten, Please review following threads which covers chatty shard-shard and shard-replica conversations, and since you index large volumes of data it can be a potential bottleneck in your case. http://lucene.472066.n3.nabble.com/Sharding-and-Replication-td4071614.html http://lucene.472066.n3.nabble.com/Performance-vs-maxBufferedAddsPerServer-10-td4080283.html https://issues.apache.org/jira/browse/SOLR-4956 -Original Message- From: Tim Vaillancourt [mailto:t...@elementspace.com] Sent: Monday, August 12, 2013 08:19 To: solr-user@lucene.apache.org Subject: Re: Internal shard communication - performance? For me the biggest deal with increased chatter between SolrCloud is object creation and GCs. The resulting CPU load from the increase GCing seems to affect performance for me in some load tests, but I'm still trying to gather hard numbers on it. Cheers, Tim On 07/08/13 04:05 PM, Shawn Heisey wrote: > On 8/7/2013 2:45 PM, Torsten Albrecht wrote: >> I would like to run zookeeper external at my old master server. >> >> So I have two zookeeper to control my cloud. The third and fourth >> zookeeper will be a virtual machine. > > For true HA with zookepeer, you need at least three instances on > separate physical hardware. If you want to use VMs, that would be > fine, but you must ensure that you aren't running more than one > instance on the same physical server. > > For best results, use an odd number of ZK instances. With three ZK > instances, one can go down and everything still works. With five, two > can go down and everything still works. > > If you've got a fully switched network that's at least gigabit speed, > then the network latency involved in internal communication shouldn't > really matter. > > Thanks, > Shawn >
Re: Internal shard communication - performance?
For me the biggest deal with increased chatter between SolrCloud is object creation and GCs. The resulting CPU load from the increase GCing seems to affect performance for me in some load tests, but I'm still trying to gather hard numbers on it. Cheers, Tim On 07/08/13 04:05 PM, Shawn Heisey wrote: On 8/7/2013 2:45 PM, Torsten Albrecht wrote: I would like to run zookeeper external at my old master server. So I have two zookeeper to control my cloud. The third and fourth zookeeper will be a virtual machine. For true HA with zookepeer, you need at least three instances on separate physical hardware. If you want to use VMs, that would be fine, but you must ensure that you aren't running more than one instance on the same physical server. For best results, use an odd number of ZK instances. With three ZK instances, one can go down and everything still works. With five, two can go down and everything still works. If you've got a fully switched network that's at least gigabit speed, then the network latency involved in internal communication shouldn't really matter. Thanks, Shawn
Re: Internal shard communication - performance?
On 8/7/2013 2:45 PM, Torsten Albrecht wrote: I would like to run zookeeper external at my old master server. So I have two zookeeper to control my cloud. The third and fourth zookeeper will be a virtual machine. For true HA with zookepeer, you need at least three instances on separate physical hardware. If you want to use VMs, that would be fine, but you must ensure that you aren't running more than one instance on the same physical server. For best results, use an odd number of ZK instances. With three ZK instances, one can go down and everything still works. With five, two can go down and everything still works. If you've got a fully switched network that's at least gigabit speed, then the network latency involved in internal communication shouldn't really matter. Thanks, Shawn
Re: Internal shard communication - performance?
Hi Jack, I would like to run zookeeper external at my old master server. So I have two zookeeper to control my cloud. The third and fourth zookeeper will be a virtual machine. Torsten Von: Jack Krupansky Gesendet: ?Mittwoch?, ?7?. ?August? ?2013 ?20?:?05 An: solr-user@lucene.apache.org Three zookeepers give you bare minimum high availability - one can go down. But... I would personally assert that running embedded zookeeper is inherently not "high availability", just by definition (okay, by MY definition.) You didn't say whether you were running embedded zookeeper or not. But if you were, to be HA, your cluster should be able to have all but one node per shard go down and your cluster should still service both queries and updates. But with embedded zookeeper on a four-node cluster, taking down two of the nodes running embedded zookeeper would make zookeeper no longer usable, and hence your cluster would not be HA. -- Jack Krupansky -Original Message- From: Torsten Albrecht Sent: Wednesday, August 07, 2013 1:15 PM To: solr-user Subject: Internal shard communication - performance? Hi, I use a system with solr 3 and 20 shards (3 million docs per shard). At a testsystem with one shard (60 million docs) I get 750 requests per second. At my live system (20 shards) I get 200 requests per second. Is the internal communication between the 20 shards a performance killer? Another question. Is a solr 4 system with solrcloud and Zookeeper a high availability system? Regards, Torsten
Re: Internal shard communication - performance?
Three zookeepers give you bare minimum high availability - one can go down. But... I would personally assert that running embedded zookeeper is inherently not "high availability", just by definition (okay, by MY definition.) You didn't say whether you were running embedded zookeeper or not. But if you were, to be HA, your cluster should be able to have all but one node per shard go down and your cluster should still service both queries and updates. But with embedded zookeeper on a four-node cluster, taking down two of the nodes running embedded zookeeper would make zookeeper no longer usable, and hence your cluster would not be HA. -- Jack Krupansky -Original Message- From: Torsten Albrecht Sent: Wednesday, August 07, 2013 1:15 PM To: solr-user Subject: Internal shard communication - performance? Hi, I use a system with solr 3 and 20 shards (3 million docs per shard). At a testsystem with one shard (60 million docs) I get 750 requests per second. At my live system (20 shards) I get 200 requests per second. Is the internal communication between the 20 shards a performance killer? Another question. Is a solr 4 system with solrcloud and Zookeeper a high availability system? Regards, Torsten
Internal shard communication - performance?
Hi, I use a system with solr 3 and 20 shards (3 million docs per shard). At a testsystem with one shard (60 million docs) I get 750 requests per second. At my live system (20 shards) I get 200 requests per second. Is the internal communication between the 20 shards a performance killer? Another question. Is a solr 4 system with solrcloud and Zookeeper a high availability system? Regards, Torsten
