I started with the same three-node 15-shard configuration I’d been used to, in
an RF1 cluster. (the index is almost 700G so this takes three r4.8xlarge’s if I
want to be entirely memory-resident) I eventually dropped down to a 1/3rd size
index on a single node (so 5 shards, 100M docs each) so I could test
configurations more quickly. The system time usage was present on all solr
nodes regardless. I adjusted for a difference in the CPU count on the EC2 nodes
when I picked my load testing rates.
Zookeeper is a separate cluster on separate nodes. It is NOT collocated with
Solr, although it’s dedicated exclusively to Solr’s use.
I specify a timeout on all queries, and as mentioned, use SOLR-4449. So there’s
possibly an argument I’m doing a lot more timing related calls than most.
There’s nothing particularly exotic there though, just another Executor
Service, and you’ll never get a backup request on an RF1 cluster because
there’s no alternate to try.
On 5/1/17, 6:28 PM, "Walter Underwood" <wun...@wunderwood.org> wrote:
Might want to measure the single CPU performance of your EC2 instance. The
last time I checked, my MacBook was twice as fast as the EC2 instance I was
using.
wunder
Walter Underwood
wun...@wunderwood.org
https://linkprotect.cudasvc.com/url?a=http://observer.wunderwood.org/&c=E,1,L0yDngRyy1MwN7dh5tRFW86sVcn6tcLZH4c03j0EdQSsGBMn0SLDqeB_sHQjB4DdbRMOLka5MnyeXnKS_CEUEv4qIgU5wuyhZBMHciVoH6e8uo7KGr09mXTtDw,,&typo=0
(my blog)
> On May 1, 2017, at 6:24 PM, Chris Hostetter <hossman_luc...@fucit.org>
wrote:
>
>
> : tldr: Recently, I tried moving an existing solrcloud configuration from
> : a local datacenter to EC2. Performance was roughly 1/10th what I’d
> : expected, until I applied a bunch of linux tweaks.
>
> How many total nodes in your cluster? How many of them running ZooKeeper?
>
> Did you observe the heavy increase in system time CPU usage on all nodes,
> or just the ones running zookeeper?
>
> I ask because if your speculation is correct and it is an issue of
> clocksource, then perhaps ZK is where the majority of those system calls
> are happening, and perhaps that's why you didn't see any similar heavy
> system CPU load in ES?
>
> (Then again: at the lowest levels "lucene" really shouldn't care about
> anything clock related at all Any "time" realted code would live in the
> Solr level ... hmmm.)
>
>
> -Hoss
>
https://linkprotect.cudasvc.com/url?a=http://www.lucidworks.com/&c=E,1,ooHM-f4KYxxASNvbLSSYXKwDzWVBK-9orXh84oAZsxzfcPKZ8AF2m_U8K7wc8D5EUvaoHJCrb3O6BPCQIJucBxQaqJMOakPTxCnMW1BDHsyBf13HxMyCeEM_&typo=0
