I started with the same three-node 15-shard configuration I’d been used to, in an RF1 cluster. (the index is almost 700G so this takes three r4.8xlarge’s if I want to be entirely memory-resident) I eventually dropped down to a 1/3rd size index on a single node (so 5 shards, 100M docs each) so I could test configurations more quickly. The system time usage was present on all solr nodes regardless. I adjusted for a difference in the CPU count on the EC2 nodes when I picked my load testing rates.
Zookeeper is a separate cluster on separate nodes. It is NOT collocated with Solr, although it’s dedicated exclusively to Solr’s use. I specify a timeout on all queries, and as mentioned, use SOLR-4449. So there’s possibly an argument I’m doing a lot more timing related calls than most. There’s nothing particularly exotic there though, just another Executor Service, and you’ll never get a backup request on an RF1 cluster because there’s no alternate to try. On 5/1/17, 6:28 PM, "Walter Underwood" <wun...@wunderwood.org> wrote: Might want to measure the single CPU performance of your EC2 instance. The last time I checked, my MacBook was twice as fast as the EC2 instance I was using. wunder Walter Underwood wun...@wunderwood.org https://linkprotect.cudasvc.com/url?a=http://observer.wunderwood.org/&c=E,1,L0yDngRyy1MwN7dh5tRFW86sVcn6tcLZH4c03j0EdQSsGBMn0SLDqeB_sHQjB4DdbRMOLka5MnyeXnKS_CEUEv4qIgU5wuyhZBMHciVoH6e8uo7KGr09mXTtDw,,&typo=0 (my blog) > On May 1, 2017, at 6:24 PM, Chris Hostetter <hossman_luc...@fucit.org> wrote: > > > : tldr: Recently, I tried moving an existing solrcloud configuration from > : a local datacenter to EC2. Performance was roughly 1/10th what I’d > : expected, until I applied a bunch of linux tweaks. > > How many total nodes in your cluster? How many of them running ZooKeeper? > > Did you observe the heavy increase in system time CPU usage on all nodes, > or just the ones running zookeeper? > > I ask because if your speculation is correct and it is an issue of > clocksource, then perhaps ZK is where the majority of those system calls > are happening, and perhaps that's why you didn't see any similar heavy > system CPU load in ES? > > (Then again: at the lowest levels "lucene" really shouldn't care about > anything clock related at all Any "time" realted code would live in the > Solr level ... hmmm.) > > > -Hoss > https://linkprotect.cudasvc.com/url?a=http://www.lucidworks.com/&c=E,1,ooHM-f4KYxxASNvbLSSYXKwDzWVBK-9orXh84oAZsxzfcPKZ8AF2m_U8K7wc8D5EUvaoHJCrb3O6BPCQIJucBxQaqJMOakPTxCnMW1BDHsyBf13HxMyCeEM_&typo=0