Hi Ere,
Thanks for your advice! I'm aware of the performance problems with deep
paging and unfortunately it is not the case here, as the rows number is
always 24 and next pages are hardly ever requested from what i see in
the logs.
On 29.10.18 11:19, Ere Maijala wrote:
Hi Sofiya,
You've already received a lot of ideas, but I think this wasn't yet
mentioned: You didn't specify the number of rows your queries fetch or
whether you're using deep paging in the queries. Both can be real
perfomance killers in a sharded index because a large set of records
have to be fetched from all shards. This consumes a relatively high
amount of memory, and even if the servers are able to handle a certain
number of these queries simultaneously, you'd run into garbage
collection trouble with more queries being served. So just one more
thing to be aware of!
Regards,
Ere
Sofiya Strochyk kirjoitti 26.10.2018 klo 18.55:
Hi everyone,
We have a SolrCloud setup with the following configuration:
* 4 nodes (3x128GB RAM Intel Xeon E5-1650v2, 1x64GB RAM Intel Xeon
E5-1650v2, 12 cores, with SSDs)
* One collection, 4 shards, each has only a single replica (so 4
replicas in total), using compositeId router
* Total index size is about 150M documents/320GB, so about 40M/80GB
per node
* Zookeeper is on a separate server
* Documents consist of about 20 fields (most of them are both stored
and indexed), average document size is about2kB
* Queries are mostly 2-3 words in the q field, with 2 fq parameters,
with complex sort expression (containing IF functions)
* We don't use faceting due to performance reasons but need to add it
in the future
* Majority of the documents are reindexed 2 times/day, as fast as the
SOLR allows, in batches of 1000-10000 docs. Some of the documents
are also deleted (by id, not by query)
* autoCommit is set to maxTime of 1 minute with openSearcher=false and
autoSoftCommit maxTime is 30 minutes with openSearcher=true. Commits
from clients are ignored.
* Heap size is set to 8GB.
Target query rate is up to 500 qps, maybe 300, and we need to keep
response time at <200ms. But at the moment we only see very good
search performance with up to 100 requests per second. Whenever it
grows to about 200, average response time abruptly increases to 0.5-1
second. (Also it seems that request rate reported by SOLR in admin
metrics is 2x higher than the real one, because for every query,
every shard receives 2 requests: one to obtain IDs and second one to
get data by IDs; so target rate for SOLR metrics would be 1000 qps).
During high request load, CPU usage increases dramatically on the
SOLR nodes. It doesn't reach 100% but averages at 50-70% on 3 servers
and about 93% on 1 server (random server each time, not the smallest
one).
The documentation mentions replication to spread the load between the
servers. We tested replicating to smaller servers (32GB RAM, Intel
Core i7-4770). However, when we tested it, the replicas were going
out of sync all the time (possibly during commits) and reported
errors like "PeerSync Recovery was not successful - trying
replication." Then they proceed with replication which takes hours
and the leader handles all requests singlehandedly during that time.
Also both leaders and replicas started encountering OOM errors (heap
space) for unknown reason. Heap dump analysis shows that most of the
memory is consumed by [J (array of long) type, my best guess would be
that it is "_version_" field, but it's still unclear why it happens.
Also, even though with replication request rate and CPU usage drop 2
times, it doesn't seem to affect mean_ms, stddev_ms or p95_ms numbers
(p75_ms is much smaller on nodes with replication, but still not as
low as under load of <100 requests/s).
Garbage collection is much more active during high load as well. Full
GC happens almost exclusively during those times. We have tried
tuning GC options like suggested here
<https://wiki.apache.org/solr/ShawnHeisey#CMS_.28ConcurrentMarkSweep.29_Collector>
and it didn't change things though.
My questions are
* How do we increase throughput? Is replication the only solution?
* if yes - then why doesn't it affect response times, considering that
CPU is not 100% used and index fits into memory?
* How to deal with OOM and replicas going into recovery?
* Is memory or CPU the main problem? (When searching on the internet,
i never see CPU as main bottleneck for SOLR, but our case might be
different)
* Or do we need smaller shards? Could segments merging be a problem?
* How to add faceting without search queries slowing down too much?
* How to diagnose these problems and narrow down to the real reason in
hardware or setup?
Any help would be much appreciated.
Thanks!
--
Email Signature
*Sofiia Strochyk
*
s...@interlogic.com.ua <mailto:s...@interlogic.com.ua>
InterLogic
www.interlogic.com.ua <https://www.interlogic.com.ua>
Facebook icon <https://www.facebook.com/InterLogicOfficial> LinkedIn
icon <https://www.linkedin.com/company/interlogic>
--
Email Signature
*Sofiia Strochyk
*
s...@interlogic.com.ua <mailto:s...@interlogic.com.ua>
InterLogic
www.interlogic.com.ua <https://www.interlogic.com.ua>
Facebook icon <https://www.facebook.com/InterLogicOfficial> LinkedIn
icon <https://www.linkedin.com/company/interlogic>
