The most basic question is how you are load-testing it? Assuming you have some kind of client firing queries at Solr, keep adding threads so Solr is handling more and more queries in parallel. If you start to see the response time at the client get longer _and_ the QTime in Solr’s response stays about the same, then the queries are queueing up and you need to see about increasing the Jetty threads handling queries.
Second is whether you’re hitting GC pauses, look at the GC logs, especially for “stop the world” pauses. This is unlikely as you’re still getting 60 qps, but something to check. Setting your heap to 31G is good advice, but it won’t dramatically increase the throughput I’d guess. If your I/O isn’t very high, then your index is mostly memory-resident. A general bit of tuning advice is to _reduce_ the heap size, leaving OS memory for the index. See Uwe’s blog: https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html. There’s a sweet spot between having too much heap and too little, and unfortunately you have to experiment to find out. But given the numbers you’re talking about, you won’t be getting 1,000 QPS on a single box and you’ll have to scale out with replicas to hit that number. Getting all the QPS you can out of the box is important, of course. Do be careful to use enough different queries that you don’t get them from the queryResultCache. I had one client who was thrilled they were getting 3ms response times…. by firing the same query over and over and hitting the queryResultCache 99.9999% of the time ;). Best, Erick > On Sep 30, 2019, at 4:28 AM, Yasufumi Mizoguchi <yasufumi0...@gmail.com> > wrote: > > Hi, Ere. > > Thank you for valuable feedback. > I will try Xmx31G and Xms31G instead of current ones. > > Thanks and Regards, > Yasufumi. > > 2019年9月30日(月) 17:19 Ere Maijala <ere.maij...@helsinki.fi>: > >> Just a side note: -Xmx32G is really bad for performance as it forces >> Java to use non-compressed pointers. You'll actually get better results >> with -Xmx31G. For more information, see e.g. >> >> https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/ >> >> Regards, >> Ere >> >> Yasufumi Mizoguchi kirjoitti 30.9.2019 klo 11.05: >>> Hi, Deepak. >>> Thank you for replying me. >>> >>> JVM settings from solr.in.sh file are as follows. (Sorry, I could not >> show >>> all due to our policy) >>> >>> -verbose:gc >>> -XX:+PrintHeapAtGC >>> -XX:+PrintGCDetails >>> -XX:+PrintGCDateStamps >>> -XX:+PrintGCTimeStamps >>> -XX:+PrintTenuringDistribution >>> -XX:+PrintGCApplicationStoppedTime >>> -Dcom.sun.management.jmxremote.ssl=false >>> -Dcom.sun.management.jmxremote.authenticate=false >>> -Dcom.sun.management.jmxremote.port=18983 >>> -XX:OnOutOfMemoryError=/home/solr/solr-6.2.1/bin/oom_solr.sh >>> -XX:NewSize=128m >>> -XX:MaxNewSize=128m >>> -XX:+UseG1GC >>> -XX:+PerfDisableSharedMem >>> -XX:+ParallelRefProcEnabled >>> -XX:G1HeapRegionSize=8m >>> -XX:MaxGCPauseMillis=250 >>> -XX:InitiatingHeapOccupancyPercent=75 >>> -XX:+UseLargePages >>> -XX:+AggressiveOpts >>> -Xmx32G >>> -Xms32G >>> -Xss256k >>> >>> >>> Thanks & Regards >>> Yasufumi. >>> >>> 2019年9月30日(月) 16:12 Deepak Goel <deic...@gmail.com>: >>> >>>> Hello >>>> >>>> Can you please share the JVM heap settings in detail? >>>> >>>> Deepak >>>> >>>> On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, <yasufumi0...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I am trying some tests to confirm if single Solr instance can perform >>>> over >>>>> 1000 queries per second(!). >>>>> >>>>> But now, although CPU usage is 40% or so and iowait is almost 0%, >>>>> throughput does not increase over 60 queries per second. >>>>> >>>>> I think there are some bottlenecks around Kernel, JVM, or Solr >> settings. >>>>> >>>>> The values we already checked and configured are followings. >>>>> >>>>> * Kernel: >>>>> file descriptor >>>>> net.ipv4.tcp_max_syn_backlog >>>>> net.ipv4.tcp_syncookies >>>>> net.core.somaxconn >>>>> net.core.rmem_max >>>>> net.core.wmem_max >>>>> net.ipv4.tcp_rmem >>>>> net.ipv4.tcp_wmem >>>>> >>>>> * JVM: >>>>> Heap [ -> 32GB] >>>>> G1GC settings >>>>> >>>>> * Solr: >>>>> (Jetty) MaxThreads [ -> 20000] >>>>> >>>>> >>>>> And the other info is as follows. >>>>> >>>>> CPU : 16 cores >>>>> RAM : 128 GB >>>>> Disk : SSD 500GB >>>>> NIC : 10Gbps(maybe) >>>>> OS : Ubuntu 14.04 >>>>> JVM : OpenJDK 1.8.0u191 >>>>> Solr : 6.2.1 >>>>> Index size : about 60GB >>>>> >>>>> Any insights will be appreciated. >>>>> >>>>> Thanks and regards, >>>>> Yasufumi. >>>>> >>>> >>> >> >> -- >> Ere Maijala >> Kansalliskirjasto / The National Library of Finland >>
