Ah, sorry. Not JUnit, we use JMeter.
Thanks, Yasufumi 2019年10月1日(火) 19:08 Yasufumi Mizoguchi <yasufumi0...@gmail.com>: > Thank you for replying me. > > Followings are current load test set up. > > * Load test program : JUnit > * The number of Firing hosts : 6 > * [Each host]ThreadGroup.num_threads : 200 > * [Each host]ThreadGroup.ramp_time : 600 > * [Each host]ThreadGroup.duration: 1800 > * [Each host]ThreadGroup.delay: 0 > * The number of sample queries: 20,000,000 > > And we confirmed that Jetty threads was increasing and reached the limit > (10000). > Therefore, we raised MaxThreads value. > > I checked GC logs and found that it happened no major GC, and almost all > minor GC were finished by 200ms. > > Cache hit rate is not checked now, but I think those are extremely low all > kinds of cache. > Because the number of sample query is big(20,000,000) compared to > queryResult and filter cache size(both 512) and there are few duplication > in fq and q parameter. > Those small cache size are intended, but I want to change those.... > > Thanks, > Yasufumi > > > > 2019年9月30日(月) 20:49 Erick Erickson <erickerick...@gmail.com>: > >> The most basic question is how you are load-testing it? Assuming you have >> some kind of client firing queries at Solr, keep adding threads so Solr is >> handling more and more queries in parallel. If you start to see the >> response time at the client get longer _and_ the QTime in Solr’s response >> stays about the same, then the queries are queueing up and you need to see >> about increasing the Jetty threads handling queries. >> >> Second is whether you’re hitting GC pauses, look at the GC logs, >> especially for “stop the world” pauses. This is unlikely as you’re still >> getting 60 qps, but something to check. >> >> Setting your heap to 31G is good advice, but it won’t dramatically >> increase the throughput I’d guess. >> >> If your I/O isn’t very high, then your index is mostly memory-resident. A >> general bit of tuning advice is to _reduce_ the heap size, leaving OS >> memory for the index. See Uwe’s blog: >> https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html. >> There’s a sweet spot between having too much heap and too little, and >> unfortunately you have to experiment to find out. >> >> But given the numbers you’re talking about, you won’t be getting 1,000 >> QPS on a single box and you’ll have to scale out with replicas to hit that >> number. Getting all the QPS you can out of the box is important, of course. >> Do be careful to use enough different queries that you don’t get them from >> the queryResultCache. I had one client who was thrilled they were getting >> 3ms response times…. by firing the same query over and over and hitting the >> queryResultCache 99.9999% of the time ;). >> >> Best, >> Erick >> >> > On Sep 30, 2019, at 4:28 AM, Yasufumi Mizoguchi <yasufumi0...@gmail.com> >> wrote: >> > >> > Hi, Ere. >> > >> > Thank you for valuable feedback. >> > I will try Xmx31G and Xms31G instead of current ones. >> > >> > Thanks and Regards, >> > Yasufumi. >> > >> > 2019年9月30日(月) 17:19 Ere Maijala <ere.maij...@helsinki.fi>: >> > >> >> Just a side note: -Xmx32G is really bad for performance as it forces >> >> Java to use non-compressed pointers. You'll actually get better results >> >> with -Xmx31G. For more information, see e.g. >> >> >> >> >> https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/ >> >> >> >> Regards, >> >> Ere >> >> >> >> Yasufumi Mizoguchi kirjoitti 30.9.2019 klo 11.05: >> >>> Hi, Deepak. >> >>> Thank you for replying me. >> >>> >> >>> JVM settings from solr.in.sh file are as follows. (Sorry, I could not >> >> show >> >>> all due to our policy) >> >>> >> >>> -verbose:gc >> >>> -XX:+PrintHeapAtGC >> >>> -XX:+PrintGCDetails >> >>> -XX:+PrintGCDateStamps >> >>> -XX:+PrintGCTimeStamps >> >>> -XX:+PrintTenuringDistribution >> >>> -XX:+PrintGCApplicationStoppedTime >> >>> -Dcom.sun.management.jmxremote.ssl=false >> >>> -Dcom.sun.management.jmxremote.authenticate=false >> >>> -Dcom.sun.management.jmxremote.port=18983 >> >>> -XX:OnOutOfMemoryError=/home/solr/solr-6.2.1/bin/oom_solr.sh >> >>> -XX:NewSize=128m >> >>> -XX:MaxNewSize=128m >> >>> -XX:+UseG1GC >> >>> -XX:+PerfDisableSharedMem >> >>> -XX:+ParallelRefProcEnabled >> >>> -XX:G1HeapRegionSize=8m >> >>> -XX:MaxGCPauseMillis=250 >> >>> -XX:InitiatingHeapOccupancyPercent=75 >> >>> -XX:+UseLargePages >> >>> -XX:+AggressiveOpts >> >>> -Xmx32G >> >>> -Xms32G >> >>> -Xss256k >> >>> >> >>> >> >>> Thanks & Regards >> >>> Yasufumi. >> >>> >> >>> 2019年9月30日(月) 16:12 Deepak Goel <deic...@gmail.com>: >> >>> >> >>>> Hello >> >>>> >> >>>> Can you please share the JVM heap settings in detail? >> >>>> >> >>>> Deepak >> >>>> >> >>>> On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, < >> yasufumi0...@gmail.com> >> >>>> wrote: >> >>>> >> >>>>> Hi, >> >>>>> >> >>>>> I am trying some tests to confirm if single Solr instance can >> perform >> >>>> over >> >>>>> 1000 queries per second(!). >> >>>>> >> >>>>> But now, although CPU usage is 40% or so and iowait is almost 0%, >> >>>>> throughput does not increase over 60 queries per second. >> >>>>> >> >>>>> I think there are some bottlenecks around Kernel, JVM, or Solr >> >> settings. >> >>>>> >> >>>>> The values we already checked and configured are followings. >> >>>>> >> >>>>> * Kernel: >> >>>>> file descriptor >> >>>>> net.ipv4.tcp_max_syn_backlog >> >>>>> net.ipv4.tcp_syncookies >> >>>>> net.core.somaxconn >> >>>>> net.core.rmem_max >> >>>>> net.core.wmem_max >> >>>>> net.ipv4.tcp_rmem >> >>>>> net.ipv4.tcp_wmem >> >>>>> >> >>>>> * JVM: >> >>>>> Heap [ -> 32GB] >> >>>>> G1GC settings >> >>>>> >> >>>>> * Solr: >> >>>>> (Jetty) MaxThreads [ -> 20000] >> >>>>> >> >>>>> >> >>>>> And the other info is as follows. >> >>>>> >> >>>>> CPU : 16 cores >> >>>>> RAM : 128 GB >> >>>>> Disk : SSD 500GB >> >>>>> NIC : 10Gbps(maybe) >> >>>>> OS : Ubuntu 14.04 >> >>>>> JVM : OpenJDK 1.8.0u191 >> >>>>> Solr : 6.2.1 >> >>>>> Index size : about 60GB >> >>>>> >> >>>>> Any insights will be appreciated. >> >>>>> >> >>>>> Thanks and regards, >> >>>>> Yasufumi. >> >>>>> >> >>>> >> >>> >> >> >> >> -- >> >> Ere Maijala >> >> Kansalliskirjasto / The National Library of Finland >> >> >> >>
