We are trying to migrate our SOLR 8.1.1 cluster from OpenJDK Java 8 to OpenJDK
Java 11 and are facing issues with Indexing. While our indexing is happening
flawlessly on Java 8, it crawls or maybe I should say it stalls with Java 11.
Any pointers/help is appreciated.
Symptoms
* With OpenJDK 11 and SOLR 8.1.1 we see that for the first 30 minutes
response times for updates similar to our current implementation (OpenJDK 8 and
SOLR 8.1.1). It has to be noted that there are no read queries being executed
at the time of indexing
* On the OpenJDK 11 implementation, the qtp active threads continuously
increasing to thousands while OpenJDK 8 implementation stops after
approximately going up to 150.
* On the On the OpenJDK 11 implementation, the classes loaded start at a
very high number and stay there as opposed to OpenJDK 8 implementation where
the number of classes loaded are small to begin with and remains under control.
I believe the qtp threads in wait state mentioned above are causing this symptom
* Attachment 1: OpenJDK 8 vs OpenJDK 8 key metrics
* Attachment 2: OpenJDK 8 vs OpenJDK 8 waiting QTP Threads
* Attachment 3: OpenJDK 11 Thread dump
Following are the key metrics/configuration of our application.
Index Size: 8 GB/shard
Total no of Documents in Solr cluster: 70 Million
Average Document size: 15 KB
JSON Payload for each update contains: 50 docs
Average Time Taken to post for 50 Docs: 300 milli seconds
Average Rate at which documents are being posted to SOLR: 7500 requests per
second
No of shards in the Cluster: 10 (No Replicas)
CPUs: 32
Memory: 128 GB
Heap allocated: 32 GB
SOLR Client: 8.1.1
ZK Ensemble: 3
<mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
<int name="maxMergeAtOnce">100</int>
<int name="segmentsPerTier">150</int>
</mergePolicyFactory>
<autoSoftCommit>
<maxTime>480000</maxTime>
</autoSoftCommit>
<autoCommit>
<maxTime>180000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
GC and ZK Settings
-DzkClientTimeout=300000
-DzkHost= XXXX,YYYY,ZZZZ
-XX:+PrintGCDetails
-XX:+UseG1GC
-XX:+UseStringDeduplication
-XX:ConcGCThreads=8
-XX:InitiatingHeapOccupancyPercent=70
-XX:MaxGCPauseMillis=200
-XX:ParallelGCThreads=32
-XX:PermSize=512m
-Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,
uptime:filecount=9,
filesize=20M
-Xms32g-Xmx32g
-Xss256k-Xss256k
-verbose:gc
Thanks,
Anup
This message, including any attachments, is the property of Transform HoldCo
LLC and/or one of its subsidiaries. It is confidential and may contain
proprietary or legally privileged information. If you are not the intended
recipient, please delete it without reading the contents. Thank you.
