Hi, We are using Solr 7.2.1 with 2 nodes (245GB RAM each) and 3 node ZK cluster in production. We are using Java 8 with default GC settings (with NewRatio=3) with 15GB max heap, changed to 16 GB after the performance issue mentioned below.
We have about 90 collections in this (~8 shards each), about 50 of them are actively being used. About 3 collections are being actively updated using SolrJ update query with soft commit of 30 secs. Other collection go through update handler batch CSV update. We had read timeout/slowness issue when Young Collection size usage peaked. As you can see in the GC Graph below during the problem time. After that we increased the overall heap size to 16GB (from 15 GB) and as you can see that we did not see any read issue. 1. I see our Heap is very large, we are seeing higher usage of young collection, is this due to solrj updates (concurrent one record update)? 2. Should we change the NewRatio to 2 (so that young size increases more)? as we are seeing only 58% usage of old gen 3. We are also seeing a behavior that if we restart the Solr in production, when updates are happening, one server starts up, but does not have all collections and shards up, and when we restart both the server up, it comes up fine, is this behavior also related to the Solrj updates? Problem GC Report https://gceasy.io/my-gc-report.jsp?p=YXJjaGl2ZWQvMjAxOS8xMC83Ly0tMDJfc29scl9nYy5sb2cuNi5jdXJyZW50LS0xNC00My01OA==&channel=WEB<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgceasy.io%2Fmy-gc-report.jsp%3Fp%3DYXJjaGl2ZWQvMjAxOS8xMC83Ly0tMDJfc29scl9nYy5sb2cuNi5jdXJyZW50LS0xNC00My01OA%3D%3D%26channel%3DWEB&data=02%7C01%7Csethuramang%40dnb.com%7Cc0086a8014ba4c980faa08d752440eaf%7C19e2b708bf12437597198dec42771b3e%7C0%7C0%7C637068324577120430&sdata=TOum%2Fqf8ast2OAnFDpqumRniODNIl1elfBGYA%2BSgzYE%3D&reserved=0> No Problem GC Report (still see higher Young collection use) https://gceasy.io/my-gc-report.jsp?p=YXJjaGl2ZWQvMjAxOS8xMC85Ly0tMDJfX3NvbHJfZ2MubG9nLjIuY3VycmVudC0tMjAtNDQtMjY=&channel=WEB<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgceasy.io%2Fmy-gc-report.jsp%3Fp%3DYXJjaGl2ZWQvMjAxOS8xMC85Ly0tMDJfX3NvbHJfZ2MubG9nLjIuY3VycmVudC0tMjAtNDQtMjY%3D%26channel%3DWEB&data=02%7C01%7Csethuramang%40dnb.com%7Cc0086a8014ba4c980faa08d752440eaf%7C19e2b708bf12437597198dec42771b3e%7C0%7C0%7C637068324577120430&sdata=AOqgAgpgBWri3a77%2F9I6JdrGzNkUpLltUdiZA056zPc%3D&reserved=0> Any help on the above question appreciated. Thanks &Regards, Ganesh
