RE: Solr JVM Turning - 7.2.1 - with solrj updates

2019-10-21 Thread Sethuraman, Ganesh 
Dear Solr Users, appreciate any insight on this performance issue below.






-Original Message-
From: Sethuraman, Ganesh 
Sent: Wednesday, October 16, 2019 9:25 PM
To: solr-user@lucene.apache.org
Subject: Solr JVM Turning - 7.2.1

CAUTION: This email originated from outside of D&B. Please do not click links 
or open attachments unless you recognize the sender and know the content is 
safe.


Hi,

We are using Solr 7.2.1 with 2 nodes (245GB RAM each) and 3 node ZK cluster in 
production. We are using Java 8 with default GC settings (with NewRatio=3) with 
15GB max heap, changed to 16 GB after the performance issue mentioned below.

We have about 90 collections in this (~8  shards each), about 50 of them are 
actively being used. About 3 collections are being actively updated using SolrJ 
update query with soft commit of 30 secs. Other collection go through update 
handler batch CSV update.

We had read timeout/slowness issue when Young Collection size usage peaked. As 
you can see in the GC Graph below during the problem time. After that we 
increased the overall heap size to 16GB (from 15 GB) and as you can see that we 
did not see any read issue.

  1.  I see our Heap is very large, we are seeing higher usage of young 
collection, is this due to solrj updates (concurrent one record update)?
  2.  Should we change the NewRatio to 2 (so that young size increases more)? 
as we are seeing only 58% usage of old gen
  3.  We are also seeing a behavior that if we restart the Solr in production, 
when updates are happening, one server starts up, but does not have all 
collections and shards up, and when we restart both the server up, it comes up 
fine, is this behavior also related to the Solrj updates?



Problem GC Report  
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgceasy.io%2Fmy-gc-report.jsp%3Fp%3DYXJjaGl2ZWQvMjAxOS8xMC83Ly0tMDJfc29scl9nYy5sb2cuNi5jdXJyZW50LS0xNC00My01OA%3D%3D%26channel%3DWEB&data=02%7C01%7CSethuramanG%40dnb.com%7C0a3a55d0df6942c2ed0608d753dacc70%7C19e2b708bf12437597198dec42771b3e%7C0%7C1%7C637070071509525306&sdata=%2FL12%2F00s9sq50mQP5YYquwxMGfZSesibCH3YkUzMA18%3D&reserved=0

No Problem GC Report (still see higher Young collection use)  
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgceasy.io%2Fmy-gc-report.jsp%3Fp%3DYXJjaGl2ZWQvMjAxOS8xMC85Ly0tMDJfX3NvbHJfZ2MubG9nLjIuY3VycmVudC0tMjAtNDQtMjY%3D%26channel%3DWEB&data=02%7C01%7CSethuramanG%40dnb.com%7C0a3a55d0df6942c2ed0608d753dacc70%7C19e2b708bf12437597198dec42771b3e%7C0%7C1%7C637070071509535302&sdata=CpBKvsZKcBIJaWT2WcEVLG66594oIOSoXTZu82lLtlg%3D&reserved=0

 Any help on the above question appreciated.

Thanks &Regards,

Ganesh

RE: Solr JVM Turning - 7.2.1

2019-10-18 Thread Sethuraman, Ganesh 
Solr Users,

Any suggestion or insights on the Solr behavior will help. 

Regards
Ganesh

-Original Message-
From: Sethuraman, Ganesh  
Sent: Wednesday, October 16, 2019 9:25 PM
To: solr-user@lucene.apache.org
Subject: Solr JVM Turning - 7.2.1

CAUTION: This email originated from outside of D&B. Please do not click links 
or open attachments unless you recognize the sender and know the content is 
safe.


Hi,

We are using Solr 7.2.1 with 2 nodes (245GB RAM each) and 3 node ZK cluster in 
production. We are using Java 8 with default GC settings (with NewRatio=3) with 
15GB max heap, changed to 16 GB after the performance issue mentioned below.

We have about 90 collections in this (~8  shards each), about 50 of them are 
actively being used. About 3 collections are being actively updated using SolrJ 
update query with soft commit of 30 secs. Other collection go through update 
handler batch CSV update.

We had read timeout/slowness issue when Young Collection size usage peaked. As 
you can see in the GC Graph below during the problem time. After that we 
increased the overall heap size to 16GB (from 15 GB) and as you can see that we 
did not see any read issue.

  1.  I see our Heap is very large, we are seeing higher usage of young 
collection, is this due to solrj updates (concurrent one record update)?
  2.  Should we change the NewRatio to 2 (so that young size increases more)? 
as we are seeing only 58% usage of old gen
  3.  We are also seeing a behavior that if we restart the Solr in production, 
when updates are happening, one server starts up, but does not have all 
collections and shards up, and when we restart both the server up, it comes up 
fine, is this behavior also related to the Solrj updates?



Problem GC Report  
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgceasy.io%2Fmy-gc-report.jsp%3Fp%3DYXJjaGl2ZWQvMjAxOS8xMC83Ly0tMDJfc29scl9nYy5sb2cuNi5jdXJyZW50LS0xNC00My01OA%3D%3D%26channel%3DWEB&data=02%7C01%7CSethuramanG%40dnb.com%7C0cd17ad89bdd4227909108d752a0d0ef%7C19e2b708bf12437597198dec42771b3e%7C0%7C1%7C637068722973512337&sdata=TrrmhXqtgbv9%2BKbcNulKw%2FrOCzTf9%2FSxO3JStWRlNG8%3D&reserved=0

No Problem GC Report (still see higher Young collection use)  
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgceasy.io%2Fmy-gc-report.jsp%3Fp%3DYXJjaGl2ZWQvMjAxOS8xMC85Ly0tMDJfX3NvbHJfZ2MubG9nLjIuY3VycmVudC0tMjAtNDQtMjY%3D%26channel%3DWEB&data=02%7C01%7CSethuramanG%40dnb.com%7C0cd17ad89bdd4227909108d752a0d0ef%7C19e2b708bf12437597198dec42771b3e%7C0%7C1%7C637068722973517327&sdata=MolLF1OSc8SBsx9rQqSfaHgwtzd534q%2B1Zqoc4W44QY%3D&reserved=0

 Any help on the above question appreciated.

Thanks &Regards,

Ganesh