Re: SOLR Cloud: Few cores goes to recovery mode all of a sudden

2019-09-03 Thread Doss
Thanks Erick, ulimit in all three lodes are more than 65K including max process list. If you look at the timestamp the core down error happened ahead of unable to create thread error, and also core down error took place in node1 and unable to create thread error took place in node3. BTW we are r

Re: SOLR Cloud: Few cores goes to recovery mode all of a sudden

2019-09-03 Thread Erick Erickson
The “unable to create new thread” is where I’d focus first. It means you’re running out of some system resources and it’s quite possible that your other problems are arising from that root cause. What are you “ulimit” settings? the number of file handles and processes should be set to 65k at le

SOLR Cloud: Few cores goes to recovery mode all of a sudden

2019-09-03 Thread Doss
Hi, We are using 3 node SOLR (7.0.1) cloud setup 1 node zookeeper ensemble. Each system has 16CPUs, 90GB RAM (14GB HEAP), 130 cores (3 replicas NRT) with index size ranging from 700MB to 20GB. autoCommit - 10 minutes once softCommit - 30 Sec Once We are facing the following problems in recent ti