If you are 100% sure no machines or solr software were restarted, you had a networking issue that broke communications, perhaps a switch or router restart, someone re-plugging cables, or a router that got overwhelmed, (or is failing)?
Unrelated side note: 48GB heap is not usually a great setting. @32GB the JVM switches to 64 bit pointers, and so going above 31GB is actually less memory until somewhere around 38-40GB. Meanwhile you operating system can't use that memory for disk caching. Of course if you had heavy use cases that were causing you to see OutOfMemoryError that forced you into that setting nevermind. Certainly sometimes super-sized heap is necessary, but usually not. If that setting was chosen via "hmmm, this looks good" ... it doesn't look good to me ;) On Thu, Feb 16, 2023 at 6:21 AM HariBabu kuruva <[email protected]> wrote: > Hi, > > Thanks for your reply. > > But, I can see all the solr nodes are up and running, and I have manually > accessed the solr url with each node , it's accessible. > > On Tue, Feb 14, 2023 at 6:36 PM Markus Jelsma <[email protected]> > wrote: > > > Hello Hari, > > > > One or more Solr servers was down when sending your data. The broken pipe > > messages indicate the same, it happens when connections are broken. > > > > Find out which servers went offline/unavailable, and see what they wrote > in > > their logs. > > > > Regards, > > Markus > > > > Op di 14 feb. 2023 om 13:00 schreef HariBabu kuruva < > > [email protected]>: > > > > > Hi All, > > > > > > We are receiving below errors when the manual indexing is initialized > for > > > one of the stores, and its working fine for all other stores. The certs > > are > > > updated and in place. > > > > > > *Architecture*: 10 solr nodes and 5 zookeepers. 60 GB Ram and 4 cpu for > > all > > > 10 nodes. ~48 GB Heap is assigned. > > > > > > ****************************************** > > > org.apache.solr.client.solrj.SolrServerException: No live SolrServers > > > available to handle this request:[https:// > > *****.com:8981/solr/s_doc_quote, > > > https://l******.com:8981/solr/s_doc_quote] > > > at > > > > > > > > > org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:346) > > > at > > > > > > > > > org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1171) > > > at > > > > > > > > > org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934) > > > at > > > > > > > > > org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:866) > > > at > > > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214) > > > at > > > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:231) > > > at > > > > > > > > > eisp.cdc.data.indexer.adapter.EispSolrCloudServiceImpl.getCursorSolrResponse(EispSolrCloudServiceImpl.java:197) > > > at > > > > > > > > > eisp.kafka.cdc.task.MainIndexerTask.dataIndexerProcess(MainIndexerTask.java:208) > > > at > > > > > > > > > eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest(MainIndexerTask.java:147) > > > at > > > > > > > > > eisp.kafka.cdc.task.MainIndexerTask$$FastClassBySpringCGLIB$$88fcf6c3.invoke(<generated>) > > > at > > > > org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) > > > at > > > > > > > > > org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783) > > > at > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > at > > > > > > > > > org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) > > > at > > > > > > > > > org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115) > > > at > > > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > > at > > > > > > > > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > > at > > > > > > > > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > > at java.base/java.lang.Thread.run(Thread.java:834) > > > Caused by: org.apache.solr.client.solrj.SolrServerException: > IOException > > > occurred when talking to server at: https:// > > > *****.***.com:8981/solr/s_doc_quote > > > at > > > > > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:695) > > > at > > > > > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266) > > > at > > > > > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248) > > > at > > > > > > > > > org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:369) > > > at > > > > > > > > > org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:297) > > > ... 18 common frames omitted > > > Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not > > authenticated > > > at > > > > > > > > > java.base/sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:526) > > > at > > > > > > > > > org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:504) > > > at > > > > > > > > > org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437) > > > at > > > > > > > > > org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384) > > > at > > > > > > > > > org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) > > > at > > > > > > > > > org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) > > > at > > > > > > > > > org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) > > > at > > > > > > > > > org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) > > > at > > > > > > org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) > > > at > > > org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) > > > at > > > > > > org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) > > > at > > > > > > > > > org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) > > > at > > > > > > > > > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) > > > at > > > > > > > > > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) > > > at > > > > > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571) > > > ... 22 common frames omitted > > > ERROR <hostname) 14-02-2023 07:37:28.954 [mainRequestHandler-1] > > > eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest - Error in > > range > > > data processing -> > > > java.lang.NullPointerException: null > > > > > > ************************************************* > > > > > > Also we could see Broken pipe errors in 4-5 nodes. > > > *Suppressed: java.io.IOException: Broken pipe* > > > > > > *Caused by: javax.net.ssl.SSLHandshakeException: Broken pipeCaused by: > > > java.io.IOException: Broken pipe* > > > > > > > > > Could you please advise. > > > > > > -- > > > > > > Thanks and Regards, > > > Hari > > > Mobile:9790756568 > > > > > > > > -- > > Thanks and Regards, > Hari > Mobile:9790756568 > -- http://www.needhamsoftware.com (work) http://www.the111shift.com (play)
