Hi Thank you for your reply. I will try to check if there was any network issue.
Yes we were facing OOM issues and we had to increase the HEAP. We will try to see if we can reduce it. On Thu, Feb 16, 2023 at 10:12 PM Gus Heck <[email protected]> wrote: > If you are 100% sure no machines or solr software were restarted, you had a > networking issue that broke communications, perhaps a switch or router > restart, someone re-plugging cables, or a router that got overwhelmed, (or > is failing)? > > Unrelated side note: > > 48GB heap is not usually a great setting. @32GB the JVM switches to 64 bit > pointers, and so going above 31GB is actually less memory until somewhere > around 38-40GB. Meanwhile you operating system can't use that memory for > disk caching. Of course if you had heavy use cases that were causing you to > see OutOfMemoryError that forced you into that setting nevermind. Certainly > sometimes super-sized heap is necessary, but usually not. If that setting > was chosen via "hmmm, this looks good" ... it doesn't look good to me ;) > > On Thu, Feb 16, 2023 at 6:21 AM HariBabu kuruva <[email protected] > > > wrote: > > > Hi, > > > > Thanks for your reply. > > > > But, I can see all the solr nodes are up and running, and I have manually > > accessed the solr url with each node , it's accessible. > > > > On Tue, Feb 14, 2023 at 6:36 PM Markus Jelsma < > [email protected]> > > wrote: > > > > > Hello Hari, > > > > > > One or more Solr servers was down when sending your data. The broken > pipe > > > messages indicate the same, it happens when connections are broken. > > > > > > Find out which servers went offline/unavailable, and see what they > wrote > > in > > > their logs. > > > > > > Regards, > > > Markus > > > > > > Op di 14 feb. 2023 om 13:00 schreef HariBabu kuruva < > > > [email protected]>: > > > > > > > Hi All, > > > > > > > > We are receiving below errors when the manual indexing is initialized > > for > > > > one of the stores, and its working fine for all other stores. The > certs > > > are > > > > updated and in place. > > > > > > > > *Architecture*: 10 solr nodes and 5 zookeepers. 60 GB Ram and 4 cpu > for > > > all > > > > 10 nodes. ~48 GB Heap is assigned. > > > > > > > > ****************************************** > > > > org.apache.solr.client.solrj.SolrServerException: No live SolrServers > > > > available to handle this request:[https:// > > > *****.com:8981/solr/s_doc_quote, > > > > https://l******.com:8981/solr/s_doc_quote] > > > > at > > > > > > > > > > > > > > org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:346) > > > > at > > > > > > > > > > > > > > org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1171) > > > > at > > > > > > > > > > > > > > org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934) > > > > at > > > > > > > > > > > > > > org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:866) > > > > at > > > > > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214) > > > > at > > > > > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:231) > > > > at > > > > > > > > > > > > > > eisp.cdc.data.indexer.adapter.EispSolrCloudServiceImpl.getCursorSolrResponse(EispSolrCloudServiceImpl.java:197) > > > > at > > > > > > > > > > > > > > eisp.kafka.cdc.task.MainIndexerTask.dataIndexerProcess(MainIndexerTask.java:208) > > > > at > > > > > > > > > > > > > > eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest(MainIndexerTask.java:147) > > > > at > > > > > > > > > > > > > > eisp.kafka.cdc.task.MainIndexerTask$$FastClassBySpringCGLIB$$88fcf6c3.invoke(<generated>) > > > > at > > > > > > org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) > > > > at > > > > > > > > > > > > > > org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783) > > > > at > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > at > > > > > > > > > > > > > > org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) > > > > at > > > > > > > > > > > > > > org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115) > > > > at > > > > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > > > at > > > > > > > > > > > > > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > > > at > > > > > > > > > > > > > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > > > at java.base/java.lang.Thread.run(Thread.java:834) > > > > Caused by: org.apache.solr.client.solrj.SolrServerException: > > IOException > > > > occurred when talking to server at: https:// > > > > *****.***.com:8981/solr/s_doc_quote > > > > at > > > > > > > > > > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:695) > > > > at > > > > > > > > > > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266) > > > > at > > > > > > > > > > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248) > > > > at > > > > > > > > > > > > > > org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:369) > > > > at > > > > > > > > > > > > > > org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:297) > > > > ... 18 common frames omitted > > > > Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not > > > authenticated > > > > at > > > > > > > > > > > > > > java.base/sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:526) > > > > at > > > > > > > > > > > > > > org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:504) > > > > at > > > > > > > > > > > > > > org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437) > > > > at > > > > > > > > > > > > > > org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384) > > > > at > > > > > > > > > > > > > > org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) > > > > at > > > > > > > > > > > > > > org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) > > > > at > > > > > > > > > > > > > > org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) > > > > at > > > > > > > > > > > > > > org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) > > > > at > > > > > > > > > > org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) > > > > at > > > > org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) > > > > at > > > > > > > > > > org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) > > > > at > > > > > > > > > > > > > > org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) > > > > at > > > > > > > > > > > > > > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) > > > > at > > > > > > > > > > > > > > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) > > > > at > > > > > > > > > > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571) > > > > ... 22 common frames omitted > > > > ERROR <hostname) 14-02-2023 07:37:28.954 [mainRequestHandler-1] > > > > eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest - Error in > > > range > > > > data processing -> > > > > java.lang.NullPointerException: null > > > > > > > > ************************************************* > > > > > > > > Also we could see Broken pipe errors in 4-5 nodes. > > > > *Suppressed: java.io.IOException: Broken pipe* > > > > > > > > *Caused by: javax.net.ssl.SSLHandshakeException: Broken pipeCaused > by: > > > > java.io.IOException: Broken pipe* > > > > > > > > > > > > Could you please advise. > > > > > > > > -- > > > > > > > > Thanks and Regards, > > > > Hari > > > > Mobile:9790756568 > > > > > > > > > > > > > -- > > > > Thanks and Regards, > > Hari > > Mobile:9790756568 > > > > > -- > http://www.needhamsoftware.com (work) > http://www.the111shift.com (play) > -- Thanks and Regards, Hari Mobile:9790756568
