Jason Gerlowski created SOLR-13038: -------------------------------------- Summary: Overseer actions fail with NoHttpResponseException following a node restart Key: SOLR-13038 URL: https://issues.apache.org/jira/browse/SOLR-13038 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: SolrCloud Affects Versions: master (8.0) Reporter: Jason Gerlowski Assignee: Jason Gerlowski
I noticed recently that a lot of overseer operations fail if they're executed right after a restart of a Solr node. The failure returns a message like "org.apache.solr.client.solrj.SolrServerException:IOException occured when talking to server at: https://127.0.0.1:62253/solr". The logs are a bit more helpful: {code} org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:62253/solr at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657) ~[java/:?] at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255) ~[java/:?] at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) ~[java/:?] at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260) ~[java/:?] at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:172) ~[java/:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_172] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172] at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) ~[metrics-core-3.2.6.jar:3.2.6] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172] Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:62253 failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141) ~[httpclient-4.5.6.jar:4.5.6] at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.6.jar:4.5.6] at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.10.jar:4.4.10] at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.10.jar:4.4.10] at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) ~[httpclient-4.5.6.jar:4.5.6] at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.10.jar:4.4.10] at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.10.jar:4.4.10] at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:120) ~[java/:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.6.jar:4.5.6] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.6.jar:4.5.6] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.6.jar:4.5.6] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.6.jar:4.5.6] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.6.jar:4.5.6] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.6.jar:4.5.6] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.6.jar:4.5.6] at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542) ~[java/:?] ... 12 more {code} After a bit of debugging I was able to confirm the problem: when some non-overseer node gets restarted, the overseer never notices that its connections are invalid and will try to reuse them for subsequent requests that happen right after the restart. There's a few ways we might be able to tackle this: * we could look at adding logic to {{SolrHttpRequestRetryHandler}} to retry when this happens. SHRRH already retries NoHttpResponseException generally, but has other logic which prevents any retries on collection/core-admin APIs. Maybe we could elaborate this a bit. * we could add retry logic to the {{HttpShardHandler}} code that makes these requests. We could do this across the board, or more selectively for only the overseer commands that are "retry-able". * We could tweak how our connection pool is managed so that it evicts these idle connections more aggressively. It seems like something similar has already been tried (without success) on SOLR-6944 Not sure what the right approach is. Seems like intermittent NoHttpResponseExceptions have been a problem in Solr (and its tests) going back at least 5 years or so. Several JIRAs suggested adding retries for NHRE in the past but have been killed since not all APIs are idempotent and other JIRAs have been concerned with fixing this at the (very broad) SolrClient level. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org