On Sun, 2015-08-30 at 20:39 -0600, Shawn Heisey wrote:
> I am seeing SocketTimeoutException in the log for my SolrJ program.
> SolrJ uses HttpClient.
>
> I'm starting with the HC list for this problem because that seems like
> the most likely place for a problem, but I know that it could be in
> SolrJ or my own code.
>
> The index cycle that timed out began at 06:13:30.003. This starts out
> talking to a MySQL database to work out any index changes that need to
> be made, and if any are found, proceeds to make them. The loop failed
> with a socket timeout error at 2015-08-30 06:14:00.232, so it looks like
> it only waited 30 seconds ... but the SO_TIMEOUT on the httpclient
> should have been set to 15 minutes.
>
> Here is the full exception, slightly redacted to obscure my company name:
>
> ERROR - 2015-08-30 06:14:00.232; 977; 2; chain.b: error committing shard 2
> com.REDACTED.idxbuild.util.BuildException: shard.b.2.spark2live commit
> failed
> at com.REDACTED.idxbuild.solr.Core.commit(Core.java:496)
> at com.REDACTED.idxbuild.index.Shard.commit(Shard.java:454)
> at com.REDACTED.idxbuild.index.Chain$2.run(Chain.java:969)
> Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout
> occured while waiting response from server at:
> http://idxb3.REDACTED.com:8981/solr
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:570)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
> at
> org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:523)
> at com.REDACTED.idxbuild.solr.Core.commit(Core.java:484)
> ... 2 more
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at
> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
> at
> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
> at
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> at
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> at
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
> at
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
> at
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
> at
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
> at
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
> at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
> at
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:466)
> ... 7 more
>
> Here's my code that creates the two HttpClient objects which are shared
> among multiple HttpSolrClient instances. I create a client with a 15
> minute socket timeout for general use, and a client with a two hour
> timeout for doing index optimizes:
>
> synchronized (firstInstance)
> {
> if (firstInstance)
> {
> ModifiableSolrParams params = new ModifiableSolrParams();
> params.add(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, "300");
> params.add(HttpClientUtil.PROP_MAX_CONNECTIONS, "5000");
> params.add(HttpClientUtil.PROP_CONNECTION_TIMEOUT, "15000");
> params.add(HttpClientUtil.PROP_SO_TIMEOUT, "900000");
> httpClient = HttpClientUtil.createClient(params);
> params.clear();
> params.add(HttpClientUtil.PROP_CONNECTION_TIMEOUT, "15000");
> params.add(HttpClientUtil.PROP_SO_TIMEOUT, "7200000");
> optimizeHttpClient = HttpClientUtil.createClient(params);
> firstInstance = false;
> }
> }
>
> httpClient and optimizeHttpClient are static instances of HttpClient.
>
> The code for SolrJ's HttpClientUtil is in Apache SVN. It is still using
> a lot of deprecated methods. There were a lot of deprecations in
> HttpClient 4.3 that have not yet been removed from SolrJ.
>
> Below is the list of jars that I have available to my SolrJ program.
> One of them is the 5.2.1 version of SolrJ. The solrj dependencies that
> are shipped with Solr include HttpClient 4.4.1. I have upgraded
> versions of some components, including HttpClient. Most of these
> dependencies are for SolrJ, but there are some that are only for my program:
>
> commons-dbcp2-2.1.1.jar
> commons-io-2.4.jar
> commons-pool2-2.4.2.jar
> httpclient-4.5.jar
> httpcore-4.4.1.jar
> httpmime-4.5.jar
> javax.mail.jar
> jcl-over-slf4j-1.7.12.jar
> jrobin-1.5.14.jar
> jul-to-slf4j-1.7.12.jar
> log4j-1.2.17.jar
> mysql-connector-java-5.1.35-bin.jar
> noggit-0.7.jar
> slf4j-api-1.7.12.jar
> slf4j-log4j12-1.7.12.jar
> solr-solrj-5.2.1.jar
> stax2-api-3.1.1.jar
> woodstox-core-asl-4.2.0.jar
> zookeeper-3.4.6.jar
>
> Where should I start in the processof getting to the bottom of this
> problem? The first thing I am going to try is creating separate
> instances of ModifiableSolrParams for each HttpClient instance. If
> that's the problem, I would expect different symptoms, but it's a
> potential code mistake that I did notice.
>
> If you have questions about how HttpClientUtil works, I can attempt to
> decipher the code.
>
> Thanks,
> Shawn
>
Shawn,
The socket timeout in blocking i/o is handled by the JRE. We have no
control over it.
---
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
---
There likely to be something that sets SO_TIMEOUT to a lower value. You
should be able to trace it by placing a breakpoint or good ol'
System.out.println at this line:
http://hc.apache.org/httpcomponents-core-4.4.x/httpcore/xref/org/apache/http/impl/BHttpConnectionBase.html#277
Naturally it would make things easier if Solr used no deprecated
methods.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
