I am seeing SocketTimeoutException in the log for my SolrJ program.
SolrJ uses HttpClient.
I'm starting with the HC list for this problem because that seems like
the most likely place for a problem, but I know that it could be in
SolrJ or my own code.
The index cycle that timed out began at 06:13:30.003. This starts out
talking to a MySQL database to work out any index changes that need to
be made, and if any are found, proceeds to make them. The loop failed
with a socket timeout error at 2015-08-30 06:14:00.232, so it looks like
it only waited 30 seconds ... but the SO_TIMEOUT on the httpclient
should have been set to 15 minutes.
Here is the full exception, slightly redacted to obscure my company name:
ERROR - 2015-08-30 06:14:00.232; 977; 2; chain.b: error committing shard 2
com.REDACTED.idxbuild.util.BuildException: shard.b.2.spark2live commit
failed
at com.REDACTED.idxbuild.solr.Core.commit(Core.java:496)
at com.REDACTED.idxbuild.index.Shard.commit(Shard.java:454)
at com.REDACTED.idxbuild.index.Chain$2.run(Chain.java:969)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout
occured while waiting response from server at:
http://idxb3.REDACTED.com:8981/solr
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:570)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
at
org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:523)
at com.REDACTED.idxbuild.solr.Core.commit(Core.java:484)
... 2 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
at
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
at
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:466)
... 7 more
Here's my code that creates the two HttpClient objects which are shared
among multiple HttpSolrClient instances. I create a client with a 15
minute socket timeout for general use, and a client with a two hour
timeout for doing index optimizes:
synchronized (firstInstance)
{
if (firstInstance)
{
ModifiableSolrParams params = new ModifiableSolrParams();
params.add(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, "300");
params.add(HttpClientUtil.PROP_MAX_CONNECTIONS, "5000");
params.add(HttpClientUtil.PROP_CONNECTION_TIMEOUT, "15000");
params.add(HttpClientUtil.PROP_SO_TIMEOUT, "900000");
httpClient = HttpClientUtil.createClient(params);
params.clear();
params.add(HttpClientUtil.PROP_CONNECTION_TIMEOUT, "15000");
params.add(HttpClientUtil.PROP_SO_TIMEOUT, "7200000");
optimizeHttpClient = HttpClientUtil.createClient(params);
firstInstance = false;
}
}
httpClient and optimizeHttpClient are static instances of HttpClient.
The code for SolrJ's HttpClientUtil is in Apache SVN. It is still using
a lot of deprecated methods. There were a lot of deprecations in
HttpClient 4.3 that have not yet been removed from SolrJ.
Below is the list of jars that I have available to my SolrJ program.
One of them is the 5.2.1 version of SolrJ. The solrj dependencies that
are shipped with Solr include HttpClient 4.4.1. I have upgraded
versions of some components, including HttpClient. Most of these
dependencies are for SolrJ, but there are some that are only for my program:
commons-dbcp2-2.1.1.jar
commons-io-2.4.jar
commons-pool2-2.4.2.jar
httpclient-4.5.jar
httpcore-4.4.1.jar
httpmime-4.5.jar
javax.mail.jar
jcl-over-slf4j-1.7.12.jar
jrobin-1.5.14.jar
jul-to-slf4j-1.7.12.jar
log4j-1.2.17.jar
mysql-connector-java-5.1.35-bin.jar
noggit-0.7.jar
slf4j-api-1.7.12.jar
slf4j-log4j12-1.7.12.jar
solr-solrj-5.2.1.jar
stax2-api-3.1.1.jar
woodstox-core-asl-4.2.0.jar
zookeeper-3.4.6.jar
Where should I start in the processof getting to the bottom of this
problem? The first thing I am going to try is creating separate
instances of ModifiableSolrParams for each HttpClient instance. If
that's the problem, I would expect different symptoms, but it's a
potential code mistake that I did notice.
If you have questions about how HttpClientUtil works, I can attempt to
decipher the code.
Thanks,
Shawn
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
