I'm working with a client that's trying to process a lot of data (billions
of docs) via a streaming expression, and the initial query is (not
surprisingly) taking a long time. Lots of various types of timeouts have
been cropping up and I've found myself thinking I solved some only to
discover that the settings in solr.xml are far less wide reaching than I
thought initially. The present 5% scale cluster seems to hit one particular
time out about 50% of the time which has made it particularly confusing.
I'm guessing it's probably depending on something like how busy the
virtualization in Amazon is, just barely making it when it gets more
resources and timing out if anything is starved.
As I look around the code base I'm finding a LOT of places where timeouts
on SolrClients and CloudSolrClients are just arbitrarily set to one-off
constant values. The one bugging me right now is
public abstract class SolrClientBuilder<B extends SolrClientBuilder<B>> {
protected HttpClient httpClient;
protected ResponseParser responseParser;
protected Integer connectionTimeoutMillis = 15000;
protected Integer socketTimeoutMillis = 120000;
Which I am unable to change because of this code in SolrStream:
/**
* Opens the stream to a single Solr instance.
**/
public void open() throws IOException {
if(cache == null) {
client = new HttpSolrClient.Builder(baseUrl).build();
} else {
client = cache.getHttpSolrClient(baseUrl);
}
I need to make this particular case configurable, so that I can get results
from a very long running query, but I sense that there is a much wider
problem in that we don't seem to have any organized plan for how socket
timeouts are set/managed in the code.
What thoughts have people had on this front?
-Gus
--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)
