The problem was never resolved but Shawn asked for the stack trace, here it is:
org.apache.solr.client.solrj.SolrServerException: java.lang.IllegalStateException: Connection pool shut down at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:485) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:388) at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1383) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134) at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942) at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1173) at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1090) at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1110) ..internal method calling getById().. at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalStateException: Connection pool shut down at org.apache.http.util.Asserts.check(Asserts.java:34) at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184) at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:217) at org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection(PoolingClientConnectionManager.java:184) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447) ... 24 more So, to summarize, we have a program checking presence of documents in Solr using getById() and we don't want this exception to bubble up, we want SolrJ to restore the connection pool just as CloudSolrClient would move on to another node if one went down in the mean time. Is this possible? How? Many thanks, Markus -----Original message----- > From:Markus Jelsma <markus.jel...@openindex.io> > Sent: Thursday 29th June 2017 16:38 > To: solr-user@lucene.apache.org > Subject: RE: SolrJ 6.6.0 Connection pool shutdown > > Thanks. I probably should have mentioned there is no firewall limiting > connections between those hosts. Actually, the processes run on the same > hosts as the Solr cluster is running on. > > Thanks, > Markus > > > > -----Original message----- > > From:Alexandre Rafalovitch <arafa...@gmail.com> > > Sent: Thursday 29th June 2017 15:38 > > To: solr-user <solr-user@lucene.apache.org> > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown > > > > One thing to check is whether there is a firewall between the client > > and the server. They - sometimes - cut the silent connections in the > > _middle_ (at the firewall). The usual solution is keepAlive request of > > some kind or not using the connection pool. > > > > One way to check is with network tracer like Wireshark and checking > > whether the actual hardware at the other end of the connection is a > > normal server or some sort of unexpected hardware piece of equipment > > (firewall). Yes, that's using the hammer to swat a fly :-) > > > > Regards, > > Alex. > > ---- > > http://www.solr-start.com/ - Resources for Solr users, new and experienced > > > > > > On 29 June 2017 at 08:21, Markus Jelsma <markus.jel...@openindex.io> wrote: > > > Hi, > > > > > > Everything is 6.6.0. I could include a stack trace (i don't print them in > > > my program), but that would only be the the trace from getById() to > > > CloudSolrClient.requestWithRetryOnStaleState() and little deeper, that > > > what you're looking for? > > > > > > We haven't called close() in that particular part of the program. > > > > > > Method requestWithRetryOnStaleState has some retry logic built-in but > > > doesn't seem to work for the exception i got. > > > > > > I'll let it print the stack trace and get back if it happens again. > > > > > > Thanks, > > > Markus > > > > > > -----Original message----- > > >> From:Shawn Heisey <apa...@elyograg.org> > > >> Sent: Tuesday 27th June 2017 23:02 > > >> To: solr-user@lucene.apache.org > > >> Subject: Re: SolrJ 6.6.0 Connection pool shutdown > > >> > > >> On 6/27/2017 6:50 AM, Markus Jelsma wrote: > > >> > We have a proces checking presence of many documents in a collection, > > >> > just a simple client.getById(id). It sometimes begins throwing lots of > > >> > these exceptions in a row: > > >> > > > >> > org.apache.solr.client.solrj.SolrServerException: > > >> > java.lang.IllegalStateException: Connection pool shut down > > >> > > > >> > Then, as suddenly as it appeared, it's gone again a no longer a > > >> > problem. I would expect SolrJ not to throw this but to wait until it > > >> > the connection pool, or whatever mechanism is there, to recover. > > >> > > > >> > Did i miss a magic parameter for SolrJ?\ > > >> > > >> That error message will be much longer than what you've provided here. > > >> It will have a java stacktrace that's typically a dozen or so lines > > >> long. There may also be one or more "Caused by" sections after the > > >> stacktrace, each with a stacktrace of its own. Can you share the full > > >> error message? Is the server also running 6.6.0, or a different version? > > >> > > >> It would also be helpful if you can share the SolrJ code you've written, > > >> cleanly redacted to remove anything sensitive. > > >> > > >> That particular message ("Connection pool shut down") sounds like it > > >> probably came from HttpClient, which SolrJ uses ... and I would expect > > >> that to only happen if you close/shutdown the HttpClient or the > > >> SolrClient. After closing, a client can't be used any more. Normally > > >> the only time you should close a client is right before exiting the > > >> program, although if the program's about to exit, it's generally > > >> unnecessary, so in my opinion for *most* usages, closing the client > > >> likely never needs to happen. > > >> > > >> Thanks, > > >> Shawn > > >> > > >> > > >
