Hello Susheel, Yes, the closing happens only at the end of the checking cycle. I asked my colleague about the firewall and he is positive everything is allowed between those nodes. I also cannot completely drop the firewall between those nodes to be sure, because the problem is very hard to reproduce; it pops up once in a while, sometimes not for weeks, today already a couple of times.
It is locally unreproducible but we're going to try to reproduce it in our development environment. So i have to get back to the problem in a few weeks from now. Number of requests is sometimes, briefly, very high. Usually very low. These specific checks are executed in order, not concurrently. Thanks, Markus -----Original message----- > From:Susheel Kumar <susheel2...@gmail.com> > Sent: Tuesday 18th July 2017 15:17 > To: solr-user@lucene.apache.org > Subject: Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace > > Then most likely its due to closing of connection as mentioned above though > you said it's not happening in that part of your code. To rule out > firewall possibility, you can test in some other/local env. Also how many > requests/client/connections happening concurrently. > > Thanks, > Susheel > > On Tue, Jul 18, 2017 at 8:43 AM, Markus Jelsma <markus.jel...@openindex.io> > wrote: > > > Hello Susheel, > > > > No, nothing at all. I've check all six nodes, they are clean. > > > > Thanks, > > Markus > > > > > > > > -----Original message----- > > > From:Susheel Kumar <susheel2...@gmail.com> > > > Sent: Tuesday 18th July 2017 14:30 > > > To: solr-user@lucene.apache.org > > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace > > > > > > Do you see any errors etc. in solr.log during this time? > > > > > > On Tue, Jul 18, 2017 at 7:10 AM, Markus Jelsma < > > markus.jel...@openindex.io> > > > wrote: > > > > > > > The problem was never resolved but Shawn asked for the stack trace, > > here > > > > it is: > > > > > > > > org.apache.solr.client.solrj.SolrServerException: java.lang. > > IllegalStateException: > > > > Connection pool shut down > > > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient. > > > > doRequest(LBHttpSolrClient.java:485) > > > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request( > > > > LBHttpSolrClient.java:388) > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient. > > > > sendRequest(CloudSolrClient.java:1383) > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient. > > > > requestWithRetryOnStaleState(CloudSolrClient.java:1134) > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request( > > > > CloudSolrClient.java:1073) > > > > at org.apache.solr.client.solrj.SolrRequest.process( > > SolrRequest.java:160) > > > > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942) > > > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient. > > java:1173) > > > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient. > > java:1090) > > > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient. > > java:1110) > > > > ..internal method calling getById().. > > > > at java.lang.Thread.run(Thread.java:748) > > > > Caused by: java.lang.IllegalStateException: Connection pool shut down > > > > at org.apache.http.util.Asserts.check(Asserts.java:34) > > > > at org.apache.http.pool.AbstractConnPool.lease( > > AbstractConnPool.java:184) > > > > at org.apache.http.pool.AbstractConnPool.lease( > > AbstractConnPool.java:217) > > > > at org.apache.http.impl.conn.PoolingClientConnectionManager > > > > .requestConnection(PoolingClientConnectionManager.java:184) > > > > at org.apache.http.impl.client.DefaultRequestDirector.execute( > > > > DefaultRequestDirector.java:415) > > > > at org.apache.http.impl.client.AbstractHttpClient.doExecute( > > > > AbstractHttpClient.java:882) > > > > at org.apache.http.impl.client.CloseableHttpClient.execute( > > > > CloseableHttpClient.java:82) > > > > at org.apache.http.impl.client.CloseableHttpClient.execute( > > > > CloseableHttpClient.java:55) > > > > at org.apache.solr.client.solrj.impl.HttpSolrClient. > > > > executeMethod(HttpSolrClient.java:515) > > > > at org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > > > HttpSolrClient.java:279) > > > > at org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > > > HttpSolrClient.java:268) > > > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient. > > > > doRequest(LBHttpSolrClient.java:447) > > > > ... 24 more > > > > > > > > So, to summarize, we have a program checking presence of documents in > > Solr > > > > using getById() and we don't want this exception to bubble up, we want > > > > SolrJ to restore the connection pool just as CloudSolrClient would > > move on > > > > to another node if one went down in the mean time. > > > > > > > > Is this possible? How? > > > > > > > > Many thanks, > > > > Markus > > > > > > > > -----Original message----- > > > > > From:Markus Jelsma <markus.jel...@openindex.io> > > > > > Sent: Thursday 29th June 2017 16:38 > > > > > To: solr-user@lucene.apache.org > > > > > Subject: RE: SolrJ 6.6.0 Connection pool shutdown > > > > > > > > > > Thanks. I probably should have mentioned there is no firewall > > limiting > > > > connections between those hosts. Actually, the processes run on the > > same > > > > hosts as the Solr cluster is running on. > > > > > > > > > > Thanks, > > > > > Markus > > > > > > > > > > > > > > > > > > > > -----Original message----- > > > > > > From:Alexandre Rafalovitch <arafa...@gmail.com> > > > > > > Sent: Thursday 29th June 2017 15:38 > > > > > > To: solr-user <solr-user@lucene.apache.org> > > > > > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown > > > > > > > > > > > > One thing to check is whether there is a firewall between the > > client > > > > > > and the server. They - sometimes - cut the silent connections in > > the > > > > > > _middle_ (at the firewall). The usual solution is keepAlive > > request of > > > > > > some kind or not using the connection pool. > > > > > > > > > > > > One way to check is with network tracer like Wireshark and checking > > > > > > whether the actual hardware at the other end of the connection is a > > > > > > normal server or some sort of unexpected hardware piece of > > equipment > > > > > > (firewall). Yes, that's using the hammer to swat a fly :-) > > > > > > > > > > > > Regards, > > > > > > Alex. > > > > > > ---- > > > > > > http://www.solr-start.com/ - Resources for Solr users, new and > > > > experienced > > > > > > > > > > > > > > > > > > On 29 June 2017 at 08:21, Markus Jelsma < > > markus.jel...@openindex.io> > > > > wrote: > > > > > > > Hi, > > > > > > > > > > > > > > Everything is 6.6.0. I could include a stack trace (i don't print > > > > them in my program), but that would only be the the trace from > > getById() to > > > > CloudSolrClient.requestWithRetryOnStaleState() and little deeper, that > > > > what you're looking for? > > > > > > > > > > > > > > We haven't called close() in that particular part of the program. > > > > > > > > > > > > > > Method requestWithRetryOnStaleState has some retry logic built-in > > > > but doesn't seem to work for the exception i got. > > > > > > > > > > > > > > I'll let it print the stack trace and get back if it happens > > again. > > > > > > > > > > > > > > Thanks, > > > > > > > Markus > > > > > > > > > > > > > > -----Original message----- > > > > > > >> From:Shawn Heisey <apa...@elyograg.org> > > > > > > >> Sent: Tuesday 27th June 2017 23:02 > > > > > > >> To: solr-user@lucene.apache.org > > > > > > >> Subject: Re: SolrJ 6.6.0 Connection pool shutdown > > > > > > >> > > > > > > >> On 6/27/2017 6:50 AM, Markus Jelsma wrote: > > > > > > >> > We have a proces checking presence of many documents in a > > > > collection, just a simple client.getById(id). It sometimes begins > > throwing > > > > lots of these exceptions in a row: > > > > > > >> > > > > > > > >> > org.apache.solr.client.solrj.SolrServerException: java.lang. > > IllegalStateException: > > > > Connection pool shut down > > > > > > >> > > > > > > > >> > Then, as suddenly as it appeared, it's gone again a no longer > > a > > > > problem. I would expect SolrJ not to throw this but to wait until it > > the > > > > connection pool, or whatever mechanism is there, to recover. > > > > > > >> > > > > > > > >> > Did i miss a magic parameter for SolrJ?\ > > > > > > >> > > > > > > >> That error message will be much longer than what you've provided > > > > here. > > > > > > >> It will have a java stacktrace that's typically a dozen or so > > lines > > > > > > >> long. There may also be one or more "Caused by" sections after > > the > > > > > > >> stacktrace, each with a stacktrace of its own. Can you share > > the > > > > full > > > > > > >> error message? Is the server also running 6.6.0, or a different > > > > version? > > > > > > >> > > > > > > >> It would also be helpful if you can share the SolrJ code you've > > > > written, > > > > > > >> cleanly redacted to remove anything sensitive. > > > > > > >> > > > > > > >> That particular message ("Connection pool shut down") sounds > > like it > > > > > > >> probably came from HttpClient, which SolrJ uses ... and I would > > > > expect > > > > > > >> that to only happen if you close/shutdown the HttpClient or the > > > > > > >> SolrClient. After closing, a client can't be used any more. > > > > Normally > > > > > > >> the only time you should close a client is right before exiting > > the > > > > > > >> program, although if the program's about to exit, it's generally > > > > > > >> unnecessary, so in my opinion for *most* usages, closing the > > client > > > > > > >> likely never needs to happen. > > > > > > >> > > > > > > >> Thanks, > > > > > > >> Shawn > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
