Dave-- Jena switched HTTP Commons versions a good bit from 3.1.1 to now, so that may be part of it. I will look into whether HTTP Commons Client changed its behavior under us.
I agree that just upping the maxes and hoping for the best isn't the best outcome at all. Otherwise, this sounds either buggy (if all the execs really are getting closed) or at least not very ergonomic (since you aren't seeing any warnings). Do I understand correctly that the behavior you would want would be that 1) after the max number of connections have been drawn out of the pool and used, the next request should block only until a connection is released, 2) and that closing a query execution should definitely return the connection underneath it to the pool, more or less immediately? It's almost like something above the client is blocking and not letting the connections get released... ajs6f > On Dec 8, 2017, at 12:55 PM, Dave Reynolds <[email protected]> wrote: > > Hi Rob, > > Thanks, that's useful to know but just raising the limit will just make it > less frequent but not cure it. > > The issue is that if you ever get more in-flight than the limit then all > future requests are blocked. The in-flight requests return fine, the execs > are closed but the httpclient never recovers. > > I will, of course, check again that I'm successfully closing all the execs. > However, with Jena 3.1.1 this code has successfully run for months between > reboots with request rates in the millions per week (with occasional high > bursts). No lock ups at all. As it stands I can't switch it to a newer Jena > unless I can absolutely guarantee an upper limit on the number of concurrent > requests. That is often possible (through an apache reverse proxy front end > with request throttling) but feels like an unsatisfying resolution. > > Dave > > On 08/12/17 16:56, Rob Vesse wrote: >> I think this relates to a HttpClient behaviour that limits the maximum >> connections to a given service >> At least in how Jena sets it up the default is 5 connections per service >> which is more generous than the HTTP client defaults. Jena appears to read >> this from a JVM property http.maxConnections OR you can construct your own >> client by calling setMaxConnPerRoute() and setMaxConnTotal() on the builder >> to set your desired settings. >> Rob >> On 08/12/2017, 16:44, "Dave Reynolds" <[email protected]> wrote: >> Hi, >> I've being updating some rather old libraries that use Jena to issue >> sparql requests to a remote endpoint and pull back results. >> These work under Jena 3.1.1 but there's a fatal problem under Jena >> 3.2.0 >> and later ... >> If I issue 6 concurrent execSelect calls to a remote sparql endpoint >> (happens to be fuseki) then 5 will get issued and return correctly but >> #6 will not and from then onwards no further remote calls will go >> through. This only happens if at least 5 requests are in flight with no >> response yet from the remote endpoint before the final one is issued. >> It's hard to produce a deterministic, standalone test case because >> by >> definition it depends on an external sparql endpoint being reliably >> there and reliably not too fast :) >> Looking at the stack trace I see: >> Unsafe.park(boolean, long) line: not available [native method] >> LockSupport.park(Object) line: 175 >> AbstractQueuedSynchronizer$ConditionObject.await() line: 2039 >> CPool(AbstractConnPool<T,C,E>).getPoolEntryBlocking(T, Object, long, >> TimeUnit, Future<E>) line: 377 >> AbstractConnPool<T,C,E>.access$200(AbstractConnPool, Object, Object, >> long, TimeUnit, Future) line: 67 >> AbstractConnPool$2.get(long, TimeUnit) line: 243 >> AbstractConnPool$2.get(long, TimeUnit) line: 191 >> PoolingHttpClientConnectionManager.leaseConnection(Future<CPoolEntry>, >> long, TimeUnit) line: 282 >> PoolingHttpClientConnectionManager$1.get(long, TimeUnit) line: 269 >> MainClientExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext, >> HttpExecutionAware) line: 191 >> ProtocolExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext, >> HttpExecutionAware) line: 185 >> RetryExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext, >> HttpExecutionAware) line: 89 >> RedirectExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext, >> HttpExecutionAware) line: 111 >> InternalHttpClient.doExecute(HttpHost, HttpRequest, HttpContext) line: >> 185 >> InternalHttpClient(CloseableHttpClient).execute(HttpUriRequest, >> HttpContext) line: 83 >> InternalHttpClient(CloseableHttpClient).execute(HttpUriRequest, >> HttpContext) line: 56 >> HttpOp.exec(String, HttpUriRequest, String, HttpResponseHandler, >> HttpClient, HttpContext) line: 1081 >> HttpOp.execHttpGet(String, String, HttpResponseHandler, HttpClient, >> HttpContext) line: 308 >> HttpOp.execHttpGet(String, String, HttpClient, HttpContext) line: 367 >> HttpQuery.execGet() line: 326 >> HttpQuery.exec() line: 288 >> QueryEngineHTTP.execResultSetInner() line: 348 >> QueryEngineHTTP.execSelect() line: 340 >> SSResultSet.<init>(BaseSparqlSource, String) line: 33 >> RemoteSparqlSource(BaseSparqlSource).streamableSelect(String) line: 59 >> Which suggests some problem with the connection pool locking. >> I've tried 3.4.0 and 3.5.0 which incorporate the fix to JENA-1335 >> (in >> case that's related) but no luck. >> I've tried both: >> HttpOp.setDefaultHttpClient(HttpClients.createMinimal()); >> and >> HttpOp.setDefaultHttpClient(HttpClients.createDefault()); >> right at the start of application startup. >> Which don't fix the problem. Though perhaps I'm getting the >> sequence of >> where to set those wrong. >> Any suggestions? >> Any pointers to documentation on how to configure the http client >> set up >> in current Jena versions? >> Thanks, >> Dave >> >
