Re: Pooling Http Client lockup

[Dave Reynolds]

Hi Rob,

Thanks, that's useful to know but just raising the limit will just make 
it less frequent but not cure it.

The issue is that if you ever get more in-flight than the limit then all 
future requests are blocked. The in-flight requests return fine, the 
execs are closed but the httpclient never recovers.

I will, of course, check again that I'm successfully closing all the 
execs. However, with Jena 3.1.1 this code has successfully run for 
months between reboots with request rates in the millions per week (with 
occasional high bursts). No lock ups at all. As it stands I can't switch 
it to a newer Jena unless I can absolutely guarantee an upper limit on 
the number of concurrent requests. That is often possible (through an 
apache reverse proxy front end with request throttling) but feels like 
an unsatisfying resolution.

Dave

On 08/12/17 16:56, Rob Vesse wrote:
I think this relates to a HttpClient behaviour that limits the maximum 
connections to a given service

At least in how Jena sets it up the default is 5 connections per service which 
is more generous than the HTTP client defaults.  Jena appears to read this from 
a JVM property http.maxConnections OR you can construct your own client by 
calling setMaxConnPerRoute() and setMaxConnTotal() on the builder to set your 
desired settings.

Rob

On 08/12/2017, 16:44, "Dave Reynolds" <dave.e.reyno...@gmail.com> wrote:

     Hi,
     
     I've being updating some rather old libraries that use Jena to issue
     sparql requests to a remote endpoint and pull back results.
     
     These work under Jena 3.1.1 but there's a fatal problem under Jena 3.2.0
     and later ...
     
     If I issue 6 concurrent execSelect calls to a remote sparql endpoint
     (happens to be fuseki) then 5 will get issued and return correctly but
     #6 will not and from then onwards no further remote calls will go
     through. This only happens if at least 5 requests are in flight with no
     response yet from the remote endpoint before the final one is issued.
     
     It's hard to produce a deterministic, standalone test case because by
     definition it depends on an external sparql endpoint being reliably
     there and reliably not too fast :)
     
     Looking at the stack trace I see:
     
     Unsafe.park(boolean, long) line: not available [native method]
     LockSupport.park(Object) line: 175
     AbstractQueuedSynchronizer$ConditionObject.await() line: 2039
     CPool(AbstractConnPool<T,C,E>).getPoolEntryBlocking(T, Object, long,
     TimeUnit, Future<E>) line: 377
     AbstractConnPool<T,C,E>.access$200(AbstractConnPool, Object, Object,
     long, TimeUnit, Future) line: 67
     AbstractConnPool$2.get(long, TimeUnit) line: 243
     AbstractConnPool$2.get(long, TimeUnit) line: 191
     PoolingHttpClientConnectionManager.leaseConnection(Future<CPoolEntry>,
     long, TimeUnit) line: 282
     PoolingHttpClientConnectionManager$1.get(long, TimeUnit) line: 269
     MainClientExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext,
     HttpExecutionAware) line: 191
     ProtocolExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext,
     HttpExecutionAware) line: 185
     RetryExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext,
     HttpExecutionAware) line: 89
     RedirectExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext,
     HttpExecutionAware) line: 111
     InternalHttpClient.doExecute(HttpHost, HttpRequest, HttpContext) line: 185
     InternalHttpClient(CloseableHttpClient).execute(HttpUriRequest,
     HttpContext) line: 83
     InternalHttpClient(CloseableHttpClient).execute(HttpUriRequest,
     HttpContext) line: 56
     HttpOp.exec(String, HttpUriRequest, String, HttpResponseHandler,
     HttpClient, HttpContext) line: 1081
     HttpOp.execHttpGet(String, String, HttpResponseHandler, HttpClient,
     HttpContext) line: 308
     HttpOp.execHttpGet(String, String, HttpClient, HttpContext) line: 367
     HttpQuery.execGet() line: 326
     HttpQuery.exec() line: 288
     QueryEngineHTTP.execResultSetInner() line: 348
     QueryEngineHTTP.execSelect() line: 340
     SSResultSet.<init>(BaseSparqlSource, String) line: 33
     RemoteSparqlSource(BaseSparqlSource).streamableSelect(String) line: 59
     
     Which suggests some problem with the connection pool locking.
     
     I've tried 3.4.0  and 3.5.0 which incorporate the fix to JENA-1335 (in
     case that's related) but no luck.
     
     I've tried both:
     
         HttpOp.setDefaultHttpClient(HttpClients.createMinimal());
     
     and
     
         HttpOp.setDefaultHttpClient(HttpClients.createDefault());
     
     right at the start of application startup.
     
     Which don't fix the problem. Though perhaps I'm getting the sequence of
     where to set those wrong.
     
     
     Any suggestions?
     
     Any pointers to documentation on how to configure the http client set up
     in current Jena versions?
     
     Thanks,
     Dave