Re: Pooling Http Client lockup

[Dave Reynolds]

Thanks Andy. Yes, I remember mention of things like that on the list 
before but hadn't done a systematic search through.

In this instance the server was localhost which won't have helped but at 
least part of it is/was a straight code defect on my part which allowed 
one class of connections to potentially escape the close().

Dave


On 09/12/17 10:06, Andy Seaborne wrote:
In testing, connection cycling and running out of connections has been a 
recurring and non-deterministic issue.

Under load, the kernel (linux at least) does not clear up connections 
and sockets fast enough.  Its worse when the server is localhost and if 
servers are starting and stopping but it can happen in less extreme, 
more realistic cases, as well.

Th effect can be seen with "netstat -t" - lots of connections in one of 
the wait states.  And this is with all the close() done.

An unclosed connection hangs around for IIRC at least 2 minutes.

The improvements in HttpClient usage don't change this although pooling 
might make it worse.

Makes me appreciate the GC even more.

     Andy

On 08/12/17 23:52, Dave Reynolds wrote:
Hi,

Several test cases later I'm now sure this is my fault, not Jena's.

Thanks for the help, and apologies for the noise.

[There's some code path that can fail to close an exec under certain 
conditions. I can't yet explain why the switch from Jena 3.1.1 to Jena 
3.5.0 would have provoked it when it was so rock solid under Jena 
3.1.1 but I'm sure the problem is on my side.]

Dave

On 08/12/17 22:06, Dave Reynolds wrote:
Hi,

On 08/12/17 19:39, ajs6f wrote:
Dave--

Jena switched HTTP Commons versions a good bit from 3.1.1 to now, so 
that may be part of it. I will look into whether HTTP Commons Client 
changed its behavior under us.

Thanks.

I agree that just upping the maxes and hoping for the best isn't the 
best outcome at all.

Otherwise, this sounds either buggy (if all the execs really are 
getting closed) or at least not very ergonomic (since you aren't 
seeing any warnings).

Do I understand correctly that the behavior you would want would be 
that

1) after the max number of connections have been drawn out of the 
pool and used, the next request should block only until a connection 
is released,
2) and that closing a query execution should definitely return the 
connection underneath it to the pool, more or less immediately?

Yes.

It's almost like something above the client is blocking and not 
letting the connections get released...

Indeed.

I'll attempt to recreate the behaviour with a minimal test case with 
none of our other code stack in the way. Should be able isolate 
whether it some bad practice at our (well, my) end which just happens 
to be safe in earlier versions of http client/jena or whether it 
really is Jena.

If I can get it pinned down to the latter then I'll open a JIRA.

Dave

On Dec 8, 2017, at 12:55 PM, Dave Reynolds 
<dave.e.reyno...@gmail.com> wrote:

Hi Rob,

Thanks, that's useful to know but just raising the limit will just 
make it less frequent but not cure it.

The issue is that if you ever get more in-flight than the limit 
then all future requests are blocked. The in-flight requests return 
fine, the execs are closed but the httpclient never recovers.

I will, of course, check again that I'm successfully closing all 
the execs. However, with Jena 3.1.1 this code has successfully run 
for months between reboots with request rates in the millions per 
week (with occasional high bursts). No lock ups at all. As it 
stands I can't switch it to a newer Jena unless I can absolutely 
guarantee an upper limit on the number of concurrent requests. That 
is often possible (through an apache reverse proxy front end with 
request throttling) but feels like an unsatisfying resolution.

Dave

On 08/12/17 16:56, Rob Vesse wrote:
I think this relates to a HttpClient behaviour that limits the 
maximum connections to a given service
At least in how Jena sets it up the default is 5 connections per 
service which is more generous than the HTTP client defaults.  
Jena appears to read this from a JVM property http.maxConnections 
OR you can construct your own client by calling 
setMaxConnPerRoute() and setMaxConnTotal() on the builder to set 
your desired settings.
Rob
On 08/12/2017, 16:44, "Dave Reynolds" <dave.e.reyno...@gmail.com> 
wrote:
     Hi,
          I've being updating some rather old libraries that use 
Jena to issue
     sparql requests to a remote endpoint and pull back results.
          These work under Jena 3.1.1 but there's a fatal problem 
under Jena 3.2.0
     and later ...
          If I issue 6 concurrent execSelect calls to a remote 
sparql endpoint
     (happens to be fuseki) then 5 will get issued and return 
correctly but
     #6 will not and from then onwards no further remote calls 
will go
     through. This only happens if at least 5 requests are in 
flight with no
     response yet from the remote endpoint before the final one is 
issued.
          It's hard to produce a deterministic, standalone test 
case because by
     definition it depends on an external sparql endpoint being 
reliably
     there and reliably not too fast :)
          Looking at the stack trace I see:
          Unsafe.park(boolean, long) line: not available [native 
method]
     LockSupport.park(Object) line: 175
     AbstractQueuedSynchronizer$ConditionObject.await() line: 2039
     CPool(AbstractConnPool<T,C,E>).getPoolEntryBlocking(T, 
Object, long,
     TimeUnit, Future<E>) line: 377
     AbstractConnPool<T,C,E>.access$200(AbstractConnPool, Object, 
Object,
     long, TimeUnit, Future) line: 67
     AbstractConnPool$2.get(long, TimeUnit) line: 243
     AbstractConnPool$2.get(long, TimeUnit) line: 191
PoolingHttpClientConnectionManager.leaseConnection(Future<CPoolEntry>, 

     long, TimeUnit) line: 282
     PoolingHttpClientConnectionManager$1.get(long, TimeUnit) 
line: 269
     MainClientExec.execute(HttpRoute, HttpRequestWrapper, 
HttpClientContext,
     HttpExecutionAware) line: 191
     ProtocolExec.execute(HttpRoute, HttpRequestWrapper, 
HttpClientContext,
     HttpExecutionAware) line: 185
     RetryExec.execute(HttpRoute, HttpRequestWrapper, 
HttpClientContext,
     HttpExecutionAware) line: 89
     RedirectExec.execute(HttpRoute, HttpRequestWrapper, 
HttpClientContext,
     HttpExecutionAware) line: 111
     InternalHttpClient.doExecute(HttpHost, HttpRequest, 
HttpContext) line: 185
     InternalHttpClient(CloseableHttpClient).execute(HttpUriRequest,
     HttpContext) line: 83
     InternalHttpClient(CloseableHttpClient).execute(HttpUriRequest,
     HttpContext) line: 56
     HttpOp.exec(String, HttpUriRequest, String, HttpResponseHandler,
     HttpClient, HttpContext) line: 1081
     HttpOp.execHttpGet(String, String, HttpResponseHandler, 
HttpClient,
     HttpContext) line: 308
     HttpOp.execHttpGet(String, String, HttpClient, HttpContext) 
line: 367
     HttpQuery.execGet() line: 326
     HttpQuery.exec() line: 288
     QueryEngineHTTP.execResultSetInner() line: 348
     QueryEngineHTTP.execSelect() line: 340
     SSResultSet.<init>(BaseSparqlSource, String) line: 33
     RemoteSparqlSource(BaseSparqlSource).streamableSelect(String) 
line: 59
          Which suggests some problem with the connection pool 
locking.
          I've tried 3.4.0  and 3.5.0 which incorporate the fix to 
JENA-1335 (in
     case that's related) but no luck.
          I've tried both:
HttpOp.setDefaultHttpClient(HttpClients.createMinimal());
          and
HttpOp.setDefaultHttpClient(HttpClients.createDefault());
          right at the start of application startup.
          Which don't fix the problem. Though perhaps I'm getting 
the sequence of
     where to set those wrong.
               Any suggestions?
          Any pointers to documentation on how to configure the 
http client set up
     in current Jena versions?
          Thanks,
     Dave