On 05/07/2021 14:02, Martynas Jusevičius wrote:
> HTTPClient is not running out of connections? It is known to hang in
such cases.
It is not reopening connections for one request.
If parsing starts, it has the connection until the end.
On 05/07/2021 14:10, Rob Vesse wrote:
That's a really good suggestion. In the normal code flow do you ever call
stream.close() ? And is createHttpClient() re-using an existing HttpClient
object ? And is the hang only happening after some requests have succeeded ?
It is possible what is happening is that you aren't closing the stream (and I
don't believe Jena's parsers ever close the stream for you) so after so many
requests (10 I think by default) you are exhausting the max connections per
route for the HTTP Client. If that is the case wrapping the use of the stream
in a try-with-resources block may be the solution.
Yes - try-with-resources.
(And a transaction.)
If the code is in a loop, and connections are not managed properly
HttpClient can run out of connections. And the OS kernel can run out of
sockets for that matter if too many connections happen in too short a
time. (Jena's test suite has to deal with this - and it's easier case.)
This is not HttpClient's fault.
Rob's right - RDFParser does not close a resource that is passed into
the parser. If given a resource, the caller is responsible -
try-with-resources or similar is expected.
We are missing some facts.
1/ What's sub.getGraph()?
2/ (if reading to memory) Is the GC going crazy trying to free heap
space because of other earlier work?
3/ Does any data get processed then stop, or does it actually never start?
4/ We haven't seen what the HttpClient setup is. It isn't the default.
Andy
Rob
On 05/07/2021, 14:03, "Martynas Jusevičius" <[email protected]> wrote:
HTTPClient is not running out of connections? It is known to hang in such
cases.
On Mon, Jul 5, 2021 at 2:58 PM james anderson <[email protected]> wrote:
>
> good afternoon;
>
> > On 2021-07-05, at 12:36:20, Andy Seaborne <[email protected]> wrote:
> >
> >
> >
> > On 05/07/2021 10:01, Ivan Lagunov wrote:
> >> Hello,
> >> We’re facing an issue with Jena reading n-triples stream over HTTP.
In fact, our application hangs entirely while executing this piece of code:
> >> Model sub = ModelFactory.createDefaultModel();
> >> TypedInputStream stream = HttpOp.execHttpGet(requestURL,
WebContent.contentTypeNTriples, createHttpClient(auth), null)
> >> // The following part sometimes hangs:
> >> RDFParser.create()
> >> .source(stream)
> >> .lang(Lang.NTRIPLES)
> >> .errorHandler(ErrorHandlerFactory.errorHandlerStrict)
> >> .parse(sub.getGraph());
> >> // This point is not reached
> > >
> >> The issue is not persistent, moreover it happens infrequently.
> >
> > Then it looks like the data has stopped arriving but the connection is
still open. (or the system has gone in GC overload due to heap pressure.)
> >
> > Is intermittent on the same data? Or is the data changing? because
maybe the data can't be written properly and the sender stops sending, though I'd
expect the sender to close the connection (it's now in an unknown state and can't be
reused).
> >
> >> When it occurs, the RDF store server (we use Dydra for that) logs a
successful HTTP 200 response for our call (truncated for readability):
> >> HTTP/1.1" 200 3072/55397664 10.676/10.828 "application/n-triples" "-" "-"
"Apache-Jena-ARQ/3.17.0" "127.0.0.1:8104"
>
> the situation involves an nginx proxy and an upstream sparql processor.
>
> >
> > What do the fields mean?
>
> the line is an excerpt from an entry from the nginx request log. that
line contains:
>
> protocol code requestLength/responseLength
upstreamElapsedTime/clientElapsedTIme acceptType - - clientAgemt upstreamPort
>
> >
> > Is that 3072 bytes sent (so far) of 55397664?
> >
> > If so, is Content-Length set (and then chunk encoding isn't needed).
>
> likely not, as the response is (i believe) that from a sparql request,
which is emitted as it is generated.
>
> >
> > Unfortunately, in HTTP, 200 really means "I started to send stuff", not "I
completed sending stuff". There is no way in HTTP 1/1 to signal an error after starting the response.
>
> that is true, but there are indications in other logs which imply that
the sparql processor believes the response to have been completely sent to nginx.
> there are several reasons to believe this.
> the times and the 200 response code in the nginx log indicate completion.
> otherwise, it would either indicate that it timed out, or would include
a 499 code, to the effect that the client closed the connection before the
response was sent.
> neither is the case.
> in addition, the elapsed time is well below that for which nginx would
time out an upstream connection.
>
> >
> > The HttpClient - how is it configured?
> >
> >> So it looks like the RDF store successfully executes the SPARQL
query, responds with HTTP 200 and starts transferring the data with the chunked
encoding. Then something goes wrong when Jena processes the input stream. I expect there
might be some timeout behind the scenes while Jena reads the stream
> >
> > Does any data reach the graph?
> >
> > There is no timeout at the client end - otherwise you would get an
exception. The parser is reading the input stream from Apache HttpClient. If it
hangs, it's because the data has stopped arriving but the connection is still open.
> >
> > You could try replacing .parse(graph) with parse(StreamRDF) and plug
in a logging StreamRDF so you can see the progress, either sending on data to the
graph or for investigation, merely logging.
> >
> > In HTTP 1.1, a streamed response requires chunk encoding only when the
Content-Length isn't given.
>
> i believe, the content length is not given.
>
> >
> > >
> > , and it causes it to wait indefinitely. At the same time
ErrorHandlerFactory.errorHandlerStrict does not help at all – no errors are logged.
> >> Is there a way to configure the timeout behavior for the underlying
Jena logic of processing HTTP stream? Ideally we want to abort the request if it times
out and then retry it a few times until it succeeds.
> >
> > The HttpClient determines the transfer.
> >
> > Andy
> >
> > FYI: RDFConnectionRemote is an abstraction to make this a little
easier. No need to go to the low-level HttpOp.
> >
> >
> > FYI: Jena 4.mumble.0 is likely to change to using jena.net.http as the
HTTP code. There has to be some change anyway to get HTTP/2 (Apache HttpClient v5+,
not v4, has HTTP/2 support).
> >
> > This will include a new Graph Store Protocol client.
> >
> >> Met vriendelijke groet, with kind regards,
> >> Ivan Lagunov
> >> Technical Lead / Software Architect
> >> Skype: lagivan
> >> Semaku B.V.
> >> Torenallee 20 (SFJ3D) • 5617 BC Eindhoven • www.semaku.com
>
