The following may be the culprit in JDBC's case: The PoolingClientConnectionManager will allocate connections based on its configuration. If all connections for a given route have already been leased, a request for a connection will block until a connection is released back to the pool. One can ensure the connection manager does not block indefinitely in the connection request operation by setting 'http.conn-manager.timeout' to a positive value. If the connection request cannot be serviced within the given time period ConnectionPoolTimeoutException will be thrown.
So HttpClient will block indefinitely until a connection is available. We likely want to turn off that behaviour so that when we hit this state things get a useful error rather than an infinite hang. Rob On 8/8/13 4:11 PM, "Andy Seaborne" <[email protected]> wrote: >Maybe related to JENA-498 (many HttpOps overwhelming the system). > >But if HttOp uses a shared HttpClient, I was getting lockups. It does >appear to be HTTP error handling (failing to close the input stream of >the response when it's 4xx or 5xx - there may be a body still). > >The other part of a shared HttpClient is the authenticator. I haven't >check that yet. I wonder if we need to make it only the HttpClient is >passed in with a HttpAuthenticator alreay set. The DatasetAccessorHttp >could do that. I haven't check the other uses yet; I doubt it's as >clear cut for SPARQL Query etc. > >With the old code, creating new SystemDefaultHttpClient was not giving >connection pooling and reuse; only a fast loop caused a problem (20k-40k >iterators). > >But I don't know why it works on your interval system and not >AFS/Jenkins. Different versions of ARQ/HttpOp? > > Andy > >On 08/08/13 23:44, Rob Vesse wrote: >> Yes the module that hangs is the driver for remote endpoints and stands >>up >> a Fuseki server and communicates with it using HTTP which of course now >> all goes through HttpOp >> >> Problem is that I never seem to get an actual exception just hangs on >>the >> build server. >> >> This might also explain why DEBUG level logging makes the build succeed >> because HttpClient is very noisy at DEBUG level and all that logging >> likely introduces the delays in the right parts of the code to allow >> resources to be freed up. >> >> Rob >> >> >> On 8/8/13 3:40 PM, "Andy Seaborne" <[email protected]> wrote: >> >>> On 08/08/13 19:42, Rob Vesse wrote: >>>> So I am officially stumped >>>> >>>> Adding the delay still causes the builds to hang so I really don't >>>> understand why the builds fail on Apache Jenkins. Note that I've been >>>> building the JDBC module on our internal Jenkins server for some time >>>> and >>>> never had an issue there. Plus the builds run fine on a local >>>>machine. >>>> >>>> If anyone else can take a look or has any suggestions please jump in >>> >>> <straw-grasping mode> >>> >>> Are you using HttpOp? Apache HttpClient? >>> >>> I'm fairly certain HttpOp can cause resource starvation by improper use >>> of HttpClient. However, I haven't managed to find out where for >>>certain >>> [HTTP Exceptions are my current best guess]. (I can perturb the >>> situation by tweaking pooling numbers.) >>> >>> Andy >>> >>>> >>>> Rob >>>> >>>> >>>> On 8/8/13 11:12 AM, "Rob Vesse" <[email protected]> wrote: >>>> >>>>> Ok, so turning the log level back down causes the build to go back to >>>>> failing >>>>> >>>>> This starts to look like some kind of timing issue manifesting on the >>>>> build server causing the tests to get into a hung state. Apparently >>>>> having the high log level adds sufficient delay into the process to >>>>> avoid >>>>> this. >>>>> >>>>> My next idea is to simply insert a delay between the tests in >>>>>question >>>>> and >>>>> see if that solves things. >>>>> >>>>> Rob >>>>> >>>>> >>>>> >>>>> On 8/8/13 10:55 AM, "Rob Vesse" <[email protected]> wrote: >>>>> >>>>>> Ok that is very very weird, after turning up the logging for that >>>>>> module >>>>>> the build ran through to success (and generated a ridiculously large >>>>>> log >>>>>> file at the same time). >>>>>> >>>>>> Next step is to try turning down the log level and see if the build >>>>>> still >>>>>> succeeds. >>>>>> >>>>>> Rob >>>>>> >>>>>> >>>>>> >>>>>> On 8/8/13 10:35 AM, "Rob Vesse" <[email protected]> wrote: >>>>>> >>>>>>> The problem is that nothing is blowing up, the build just gets >>>>>>>stuck >>>>>>> and >>>>>>> hangs until the build timeout plugin steps in and aborts the build >>>>>>> >>>>>>> >>>>>>> The hang is in the tests for the remote endpoint driver which are >>>>>>> standing >>>>>>> up Fuseki instances. However if there was some contention for >>>>>>>ports >>>>>>> in >>>>>>> the tests I would expect the tests to just plain fail. >>>>>>> >>>>>>> I suspect there may be some deadlock of some sort happening when >>>>>>> running >>>>>>> the tests on the server but it's hard to tell where/what the >>>>>>>deadlock >>>>>>> is. >>>>>>> I am turning the log level for the tests in question to DEBUG and >>>>>>> will >>>>>>> re-run a build to see if that yields anything more useful. >>>>>>> >>>>>>> Rob >>>>>>> >>>>>>> On 8/8/13 6:53 AM, "Andy Seaborne" <[email protected]> wrote: >>>>>>> >>>>>>>> On 01/08/13 20:56, Rob Vesse wrote: >>>>>>>>> I've removed it from the main build for now. For some reason it >>>>>>>>>is >>>>>>>>> getting stuck (but not crashing) on the Apache build server. >>>>>>>>>This >>>>>>>>> is >>>>>>>>> despite it building fine locally and on our internal build >>>>>>>>>servers. >>>>>>>>> >>>>>>>>> Not sure how to proceed on this - is it worth setting up a >>>>>>>>>separate >>>>>>>>> build >>>>>>>>> for JDBC on the Apache build servers to help try and isolate the >>>>>>>>> problem? >>>>>>>>> >>>>>>>>> Rob >>>>>>>> >>>>>>>> What exactly is blowing up? >>>>>>>> >>>>>>>> The Apache build servers have all sorts of things on them and a >>>>>>>>wide >>>>>>>> range of plugins, which itself can a problem. >>>>>>>> >>>>>>>> Andy >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 8/1/13 11:45 AM, "Rob Vesse" <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> I've moved Jena JDBC from Experimental into Trunk and added it >>>>>>>>>>to >>>>>>>>>> the >>>>>>>>>> main build. The builds are a little nosier that some of the >>>>>>>>>>other >>>>>>>>>> modules so may want some tweaking to avoid spurious build >>>>>>>>>>output. >>>>>>>>>> >>>>>>>>>> I haven't attempted to figure out how to add it to the distro >>>>>>>>>> because >>>>>>>>>> I >>>>>>>>>> know nothing about Maven Assembly plugin >>>>>>>>>> >>>>>>>>>> Rob >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
