Mark
Answers inline:
On 02/06/2014 19:14, "Mark Feblowitz" <markfeblow...@comcast.net> wrote:
>Ok -
>
>It’s another day… after a weekend’s rest. And now I have a few
>observations and many questions:
>
>So, a bit more about what I’m doing (much of it likely naive):
>
>I create a local ontmodel, create several statements in that model,
>convert the model to a string, wrap it with INSERT DATA { }, then post it
>to Fuseki using an Update Request:
>
> UpdateRequest ur = new UpdateRequest();
> ur.add(payloadContent);
> UpdateProcessor up =
> UpdateExecutionFactory.createRemote(ur, service);
> up.execute();
> } catch (Exception e) {
> e.printStackTrace();
> } finally {
> }
If your graphs are relatively small then it might be more efficient to
look at using DatasetAccessor
(https://jena.apache.org/documentation/javadoc/arq/com/hp/hpl/jena/query/Da
tasetAccessor.html) to PUT the entire updated graph to the server via the
Graph Store Protocol (SPARQL 1.1 Graph Store HTTP Protocol
<http://www.w3.org/TR/sparql11-http-rdf-update/>) e.g.
DatasetAccessor accessor =
DatasetAccessorFactory.createHTTP("http://localhost:3030/ds/data";);
accessor.putModel(model, "http://example.org/graph";);
>
>
>I have standalone code that sends SPARQL queries to the same Fuseki
>server, using the usual methods.
>
>So, before I go deeper into the questions, it occurs to me that I don’t
>necessarily need to post remotely, and removing that overhead would be an
>obvious improvement . I’m wondering whether my update code could interact
>directly with TDB while also having the content be queriable via Fuseki.
>I recall trying that a while ago and noting that direct posts to TDB were
>not available for query until restarting Fuseki. Was that a (repaired)
>bug or a feature?
DO NOT do this!
TDB is only designed for single JVM usage and attaching multiple JVMs to a
dataset runs the risk of irrevocably corrupting your dataset. Work is
actually under way to add functionality to TDB to prevent users doing this
because we've seen far too many cases of data corruption resulting from
this.
>
>If I can’t do this, do you know of any projects that have created a
>server that can be run with my application and share the TDB connection?
You want a triple store that can run in both embedded and remote mode
simultaneously? No I don't know of any
There are stores that support non-HTTP protocols (ODBC, custom protocol
buffer based etc) that may offer better performance and allow you to have
multiple applications talking to the server over different protocols but
they still require a single server. Most of these are however commercial
options but we can provide your our recommendations if you want them.
>
>Assuming that I do stick with remote updates, I have some questions:
>
>First, note that I create a new Update processor for each posted update.
>I don’t see any up.close() so I’m guessing I’m not creating any leakage
>here. Would it be better to create one update processor and reuse it
>across updates?
Yes each remote update request is a self contained HTTP request with no
response body expected so the connection is closed automatically after the
request is made and the response headers read.
>
>Next, I’m seeing a lot of notices like the following in the Fuseki
>console:
>
>13:40:07 INFO [19125] 204 No Content (21 ms)
>13:40:07 INFO [19126] POST http://localhost:3030/km4sp/update
>13:40:07 INFO [19126] 204 No Content (13 ms)
>13:40:07 INFO [19127] POST http://localhost:3030/km4sp/update
>13:40:07 INFO [19127] 204 No Content (16 ms)
>13:40:07 INFO [19128] POST http://localhost:3030/km4sp/update
>13:40:07 INFO [19128] 204 No Content (14 ms)
>13:40:07 INFO [19129] POST http://localhost:3030/km4sp/update
>13:40:07 INFO [19129] 204 No Content (8 ms)
>13:40:07 INFO [19130] POST http://localhost:3030/km4sp/update
>
>I can't tell whether this reflects
> 1) a high overall number of update calls from my code, or
> 2) that the Fuseki side processes each of the triples for an update as
>an http post.
It reflects both
A remote update request is executed by making a HTTP POST to the /update
endpoint seen in your logs
204 No Content is the standard Fuseki response to a successful update
since there is not expected to be any response body from an update
>
>If it’s the latter, I don’t think I can batch up more updates to improve
>overall performance.
>
>I guess I can tell by inserting delays between my posts and capturing
>timestamps in my log that can be compared against what I’m seeing in the
>Fuseki console. I’ll try that next.
The number in square brackets is the request ID and is included in the
response as a Fuseki-Request-ID header. Returned headers aren't exposed
to the ARQ code per se but you can up the debugging level for the
org.apache.http packages for your client application to see detailed HTTP
traces in the logs of the communications between your application and
Fuseki. You'll need DEBUG or TRACE level depending on the version of ARQ,
more recent versions of ARQ include more recent versions of Apache
HttpClient which need you to turn the log level all the way up to TRACE
while older ARQ versions only needed the log level up to DEBUG to see the
traces.
Rob
>
>I guess that’s it for the questions for now.
>
>Any recommendations?
>
>Thanks again,
>
>Mark
>
>
>
>On May 31, 2014, at 12:22 PM, Andy Seaborne <a...@apache.org> wrote:
>
>> Hi Mark,
>>
>> The long running query is quite significant.
>>
>> On 30/05/14 18:26, Mark Feblowitz wrote:
>>> That’s a good idea.
>>>
>>> One improvement I’ve already made was to relocate the DB to local
>>> disk - having it on a shared filesystem is an even worse idea.
>>>
>>> The updates tend to be on the order of 5-20 triples at a time.
>>
>> If you could batch up changes, that will help for all sorts of reasons.
>> c.f. autocommit and JDBC where many small changes run really slowly.
>>
>> This is part of the issue - write transactions have a significant fixed
>> cost that you get for even a (theoretical) transaction of no changes. It
>> has to write a few bytes and do a disk-sync. Reads continue during this
>> time but longer write times means there is less chance of system being
>> able to write the journal to the main database. JENA-567 may help but
>> isn't faster (it's slower) but it saves memory.
>>
>> Read transactions have near zero cost in TDB - Fuseki/TDB is
>>read-centric.
>>
>> What's more, TDB block size is 8Kbytes so one change in a block is 8K
>>of transaction state. Multiple times for multiple indexes. So 5
>>triples of change get every little shared block effect and the memory
>>footprint is disproportionally large.
>>
>> <thinking out loud id=1>
>>
>> A block size of 1 or 2k for the leaf blocks in TDB, leaving the branch
>>blocks at 8k (they have in different block managers = files) would be
>>worth experimenting with.
>>
>> </thinking out loud>
>>
>> <thinking out loud id=2>
>>
>> We could provide some netty/mima/... based server that did the moral
>> equivalent of the SPARQL Protocol (cf jena-jdbc, jena-client). HTTP is
>>said to be an appreciable cost. This is no judgement of Jetty/tomcat,
>>it is the nature of HTTP; it is cautious wording because I haven't
>>observed it myself - Jetty locally, together with careful streaming of
>>results seems to be quite effective. Fast encoding of results would be
>>good for both.
>>
>> </thinking out loud>
>>
>>> I believe I identified the worst culprit, and that was using
>>> OWLFBRuleReasoner rather than RDFSExptRuleReasoner or
>>> TransitiveReasoner. My guess is that the longish query chain over a
>>> large triplestore, using the Owl reasoner was leading to very long
>>> query times and lots of memory consumption. Do you think that’s a
>>> reasonable guess?
>>
>> That does look right. Long running queries, or the effect of an
>>intense stream of small back-to-back queries combined with the update
>>pattern, leave no time for the system to flush the journal back to the
>>main database. This leads to memory usage and eventually OOME.
>>
>>> How I reached that conclusion was to kill the non-responsive (even
>>> for a small query) Fuseki and restart with RDFSExptRuleReasoner (same
>>> DB, with many triples). After that, both the small query and the
>>> multi-join query responded quite quickly.
>>>
>>> If necessary, I’ll try to throttle the posts, since I’m in complete
>>> control of the submissions.
>>
>> That should at least prove whether this discussion has correctly
>>diagnosed the interactions leading to OOME. What we have is ungrace-ful
>>("disgraceful") behaviour as the load reaches system saturation. It
>>ought to be more graceful but, fundamentally, its always going to be
>>possible to flood a system, any system, with more work than it is
>>capable of.
>>
>> Andy
>>>
>>> Thanks,
>>>
>>> Mark
>>>
>>> On May 30, 2014, at 12:52 PM, Andy Seaborne <a...@apache.org> wrote:
>>>
>>>> Mark,
>>>>
>>>> How big are the updates?
>>>>
>>>> An SSD for the database and the journal will help.
>>>>
>>>> Every transaction is a commit, and a commit is a disk operation to
>>>> ensure the commit record is permanent. That is not cheap with a
>>>> rotational disk (seek time), and much better with an SSD.
>>>>
>>>> If you are driving Fuseki as hard as possible, something will break
>>>> - the proposal in JENA-703 amounts to slowing the clients down as
>>>> well as being more defensive.
>>>>
>>>> Andy
>>>>
>>>> On 30/05/14 15:39, Rob Vesse wrote:
>>>>> Mark
>>>>>
>>>>> This sounds like the same problem described in
>>>>> https://issues.apache.org/jira/browse/JENA-689
>>>>>
>>>>> TL;DR
>>>>>
>>>>> For a system with no quiescent periods continually receiving
>>>>> updates the in-memory journal continues to expand until such time
>>>>> as an OOM occurs. There will be little/no data loss because the
>>>>> journal is a write ahead log and is first written to disk (you
>>>>> will lose at most the data from the transaction that encountered
>>>>> the OOM). Therefore once the system is restarted the journal
>>>>> will be replayed and flushed.
>>>>>
>>>>> See https://issues.apache.org/jira/browse/JENA-567 for an
>>>>> experimental feature that may mitigate this and see
>>>>> https://issues.apache.org/jira/browse/JENA-703 for the issue
>>>>> tracking the work to remove this limitation
>>>>>
>>>>> Rob
>>
>
