Hi Martynas,
On 20/06/16 22:18, Martynas Jusevičius wrote:
Hey,
after using GenericRuleReasoner and InfModel more extensively, we
started experiencing memory leaks that eventually kill our webapp
because it runs out of heap space. Jena version is 2.11.0.
After some profiling, it seems that RETEEngine.clauseIndex and/or
RETEEngine.infGraph are retaining a lot of references. It might be
related to this report, but I'm not sure:
https://mail-archives.apache.org/mod_mbox/jena-users/201403.mbox/%3c5319b4e0.4060...@gmail.com%3E
If it is related to that then it is not a leak it is "just" memory use.
A leak implies that when you turn over data then unused internal state
objects are not reclaimed. Are you continuously adding and deleting data? If
so then the delete should release the whole of the RETEEngine state and
start over. If that isn't happening then that's a bug but you could work
around with an explicit reset() or even delete and recreate your InfGraph at
that stage. A delete loses all the state anyway.
The suggestion was to use use backward rules instead of forward rules.
I have read the following:
https://jena.apache.org/documentation/inference/#rules
But still I fail to understand in which situations backward rules
can/should be used instead of forward rules?
Forward rules are generally faster because they keep all that partially
matched state. So if you have stable data or just add triples monotonically,
and have a lot of queries, then generally use forward rules for performance.
Backward rules (without tabling) keep no state so there's less memory
overhead and no cost for delete but they are slow and have to redo the work
for every query.
Strictly the performance trade-off is a bit more subtle than that. Forward
rules will try to work out all the entailments whereas backward rules are
just responding to specific queries. So if your queries only touch a small
part of the possible space then backward rules could be more efficient.
However in practice RDF rules seem involve a lot of unground terms and lots
of rules match nearly every query.
Tabling allows you to selectively cache certain predicates which can enable
you to get more reasonable performance while keeping memory use under
control. You can also do some tuning of how the rules execute by testing if
variables are bound or not and using different clause orderings for
different query patterns.
I guess simply replacing
-> with <- will not be enough?
Unless you use non-monotonic predicates (which, sadly, you do) then that
would be enough to get something working. In fact you don't even need to do
that. If you create a pure backward reasoner instances (as opposed to the
hybrid) reasoner it'll read forward syntax rules but treat them as backward.
The actual rules in question look like
this:
[gp: (?class rdf:type
<http://www.w3.org/2000/01/rdf-schema#Class>), (?class ?p ?o), (?p
rdf:type owl:AnnotationProperty), (?p rdfs:isDefinedBy
<http://graphity.org/gp#>), (?subClass rdfs:subClassOf ?class),
(?subClass rdf:type <http://www.w3.org/2000/01/rdf-schema#Class>),
noValue(?subClass ?p) -> (?subClass ?p ?o) ]
That's a horrible rule from the engine's point of view. The head is
completely ungrounded so when running backwards then it will need to run for
*every* triple pattern. [It also makes no sense to me as a use of
owl:AnnotationProperty but whatever.] You could try it backwards but put the
clauses in a more efficient order:
(?subClass ?p ?o) <-
(?p rdf:type owl:AnnotationProperty),
(?p rdfs:isDefinedBy <http://graphity.org/gp#>),
(?subClass rdfs:subClassOf ?class), (?class ?p ?o) .
The rdf:type rdfs:Class constraints are pointless since those are implied by
rdfs:subClassOf anyway. The noValue check is probably best avoided for both
cases.
Alternatively, depending on the nature of your space leak you could use
hybrid rules:
(?p rdf:type owl:AnnotationProperty),
(?p rdfs:isDefinedBy <http://graphity.org/gp#>)
->
[ (?subClass ?p ?o) <- (?subClass rdfs:subClassOf ?class),
(?class ?p ?o) ]
That way the forward engine is only looking at your annotations and the
backward engine then has rules that have grounded predicates. You could also
table those predicates:
(?p rdf:type owl:AnnotationProperty),
(?p rdfs:isDefinedBy <http://graphity.org/gp#>)
->
table(?p),
[ (?subClass ?p ?o) <- (?subClass rdfs:subClassOf ?class),
(?class ?p ?o) ]
[gcdm: (?template rdf:type <http://graphity.org/gp#Template>),
(?template <http://graphity.org/gc#defaultMode> ?o), (?subClass
rdfs:subClassOf ?template), (?subClass rdf:type
<http://graphity.org/gp#Template>), noValue(?subClass
<http://graphity.org/gc#defaultMode>) -> (?subClass
<http://graphity.org/gc#defaultMode> ?o) ]
[gcsm: (?template rdf:type <http://graphity.org/gp#Template>),
(?template <http://graphity.org/gc#supportedMode> ?supportedMode),
(?subClass rdfs:subClassOf ?template), (?subClass rdf:type
<http://graphity.org/gp#Template>) -> (?subClass
<http://graphity.org/gc#supportedMode> ?supportedMode) ]
These two are more reasonable and could be used backwards or hybrid.
[rdfs9: (?x rdfs:subClassOf ?y), (?a rdf:type ?x) -> (?a rdf:type ?y)]
That would work backwards. Depending on the scale of your data you might
want to table rdf:type for performance/space tradeoff.
Can these be rewritten as backward rules instead?
Sure, the challenge is performance tuning as noted above.
Does it involve code changes, such as calling reset() etc?
Shouldn't do.
Dave
