[jira] [Comment Edited] (JENA-2107) RDF Star performance issue with non-concrete node triples
[ https://issues.apache.org/jira/browse/JENA-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17346211#comment-17346211 ] Andy Seaborne edited comment on JENA-2107 at 5/17/21, 3:04 PM: --- {{tdbquery}} processing will be in SolverRX (one each for TDB1, TDB2). was (Author: andy.seaborne): {{tdbquery}} with be in SolverRX (one each for TDB1, TDB2). > RDF Star performance issue with non-concrete node triples > - > > Key: JENA-2107 > URL: https://issues.apache.org/jira/browse/JENA-2107 > Project: Apache Jena > Issue Type: Improvement > Components: ARQ >Affects Versions: Jena 3.17.0, Jena 4.0.0 >Reporter: Lorenz Bühmann >Priority: Critical > Fix For: Jena 4.1.0 > > > the following graph pattern is not evaluated efficiently (results in > full-scan per binding) because the second triple pattern doesn't take > advantage of the bindings generated by evaluation of the first one: > {code:java} > ?s ?o . > << ?s ?o >> ?v . > {code} > A possible fix would be to adapt the method {{rdfStarTripleSub()}} in class > > [SolverRX3.java|https://github.com/apache/jena/blob/2efff8a00b4ffa82751cf46c8a3fed84b6ff3090/jena-arq/src/main/java/org/apache/jena/sparql/engine/main/solver/SolverRX3.java#L63-L71] > by changing the beginning to > {code:java} > private static Iterator rdfStarTripleSub(Binding input, Triple > xPattern, ExecutionContext execCxt) { > Triple tPattern = Substitute.substitute(xPattern, input); > {code} > We went from 75s for a very small dataset (50k triples) to near instant > response times. > If this fix is correct and doesn't break anything, it might be the same way > to fix for its quads counterpart in {{SolverRX4}} class. > > Note, for tdbquery, this seems to be evaluated at a different place? At > least, we couldn't find any performance improvement, it's still horribly slow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (JENA-2107) RDF Star performance issue with non-concrete node triples
[ https://issues.apache.org/jira/browse/JENA-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17346371#comment-17346371 ] Andy Seaborne edited comment on JENA-2107 at 5/17/21, 10:32 PM: All the solvers need this fix. I've done (locally) RX4 and the TDB SolverRX's following the pattern from the PR. [~LorenzB] if you adjust the commit message on the PR, I'll do the other three. Would you be able to test the TDB (either one, ideally both)? I can't think of a test for the situation other than timing because the core of the RDF-star triple solver code is, by design, more general than assuming substitution has been done, which is why it didn't show except from a performance effect. was (Author: andy.seaborne): All the solvers need this fix. I've don RX4 and the TDB SolverRX's following the pattern from the PR. [~LorenzB] if you adjust he commit message on the PR, I'll do the other three. Would you be able to test the TDB (either one, ideally both)? I can't think of a test for the situation other than timing because the core of the RDF-star triple solver code is, by design, more general than assuming substitution has been done, which is why it didn't show except from a performance effect. > RDF Star performance issue with non-concrete node triples > - > > Key: JENA-2107 > URL: https://issues.apache.org/jira/browse/JENA-2107 > Project: Apache Jena > Issue Type: Improvement > Components: ARQ >Affects Versions: Jena 3.17.0, Jena 4.0.0 >Reporter: Lorenz Bühmann >Priority: Critical > Fix For: Jena 4.1.0 > > > the following graph pattern is not evaluated efficiently (results in > full-scan per binding) because the second triple pattern doesn't take > advantage of the bindings generated by evaluation of the first one: > {code:java} > ?s ?o . > << ?s ?o >> ?v . > {code} > A possible fix would be to adapt the method {{rdfStarTripleSub()}} in class > > [SolverRX3.java|https://github.com/apache/jena/blob/2efff8a00b4ffa82751cf46c8a3fed84b6ff3090/jena-arq/src/main/java/org/apache/jena/sparql/engine/main/solver/SolverRX3.java#L63-L71] > by changing the beginning to > {code:java} > private static Iterator rdfStarTripleSub(Binding input, Triple > xPattern, ExecutionContext execCxt) { > Triple tPattern = Substitute.substitute(xPattern, input); > {code} > We went from 75s for a very small dataset (50k triples) to near instant > response times. > If this fix is correct and doesn't break anything, it might be the same way > to fix for its quads counterpart in {{SolverRX4}} class. > > Note, for tdbquery, this seems to be evaluated at a different place? At > least, we couldn't find any performance improvement, it's still horribly slow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (JENA-2107) RDF Star performance issue with non-concrete node triples
[ https://issues.apache.org/jira/browse/JENA-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17346475#comment-17346475 ] Claus Stadler edited comment on JENA-2107 at 5/17/21, 11:16 PM: For the Dataset-based implementation we could subclass the find methods of DatasetGraphWrapper to keep track of the internal iterator sizes. After running a query on such an dataset instance one could then check whether only a specific number of tuples have been touched Alternatively, one could track the arguments passed to find and check whether those match an expected sequence (or set) of reference arguments - which would be more traceable than mere counts. Sketch: {code:java} class TrackingDatasetGraph extends DatasetGraphWrapper { protected long numSeenTuples = 0; protected Collection seenArgs = new LinkedHashSet<>(); // or ArrayList @Override public Iterator find(Node g, Node s, Node p, Node o) { seenArgs.add(Arrays.asList(g, s, p, o)); try { Iterator it = getR().find() List materialized = Iter.toList(it); numSeenTuples += materialized.size(); return materialized.iterator(); } } {code} It's just somewhat cumbersome having to repeat the same pattern for NodeTupleTable(Wrapper) ; and I am not sure about how accessible this engine is for testing these internals. Having at least a single test case would already be beneficial for detecting regressions in this regard while work on RDF star progresses. was (Author: aklakan): For the Dataset-based implementation we could subclass the find methods of DatasetGraphWrapper to keep track of the internal iterator sizes. After running a query on such an dataset instance one could then check whether only a specific number of tuples have been touched Alternatively, one could track the arguments passed to find and check whether those match an expected sequence (or set) of reference arguments - which would be more traceable than mere counts. Sketch: {code:java} class TrackingDatasetGraph extends DatasetGraphWrapper { protected long numSeenTuples = 0; protected Collection seenArgs = new LinkedHashSet<>(); // or ArrayList @Override public Iterator find(Node g, Node s, Node p, Node o) { seenArgs.add(Arrays.asList(g, s, p, o)); try { Iterator it = getR().find() List materialized = Iter.toList(it); numSeenTuples += materialized.size(); return materialized.iterator(); } } {code} It's just somewhat cumbersome having to repeat the same pattern for NodeTupleTable(Wrapper). Having at least a single test case would already be beneficial for detecting regressions in this regard while work on RDF star progresses. > RDF Star performance issue with non-concrete node triples > - > > Key: JENA-2107 > URL: https://issues.apache.org/jira/browse/JENA-2107 > Project: Apache Jena > Issue Type: Improvement > Components: ARQ >Affects Versions: Jena 3.17.0, Jena 4.0.0 >Reporter: Lorenz Bühmann >Priority: Critical > Fix For: Jena 4.1.0 > > > the following graph pattern is not evaluated efficiently (results in > full-scan per binding) because the second triple pattern doesn't take > advantage of the bindings generated by evaluation of the first one: > {code:java} > ?s ?o . > << ?s ?o >> ?v . > {code} > A possible fix would be to adapt the method {{rdfStarTripleSub()}} in class > > [SolverRX3.java|https://github.com/apache/jena/blob/2efff8a00b4ffa82751cf46c8a3fed84b6ff3090/jena-arq/src/main/java/org/apache/jena/sparql/engine/main/solver/SolverRX3.java#L63-L71] > by changing the beginning to > {code:java} > private static Iterator rdfStarTripleSub(Binding input, Triple > xPattern, ExecutionContext execCxt) { > Triple tPattern = Substitute.substitute(xPattern, input); > {code} > We went from 75s for a very small dataset (50k triples) to near instant > response times. > If this fix is correct and doesn't break anything, it might be the same way > to fix for its quads counterpart in {{SolverRX4}} class. > > Note, for tdbquery, this seems to be evaluated at a different place? At > least, we couldn't find any performance improvement, it's still horribly slow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (JENA-2107) RDF Star performance issue with non-concrete node triples
[ https://issues.apache.org/jira/browse/JENA-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17346475#comment-17346475 ] Claus Stadler edited comment on JENA-2107 at 5/17/21, 11:17 PM: For the Dataset-based implementation we could subclass the find methods of DatasetGraphWrapper to keep track of the internal iterator sizes. After running a query on such an dataset instance one could then check whether only a specific number of tuples have been touched Alternatively, one could track the arguments passed to find and check whether those match an expected sequence (or set) of reference arguments - which would be more traceable than mere counts. Sketch: {code:java} class TrackingDatasetGraph extends DatasetGraphWrapper { protected long numSeenTuples = 0; protected Collection seenArgs = new LinkedHashSet<>(); // or ArrayList @Override public Iterator find(Node g, Node s, Node p, Node o) { seenArgs.add(Arrays.asList(g, s, p, o)); Iterator it = getR().find() List materialized = Iter.toList(it); numSeenTuples += materialized.size(); return materialized.iterator(); } } {code} It's just somewhat cumbersome having to repeat the same pattern for NodeTupleTable(Wrapper) ; and I am not sure about how accessible this engine is for testing these internals. Having at least a single test case would already be beneficial for detecting regressions in this regard while work on RDF star progresses. was (Author: aklakan): For the Dataset-based implementation we could subclass the find methods of DatasetGraphWrapper to keep track of the internal iterator sizes. After running a query on such an dataset instance one could then check whether only a specific number of tuples have been touched Alternatively, one could track the arguments passed to find and check whether those match an expected sequence (or set) of reference arguments - which would be more traceable than mere counts. Sketch: {code:java} class TrackingDatasetGraph extends DatasetGraphWrapper { protected long numSeenTuples = 0; protected Collection seenArgs = new LinkedHashSet<>(); // or ArrayList @Override public Iterator find(Node g, Node s, Node p, Node o) { seenArgs.add(Arrays.asList(g, s, p, o)); try { Iterator it = getR().find() List materialized = Iter.toList(it); numSeenTuples += materialized.size(); return materialized.iterator(); } } {code} It's just somewhat cumbersome having to repeat the same pattern for NodeTupleTable(Wrapper) ; and I am not sure about how accessible this engine is for testing these internals. Having at least a single test case would already be beneficial for detecting regressions in this regard while work on RDF star progresses. > RDF Star performance issue with non-concrete node triples > - > > Key: JENA-2107 > URL: https://issues.apache.org/jira/browse/JENA-2107 > Project: Apache Jena > Issue Type: Improvement > Components: ARQ >Affects Versions: Jena 3.17.0, Jena 4.0.0 >Reporter: Lorenz Bühmann >Priority: Critical > Fix For: Jena 4.1.0 > > > the following graph pattern is not evaluated efficiently (results in > full-scan per binding) because the second triple pattern doesn't take > advantage of the bindings generated by evaluation of the first one: > {code:java} > ?s ?o . > << ?s ?o >> ?v . > {code} > A possible fix would be to adapt the method {{rdfStarTripleSub()}} in class > > [SolverRX3.java|https://github.com/apache/jena/blob/2efff8a00b4ffa82751cf46c8a3fed84b6ff3090/jena-arq/src/main/java/org/apache/jena/sparql/engine/main/solver/SolverRX3.java#L63-L71] > by changing the beginning to > {code:java} > private static Iterator rdfStarTripleSub(Binding input, Triple > xPattern, ExecutionContext execCxt) { > Triple tPattern = Substitute.substitute(xPattern, input); > {code} > We went from 75s for a very small dataset (50k triples) to near instant > response times. > If this fix is correct and doesn't break anything, it might be the same way > to fix for its quads counterpart in {{SolverRX4}} class. > > Note, for tdbquery, this seems to be evaluated at a different place? At > least, we couldn't find any performance improvement, it's still horribly slow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (JENA-2107) RDF Star performance issue with non-concrete node triples
[ https://issues.apache.org/jira/browse/JENA-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17346475#comment-17346475 ] Claus Stadler edited comment on JENA-2107 at 5/17/21, 11:17 PM: For the Dataset-based implementation we could subclass the find methods of DatasetGraphWrapper to keep track of the internal iterator sizes. After running a query on such a dataset instance one could then check whether only a specific number of tuples have been touched Alternatively, one could track the arguments passed to find and check whether those match an expected sequence (or set) of reference arguments - which would be more traceable than mere counts. Sketch: {code:java} class TrackingDatasetGraph extends DatasetGraphWrapper { protected long numSeenTuples = 0; protected Collection seenArgs = new LinkedHashSet<>(); // or ArrayList @Override public Iterator find(Node g, Node s, Node p, Node o) { seenArgs.add(Arrays.asList(g, s, p, o)); Iterator it = getR().find() List materialized = Iter.toList(it); numSeenTuples += materialized.size(); return materialized.iterator(); } } {code} It's just somewhat cumbersome having to repeat the same pattern for NodeTupleTable(Wrapper) ; and I am not sure about how accessible this engine is for testing these internals. Having at least a single test case would already be beneficial for detecting regressions in this regard while work on RDF star progresses. was (Author: aklakan): For the Dataset-based implementation we could subclass the find methods of DatasetGraphWrapper to keep track of the internal iterator sizes. After running a query on such an dataset instance one could then check whether only a specific number of tuples have been touched Alternatively, one could track the arguments passed to find and check whether those match an expected sequence (or set) of reference arguments - which would be more traceable than mere counts. Sketch: {code:java} class TrackingDatasetGraph extends DatasetGraphWrapper { protected long numSeenTuples = 0; protected Collection seenArgs = new LinkedHashSet<>(); // or ArrayList @Override public Iterator find(Node g, Node s, Node p, Node o) { seenArgs.add(Arrays.asList(g, s, p, o)); Iterator it = getR().find() List materialized = Iter.toList(it); numSeenTuples += materialized.size(); return materialized.iterator(); } } {code} It's just somewhat cumbersome having to repeat the same pattern for NodeTupleTable(Wrapper) ; and I am not sure about how accessible this engine is for testing these internals. Having at least a single test case would already be beneficial for detecting regressions in this regard while work on RDF star progresses. > RDF Star performance issue with non-concrete node triples > - > > Key: JENA-2107 > URL: https://issues.apache.org/jira/browse/JENA-2107 > Project: Apache Jena > Issue Type: Improvement > Components: ARQ >Affects Versions: Jena 3.17.0, Jena 4.0.0 >Reporter: Lorenz Bühmann >Priority: Critical > Fix For: Jena 4.1.0 > > > the following graph pattern is not evaluated efficiently (results in > full-scan per binding) because the second triple pattern doesn't take > advantage of the bindings generated by evaluation of the first one: > {code:java} > ?s ?o . > << ?s ?o >> ?v . > {code} > A possible fix would be to adapt the method {{rdfStarTripleSub()}} in class > > [SolverRX3.java|https://github.com/apache/jena/blob/2efff8a00b4ffa82751cf46c8a3fed84b6ff3090/jena-arq/src/main/java/org/apache/jena/sparql/engine/main/solver/SolverRX3.java#L63-L71] > by changing the beginning to > {code:java} > private static Iterator rdfStarTripleSub(Binding input, Triple > xPattern, ExecutionContext execCxt) { > Triple tPattern = Substitute.substitute(xPattern, input); > {code} > We went from 75s for a very small dataset (50k triples) to near instant > response times. > If this fix is correct and doesn't break anything, it might be the same way > to fix for its quads counterpart in {{SolverRX4}} class. > > Note, for tdbquery, this seems to be evaluated at a different place? At > least, we couldn't find any performance improvement, it's still horribly slow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (JENA-2107) RDF Star performance issue with non-concrete node triples
[ https://issues.apache.org/jira/browse/JENA-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17346475#comment-17346475 ] Claus Stadler edited comment on JENA-2107 at 5/17/21, 11:22 PM: For the Dataset-based implementation we could subclass the find methods of DatasetGraphWrapper to keep track of the internal iterator sizes. After running a query on such a dataset instance one could then check whether only a specific number of tuples have been touched Alternatively, one could track the arguments passed to find and check whether those match an expected sequence (or set) of reference arguments - which would be more traceable than mere counts. Sketch: {code:java} class TrackingDatasetGraph extends DatasetGraphWrapper { protected long numSeenTuples = 0; protected Collection seenArgs = new LinkedHashSet<>(); // or ArrayList @Override public Iterator find(Node g, Node s, Node p, Node o) { seenArgs.add(Arrays.asList(g, s, p, o)); Iterator it = getR().find() List materialized = Iter.toList(it); numSeenTuples += materialized.size(); return materialized.iterator(); } } {code} It's just somewhat cumbersome if we had to repeat the same pattern for NodeTupleTable(Wrapper) ; and I am not sure about how accessible this engine is for testing these internals. I know that TDB2 has a DatasetGraph abstraction - and I am assuming TDB1 has one too - so above sketch might already be sufficient to test all RDF star implementations. Having at least a single test case would already be beneficial for detecting regressions in this regard while work on RDF star progresses. was (Author: aklakan): For the Dataset-based implementation we could subclass the find methods of DatasetGraphWrapper to keep track of the internal iterator sizes. After running a query on such a dataset instance one could then check whether only a specific number of tuples have been touched Alternatively, one could track the arguments passed to find and check whether those match an expected sequence (or set) of reference arguments - which would be more traceable than mere counts. Sketch: {code:java} class TrackingDatasetGraph extends DatasetGraphWrapper { protected long numSeenTuples = 0; protected Collection seenArgs = new LinkedHashSet<>(); // or ArrayList @Override public Iterator find(Node g, Node s, Node p, Node o) { seenArgs.add(Arrays.asList(g, s, p, o)); Iterator it = getR().find() List materialized = Iter.toList(it); numSeenTuples += materialized.size(); return materialized.iterator(); } } {code} It's just somewhat cumbersome having to repeat the same pattern for NodeTupleTable(Wrapper) ; and I am not sure about how accessible this engine is for testing these internals. Having at least a single test case would already be beneficial for detecting regressions in this regard while work on RDF star progresses. > RDF Star performance issue with non-concrete node triples > - > > Key: JENA-2107 > URL: https://issues.apache.org/jira/browse/JENA-2107 > Project: Apache Jena > Issue Type: Improvement > Components: ARQ >Affects Versions: Jena 3.17.0, Jena 4.0.0 >Reporter: Lorenz Bühmann >Priority: Critical > Fix For: Jena 4.1.0 > > > the following graph pattern is not evaluated efficiently (results in > full-scan per binding) because the second triple pattern doesn't take > advantage of the bindings generated by evaluation of the first one: > {code:java} > ?s ?o . > << ?s ?o >> ?v . > {code} > A possible fix would be to adapt the method {{rdfStarTripleSub()}} in class > > [SolverRX3.java|https://github.com/apache/jena/blob/2efff8a00b4ffa82751cf46c8a3fed84b6ff3090/jena-arq/src/main/java/org/apache/jena/sparql/engine/main/solver/SolverRX3.java#L63-L71] > by changing the beginning to > {code:java} > private static Iterator rdfStarTripleSub(Binding input, Triple > xPattern, ExecutionContext execCxt) { > Triple tPattern = Substitute.substitute(xPattern, input); > {code} > We went from 75s for a very small dataset (50k triples) to near instant > response times. > If this fix is correct and doesn't break anything, it might be the same way > to fix for its quads counterpart in {{SolverRX4}} class. > > Note, for tdbquery, this seems to be evaluated at a different place? At > least, we couldn't find any performance improvement, it's still horribly slow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (JENA-2107) RDF Star performance issue with non-concrete node triples
[ https://issues.apache.org/jira/browse/JENA-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17346475#comment-17346475 ] Claus Stadler edited comment on JENA-2107 at 5/17/21, 11:25 PM: For the Dataset-based implementation we could subclass the find methods of DatasetGraphWrapper to keep track of the internal iterator sizes. After running a query on such a dataset instance one could then check whether only a specific number of tuples have been touched Alternatively, one could track the arguments passed to find and check whether those match an expected sequence (or set) of reference arguments - which would be more traceable than mere counts. Sketch: {code:java} class TrackingDatasetGraph extends DatasetGraphWrapper { protected long numSeenTuples = 0; protected Collection seenArgs = new LinkedHashSet<>(); // or ArrayList @Override public Iterator find(Node g, Node s, Node p, Node o) { seenArgs.add(Arrays.asList(g, s, p, o)); Iterator it = getR().find() List materialized = Iter.toList(it); numSeenTuples += materialized.size(); return materialized.iterator(); } } {code} It's just somewhat cumbersome having to repeat the same pattern for NodeTupleTable(Wrapper) ; and I am not sure about how accessible this part of the engine is for testing these internals. Having at least a single test case would already be beneficial for detecting regressions in this regard while work on RDF star progresses. was (Author: aklakan): For the Dataset-based implementation we could subclass the find methods of DatasetGraphWrapper to keep track of the internal iterator sizes. After running a query on such a dataset instance one could then check whether only a specific number of tuples have been touched Alternatively, one could track the arguments passed to find and check whether those match an expected sequence (or set) of reference arguments - which would be more traceable than mere counts. Sketch: {code:java} class TrackingDatasetGraph extends DatasetGraphWrapper { protected long numSeenTuples = 0; protected Collection seenArgs = new LinkedHashSet<>(); // or ArrayList @Override public Iterator find(Node g, Node s, Node p, Node o) { seenArgs.add(Arrays.asList(g, s, p, o)); Iterator it = getR().find() List materialized = Iter.toList(it); numSeenTuples += materialized.size(); return materialized.iterator(); } } {code} It's just somewhat cumbersome if we had to repeat the same pattern for NodeTupleTable(Wrapper) ; and I am not sure about how accessible this engine is for testing these internals. I know that TDB2 has a DatasetGraph abstraction - and I am assuming TDB1 has one too - so above sketch might already be sufficient to test all RDF star implementations. Having at least a single test case would already be beneficial for detecting regressions in this regard while work on RDF star progresses. > RDF Star performance issue with non-concrete node triples > - > > Key: JENA-2107 > URL: https://issues.apache.org/jira/browse/JENA-2107 > Project: Apache Jena > Issue Type: Improvement > Components: ARQ >Affects Versions: Jena 3.17.0, Jena 4.0.0 >Reporter: Lorenz Bühmann >Priority: Critical > Fix For: Jena 4.1.0 > > > the following graph pattern is not evaluated efficiently (results in > full-scan per binding) because the second triple pattern doesn't take > advantage of the bindings generated by evaluation of the first one: > {code:java} > ?s ?o . > << ?s ?o >> ?v . > {code} > A possible fix would be to adapt the method {{rdfStarTripleSub()}} in class > > [SolverRX3.java|https://github.com/apache/jena/blob/2efff8a00b4ffa82751cf46c8a3fed84b6ff3090/jena-arq/src/main/java/org/apache/jena/sparql/engine/main/solver/SolverRX3.java#L63-L71] > by changing the beginning to > {code:java} > private static Iterator rdfStarTripleSub(Binding input, Triple > xPattern, ExecutionContext execCxt) { > Triple tPattern = Substitute.substitute(xPattern, input); > {code} > We went from 75s for a very small dataset (50k triples) to near instant > response times. > If this fix is correct and doesn't break anything, it might be the same way > to fix for its quads counterpart in {{SolverRX4}} class. > > Note, for tdbquery, this seems to be evaluated at a different place? At > least, we couldn't find any performance improvement, it's still horribly slow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (JENA-2107) RDF Star performance issue with non-concrete node triples
[ https://issues.apache.org/jira/browse/JENA-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17346874#comment-17346874 ] Andy Seaborne edited comment on JENA-2107 at 5/18/21, 1:16 PM: --- Indexing: certainly it can be added. I've kept away from changes that change-of-disk-layout. Disk changes are a more permanent commitment. Adding is a data reload; withdrawing/changing the feature is a reload and disruption. The first {{RDF*}} Jena implementation was a bit PG mode and a bit SA mode (PG mode - the triple is always also an asserted triple like annotation syntax). It exploited the existing indexes to look up {{<<...>>}} patterns up. The current {{RDF-star}} compliant is code, no disk-changes. With the fix for this JIRA, use of annotation syntax should be reasonable (the asserted triple will come first) >From the current state (4.1.0 onwards), functionally correct and complete, we >can see what the user-uptake is. One thing to avoid is making a change, then needing to make another change, ... and another ... for a feature that not everyone is going to use. The index setup is currently fixed - we can change it to look for additional indexes and have a tool to adding indexes. The bulk loaders need adjusting - TDB2 bulk loaders do work incrementally and make a difference adding a lot of data (comparable to the size of the existing data - if smaller, little point using them). was (Author: andy.seaborne): Indexing: certainly it can be added. I've kept away from changes that change-of-disk-layout. Disk changes are a more permanent commitment. Adding is a data reload; withdrawing/changing the feature is a reload and disruption. The first {{RDF*}} Jena implementation was a bit PG mode and a bit SA mode (PG mode - the triple is always also an asserted triple like annotation syntax). It exploited the existing indexes to look up {{<<...>>}} patterns up. The current \{{RDF-star} compliant is code, no disk-changes. With the fix for this JIRA, use of annotation syntax should be reasonable (the asserted triple will come first) >From the current state (4.1.0 onwards), functionally correct and complete, we >can see what the user-uptake is. One thing to avoid is making a change, then needing to make another change, ... and another ... for a feature that not everyone is going to use. The index setup is currently fixed - we can change it to look for additional indexes and have a tool to adding indexes. The bulk loaders need adjusting - TDB2 bulk loaders do work incrementally and make a difference adding a lot of data (comparable to the size of the existing data - if smaller, little point using them). > RDF Star performance issue with non-concrete node triples > - > > Key: JENA-2107 > URL: https://issues.apache.org/jira/browse/JENA-2107 > Project: Apache Jena > Issue Type: Improvement > Components: ARQ >Affects Versions: Jena 3.17.0, Jena 4.0.0 >Reporter: Lorenz Bühmann >Priority: Critical > Fix For: Jena 4.1.0 > > > the following graph pattern is not evaluated efficiently (results in > full-scan per binding) because the second triple pattern doesn't take > advantage of the bindings generated by evaluation of the first one: > {code:java} > ?s ?o . > << ?s ?o >> ?v . > {code} > A possible fix would be to adapt the method {{rdfStarTripleSub()}} in class > > [SolverRX3.java|https://github.com/apache/jena/blob/2efff8a00b4ffa82751cf46c8a3fed84b6ff3090/jena-arq/src/main/java/org/apache/jena/sparql/engine/main/solver/SolverRX3.java#L63-L71] > by changing the beginning to > {code:java} > private static Iterator rdfStarTripleSub(Binding input, Triple > xPattern, ExecutionContext execCxt) { > Triple tPattern = Substitute.substitute(xPattern, input); > {code} > We went from 75s for a very small dataset (50k triples) to near instant > response times. > If this fix is correct and doesn't break anything, it might be the same way > to fix for its quads counterpart in {{SolverRX4}} class. > > Note, for tdbquery, this seems to be evaluated at a different place? At > least, we couldn't find any performance improvement, it's still horribly slow. -- This message was sent by Atlassian Jira (v8.3.4#803005)