Thanks Holger, This has become a REALLY cogent point for us, and your great writeup here I think belongs front and center in Topbraid's, and maybe more so in Jena's documentation. For me anyway, it's very easy to get lost in all the different permutations of model views (e.g. base models, and all the myriad kinds of inference models and how each is handled by the myriad kinds of underlying databases) that we work on both within Topbraid an when using the SPIN and Jena API that we quickly lose track of some REALLY important issues related to what's really going on under the hood, espcially as both Topbraid and the Jena API (much to their credit) do their best to abstract these details away from the cognizance of their users. But understanding for example, when seemingly simple changes to code or simple selections on the TBC UI is going to cause possibly huge union graphs to be automatically read completely into memory can be REALLY important.
Anyway, I've got lots of more detailed questions and curiosities about this subject, but more than can probably be answered here or even on the Jena board. Would be great if there were some kind of discussion group at semtech setup (hint hint) hitting on some of these issues and different approaches that have been used to more scalably use the combined power of Jena/SPARQL/SPIN/TOPSPIN and other inference engines in the most scalable fashion possible. Again, this is really more a Jena issue than a Topbraid, but my guess is some of the most knowledgeable and experienced users of Jena reside at Topbraid. On May 28, 6:01 pm, Holger Knublauch <[email protected]> wrote: > Here's another hint: be careful which Graph you are targeting. If you just > press Run Query in TBC's SPARQL view, then the query will execute over the > current graph. This is however a union graph of the base graph + imports + > inferences + system triples. Whenever multiple graphs are involved, the query > engine will not be able to exploit any native optimizations of the database > (Jena: QueryHandler). As a result, complex basic graph patterns will need to > be split into many small SPO requests, and this can have a significant > performance penalty. > > To verify this, check the button in the SPARQL view to run the query on the > Base Model only. If your main file is a TDB then Jena can send the whole > query to the database where it will be handled natively (and likely much > faster than having to merge multiple sub-graphs). > > To enforce this in SPIN and elsewhere, you can fine tune which graphs you are > targeting, using the GRAPH keyword in SPARQL. Use a construct such as > > ... > WHERE { > GRAPH <http://example.org/myMainGraph> { > # All the heavy lifting goes here > } > > } > > to precisely select what graph to operate on. From 3.4 onwards we have > introduced a special named graph called <urn:tb:sessionbase> that will be the > base graph of the current file only. > > If this issue impacts you (or others), then I could think about how to > generalize this in SPIN, to make sure that constraints and rules are only > executed on the base model. > > Viele Gruesse > Holger > > On May 29, 2010, at 5:42 AM, Scott Henninger wrote:> Bad typo that needs > fixing. Let me try that again: > > > Oh, another one is that post-processing of SELECT results can be > > expensive *if* the result set is large. I.e. ORDER BY, GROUP BY, > > aggregates (count, max, min), etc. > > > -- Scott > > > On May 28, 2:36 pm, Scott Henninger <[email protected]> > > wrote: > >> Oh, another one is that post-processing of SELECT results can be > >> expensive is the result set is large. I.e. ORDER BY, GROUP BY, > >> aggregates (count, max, min), etc. > > >> -- Scott > > >> On May 28, 2:12 pm, Christian Fuerber <[email protected]> wrote: > > >>> Scott, thank you so much for your advice! I will try to optimize my > >>> queries, but I'm also eager for more performance tuning hints. > > >>> Cheers, > > >>> Christian > > >>> On 28 Mai, 20:51, Scott Henninger <[email protected]> wrote: > > >>>> Christian; In the end it's really no different than any kind of > >>>> database application. There will always be a point in > >>>> whichperformancesuffers and/or memory becomes scarce. There are also > >>>> tradeoffs for large heap spaces - garbage collection overhead > >>>> increases with the size of memory you are trying collect garbage on. > > >>>> So the first set of advice I'd give is to look carefully at your > >>>> queries. Keep in mind that ?s ?p ?o says "bring everything into > >>>> memory", thus defeating the purpose of using a back-end. Fortunately > >>>> Composer's SPARQL engine, ARQ, tends to ignore these. But the advice > >>>> is still sage - cut down the size of the graph match as soon as > >>>> possible. Note that OPTIONAL is fairly dangerous to start with - it > >>>> states "find this graph pattern or that graph pattern" and thus > >>>> increases the search space. We are working to optimize this better. > > >>>> Turn on Profiling Mode in the SPARQL View's query debugger and find > >>>> the triple pattern placements that result in the fewest match attempts > >>>> - i.e. use it to make sure you prune the search space optimally. It > >>>> will also give you the true ordering of evaluation, which is normally > >>>> top-to-bottom, but ARQ will perform some optimizations. Try turning > >>>> on filter placement and filter early, if that's possible. Use LIMIT > >>>> and OFFSET to experiment with pieces of the results. > > >>>> Avoid string matches as much as possible - FILTERs are often deadly in > >>>> this respect, so make sure you cut down your search space as much as > >>>> possible before doing this kind of string compare. Note this nearly > >>>> conflicts with the earlier statement. Where to place filters depends > >>>> on how computationally expensive the filter expression is and how much > >>>> it prunes the search space for downstream processing. > > >>>> There are probably a few other SPARQL query tips and I'd love to hear > >>>> from others on this. (Self-plug alert: We cover SPARQL > >>>> queryperformanceour Advanced Product training.) > > >>>> Divide-and-conquer is the next step. View your data back-end as a > >>>> huge well and you only want a specific bucketful at a time, being > >>>> careful to leave the rest where it is. I.e. view heap space memory as > >>>> a limited resource. SPARQLMotion is an excellent tool for this task > >>>> as it can be used to get specific sets of triples that can be combined > >>>> for further processing. Remember that ASK queries terminate when a > >>>> match is found, so if you just need to find if the graph pattern > >>>> exists, ASK is much more efficient than SELECT. > > >>>> For SPIN the advice is pretty much the same (Holger can say more). Be > >>>> sure to use ?this because the engine is optimized for that pre-bound > >>>> variable. SPIN will iterate until no new triples are found unless you > >>>> tell the engine otherwise. Turn iterations off if that makes sense > >>>> for your application. If some rules don't have value in forward- > >>>> chaining, set the rule property's spin:rulePropertyMaxIterationCount > >>>> to the maximum needed. > > >>>> Many things to try here and I hope this isn't mistaken as being the > >>>> last word on this topic! > > >>>> -- Scott > > >>>> On May 28, 12:36 pm, Christian Fuerber <[email protected]> wrote: > > >>>>> Hi Holger, > > >>>>> thank you for the information. I finally set up a TDB triplestore with > >>>>> over 53 Mio triples and it shows up perfectly in TBC. But now I'm > >>>>> havingperformanceproblems when running SPARQL queries over the > >>>>> triplestore in TBC's SPARQL view. SPIN constraints also seem to run > >>>>> forever. I'm using the 64bit version of TBC 3.3.1 SE and set the java > >>>>> heapspace limit to 6144m. > > >>>>> Is there any configurations the can speed up TBC's queries and SPIN > >>>>> constraints besides query optimization and java heapspace settings? > > >>>>> Thanks, > >>>>> Christian > > >>>>> On 24 Mai, 03:51, Holger Knublauch <[email protected]> wrote: > > >>>>>> Hi Christian, > > >>>>>> TopBraid's Sesame 2 integration is not as optimized as our other > >>>>>> database interfaces. In practice this means that TBC cannot use any > >>>>>> native optimizations that the database server (in your case Virtuoso) > >>>>>> may provide. Instead, it will break down even the most complex SPARQL > >>>>>> queries into small findSPO queries, which may lead to > >>>>>> significantperformanceproblems. Maybe that's why constraint checking > >>>>>> apparently did not work for you when you can SPIN constraints on > >>>>>> Virtuoso. With smaller databases this impact may not have been > >>>>>> sufficiently severe to notice. But I am glad that you have been able > >>>>>> to confirm that Virtuoso is working well with TBC in principle. > > >>>>>> Over the week end we also had some enlightening examples of SPARQL > >>>>>> queries that were not optimized: we had a query with three OPTIONAL > >>>>>> clauses, leading to a large number of potential combinations. > >>>>>> Replacing those with other SPARQL patterns were leading to many orders > >>>>>> of magnitude speed improvements. Just saying this in case you have > >>>>>> some "dangerous" queries in your constraints. I assume you have > >>>>>> executed the queries individually, e.g. from the SPARQL view, to test > >>>>>> theirperformancebefore putting them into your SPIN constraint library. > > >>>>>> Finally, I discovered aperformanceissue after running the SPIN > >>>>>> constraints from the Problems View: if hundreds or thousands of > >>>>>> violations were found, then just updating those into the Problems view > >>>>>> may freeze the system. I have just fixed this for 3.3.2 (and 3.4). > > >>>>>> Regards, > >>>>>> Holger > > >>>>>> On May 21, 2010, at 10:25 PM, Christian Fuerber wrote: > > >>>>>>> Hi Holger, > > >>>>>>> fortunately i can apply SPIN constraints now on the small graph (245 > >>>>>>> triples) in virtuoso. Although I still do not know what the problem > >>>>>>> was. Maybe. it works now because I reinstalled java. I will also send > >>>>>>> you the error log, in case you are interested. > > >>>>>>> Thanks a lot, > > >>>>>>> Christian > > >>>>>>> On 21 Mai, 00:20, Holger Knublauch <[email protected]> wrote: > >>>>>>>> On May 21, 2010, at 1:08 AM, Christian Fuerber wrote: > > >>>>>>>>> Hi Holger, > > >>>>>>>>> thank you for the quick response. Yes, I could successfully connect > >>>>>>>>> to > >>>>>>>>> a graph in virtuoso that has 245 triples. But the SPIN constraint > >>>>>>>>> checks are not working on the graph's data. I receive an error > >>>>>>>>> "Could > >>>>>>>>> not run checker" when executing "Refresh and show problems > >>>>>>>>> (constraints)". SPARQL in TBC is also not working. I just can see > >>>>>>>>> the > >>>>>>>>> classes and instances in the editor. > > >>>>>>>> Are there any more details available, e.g. the Error Log? > > >>>>>>>> And yes, TDB will almost certainly be faster for SPARQL, because it > >>>>>>>> will "live" in the same JVM, so no communication overhead is needed. > >>>>>>>> Furthermore, TDB is better optimized to work with the ARQ SPARQL > >>>>>>>> engine. > > >>>>>>>> Thanks > >>>>>>>> Holger > > >>>>>>>> -- > >>>>>>>> You received this message because you are subscribed to the Google > >>>>>>>> Group "TopBraid Suite Users", the topics of which include TopBraid > >>>>>>>> Composer, > >>>>>>>> TopBraid Live, TopBraid Ensemble, SPARQLMotion and SPIN. > >>>>>>>> To post to this group, send email to > >>>>>>>> [email protected] > >>>>>>>> To unsubscribe from this group, send email to > >>>>>>>> [email protected] > >>>>>>>> For more options, visit this group > >>>>>>>> athttp://groups.google.com/group/topbraid-users?hl=en > > >>>>>>> -- > >>>>>>> You received this message because you are subscribed to the Google > >>>>>>> Group "TopBraid Suite Users", the topics of which include TopBraid > >>>>>>> Composer, > >>>>>>> TopBraid Live, TopBraid Ensemble, SPARQLMotion and SPIN. > >>>>>>> To post to this group, send email to > >>>>>>> [email protected] > >>>>>>> To unsubscribe from this group, send email to > >>>>>>> [email protected] > >>>>>>> For more options, visit this group at > >>>>>>>http://groups.google.com/group/topbraid-users?hl=en > > >>>>>> -- > >>>>>> You received this message because you are subscribed to the Google > >>>>>> Group "TopBraid Suite Users", the topics of which include TopBraid > >>>>>> Composer, > >>>>>> TopBraid Live, TopBraid Ensemble, SPARQLMotion and SPIN. > >>>>>> To post to this group, send email to > >>>>>> [email protected] > >>>>>> To unsubscribe from this group, send email to > >>>>>> [email protected] > >>>>>> For more options, visit this group > >>>>>> athttp://groups.google.com/group/topbraid-users?hl=en > > > -- > > You received this message because you are subscribed to the Google > > Group "TopBraid Suite Users", the topics of which include TopBraid Composer, > > TopBraid Live, TopBraid Ensemble, SPARQLMotion and SPIN. > > To post to this group, send email to > > [email protected] > > To unsubscribe from this group, send email to > > [email protected] > > For more options, visit this group at > >http://groups.google.com/group/topbraid-users?hl=en -- You received this message because you are subscribed to the Google Group "TopBraid Suite Users", the topics of which include TopBraid Composer, TopBraid Live, TopBraid Ensemble, SPARQLMotion and SPIN. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/topbraid-users?hl=en
