I’m confused about two of your points here. Let me separate them out so we can discuss them easily.
1) "writes are not supported”: Writes are certainly supported in the Graph/DatasetGraph SPI. Graph::add and ::delete, DatasetGraph::add, ::delete, ::deleteAny… after all, Graph and DatasetGraph are the basic abstractions implemented by Jena’s own out-of-the-box implementations of RDF storage. Can you explain what you mean by this? 2) "methods which call find(ANY, ANY, ANY) play havoc with an on demand triple caching algorithm”: The subtypes of TupleTable with which you are working have exactly the same kinds of find() methods. Why are they not problematic in that context? --- A. Soroka The University of Virginia Library > On Mar 3, 2016, at 5:47 AM, Joint <dandh...@gmail.com> wrote: > > > > Hi Andy. > I implemented the entire SPI at the DatasetGraph and Graph level. It got to > the point where I had overridden more methods than not. In addition writes > are not supported and contains methods which call find(ANY, ANY, ANY) play > havoc with an on demand triple caching algorithm! ;-) I'm using the TriTable > because it fits and quads are spoofed via triple to quad iterator. > I have a set of filters and handles which the find triple is compared against > and either passed straight to the TriTable if the triple has been handled > before or its passed to the appropriate handle which adds the triples to the > TriTable then calls the find. As the underlying data is a tree a cache depth > can be set which allows related triples to be cached. Also the cache can be > preloaded with common triples e.g. ANY RDF:type ?. > Would you consider a generic version for the Jena code base? > > > Dick > > -------- Original message -------- > From: Andy Seaborne <a...@apache.org> > Date: 18/02/2016 6:31 pm (GMT+00:00) > To: users@jena.apache.org > Subject: Re: SPI DatasetGraph creating Triples/Quads on demand using > DatasetGraphInMemory > > Hi, > > I'm not seeing how tapping into the implementation of > DatasetGraphInMemory is going to help (through the details > > As well as the DatasetGraphMap approach, one other thought that occurred > to me is to have a dataset (DatasetGraphWrapper) over any DatasetGraph > implementation. > > It loads, and clears, the mapped graph on-demand, and passes the find() > call through to the now-setup data. > > Andy > > On 16/02/16 17:42, A. Soroka wrote: >>> Based on your description the DatasetGraphInMemory would seem to match the >>> dynamic load requirement. How did you foresee it being loaded? Is there a >>> large over head to using the add methods? >> >> No, I certainly did not mean to give that impression, and I don’t think it >> is entirely accurate. DSGInMemory was definitely not at all meant for >> dynamic loading. That doesn’t mean it can’t be used that way, but that was >> not in the design, which assumed that all tuples take about the same amount >> of time to access and that all of the same type are coming from the same >> implementation (in a QuadTable and a TripleTable). >> >> The overhead of mutating a dataset is mostly inside the implementations of >> TupleTable that are actually used to store tuples. You should be aware that >> TupleTable extends TransactionalComponent, so if you want to use it to >> create some kind of connection to your storage, you will need to make that >> connection fully transactional. That doesn’t sound at all trivial in your >> case. >> >> At this point it seems to me that extending DatasetGraphMap (and >> implementing GraphMaker and Graph instead of TupleTable) might be a more >> appropriate design for your work. You can put dynamic loading behavior in >> Graph (or a GraphView subtype) just as easily as in TupleTable subtypes. Are >> there reasons around the use of transactionality in your work that demand >> the particular semantics supported by DSGInMemory? >> >> --- >> A. Soroka >> The University of Virginia Library >> >>> On Feb 13, 2016, at 5:18 AM, Joint <dandh...@gmail.com> wrote: >>> >>> >>> >>> Hi. >>> The quick full scenario is a distributed DaaS which supports queries, >>> updates, transforms and bulkloads. Andy Seaborne knows some of the detail >>> because I spoke to him previously. We achieve multiple writes by having >>> parallel Datasets, both traditional TDB and on demand in memory. Writes are >>> sent to a free dataset, free being not in a write transaction. That's a >>> simplistic overview... >>> Queries are handled by a dataset proxy which builds a dynamic dataset based >>> on the graph URIs. For example the graph URI urn:Iungo:all causes the proxy >>> find method to issue the query to all known Datasets and return the union >>> of results. Various dataset proxies exist, some load TDBs, others load TTL >>> files into graphs, others dynamically create tuples. The common thing being >>> they are all presented as Datasets backed by DatasetGraph. Thus a SPARQL >>> query can result in multiple Datasets being loaded to satisfy the query. >>> Nodes can be preloaded which then load Datasets to satisfy finds. This way >>> the system can be scaled to handle increased work loads. Also specific >>> nodes can be targeted to specific hardware. >>> When a graph URI is encountered the proxy can interpret it's structure. So >>> urn:Iungo:sdai/foo/bar would cause the SDAI model bar in the SDAI >>> repository foo to be dynamically loaded into memory along with the quads >>> which are required to satisfy the find. >>> Typically a group of people will be working on a set of data so the first >>> to query will load the dataset then it will be accessed multiple times. >>> There will be an initial dynamic load of data which will tail off with some >>> additional loading over time. >>> Based on your description the DatasetGraphInMemory would seem to match the >>> dynamic load requirement. How did you foresee it being loaded? Is there a >>> large over head to using the add methods? >>> A typical scenario would be to search all SDAI repository's for some key >>> information then load detailed information in some, continuing to drill >>> down. >>> Hope this helps. >>> I'm going to extend the hex and tri tables and run some tests. I've already >>> shimed the DGTriplesQuads so the actual caching code already exists and >>> should bed easy to hook on. >>> Dick >>> >>> -------- Original message -------- >>> From: "A. Soroka" <aj...@virginia.edu> >>> Date: 12/02/2016 11:07 pm (GMT+00:00) >>> To: users@jena.apache.org >>> Subject: Re: SPI DatasetGraph creating Triples/Quads on demand using >>> DatasetGraphInMemory >>> >>> Okay, I’m more confident at this point that you’re not well served by >>> DatasetGraphInMemory, which has very strong assumptions about the speedy >>> reachability of data. DSGInMemory was built for situations when all of the >>> data is in core memory and multithreaded access is important. If you have a >>> lot of core memory and can load the data fully, you might want to use it, >>> but that doesn’t sound at all like your case. Otherwise, as far as what the >>> right extension point is, I will need to defer to committers or more >>> experienced devs, but I think you may need to look at DatasetGraph from a >>> more close-to-the-metal point. TDB extends DatasetGraphTriplesQuads >>> directly, for example. >>> >>> Can you tell us a bit more about your full scenario? I don’t know much >>> about STEP (sorry if others do)— is there a canonical RDF formulation? What >>> kinds of queries are you going to be using with this data? How quickly are >>> users going to need to switch contexts between datasets? >>> >>> --- >>> A. Soroka >>> The University of Virginia Library >>> >>>> On Feb 12, 2016, at 2:44 PM, Joint <dandh...@gmail.com> wrote: >>>> >>>> >>>> >>>> Thanks for the fast response! >>>> I have a set of disk based binary SDAI repository's which are based on >>>> ISO10303 parts 11/21/25/27 otherwise known as the EXPRESS/STEP/SDAI parts. >>>> In particular my files are IFC2x3 files which can be +1Gb. However after >>>> processing into a SDAI binary I typically see a size reduction e.g. 1.4Gb >>>> STEP file becomes a 1Gb SDAI repository. If I convert the STEP file into >>>> TDB I get +100M quads and a 50Gb folder. Multiplied by 1000's of similar >>>> sized STEP files... >>>> Typically only a small subset of the STEP file needs to be queried but >>>> sometimes other parts need to be queried. Hence the on demand caching and >>>> DatasetGraphInMemory. The aim is that in the find methods I check a cache >>>> and call the native SDAI find methods based on the node URI's in the case >>>> of a cache miss, calling the add methods for the minted tuples, then >>>> passing on the call to the super find. The underlying SDAI repository's >>>> are static so once a subject is cached no other work is required. >>>> As the DatasetGraphInMemory is commented as very fast quad and triple >>>> access it seemed a logical place to extend. The shim cache would be set to >>>> expire entries and limit the total number of tuples power repository. This >>>> is currently deployed on a 256Gb ram device. >>>> In the bigger picture l have a service very similar to Fuseki which allows >>>> SPARQL requests to be made against Datasets which are either TDB or SDAI >>>> cache backed. >>>> What was DatasetGraphInMemory created for..? ;-) >>>> Dick >>>> >>>> -------- Original message -------- >>>> From: "A. Soroka" <aj...@virginia.edu> >>>> Date: 12/02/2016 6:21 pm (GMT+00:00) >>>> To: users@jena.apache.org >>>> Subject: Re: SPI DatasetGraph creating Triples/Quads on demand using >>>> DatasetGraphInMemory >>>> >>>> I wrote the DatasetGraphInMemory code, but I suspect your question may be >>>> better answered by other folks who are more familiar with Jena's >>>> DatasetGraph implementations, or may actually not have anything to do with >>>> DatasetGraph (see below for why). I will try to give some background >>>> information, though. >>>> >>>> There are several paths by which where DatasetGraphInMemory can be >>>> performing finds, but they come down to two places in the code, >>>> QuadTable:: and TripleTable::find and in default operation, the concrete >>>> forms: >>>> >>>> https://github.com/apache/jena/blob/master/jena-arq/src/main/java/org/apache/jena/sparql/core/mem/PMapQuadTable.java#L100 >>>> >>>> for Quads and >>>> >>>> https://github.com/apache/jena/blob/master/jena-arq/src/main/java/org/apache/jena/sparql/core/mem/PMapTripleTable.java#L99 >>>> >>>> for Triples. Those methods are reused by all the differently-ordered >>>> indexes within Hex- or TriTable, each of which will answer a find by >>>> selecting an appropriately-ordered index based on the fixed and variable >>>> slots in the find pattern and using the concrete methods above to stream >>>> tuples back. >>>> >>>> As to why you are seeing your methods called in some places and not in >>>> others, DatasetGraphBaseFind features methods like findInDftGraph(), >>>> findInSpecificNamedGraph(), findInAnyNamedGraphs() etc. and that these are >>>> the methods that DatasetGraphInMemory is implementing. DSGInMemory does >>>> not make a selection between those methods— that is done by >>>> DatasetGraphBaseFind. So that is where you will find the logic that should >>>> answer your question. >>>> >>>> Can you say a little more about your use case? You seem to have some >>>> efficient representation in memory of your data (I hope it is in-memory— >>>> otherwise it is a very bad choice to subclass DSGInMemory) and you want to >>>> create tuples on the fly as queries are received. That is really not at >>>> all what DSGInMemory is for (DSGInMemory is using map structures for >>>> indexing and in default mode, uses persistent data structures to support >>>> transactionality). I am wondering whether you might not be much better >>>> served by tapping into Jena at a different place, perhaps implementing the >>>> Graph SPI directly. Or, if reusing DSGInMemory is the right choice, just >>>> implementing Quad- and TripleTable and using the constructor >>>> DatasetGraphInMemory(final QuadTable i, final TripleTable t). >>>> >>>> --- >>>> A. Soroka >>>> The University of Virginia Library >>>> >>>>> On Feb 12, 2016, at 12:58 PM, Dick Murray <dandh...@gmail.com> wrote: >>>>> >>>>> Hi. >>>>> >>>>> Does anyone know the "find" paths through DatasetGraphInMemory please? >>>>> >>>>> For example if I extend DatasetGraphInMemory and override >>>>> DatasetGraphBaseFind.find(node, Node, Node, Node) it breakpoints on >>>>> "select >>>>> * where {?s ?p ?o}" however if I override the other >>>>> DatasetGraphBaseFind.find(...) methods, "select * where {graph ?g {?s ?p >>>>> ?o}}" does not trigger a breakpoint i.e. I don't know what method it's >>>>> calling (but as I type I'm guessing it's optimised to return the HexTable >>>>> nodes...). >>>>> >>>>> Would I be better off overriding HexTable and TriTable classes find >>>>> methods >>>>> when I create the DatasetGraphInMemory? Are all finds guaranteed to end in >>>>> one of these methods? >>>>> >>>>> I need to know the root find methods so that I can shim them to create >>>>> triples/quads before they perform the find. >>>>> >>>>> I need to create Triples/Quads on demand (because a bulk load would create >>>>> ~100M triples but only ~1000 are ever queried) and the source binary form >>>>> is more efficient (binary ~1GB native tree versus TDB ~50GB ~100M quads) >>>>> than quads. >>>>> >>>>> Regards Dick Murray. >>>> >>> >> >
