Re: Jena-Fuseki - slow delete/insert with 30k triples.

2015-10-30 Thread Andy Seaborne 

Hi Radek,

A way to split the action but still have a single transaction would be 
to write it as two SPARQL operations in the one HTTP request:


DELETE{} WHERE {} ; INSERT DATA {  }

Caution: this always executes the INSERT DATA even if the WHERE{} does 
not match.  It's possible to get clever and mark that the insert should 
happen:


# Insert a triple to note INSERT should be done.
# EXISTS part is your WHERE pattern.
INSERT { markertriple } WHERE { FILTER EXISTS { ... } }
# Do the delete.
DELETE{} WHERE {} ;
# INSERT if an donyl if the marker triple is present.
INSERT { 30k triple ) WHERE { markertriple } ;
# Remove work item.
DELETE { markertriple }

What seems to be happening is that the code is not taking advantage of 
the fact that there are lots of ground triples (no variables) in the 
INSERT part and doing the INSERT for every WHERE solution generated as 
if it were a general template.  That makes for a lot of repeated work; 
it loops on the result of WHERE and does the INSERT over and over again.


I've created JENA-1059 to record the optimization.  (Contributions welcome!)

Thanks for the report,

Andy


On 30/10/15 15:53, Radek Hubner wrote:

Hi,

we are using jena-fuseki with TDB store and discovered some weird behaviour
for DELETE/INSERT query.

When we run query(see below) it takes too long or never finishes(we killed
it after 30min). But when we split this query to separate delete and
insert, it finishes in seconds(3-5s). The query is deleting around 30 000
triples and adding new 30 000.

Versions which we tested : jena-fuseki-0.2.6, apache-jena-fuseki-2.3.0. On
both it behaves the same.

Thanks Radek.


Query example :

PREFIX  dri:  
PREFIX  rdfs: 
PREFIX  owl:  
PREFIX  xsd:  
PREFIX  rdf:  
PREFIX  dcterms: 

WITH 
DELETE {
   ?recordList ?property ?value .
}
INSERT {
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
rdf:type dri:Export .
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:username "username"^^xsd:string .
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dcterms:created "2015-10-29T16:28:53.627Z"^^xsd:dateTime .
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportStatus <
http://nationalarchives.gov.uk/dri/catalogue/exportStatus#ReadyToProcess> .
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dcterms:identifier "b83c7512-4c51-467a-9cb7-31cb1ecb6ea0"^^xsd:string .
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/b78bbd71-5c10-4c3e-8572-e10c40ee4aad>
.
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/a2e40103-918b-4c80-9c3e-c3abec53fd03>
.
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/73ddced5-0db2-4c64-b975-d2c2954b8a0b>
.
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/e50cbd3f-ec22-4bb1-8a80-03bd55031874>
.
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/1efb14ce-7027-4d74-9b01-1d21c04ad151>
.

around 30 000 similar lines with different object


<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/5f089799-9fba-4019-9718-5fb618bf7f4d>
.
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/f8e5c82e-3821-450f-b8b3-1319a51b900a>
.
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/c4c93f1b-c5ca-465a-ae35-6f1ef0f3e57f>
.

}
WHERE
   { ?recordList dcterms:identifier
"b83c7512-4c51-467a-9cb7-31cb1ecb6ea0"^^xsd:string .
 ?recordList ?property ?value
   }

Jena-Fuseki - slow delete/insert with 30k triples.

2015-10-30 Thread Radek Hubner 
Hi,

we are using jena-fuseki with TDB store and discovered some weird behaviour
for DELETE/INSERT query.

When we run query(see below) it takes too long or never finishes(we killed
it after 30min). But when we split this query to separate delete and
insert, it finishes in seconds(3-5s). The query is deleting around 30 000
triples and adding new 30 000.

Versions which we tested : jena-fuseki-0.2.6, apache-jena-fuseki-2.3.0. On
both it behaves the same.

Thanks Radek.


Query example :

PREFIX  dri:  
PREFIX  rdfs: 
PREFIX  owl:  
PREFIX  xsd:  
PREFIX  rdf:  
PREFIX  dcterms: 

WITH 
DELETE {
  ?recordList ?property ?value .
}
INSERT {
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
rdf:type dri:Export .
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:username "username"^^xsd:string .
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dcterms:created "2015-10-29T16:28:53.627Z"^^xsd:dateTime .
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportStatus <
http://nationalarchives.gov.uk/dri/catalogue/exportStatus#ReadyToProcess> .
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dcterms:identifier "b83c7512-4c51-467a-9cb7-31cb1ecb6ea0"^^xsd:string .
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/b78bbd71-5c10-4c3e-8572-e10c40ee4aad>
.
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/a2e40103-918b-4c80-9c3e-c3abec53fd03>
.
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/73ddced5-0db2-4c64-b975-d2c2954b8a0b>
.
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/e50cbd3f-ec22-4bb1-8a80-03bd55031874>
.
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/1efb14ce-7027-4d74-9b01-1d21c04ad151>
.

around 30 000 similar lines with different object


<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/5f089799-9fba-4019-9718-5fb618bf7f4d>
.
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/f8e5c82e-3821-450f-b8b3-1319a51b900a>
.
<
http://nationalarchives.gov.uk/dri/catalogue/export/b83c7512-4c51-467a-9cb7-31cb1ecb6ea0>
dri:exportMember <
http://nationalarchives.gov.uk/dri/catalogue/file/c4c93f1b-c5ca-465a-ae35-6f1ef0f3e57f>
.

}
WHERE
  { ?recordList dcterms:identifier
"b83c7512-4c51-467a-9cb7-31cb1ecb6ea0"^^xsd:string .
?recordList ?property ?value
  }

Re: Big data vs Apachi Jena

2015-10-30 Thread Rob Walpole 
Usually when people talk about big data in anything but a very general
sense they are talking about Apache Hadoop and the MapReduce model which is
a way of parallel processing very large data sets. This is a completely
different model to the RDF graph model supported by Apache Jena. That's not
to say you can't process large data sets in parallel using Jena - but that
would be big data in the very general sense.

Rob


On Thu, Oct 29, 2015 at 9:17 PM, kumar rohit  wrote:

> Hello how can big data can be related with Jena or semantic web in general?
>