Re: [MASSMAIL]Re: Large *.dat files in Fuseki

2022-11-22 Thread Andy Seaborne
Hi Gaspar, (this is Jena 4.6.1?) Not something I recall seeing before. Are new Data-000_ directories being created? What's in the log file about backups? Backups are serialized to one at a time per dataset. Andy On 22/11/2022 14:18, Bartalus Gáspár wrote: Hi Andy, We’ve just started t

Re: [MASSMAIL]Re: Large *.dat files in Fuseki

2022-11-22 Thread Bartalus Gáspár
Hi Andy, We’ve just started to run the compaction on our database, but we are encountering that compaction doesn’t always complete. Any ideas what could cause this behaviour? We’re executing the http POST request /$/compact/database_name?deleteOld=true and subsequently checking the tasks with /

Re: [MASSMAIL]Re: Large *.dat files in Fuseki

2022-07-14 Thread Bartalus Gáspár
Hi Andy & Lorenz, Thanks for the clarification and support. Best regards, Gaspar > On 14 Jul 2022, at 19:36, Andy Seaborne wrote: > > > > On 07/07/2022 16:19, Lorenz Buehmann wrote: >> I think we should wait for Andy here with further input as he's the persons >> who basically designed and

Re: [MASSMAIL]Re: Large *.dat files in Fuseki

2022-07-14 Thread Andy Seaborne
On 07/07/2022 16:19, Lorenz Buehmann wrote: I think we should wait for Andy here with further input as he's the persons who basically designed and implemented all the fancy stuff and knows better advice for sure. @Andy Did you read the whole discussion and can you verify that it's expected

Re: Re: [MASSMAIL]Re: Large *.dat files in Fuseki

2022-07-07 Thread Lorenz Buehmann
I think we should wait for Andy here with further input as he's the persons who basically designed and implemented all the fancy stuff and knows better advice for sure. @Andy Did you read the whole discussion and can you verify that it's expected behavior that lot's of daily updates lead to su

Re: [MASSMAIL]Re: Large *.dat files in Fuseki

2022-07-07 Thread Bartalus Gáspár
Hi Lorenz, Would you recommend using tdb1 instead of tdb2 for our use case? What would be the differences? We are using fuseki 4.5.0 btw. Gaspar > On 6 Jul 2022, at 14:39, Bartalus Gáspár > wrote: > > Hi, > > Most of the updates are DELETE/INSERT queries, i.e > > DELETE {?s ?p ?oldValue} >

Re: Re: [MASSMAIL]Re: Large *.dat files in Fuseki

2022-07-06 Thread Lorenz Buehmann
Hi, you should open another thread where we can discuss your question, please don't mix up threads - makes me confused Also, did you check SPARQL 1.1 Update W3C documents? They are online and have lots of examples On 06.07.22 13:50, Dương Hồ wrote: DELETE {?s ?p ?oldValue} INSERT {?s ?p ?n

Re: [MASSMAIL]Re: Large *.dat files in Fuseki

2022-07-06 Thread Bartalus Gáspár
This is a generic shape. A real-world example would be: DELETE {?subject rdfs:label ?oldLabel} INSERT {?subject rdfs:label ?newLabel} WHERE { ?subject rdf:type SomeType. ?subject rdfs:label ?oldLabel. FILTER(?oldLabel IN (“oldLabel1”, “oldLabel2”, “oldLabel3)) BIND(CONCAT(?oldLabel, “_upda

Re: [MASSMAIL]Re: Large *.dat files in Fuseki

2022-07-06 Thread Dương Hồ
DELETE {?s ?p ?oldValue} INSERT {?s ?p ?newValue} WHERE { OPTIONAL {?s ?p ?oldValue} #derive ?newValue from somewhere } If i want update 3 triples how to use this formats? Can you help me? Vào 18:39, Th 4, 6 thg 7, 2022 Bartalus Gáspár đã viết: > Hi, > > Most of the updates are DELETE/INSER

Re: [MASSMAIL]Re: Large *.dat files in Fuseki

2022-07-06 Thread Bartalus Gáspár
Hi, Most of the updates are DELETE/INSERT queries, i.e DELETE {?s ?p ?oldValue} INSERT {?s ?p ?newValue} WHERE { OPTIONAL {?s ?p ?oldValue} #derive ?newValue from somewhere } We also have some separate DELETE queries and INSERT queries. I’ve tried HTTP POST /$/compact/db_name and as a resul

Re: Re: Re: [MASSMAIL]Re: Large *.dat files in Fuseki

2022-07-06 Thread Lorenz Buehmann
You can trigger compaction from CLI via tdb2.tdbcompact (needs Fuseki being down I think) or with Fuseki running as POST request: https://jena.apache.org/documentation/fuseki2/fuseki-server-protocol.html#datasets-and-services On 06.07.22 11:52, Lorenz Buehmann wrote: Ok, interesting so we ha

Re: Re: [MASSMAIL]Re: Large *.dat files in Fuseki

2022-07-06 Thread Lorenz Buehmann
Ok, interesting so we have - 150k triples, rather small dataset - loaded into 10MB node table files - 10 updates every 5s - which makes up to 24 * 60 * 60 / 5 * 10 ~ 200k updates per day - and leads to 10GB node table files Can you share the shape of those update queries? After doing a

Re: [MASSMAIL]Re: Large *.dat files in Fuseki

2022-07-06 Thread Bartalus Gáspár
The 3 dat files that are growing significantly are SPO.dat, OSP.dat and POS.dat ordered by size. > On 6 Jul 2022, at 11:36, Bartalus Gáspár > wrote: > > Hi Lorenz, > > Thanks for quick feedback and clarification on lucene indexes. > > Here are my answers to your questions: > - We are uploadi

Re: [MASSMAIL]Re: Large *.dat files in Fuseki

2022-07-06 Thread Bartalus Gáspár
Hi Lorenz, Thanks for quick feedback and clarification on lucene indexes. Here are my answers to your questions: - We are uploading 7 ttl files to our dataset, where 1 is larger 6Mb, the others are below 200Kb. - The overall number of triples after data upload is ~15. - We have around 10 SP