Hello Maxime, It's strange that it did not signal "40005 SR325 Transaction aborted because it's log after image size went above the limit". I've checked the generated SQL code for these inserts, it never caches more than 8000 triples in memory. It frequently flushes temp dictionaries as it is supposed to be for such a simple case (what's inserted does not affect what's being selected, so insert can run in parallel with selection). No big intermediate dictionaries --- no reasons to run out of memory :|
Could you please send the whole source code for the case? The "cheating" workaround it using "define sql:log-enable 3", as described in http://docs.openlinksw.com/virtuoso/rdfsparql.html#rdfsparqlimplementatioptragmas but 1) this may be not what you wish and 2) server should not run out of memory in any case. Best Regards, Ivan Mikhailov OpenLink Software http://virtuoso.openlinksw.com On Fri, 2011-08-05 at 17:13 +0200, Maxime Teissèdre wrote: > Hi, > I have some problem with the "insert" request. > A large insert query uses the RAM to 100% then the SWAP and finally > kill the process. > > > So i run a test who consist to insert 1.000, then 1.000.000, and then > all the graph in a new graph. > I run the test on 4 graphes. 202042 triplets, 1918433 triplets, > 13608512 triplets and 38496582 triplets. > > The procedure: > for graph in graphe1 graphe2 graphe3 graphe4 > do > replace <graphName> by $graph > exec the request > sleep 120 (to view the change in monitoring) > done > > > The 3 requests: > *request1: > SPARQL insert into <http://testgraph> { ?x ?p ?y } > where { graph <graphName> { ?x ?p ?y } } limit 1000; > > *request2: > SPARQL insert into <http://testgraph> { ?x ?p ?y } > where { graph <graphName> { ?x ?p ?y } } limit 1000000; > > *request3: > SPARQL insert into <http://testgraph> { ?x ?p ?y } > where { graph <graphName> { ?x ?p ?y } }; > > > The config server: > AMD Opteron(tm) Processor 252 > 8GO RAM > OpenLink Virtuoso Server Version 06.01.3127-pthreads for Linux as > of Aug 3 2011 > > The part of virtuoso.ini conf: > ; Server parameters > ; > [Parameters] > ServerPort = 1111 > LiteMode = 0 > DisableUnixSocket = 1 > DisableTcpSocket = 0 > ServerThreads = 5 > CheckpointInterval = 720 > O_DIRECT = 0 > NumberOfBuffers = 100000 > MaxDirtyBuffers = 75000 > CaseMode = 2 > MaxStaticCursorRows = 5000 > CheckpointAuditTrail = 1 > AllowOSCalls = 0 > SchedulerInterval = 10 > DirsAllowed = ., /usr/share/virtuoso-opensource-6.1/vad > ThreadCleanupInterval = 0 > ThreadThreshold = 10 > ResourcesCleanupInterval = 0 > FreeTextBatchSize = 100000 > SingleCPU = 0 > VADInstallDir = /usr/share/virtuoso-opensource-6.1/vad/ > PrefixResultNames = 0 > RdfFreeTextRulesSize = 100 > IndexTreeMaps = 256 > TransactionAfterImageLimit = 1500000000 > > > We have test with the configuration for our RAM system and we have > modify the buffers parameters but the system SWAP > and crash with a insert or delete request. > > > I enclose the charts of the system during the test. > > ######################################## > Fri Aug 5 10:04:57 CEST 2011 > delete http://testgraph (who is empty) > 1 Rows. -- 361669 msec. (SPARQL CLEAR GRAPH) > Done. -- 13 msec. (delete from DB.DBA.load_list where ll_graph = ) > Done. -- 0 msec. (commit WORK;) > ## INSERT ## > Fri Aug 5 10:10:59 CEST 2011 > Count triplet tetsgraph: 0 > Database Status: > File size 20900216832, 2551296 pages, 2089603 free. > 100000 buffers, 100000 used, 1982 dirty 0 wired down, repl age > 583779 0 w. io 0 w/crsr. > Disk Usage: 302560 reads avg 0 msec, 41% r 0% w last 406 s, 5718 > writes, > 719 read ahead, batch = 224. Autocompact 0 in 0 out, 0% saved. > Fri Aug 5 10:11:43 CEST 2011 > graphe1 : insert_01 : 1 Rows. -- 2136 msec. > graphe2 : insert_01 : 1 Rows. -- 114 msec. > graphe3 : insert_01 : 1 Rows. -- 100 msec. > graphe4 : insert_01 : 1 Rows. -- 207 msec. > Fri Aug 5 10:19:46 CEST 2011 > Count triplet tetsgraph: 4000 (OK) > Database Status: > File size 20900216832, 2551296 pages, 2089415 free. > 100000 buffers, 99723 used, 2169 dirty 0 wired down, repl age 578155 > 0 w. io 0 w/crsr. > Disk Usage: 305187 reads avg 0 msec, 0% r 0% w last 485 s, 5719 > writes, > 743 read ahead, batch = 220. Autocompact 1556 in 837 out, 46% > saved. > Fri Aug 5 10:19:49 CEST 2011 > graphe1 : insert_02 : 1 Rows. -- 16411 msec. > graphe2 : insert_02 : 1 Rows. -- 47227 msec. > graphe3 : insert_02 : 1 Rows. -- 76016 msec. > graphe4 : insert_02 : 1 Rows. -- 56598 msec. > Fri Aug 5 10:31:13 CEST 2011 > Count triplet tetsgraph: 3202042 (OK) > Database Status: > File size 20900216832, 2551296 pages, 2051391 free. > 100000 buffers, 100000 used, 12309 dirty 2 wired down, repl age > 693723 1841 w. io 0 w/crsr. > Disk Usage: 434339 reads avg 0 msec, 0% r 0% w last 688 s, 35627 > writes, > 1893 read ahead, batch = 143. Autocompact 20749 in 14724 out, 29% > saved. > Fri Aug 5 10:31:18 CEST 2011 > graphe1 : insert_03 : 1 Rows. -- 7842 msec. > graphe2 : insert_03 : 1 Rows. -- 96683 msec. > graphe3 : insert_03 : 1 Rows. -- 1097460 msec. > > *** Error 08S01: [Virtuoso Driver]CL065: Lost connection to server > in lines 1-2 of Top-Level: > #line 1 "(console)" > SPARQL insert into <tetsgraph> { ?x ?p ?y } > where { graph <http://graphe4> { ?x ?p ?y } } > graphe4 : insert_03 : > > Fri Aug 5 12:12:26 CEST 2011 > ######################################## > > This problem is reproducible. > > Thanks. > Maxime T. > ------------------------------------------------------------------------------ > BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA > The must-attend event for mobile developers. Connect with experts. > Get tools for creating Super Apps. See the latest technologies. > Sessions, hands-on labs, demos & much more. Register early & save! > http://p.sf.net/sfu/rim-blackberry-1 > _______________________________________________ Virtuoso-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/virtuoso-devel
