Hi Hugh,
> If all datasets are indicate to have loaded with ll_state = 2 and not errors
> reported in ll_error then the datasets have been loaded successfully.
ok, cool :)
> Did you run the checkpoint command when the bulk load completed as that
> should always be done ? Also after restart is the high CPU consumption still
> occurring ?
yes, i ran "checkpoint; commit work; checkpoint;" a couple of times already,
but after a short while the DB is busy again... the same after a full restart
of the machine.
I attached the log since restart yesterday at the end of this mail (including
some comments)... at start time the instance contained the DBpedia EN + DE +
pagelinks dumps.
I then started to load the Freebase, LinkedGeoData and Wikidata dumps with a
single loader.
Afterwards the endpoint held ~ 5.96 G triples.
> What does the “status();” command run from isql as that will show the status
> of the machine and what might be consuming resources.
Actually funny, cause when i ran that the CPU load instantly seems to have
dropped to 0 and no IO activity anymore, but the command never returned... i'm
waiting for > 10 minutes now:
OpenLink Interactive SQL (Virtuoso), version 0.9849b.
Type HELP; for help and EXIT; to exit.
SQL> status();
Connected to OpenLink Virtuoso
Driver: 07.20.3214 OpenLink Virtuoso ODBC Driver
I opened some other isql-vt sessions but since then the DB doesn't respond at
all.
It also doesn't seem to respond to "/etc/init.d/virtuoso-opensource-7 stop" or
"sudo kill ..." within 5 Minutes...
After that i tried several other kill signals like SIGHUP, SIGINT and i think
SIGUSR1 finally did the trick.
Starting took awefully long and seems to have replayed a lot of transactions
which i don't fully understand as there were several checkpoints since the
import...
Should i rather use the backup from before?
After restart I ran status(); twice, once immediately after replaying the
transactions when the DB was idle and a second time after i waited for ~5
minutes and the DB became active again:
> SQL> status();
> REPORT
> VARCHAR
> _______________________________________________________________________________
>
> OpenLink Virtuoso Server
> Version 07.20.3214-pthreads for Linux as of Nov 11 2015
> Started on: 2015-11-14 18:21 GMT+1
>
> Database Status:
> File size 0, 42128384 pages, 15652474 free.
> 10900000 buffers, 3644262 used, 0 dirty 0 wired down, repl age 0 0 w. io 0
> w/crsr.
> Disk Usage: 3613440 reads avg 0 msec, 0% r 0% w last 171 s, 1113064 writes
> flush 44.42 MB,
> 23803 read ahead, batch = 150. Autocompact 60532 in 50797 out, 16% saved.
> Gate: 1112 2nd in reads, 0 gate write waits, 0 in while read 0 busy scrap.
> Log = /var/lib/virtuoso-opensource-7/db/virtuoso.trx, 185 bytes
> 25512792 pages have been changed since last backup (in checkpoint state)
> Current backup timestamp: 0x0000-0x00-0x00
> Last backup date: unknown
> Clients: 1 connects, max 1 concurrent
> RPC: 6 calls, 1 pending, 1 max until now, 0 queued, 0 burst reads (0%), 0
> second 0M large, 10M max
> Checkpoint Remap 961058 pages, 0 mapped back. 17 s atomic time.
> DB master 42128384 total 15652474 free 961058 remap 0 mapped back
> temp 256 total 251 free
>
> Lock Status: 0 deadlocks of which 0 2r1w, 0 waits,
> Currently 1 threads running 0 threads waiting 0 threads in vdb.
> Pending:
>
> Client 1111:1: Account: dba, 316 bytes in, 2514 bytes out, 1 stmts.
> PID: 14099, OS: unix, Application: unknown, IP#: 127.0.0.1
> Transaction status: PENDING, 1 threads.
> Locks:
>
>
> Running Statements:
> Time (msec) Text
> 965 status()
>
>
> Hash indexes
>
>
> 37 Rows. -- 966 msec.
> SQL> status();
> REPORT
> VARCHAR
> _______________________________________________________________________________
>
> OpenLink Virtuoso Server
> Version 07.20.3214-pthreads for Linux as of Nov 11 2015
> Started on: 2015-11-14 18:21 GMT+1
>
> Database Status:
> File size 0, 42128384 pages, 15632400 free.
> 10900000 buffers, 3662255 used, 98085 dirty 0 wired down, repl age 0 0 w.
> io 1 w/crsr.
> Disk Usage: 3630269 reads avg 0 msec, 0% r 0% w last 395 s, 1113064 writes
> flush 44.42 MB,
> 23871 read ahead, batch = 150. Autocompact 60532 in 50797 out, 16% saved.
> Gate: 1117 2nd in reads, 0 gate write waits, 0 in while read 0 busy scrap.
> Log = /var/lib/virtuoso-opensource-7/db/virtuoso.trx, 10804859 bytes
> 25512792 pages have been changed since last backup (in checkpoint state)
> Current backup timestamp: 0x0000-0x00-0x00
> Last backup date: unknown
> Clients: 1 connects, max 1 concurrent
> RPC: 8 calls, 1 pending, 1 max until now, 0 queued, 0 burst reads (0%), 0
> second 2M large, 10M max
> Checkpoint Remap 961058 pages, 0 mapped back. 17 s atomic time.
> DB master 42128384 total 15632156 free 961058 remap 78945 mapped back
> temp 256 total 250 free
>
> Lock Status: 0 deadlocks of which 0 2r1w, 0 waits,
> Currently 6 threads running 0 threads waiting 1 threads in vdb.
> Pending:
> 23651033: IEP INTERNAL
> 23651034: IEP INTERNAL
> 23651035: IEP INTERNAL
> 23651036: IEP INTERNAL
> 42042624: IER INTERNAL
> 2: IER INTERNAL
> 6477568: IER INTERNAL
> 7: IER INTERNAL
> 18726912: IER INTERNAL
> 6: IER INTERNAL
> 39680512: IER INTERNAL
> 5: IER INTERNAL
> 40677888: IER INTERNAL
> 7: IER INTERNAL
> 9608449: IER INTERNAL
> 2: IER INTERNAL
> 37114369: IER INTERNAL
> 31: IER INTERNAL
> 9414402: IER INTERNAL
> 5: IER INTERNAL
> 41539331: IER INTERNAL
> 22: IER INTERNAL
> 42045956: IER INTERNAL
> 1: IER INTERNAL
> 22138629: IER INTERNAL
> 1: IER INTERNAL
> 39779077: IER INTERNAL
> 111: IER INTERNAL
> 17050374: IER INTERNAL
> 64: IER INTERNAL
> 40975366: IER INTERNAL
> 3: IER INTERNAL
> 27659014: IER INTERNAL
> 3: IER INTERNAL
> 41537030: IER INTERNAL
> 1: IER INTERNAL
> 42026248: IER INTERNAL
> 6: IER INTERNAL
> 41119498: IER INTERNAL
> 59: IER INTERNAL
> 29654794: IER INTERNAL
> 2: IER INTERNAL
> 34012171: IER INTERNAL
> 44: IER INTERNAL
> 13249549: IER INTERNAL
> 7: IER INTERNAL
> 38488589: IER INTERNAL
> 143: IER INTERNAL
> 41327118: IER INTERNAL
> 2: IER INTERNAL
> 40350478: IER INTERNAL
> 1: IER INTERNAL
> 39860494: IER INTERNAL
> 1: IER INTERNAL
> 26208271: IER INTERNAL
> 34: IER INTERNAL
> 40758799: IER INTERNAL
> 4: IER INTERNAL
> 39684368: IER INTERNAL
> 1: IER INTERNAL
> 38884624: IER INTERNAL
> 42: IER INTERNAL
> 30790672: IER INTERNAL
> 7: IER INTERNAL
> 18191377: IER INTERNAL
> 1: IER INTERNAL
> 41903377: IER INTERNAL
> 4: IER INTERNAL
> 41976082: IER INTERNAL
> 3: IER INTERNAL
> 40412434: IER INTERNAL
> 59: IER INTERNAL
> 39782675: IER INTERNAL
> 32: IER INTERNAL
> 8653587: IER INTERNAL
> 50: IER INTERNAL
> 41036309: IER INTERNAL
> 5: IER INTERNAL
> 15897109: IER INTERNAL
> 1: IER INTERNAL
> 10198551: IER INTERNAL
> 3: IER INTERNAL
> 41861144: IER INTERNAL
> 5: IER INTERNAL
> 38884633: IER INTERNAL
> 2: IER INTERNAL
> 5145625: IER INTERNAL
> 6: IER INTERNAL
> 16842267: IER INTERNAL
> 2: IER INTERNAL
> 12426523: IER INTERNAL
> 16: IER INTERNAL
> 40581403: IER INTERNAL
> 7: IER INTERNAL
> 18730268: IER INTERNAL
> 0: IER INTERNAL
> 41535773: IER INTERNAL
> 6: IER INTERNAL
> 22143261: IER INTERNAL
> 6: IER INTERNAL
> 26186781: IER INTERNAL
> 55: IER INTERNAL
> 35521309: IER INTERNAL
> 4: IER INTERNAL
> 40855837: IER INTERNAL
> 6: IER INTERNAL
> 41696798: IER INTERNAL
> 3: IER INTERNAL
> 12269086: IER INTERNAL
> 1: IER INTERNAL
> 29420831: IER INTERNAL
> 5: IER INTERNAL
> 39851039: IER INTERNAL
> 1: IER INTERNAL
> 41617696: IER INTERNAL
> 1: IER INTERNAL
> 20957984: IER INTERNAL
> 68: IER INTERNAL
> 41071906: IER INTERNAL
> 3: IER INTERNAL
> 39759394: IER INTERNAL
> 1: IER INTERNAL
> 18859810: IER INTERNAL
> 6: IER INTERNAL
> 16030499: IER INTERNAL
> 5: IER INTERNAL
> 40331555: IER INTERNAL
> 0: IER INTERNAL
> 39684132: IER INTERNAL
> 7: IER INTERNAL
> 6535204: IER INTERNAL
> 1: IER INTERNAL
> 40140836: IER INTERNAL
> 6: IER INTERNAL
> 40406565: IER INTERNAL
> 4: IER INTERNAL
> 6539557: IER INTERNAL
> 1: IER INTERNAL
> 39728421: IER INTERNAL
> 4: IER INTERNAL
> 40672549: IER INTERNAL
> 2: IER INTERNAL
> 38895142: IER INTERNAL
> 0: IER INTERNAL
> 40127784: IER INTERNAL
> 0: IER INTERNAL
> 40787752: IER INTERNAL
> 3: IER INTERNAL
> 21316136: IER INTERNAL
> 25: IER INTERNAL
> 21587240: IER INTERNAL
> 0: IER INTERNAL
> 16325416: IER INTERNAL
> 6: IER INTERNAL
> 40391976: IER INTERNAL
> 3: IER INTERNAL
> 40394537: IER INTERNAL
> 5: IER INTERNAL
> 33378089: IER INTERNAL
> 3: IER INTERNAL
> 40406572: IER INTERNAL
> 1: IER INTERNAL
> 33735468: IER INTERNAL
> 80: IER INTERNAL
> 37783340: IER INTERNAL
> 91: IER INTERNAL
> 24148780: IER INTERNAL
> 32: IER INTERNAL
> 40948013: IER INTERNAL
> 3: IER INTERNAL
> 40703789: IER INTERNAL
> 2: IER INTERNAL
> 39903278: IER INTERNAL
> 6: IER INTERNAL
> 40953646: IER INTERNAL
> 0: IER INTERNAL
> 38432302: IER INTERNAL
> 81: IER INTERNAL
> 82: IER INTERNAL
> 17033774: IER INTERNAL
> 1: IER INTERNAL
> 42123313: IER INTERNAL
> 5: IER INTERNAL
> 6: IER INTERNAL
> 20043825: IER INTERNAL
> 89: IER INTERNAL
> 29833523: IER INTERNAL
> 8: IER INTERNAL
> 21235765: IER INTERNAL
> 84: IER INTERNAL
> 40824373: IER INTERNAL
> 4: IER INTERNAL
> 41086518: IER INTERNAL
> 114: IER INTERNAL
> 39687222: IER INTERNAL
> 2: IER INTERNAL
> 41665335: IER INTERNAL
> 4: IER INTERNAL
> 40381496: IER INTERNAL
> 5: IER INTERNAL
> 26141241: IER INTERNAL
> 6: IER INTERNAL
> 39896635: IER INTERNAL
> 5: IER INTERNAL
> 6005564: IER INTERNAL
> 2: IER INTERNAL
> 39855676: IER INTERNAL
> 30: IER INTERNAL
> 8692540: IER INTERNAL
> 97: IER INTERNAL
> 17671230: IER INTERNAL
> 1: IER INTERNAL
> 5490239: IER INTERNAL
> 6: IER INTERNAL
> 8730688: IER INTERNAL
> 7: IER INTERNAL
> 28570689: IER INTERNAL
> 2: IER INTERNAL
> 41011266: IER INTERNAL
> 2: IER INTERNAL
> 23719747: IER INTERNAL
> 136: IER INTERNAL
> 14549315: IER INTERNAL
> 71: IER INTERNAL
> 42030917: IER INTERNAL
> 1: IER INTERNAL
> 10255685: IER INTERNAL
> 43: IER INTERNAL
> 42088773: IER INTERNAL
> 6: IER INTERNAL
> 20578886: IER INTERNAL
> 60: IER INTERNAL
> 26500422: IER INTERNAL
> 80: IER INTERNAL
> 16842310: IER INTERNAL
> 3: IER INTERNAL
> 20877638: IER INTERNAL
> 104: IER INTERNAL
> 40022342: IER INTERNAL
> 1: IER INTERNAL
> 16539206: IER INTERNAL
> 0: IER INTERNAL
> 39665223: IER INTERNAL
> 3: IER INTERNAL
> 40837449: IER INTERNAL
> 4: IER INTERNAL
> 31640394: IER INTERNAL
> 6: IER INTERNAL
> 30736458: IER INTERNAL
> 2: IER INTERNAL
> 41953355: IER INTERNAL
> 2: IER INTERNAL
> 41036876: IER INTERNAL
> 3: IER INTERNAL
> 24701516: IER INTERNAL
> 3: IER INTERNAL
> 17068879: IER INTERNAL
> 35: IER INTERNAL
> 40139601: IER INTERNAL
> 6: IER INTERNAL
> 42050641: IER INTERNAL
> 1: IER INTERNAL
> 42024530: IER INTERNAL
> 5: IER INTERNAL
> 25497683: IER INTERNAL
> 6: IER INTERNAL
> 24104787: IER INTERNAL
> 0: IER INTERNAL
> 41534547: IER INTERNAL
> 4: IER INTERNAL
> 40479063: IER INTERNAL
> 3: IER INTERNAL
> 17091416: IER INTERNAL
> 3: IER INTERNAL
> 18043994: IER INTERNAL
> 7: IER INTERNAL
> 30303322: IER INTERNAL
> 4: IER INTERNAL
> 28498779: IER INTERNAL
> 107: IER INTERNAL
> 41827931: IER INTERNAL
> 5: IER INTERNAL
> 26429019: IER INTERNAL
> 43: IER INTERNAL
> 39891549: IER INTERNAL
> 4: IER INTERNAL
> 41999965: IER INTERNAL
> 6: IER INTERNAL
> 8612446: IER INTERNAL
> 7: IER INTERNAL
> 6627168: IER INTERNAL
> 1: IER INTERNAL
> 39826277: IER INTERNAL
> 4: IER INTERNAL
> 41925735: IER INTERNAL
> 3: IER INTERNAL
> 27721319: IER INTERNAL
> 30: IER INTERNAL
> 31495527: IER INTERNAL
> 21: IER INTERNAL
> 18528616: IER INTERNAL
> 146: IER INTERNAL
> 41951080: IER INTERNAL
> 2: IER INTERNAL
> 39874664: IER INTERNAL
> 4: IER INTERNAL
> 29474666: IER INTERNAL
> 3: IER INTERNAL
> 15564650: IER INTERNAL
> 58: IER INTERNAL
> 12644203: IER INTERNAL
> 82: IER INTERNAL
> 42061676: IER INTERNAL
> 7: IER INTERNAL
> 40269421: IER INTERNAL
> 2: IER INTERNAL
> 6543726: IER INTERNAL
> 6: IER INTERNAL
> 2918255: IER INTERNAL
> 3: IER INTERNAL
> 26469488: IER INTERNAL
> 6: IER INTERNAL
> 40201585: IER INTERNAL
> 1: IER INTERNAL
> 41937522: IER INTERNAL
> 2: IER INTERNAL
> 6412915: IER INTERNAL
> 6: IER INTERNAL
> 33574771: IER INTERNAL
> 31: IER INTERNAL
> 38438260: IER INTERNAL
> 6: IER INTERNAL
> 37342324: IER INTERNAL
> 56: IER INTERNAL
> 8202101: IER INTERNAL
> 10: IER INTERNAL
> 28107132: IER INTERNAL
> 2: IER INTERNAL
> 41852028: IER INTERNAL
> 2: IER INTERNAL
> 31411325: IER INTERNAL
> 0: IER INTERNAL
> 28595581: IER INTERNAL
> 3: IER INTERNAL
> 42002814: IER INTERNAL
> 6: IER INTERNAL
> 42090879: IER INTERNAL
> 4: IER INTERNAL
> 37372288: IER INTERNAL
> 1: IER INTERNAL
> 12944512: IER INTERNAL
> 0: IER INTERNAL
> 40927105: IER INTERNAL
> 45: IER INTERNAL
> 26135425: IER INTERNAL
> 6: IER INTERNAL
> 4007042: IER INTERNAL
> 2: IER INTERNAL
> 8293762: IER NO OWNER
> 39763586: IER NO OWNER
> 39689346: IER INTERNAL
> 0: IER INTERNAL
> 27496836: IER INTERNAL
> 131: IER INTERNAL
> 35286152: IER INTERNAL
> 4: IER INTERNAL
> 11079306: IER INTERNAL
> 58: IER INTERNAL
> 6509707: IER INTERNAL
> 36: IER INTERNAL
> 42045068: IER INTERNAL
> 4: IER INTERNAL
> 17598860: IER INTERNAL
> 73: IER INTERNAL
> 27438734: IER INTERNAL
> 4: IER INTERNAL
> 26393743: IER INTERNAL
> 4: IER INTERNAL
> 21066896: IER INTERNAL
> 4: IER INTERNAL
> 21546129: IER INTERNAL
> 5: IER INTERNAL
> 41877138: IER INTERNAL
> 2: IER INTERNAL
> 32945812: IER INTERNAL
> 7: IER INTERNAL
> 25764757: IER INTERNAL
> 7: IER INTERNAL
> 6499222: IER INTERNAL
> 1: IER INTERNAL
> 16903318: IER INTERNAL
> 7: IER INTERNAL
> 41639064: IER INTERNAL
> 3: IER INTERNAL
> 40272794: IER INTERNAL
> 18: IER INTERNAL
> 33117340: IER INTERNAL
> 4: IER INTERNAL
> 8553116: IER INTERNAL
> 7: IER INTERNAL
> 23755932: IER INTERNAL
> 106: IER INTERNAL
> 30666397: IER INTERNAL
> 45: IER INTERNAL
> 28754334: IER INTERNAL
> 6: IER INTERNAL
> 22137503: IER INTERNAL
> 0: IER INTERNAL
> 1: IER INTERNAL
> 41489823: IER INTERNAL
> 5: IER INTERNAL
> 8933535: IER INTERNAL
> 5: IER INTERNAL
> 12664992: IER INTERNAL
> 48: IER INTERNAL
> 28046242: IER INTERNAL
> 64: IER INTERNAL
> 10748580: IER INTERNAL
> 29: IER INTERNAL
> 15565989: IER INTERNAL
> 1: IER INTERNAL
> 10751909: IER INTERNAL
> 16: IER INTERNAL
> 41136037: IER INTERNAL
> 6: IER INTERNAL
> 38577317: IER INTERNAL
> 6: IER INTERNAL
> 7: IER INTERNAL
> 5270183: IER INTERNAL
> 27: IER INTERNAL
> 24007593: IER INTERNAL
> 32: IER INTERNAL
> 12948395: IER INTERNAL
> 38: IER INTERNAL
> 41775020: IER INTERNAL
> 3: IER INTERNAL
> 23741869: IER INTERNAL
> 3: IER INTERNAL
> 41589421: IER INTERNAL
> 3: IER INTERNAL
> 38118831: IER INTERNAL
> 6: IER INTERNAL
> 38022831: IER INTERNAL
> 4: IER INTERNAL
> 41253551: IER INTERNAL
> 1: IER INTERNAL
> 14504623: IER INTERNAL
> 29: IER INTERNAL
> 16101040: IER INTERNAL
> 3: IER INTERNAL
> 33398193: IER INTERNAL
> 88: IER INTERNAL
> 89: IER INTERNAL
> 41133745: IER INTERNAL
> 6: IER INTERNAL
> 29420978: IER INTERNAL
> 66: IER INTERNAL
> 29386931: IER INTERNAL
> 2: IER INTERNAL
> 5189299: IER INTERNAL
> 4: IER INTERNAL
> 42091701: IER INTERNAL
> 3: IER INTERNAL
> 28825525: IER INTERNAL
> 3: IER INTERNAL
> 14906551: IER INTERNAL
> 3: IER INTERNAL
> 27380407: IER INTERNAL
> 89: IER INTERNAL
> 15719352: IER INTERNAL
> 3: IER INTERNAL
> 26073016: IER INTERNAL
> 102: IER INTERNAL
> 23861176: IER INTERNAL
> 1: IER INTERNAL
> 33113273: IER INTERNAL
> 6: IER INTERNAL
> 15204282: IER INTERNAL
> 2: IER INTERNAL
> 30134971: IER INTERNAL
> 1: IER INTERNAL
> 27763131: IER INTERNAL
> 2: IER INTERNAL
> 22576059: IER INTERNAL
> 36: IER INTERNAL
> 5582525: IER INTERNAL
> 1: IER INTERNAL
> 40227264: IER INTERNAL
> 3: IER INTERNAL
> 41456576: IER INTERNAL
> 6: IER INTERNAL
> 15722433: IER INTERNAL
> 80: IER INTERNAL
> 3672513: IER INTERNAL
> 13: IER INTERNAL
> 18126017: IER INTERNAL
> 2: IER INTERNAL
> 6493634: IER INTERNAL
> 6: IER INTERNAL
> 41456578: IER INTERNAL
> 5: IER INTERNAL
> 18474947: IER INTERNAL
> 4: IER INTERNAL
> 27533509: IER INTERNAL
> 3: IER INTERNAL
> 8841671: IER INTERNAL
> 2: IER INTERNAL
> 18859976: IER INTERNAL
> 2: IER INTERNAL
> 33121483: IER INTERNAL
> 92: IER INTERNAL
> 42095051: IER INTERNAL
> 3: IER INTERNAL
> 8468684: IER INTERNAL
> 4: IER INTERNAL
> 41879247: IER INTERNAL
> 6: IER INTERNAL
> 29157329: IER INTERNAL
> 49: IER INTERNAL
> 14339027: IER INTERNAL
> 7: IER INTERNAL
> 40562132: IER INTERNAL
> 58: IER INTERNAL
> 6607833: IER INTERNAL
> 5: IER INTERNAL
> 26198233: IER INTERNAL
> 101: IER INTERNAL
> 18824153: IER INTERNAL
> 6: IER INTERNAL
> 42083545: IER INTERNAL
> 3: IER INTERNAL
> 1511130: IER INTERNAL
> 1: IER INTERNAL
> 34099163: IER INTERNAL
> 4: IER INTERNAL
> 24007644: IER INTERNAL
> 23: IER INTERNAL
> 40089308: IER INTERNAL
> 27: IER INTERNAL
> 27229661: IER INTERNAL
> 6: IER INTERNAL
> 8639710: IER INTERNAL
> 4: IER INTERNAL
> 42100191: IER INTERNAL
> 1: IER INTERNAL
> 14747615: IER INTERNAL
> 6: IER INTERNAL
> 11154400: IER INTERNAL
> 15: IER INTERNAL
> 19888609: IER INTERNAL
> 33: IER INTERNAL
> 6701538: IER INTERNAL
> 1: IER INTERNAL
> 28059874: IER INTERNAL
> 1: IER INTERNAL
> 6539235: IER INTERNAL
> 4: IER INTERNAL
> 31681508: IER INTERNAL
> 1: IER INTERNAL
> 9065958: IER INTERNAL
> 66: IER INTERNAL
> 8842982: IER NO OWNER
> 11082471: IER INTERNAL
> 4: IER INTERNAL
> 39863528: IER INTERNAL
> 5: IER INTERNAL
> 23705576: IER NO OWNER
> 28390380: IER INTERNAL
> 3: IER INTERNAL
> 6519276: IER INTERNAL
> 1: IER INTERNAL
> 30456813: IER INTERNAL
> 3: IER INTERNAL
> 15332335: IER INTERNAL
> 68: IER INTERNAL
> 71: IER INTERNAL
> 20057327: IER INTERNAL
> 128: IER INTERNAL
> 30049008: IER INTERNAL
> 6: IER INTERNAL
> 39854066: IER INTERNAL
> 4: IER INTERNAL
> 27411958: IER INTERNAL
> 2: IER INTERNAL
> 20656631: IER INTERNAL
> 2: IER INTERNAL
> 29470711: IER INTERNAL
> 0: IER INTERNAL
> 37372410: IER INTERNAL
> 6: IER INTERNAL
> 15274747: IER INTERNAL
> 3: IER INTERNAL
> 10309372: IER INTERNAL
> 6: IER INTERNAL
> 12648958: IER INTERNAL
> 22: IER INTERNAL
> 26213886: IER INTERNAL
> 0: IER INTERNAL
>
> Client 1111:1: Account: dba, 429 bytes in, 4745 bytes out, 1 stmts.
> PID: 14099, OS: unix, Application: unknown, IP#: 127.0.0.1
> Transaction status: PENDING, 1 threads.
> Locks:
>
>
> Running Statements:
> Time (msec) Text
> 1170 status()
>
>
> Hash indexes
>
>
> 635 Rows. -- 1178 msec.
Here some values from my virtuoso.ini file:
> ;; values for 128 GB RAM
> MaxCheckpointRemap = 1250000
> NumberOfBuffers = 10900000
> MaxDirtyBuffers = 8000000
The log:
> Fri Nov 13 2015
> 12:53:55 { Loading plugin 1: Type `plain', file `wikiv' in
> `/usr/lib/virtuoso-opensource-7/hosting'
> 12:53:55 WikiV version 0.6 from OpenLink Software
> 12:53:55 Support functions for WikiV collaboration tool
> 12:53:55 SUCCESS plugin 1: loaded from
> /usr/lib/virtuoso-opensource-7/hosting/wikiv.so }
> 12:53:55 { Loading plugin 2: Type `plain', file `mediawiki' in
> `/usr/lib/virtuoso-opensource-7/hosting'
> 12:53:55 MediaWiki version 0.1 from OpenLink Software
> 12:53:55 Support functions for MediaWiki collaboration tool
> 12:53:55 SUCCESS plugin 2: loaded from
> /usr/lib/virtuoso-opensource-7/hosting/mediawiki.so }
> 12:53:55 { Loading plugin 3: Type `plain', file `creolewiki' in
> `/usr/lib/virtuoso-opensource-7/hosting'
> 12:53:55 CreoleWiki version 0.1 from OpenLink Software
> 12:53:55 Support functions for CreoleWiki collaboration tool
> 12:53:55 SUCCESS plugin 3: loaded from
> /usr/lib/virtuoso-opensource-7/hosting/creolewiki.so }
> 12:53:55 { Loading plugin 4: Type `plain', file `im' in
> `/usr/lib/virtuoso-opensource-7/hosting'
> 12:53:55 FAILED plugin 4: Unable to locate file }
> 12:53:55 OpenLink Virtuoso Universal Server
> 12:53:55 Version 07.20.3214-pthreads for Linux as of Nov 11 2015
> 12:53:55 uses parts of OpenSSL, PCRE, Html Tidy
> 12:54:17 Database version 3126
> 12:54:17 SQL Optimizer enabled (max 1000 layouts)
> 12:54:18 Compiler unit is timed at 0.000167 msec
> 12:54:29 built-in procedure "DB.DBA.RDF_PROXY_GET_HTTP_HOST" overruled by the
> RDBMS
> 12:54:29 built-in procedure "DB.DBA.RDF_SPONGE_IRI_SCH" overruled by the RDBMS
> 12:54:29 built-in procedure "DB.DBA.RDF_PROXY_ENTITY_IRI" overruled by the
> RDBMS
> 12:54:29 Roll forward started
> 12:54:29 Roll forward complete
> 12:54:32 Checkpoint started
> 12:54:51 Checkpoint removed 2004 MB of remapped pages, leaving 0 MB. Duration
> 18.78 s. To save this time, increase MaxCheckpointRemap and/or set
> Unremap quota to 0 in ini file.
> 12:55:15 Checkpoint finished, log reused
i always terminate loading with "checkpoint; commit work; checkpoint;", but
does the above somehow indicate that some cleanup of the loading of the DBpedia
dump was still pending?
> 12:55:15 HTTP/WebDAV server online at 8890
> 12:55:15 Server online at 1111 (pid 15532)
> 12:58:21 PL LOG: Loader started
> 15:00:51 * Monitor: CPU% is low while there are large numbers of runnable
> threads
> 18:06:28 The __sys_ext_map free pages incorrect 270 != 266 actually free
> 19:29:03 Write wait on column page 12272446. Waits should be on the index
> leaf page, except when col page is held for read by background write
> 19:33:30 * Monitor: CPU% is low while there are large numbers of runnable
> threads
> 20:48:31 * Monitor: CPU% is low while there are large numbers of runnable
> threads
>
> Sat Nov 14 2015
> 00:56:51 * Monitor: CPU% is low while there are large numbers of runnable
> threads
> 02:25:45 * Monitor: CPU% is low while there are large numbers of runnable
> threads
> 02:31:33 * Monitor: CPU% is low while there are large numbers of runnable
> threads
> 04:12:12 PL LOG: No more files to load. Loader has finished,
loading ~ 5 B triples in 15 hours, not bad ;) but again those log messages
worry me...
> 04:29:53 Checkpoint started
> 04:32:14 Checkpoint finished, log reused
> 04:32:47 Write load very high relative to disk write throughput. Flushing at
> 0 MB/s while application is making dirty pages at 4.2 MB/s. To
> checkpoint the database, will now pause the workload with 135 MB unflushed.
> 04:32:47 cpt/lt unusual cond rltrx.c:1437
> 04:32:47 cpt/lt unusual cond rltrx.c:1437
> 04:32:47 cpt/lt unusual cond rltrx.c:1437
i guess this is worse than a usual cond?
> 04:32:58 Checkpoint started
> 04:33:12 Checkpoint finished, log reused
> 04:33:33 Checkpoint started
> 04:33:44 Checkpoint finished, log reused
> 15:01:58 The __sys_ext_map free pages incorrect 270 != 140 actually free
something being incorrect can't be good either?
> 15:17:37 Checkpoint started
> 15:18:15 Checkpoint finished, log reused
> 15:18:43 Checkpoint started
> 15:18:58 Checkpoint finished, log reused
> 15:19:15 Checkpoint started
> 15:19:28 Checkpoint finished, log reused
> 15:19:58 Checkpoint started
> 15:20:09 Checkpoint finished, log reused
> 15:20:09 Server shutdown complete
>
> Sat Nov 14 2015
> 17:08:54 { Loading plugin 1: Type `plain', file `wikiv' in
> `/usr/lib/virtuoso-opensource-7/hosting'
> 17:08:54 WikiV version 0.6 from OpenLink Software
> 17:08:54 Support functions for WikiV collaboration tool
> 17:08:54 SUCCESS plugin 1: loaded from
> /usr/lib/virtuoso-opensource-7/hosting/wikiv.so }
> 17:08:54 { Loading plugin 2: Type `plain', file `mediawiki' in
> `/usr/lib/virtuoso-opensource-7/hosting'
> 17:08:54 MediaWiki version 0.1 from OpenLink Software
> 17:08:54 Support functions for MediaWiki collaboration tool
> 17:08:54 SUCCESS plugin 2: loaded from
> /usr/lib/virtuoso-opensource-7/hosting/mediawiki.so }
> 17:08:54 { Loading plugin 3: Type `plain', file `creolewiki' in
> `/usr/lib/virtuoso-opensource-7/hosting'
> 17:08:54 CreoleWiki version 0.1 from OpenLink Software
> 17:08:54 Support functions for CreoleWiki collaboration tool
> 17:08:54 SUCCESS plugin 3: loaded from
> /usr/lib/virtuoso-opensource-7/hosting/creolewiki.so }
> 17:08:54 { Loading plugin 4: Type `plain', file `im' in
> `/usr/lib/virtuoso-opensource-7/hosting'
> 17:08:54 FAILED plugin 4: Unable to locate file }
> 17:08:54 OpenLink Virtuoso Universal Server
> 17:08:54 Version 07.20.3214-pthreads for Linux as of Nov 11 2015
> 17:08:54 uses parts of OpenSSL, PCRE, Html Tidy
> 17:09:15 Database version 3126
> 17:09:16 SQL Optimizer enabled (max 1000 layouts)
> 17:09:17 Compiler unit is timed at 0.000175 msec
> 17:09:37 built-in procedure "DB.DBA.RDF_PROXY_GET_HTTP_HOST" overruled by the
> RDBMS
> 17:09:37 built-in procedure "DB.DBA.RDF_SPONGE_IRI_SCH" overruled by the RDBMS
> 17:09:37 built-in procedure "DB.DBA.RDF_PROXY_ENTITY_IRI" overruled by the
> RDBMS
> 17:09:38 Roll forward started
> 17:09:38 Roll forward complete
> 17:09:45 Checkpoint started
> 17:12:13 Checkpoint removed 7710 MB of remapped pages, leaving 0 MB. Duration
> 148.1 s. To save this time, increase MaxCheckpointRemap and/or set
> Unremap quota to 0 in ini file.
> 17:14:32 Checkpoint finished, log reused
again looks like some pending cleanup to me?
> 17:14:32 HTTP/WebDAV server online at 8890
> 17:14:33 Server online at 1111 (pid 2361)
> 18:06:10 Server received signal 2
> 18:07:29 Server received signal 15
> 18:08:27 Server received signal 1
>
> Sat Nov 14 2015
> 18:10:11 { Loading plugin 1: Type `plain', file `wikiv' in
> `/usr/lib/virtuoso-opensource-7/hosting'
> 18:10:11 WikiV version 0.6 from OpenLink Software
> 18:10:11 Support functions for WikiV collaboration tool
> 18:10:11 SUCCESS plugin 1: loaded from
> /usr/lib/virtuoso-opensource-7/hosting/wikiv.so }
> 18:10:11 { Loading plugin 2: Type `plain', file `mediawiki' in
> `/usr/lib/virtuoso-opensource-7/hosting'
> 18:10:11 MediaWiki version 0.1 from OpenLink Software
> 18:10:11 Support functions for MediaWiki collaboration tool
> 18:10:11 SUCCESS plugin 2: loaded from
> /usr/lib/virtuoso-opensource-7/hosting/mediawiki.so }
> 18:10:11 { Loading plugin 3: Type `plain', file `creolewiki' in
> `/usr/lib/virtuoso-opensource-7/hosting'
> 18:10:11 CreoleWiki version 0.1 from OpenLink Software
> 18:10:11 Support functions for CreoleWiki collaboration tool
> 18:10:11 SUCCESS plugin 3: loaded from
> /usr/lib/virtuoso-opensource-7/hosting/creolewiki.so }
> 18:10:11 { Loading plugin 4: Type `plain', file `im' in
> `/usr/lib/virtuoso-opensource-7/hosting'
> 18:10:11 FAILED plugin 4: Unable to locate file }
> 18:10:11 OpenLink Virtuoso Universal Server
> 18:10:11 Version 07.20.3214-pthreads for Linux as of Nov 11 2015
> 18:10:11 uses parts of OpenSSL, PCRE, Html Tidy
> 18:10:31 Database version 3126
> 18:10:32 SQL Optimizer enabled (max 1000 layouts)
> 18:10:33 Compiler unit is timed at 0.000170 msec
> 18:10:50 built-in procedure "DB.DBA.RDF_PROXY_GET_HTTP_HOST" overruled by the
> RDBMS
> 18:10:50 built-in procedure "DB.DBA.RDF_SPONGE_IRI_SCH" overruled by the RDBMS
> 18:10:50 built-in procedure "DB.DBA.RDF_PROXY_ENTITY_IRI" overruled by the
> RDBMS
> 18:10:50 Roll forward started
> 18:11:30 1000 transactions, 10239522 bytes replayed (2 %)
> 18:11:42 2000 transactions, 20296653 bytes replayed (4 %)
> 18:11:55 3000 transactions, 31179656 bytes replayed (6 %)
> 18:12:08 4000 transactions, 42019368 bytes replayed (8 %)
> 18:12:20 5000 transactions, 52612361 bytes replayed (10 %)
> 18:12:35 6000 transactions, 63142542 bytes replayed (12 %)
> 18:12:50 7000 transactions, 75266530 bytes replayed (15 %)
> 18:13:05 8000 transactions, 87221607 bytes replayed (17 %)
> 18:13:19 9000 transactions, 99026948 bytes replayed (20 %)
> 18:13:34 10000 transactions, 110393114 bytes replayed (22 %)
> 18:13:49 11000 transactions, 122245685 bytes replayed (24 %)
> 18:14:03 12000 transactions, 134188797 bytes replayed (27 %)
> 18:14:14 13000 transactions, 143472415 bytes replayed (29 %)
> 18:14:27 14000 transactions, 153949738 bytes replayed (31 %)
> 18:14:40 15000 transactions, 164727147 bytes replayed (33 %)
> 18:14:52 16000 transactions, 174723990 bytes replayed (35 %)
> 18:15:05 17000 transactions, 185699126 bytes replayed (37 %)
> 18:15:17 18000 transactions, 195343857 bytes replayed (39 %)
> 18:15:29 19000 transactions, 205297856 bytes replayed (41 %)
> 18:15:43 20000 transactions, 216900273 bytes replayed (44 %)
> 18:15:55 21000 transactions, 226721919 bytes replayed (46 %)
> 18:16:05 22000 transactions, 236472449 bytes replayed (48 %)
> 18:16:17 23000 transactions, 247365727 bytes replayed (50 %)
> 18:16:28 24000 transactions, 258184502 bytes replayed (52 %)
> 18:16:39 25000 transactions, 268717346 bytes replayed (54 %)
> 18:16:50 26000 transactions, 278737300 bytes replayed (56 %)
> 18:17:01 27000 transactions, 289454191 bytes replayed (58 %)
> 18:17:13 28000 transactions, 300148793 bytes replayed (60 %)
> 18:17:24 29000 transactions, 310627196 bytes replayed (63 %)
> 18:17:38 30000 transactions, 321418989 bytes replayed (65 %)
> 18:17:52 31000 transactions, 333264469 bytes replayed (67 %)
> 18:18:06 32000 transactions, 345197780 bytes replayed (70 %)
> 18:18:20 33000 transactions, 357019492 bytes replayed (72 %)
> 18:18:33 34000 transactions, 368099094 bytes replayed (74 %)
> 18:18:47 35000 transactions, 379862372 bytes replayed (77 %)
> 18:19:01 36000 transactions, 391750868 bytes replayed (79 %)
> 18:19:12 37000 transactions, 401260257 bytes replayed (81 %)
> 18:19:25 38000 transactions, 411422813 bytes replayed (83 %)
> 18:19:37 39000 transactions, 422202740 bytes replayed (85 %)
> 18:19:50 40000 transactions, 433077692 bytes replayed (87 %)
> 18:20:02 41000 transactions, 443763143 bytes replayed (90 %)
> 18:20:14 42000 transactions, 453022280 bytes replayed (91 %)
> 18:20:25 43000 transactions, 463022271 bytes replayed (94 %)
> 18:20:39 44000 transactions, 474772427 bytes replayed (96 %)
> 18:20:50 45000 transactions, 484674230 bytes replayed (98 %)
> 18:20:58 45827 transactions, 492428160 bytes replayed (100 %)
> 18:20:58 Roll forward complete
> 18:24:27 Checkpoint started
> 18:24:38 Checkpoint finished, log reused
> 18:24:39 HTTP/WebDAV server online at 8890
> 18:24:39 Server online at 1111 (pid 12270)
Best,
Jörn
------------------------------------------------------------------------------
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
