Hello there, 

I am currently loading 400GB of yago ontology data into Virtuoso. The data is 
split into 114 rdf/xml files and each is about 3GB. I disable the logging and 
transactions to speed up loading(log_enable(2)) and then I use the bulk load 
functionality of virtuoso. (ld_dir() then rdf_loader_run()). 

This worked well as it took about 2 hrs to load one file and Virtuoso was using 
almost 100% of CPU, suggesting it is busy with the task. But when it began to 
load the 100th file, it slowed dramatically. It was already 16 hours but the 
100th file was still not loaded. Besides, Virtuoso was only using less than 10% 
of CPU. So I restarted the server and resumed the loading process, however it 
was still very slow. 

Well, although it doesn't develop any errors or exceptions but it nows seems to 
take forever to load one file and CPU usage(10%) suggests Virtuoso is not 
actively working on it. I'm afraid the loading will ever finish. And I don't 
understand why it can just slow down dramatically while system resources(CPU, 
Memory, disk) are all still abundant. Am I running into some bottleneck of 
Virtuoso here? 

The server is running on an amazon instance with Ubuntu 12.04.01 LTS, 8GB 
Memory and more than enough disk space. Details about Virtuoso is in the status 
info from isql-v

OpenLink Virtuoso  Server
Version 06.01.3127-pthreads for Linux as of Sep 14 2012 
Started on: 2012/10/26 01:05 GMT+00
 
Database Status:
  File size 576129597440, 70328320 pages, 18725442 free.
  680000 buffers, 679997 used, 119184 dirty 2 wired down, repl age 1890649 0 w. 
io 1 w/crsr.
  Disk Usage: 8161133 reads avg 2 msec, 80% r 6% w last  2033 s, 4768340 writes,
    23476 read ahead, batch = 155.  Autocompact 4114034 in 2626277 out, 36% 
saved.
Gate:  15171 2nd in reads, 0 gate write waits, 0 in while read 0 busy scrap. 
Log = /home/ubuntu/LARGEDATA/virtuoso.trx, 4461 bytes
51482061 pages have been changed since last backup (in checkpoint state)
Current backup timestamp: 0x0000-0x00-0x00
Last backup date: unknown
Clients: 9 connects, max 2 concurrent
RPC: 49 calls, -5 pending, 2 max until now, 0 queued, 18 burst reads (36%), 0 
second brk=5826801664
Checkpoint Remap 0 pages, 0 mapped back. 4471 s atomic time.
    DB master 70328320 total 18725206 free 0 remap 0 mapped back
   temp  256 total 251 free
 
Lock Status: 0 deadlocks of which 0 2r1w, 0 waits,
   Currently 2 threads running 0 threads waiting 0 threads in vdb.
Pending:
  2591533: IER 1 
      0: IER 1 
  70112036: IER 1 
      7: IER 1 
  3174786: IER 1 
      110: IER 1 
  69942572: IER 1 
      388: IER 1 
 
Client 1111:1:  Account: dba, 626 bytes in, 12373 bytes out, 1 stmts.
PID: 29252, OS: unix, Application: unknown, IP#: 127.0.0.1
Transaction status: PENDING, 1 threads.
Locks: 
 
Client 1111:9:  Account: dba, 203 bytes in, 256 bytes out, 1 stmts.
PID: 31667, OS: unix, Application: unknown, IP#: 127.0.0.1
Transaction status: PENDING, 1 threads.
Locks: 
 
 
Running Statements:
 Time (msec) Text
        3401 status()
    38700583 rdf_loader_run()
 
 
Hash indexes
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Virtuoso-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Reply via email to