Hi Pierre,
The NumberOfBuffers parameter in the virtuoso.ini file is too low for a machine
with 16GB memory and loading large datasets like uniprot, please refer to the
following performance tuning guide document for recommended settings for best
performance with various typical memory sizes:
http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtRDFPerformanceTuning
Which for 16GB machine are:
NumberOfBuffers = 1360000
MaxDirtyBuffers = 1000000
You can also run multiple rdf_loader_run() client one per core for parallel
loading of data, with checkpointing turned off for the duration of the load and
then re-enabled with a script of the following form for a 4 core machine:
$more rdf_load.sh
isql 1111 dba dba exec="checkpoint_interval(-1);" &
isql 1111 dba dba exec="rdf_loader_run();" &
isql 1111 dba dba exec="rdf_loader_run();" &
isql 1111 dba dba exec="rdf_loader_run();" &
isql 1111 dba dba exec="rdf_loader_run();" &
wait
isql 1111 dba dba exec="checkpoint;” &
isql 1111 dba dba exec="checkpoint_interval(120);" &
Let me know if this works for you ...
Best Regards
Hugh Williams
Professional Services
OpenLink Software
Web: http://www.openlinksw.com
Support: http://support.openlinksw.com
Forums: http://boards.openlinksw.com/support
Twitter: http://twitter.com/OpenLink
On 10 Sep 2011, at 15:13, Pierre-Yves Chibon wrote:
> Dear all,
>
> I am trying to load uniprot on a quadcore machine with 16Gb of ram and 2
> hard-drives in raid 1.
> I installed virtuoso opensource from the Fedora repository, I therefore
> have the version virtuoso-opensource-6.1.3-1.fc14.x86_64.
> I generated the ~/.odbc.ini file, then I load uniprot by:
>> $ isql-vt virtuoso dba dba
>> Connected to OpenLink Virtuoso
>> Driver: 06.01.3127 OpenLink Virtuoso ODBC Driver
>> OpenLink Interactive SQL (ODBC), version 0.9849b.
>> Type HELP; for help and EXIT; to exit.
>> SQL> ld_dir ('/data/virtuoso/','uniprot.rdf','http://uniprot/');
>>
>> Done. -- 3 msec.
>> SQL> rdf_loader_run ();
>
> So for 9h virtuoso seem stuck. The database file hasn't changed since
> 7h55. I cannot connect anymore to virtuoso using isql. I could not kill
> virtuoso by ctrl+c on the terminal where I started it (the log says
> "Server received signal 2" but the server was never shutdown).
> I killed -9 the pid of virtuoso, restarted it using +foreground again.
> I loged in throught isql and ran:
> SQL> select * from DB.DBA.load_list;
> Output is at: http://fpaste.org/wswj/
> Logs are attached.
>
> It is I think the third time I try to load uniprod and face this
> problem. The max it ran was ~30h during which the memory was never full
> but the cache was very high, the load went sometime a little bit high
> but never enough to get the machine unresponsive.
> Since my first trial I tried to change the configuration, but no
> difference in the loading:
> [database]
> ;MaxCheckpointRemap = 2000
> MaxCheckpointRemap = 20000
>
> [TempDatabase]
> ;MaxCheckpointRemap = 2000
> MaxCheckpointRemap = 20000
>
> [Parameters]
> ;NumberOfBuffers = 5000
> NumberOfBuffers = 15000
> ;MaxDirtyBuffers = 3200
> MaxDirtyBuffers = 7000
>
>
> If you have suggestions or if I can help to debug this :)
>
> Thanks for your help,
> Pierre
> <virtuoso.log>------------------------------------------------------------------------------
> Malware Security Report: Protecting Your Business, Customers, and the
> Bottom Line. Protect your business and customers by understanding the
> threat from malware and how it can impact your online business.
> http://www.accelacomm.com/jaw/sfnl/114/51427462/_______________________________________________
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
