Hi Roland, You should use the Virtuoso RDF Bulik Loader for loading such large datasets as detailed at:
http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtBulkRDFLoader
From the error you report it would appear a checkpoint is being performed in
the middle of your load, so checking should be disabled which is what the
bulkloader does automatically and also turns of transaction logging with
log_enable(3) to prevent the log from running out of memory, which once again
the bulk loader does automatically and is probably the next error you would hit
...
Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc. // http://www.openlinksw.com/
Weblog -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter -- http://twitter.com/OpenLink
Google+ -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers
On 22 Apr 2013, at 13:07, Roland Cornelissen <[email protected]> wrote:
> Hi,
>
> I am trying to load a large dataset (~50GB) and bumped into:
>
> *** Error 40001: [Virtuoso Driver][Virtuoso Server]SR325: Transaction
> aborted due to a database checkpoint or database-wide atomic operation.
> Please retry transaction
> at line 10 of Top-Level:
> ttlp_mt (file_to_string_output ('/usr/viaf/viaf_fixed.nt'), '',
> 'http://viaf.org/', 255)
>
> I have 16GB memory in my box, and set:
> NumberOfBuffers = 680000
> MaxDirtyBuffers = 500000
> and:
> MaxCheckpointRemap = 2000
>
> So i guess the MaxCheckpointRemap is the problem. At what value should I
> set it?
> Here [1] it says 1/4 of the database size in pages, 8K per page.
> So when I calculate with a DB of 50GB that would be 1562500, is that
> correct?
>
> Am I missing something else I should check before kicking of another run?
>
> Thanks,
> Roland
>
> [1] http://www.openlinksw.com/OdbcRails/main/Main/VirtRDFPerformanceTuning
>
>
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Virtuoso-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
smime.p7s
Description: S/MIME cryptographic signature
------------------------------------------------------------------------------ Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis & visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________ Virtuoso-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/virtuoso-users
