...sorry, sent the INDEX information tab instead of "SPACE" info tab :
Table Index name Rows Pages Row pages Blob pages Size
DB.DBA.RDF_OBJ DB.DBA.RDF_OBJ 30443064 255445 255445 0 1833.25
MB
DB.DBA.RDF_IRI DB.DBA.RDF_IRI 59063426 206412 206412 0 1247.19
MB
DB.DBA.RDF_IRI DB_DBA_RDF_IRI_UNQC_RI_ID 59063426 183541 183541 0
1282.95 MB
DB.DBA.RDF_OBJ_RO_FLAGS_WORDS DB.DBA.RDF_OBJ_RO_FLAGS_WORDS 8341023
179875 179875 0 1332.22 MB
DB.DBA.RDF_OBJ RO_VAL 30443064 117863 117863 0 773.81 MB
DB.DBA.RO_START DB.DBA.RO_START 30443064 85319 85319
0 475.31 MB
Le 20/03/2017 à 11:34, Thomas Michaux a écrit :
Le 19/03/2017 à 16:15, Hugh Williams a écrit :
Hi Thomas,
Hi,
Is the loading of the dataset now complete or it is still in progress as you
opening statement is not clear ?
You should not need 40GB RAM for inserting and hosting 240 million triples,
which should require less then 10GB depending on how well they can be
compressed for storage in the database.
loading is complete, we finished at 243 188 427 triples , hosting
now requires 25GB ram, 15Gb disk, details :
void:triples 243188427 ;
void:classes 13 ;
void:entities 58523487 ;
void:distinctSubjects 58523514 ;
void:properties 32 ;
void:distinctObjects 73171603 .
Total pages 1925120
Free pages 607377
Buffers 2720000
Buffers used 244554
Dirty buffers 3
Wired down buffers 0
Table Index name Touches Reads Read %
DB.DBA.RDF_QUAD RDF_QUAD 1562356553 36371 0
DB.DBA.RDF_QUAD RDF_QUAD_POGS 609423455 16989 0
DB.DBA.RDF_QUAD RDF_QUAD_SP 378769255 35822 0
DB.DBA.RDF_QUAD RDF_QUAD_GS 340377017 1634 0
I assume you have set the swappiness as suggested previously ?
yes, done, $ sysctl vm.swappiness
vm.swappiness = 10
When you recompiled your Virtuoso was this done from the git stable/7 or
develop/7 branch , as I latter has a number of memory consumption fixes that
would not be in stable/7, thus I would suggest building from develop/7.
will investigate.
The two main problems we encountered while loading were :
- logs messages indicating "Flushing at 5.7 MB/s while application is
making dirty pages at 1.7 MB/s." which we interpreted as not enough
write speed while receiving lots of JDBC INSERTs (disk issue ? buffer
issue ? ...)
- high memory consumption (40GB RAM), virtuoso process never releasing
memory while loading, free RAM always going down...
Have you provided a copy of your INI file previously, if not can you provide a
copy ?
see attached (FYI QueryLog= was not active while loading)
Do ensure the following params are set to 1 in order to clean up unused
threads/resources and reduce memory consumption of the Virtuoso server, which
can otherwise be construed as memory leaks.:
ThreadCleanupInterval = 1
ResourcesCleanupInterval = 1
we have theses settings right.
Thanks for your help,
Thomas
if needed we model ORCID 2016 dataset using :
c1 c2
http://xmlns.com/foaf/0.1/Person
28021451
http://purl.org/ontology/bibo/Document
14283692
http://purl.org/ontology/bibo/Journal
9104659
http://xmlns.com/foaf/0.1/PersonalProfileDocument
2527333
http://xmlns.com/foaf/0.1/Article
974945
http://www.w3.org/ns/org#Membership
807465
http://www.w3.org/2006/vcard/ns#Address
807423
http://www.w3.org/ns/org#Organization
807418
http://purl.org/ontology/bibo/Conference
769451
http://www.w3.org/ns/org#OrganizationalUnit
649291
http://www.w3.org/2004/02/skos/core#Concept
371731
http://purl.org/ontology/bibo/Book
205493
http://www.w3.org/ns/org#Role
168423
http://www.w3.org/1999/02/22-rdf-syntax-ns#Property
170
http://www.openlinksw.com/schemas/virtrdf#QuadMapFormat
130
http://www.openlinksw.com/schemas/virtrdf#array-of-QuadMapFormat
98
http://www.w3.org/2000/01/rdf-schema#Class
56
http://www.openlinksw.com/schemas/virtrdf#QuadMapValue
8
http://www.openlinksw.com/schemas/virtrdf#array-of-QuadMapColumn
8
http://www.openlinksw.com/schemas/virtrdf#QuadMapColumn
8
Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc. //http://www.openlinksw.com/
Weblog --http://www.openlinksw.com/blogs/
LinkedIn --http://www.linkedin.com/company/openlink-software/
Twitter --http://twitter.com/OpenLink
Google+ --http://plus.google.com/100570109519069333827/
Facebook --http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers
On 15 Mar 2017, at 17:08, Thomas Michaux<mich...@abes.fr> wrote:
Hello,
FYI, virtuoso still loading but we needed to increase memory ressources,
now the process use almost 40GB of ram :
[devel@tulipe-test2 ~]$ ./memcheck-virtuoso.sh
2017-03-15T17:54 VmSize: 41273424kB 5883
stats for the graph<http://hub.abes.fr/referentiel/ORCID/2016> (forget
to mention, it's the only graph in db) :
239 451 028 triples
this:Dataset a void:Dataset ;
rdfs:seeAlso<http://hub.abes.fr/referentiel/ORCID/2016> ;
rdfs:label "" ;
void:sparqlEndpoint<http://idrefplus.v102.abes.fr:8890/sparql> ;
void:triples 239451028 ;
void:classes 13 ;
void:entities 57692917 ;
void:distinctSubjects 57650847 ;
void:properties 32 ;
void:distinctObjects 72219514 .
this:sameAsLinks a void:Linkset ;
void:inDataset this:Dataset ;
void:triples 997389 ;
void:linkPredicate owl:sameAs .
Le 14/03/2017 à 10:05, Thomas Michaux a écrit :
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
