Hi Daniel, This issue has been resolved and pushed to the develop/7 git branch:
https://github.com/openlink/virtuoso-opensource/commit/342a683c8a430dff1dbe4fe976c5994b2a259181 <https://github.com/openlink/virtuoso-opensource/commit/342a683c8a430dff1dbe4fe976c5994b2a259181> Re-testing memory consumption of repeat loads of the “mesh.nt” graph is now constant: SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', Type the rest of statement, end with a semicolon (;)> 'http://id.nlm.nih.gov/mesh'); Done. -- 91206 msec. SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', 'http://id.nlm.nih.gov/mesh'); Done. -- 73576 msec. SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', 'http://id.nlm.nih.gov/mesh'); Done. -- 73819 msec. SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', 'http://id.nlm.nih.gov/mesh'); Done. -- 73465 msec. SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', 'http://id.nlm.nih.gov/mesh'); Done. -- 73560 msec. SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', 'http://id.nlm.nih.gov/mesh'); Done. -- 73643 msec. SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', 'http://id.nlm.nih.gov/mesh'); Done. -- 73498 msec. SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', 'http://id.nlm.nih.gov/mesh'); Done. -- 73581 msec. SQL> status(''); REPORT VARCHAR _______________________________________________________________________________ OpenLink Virtuoso Server Version 07.20.3214-pthreads for Linux as of Aug 25 2015 Started on: 2015-08-25 00:48 GMT+2 [root@masala nih]# sh memcheck-virtuoso.sh 2015-08-25T00:50 VmSize: 17882728kB 25182 [root@masala nih]# sh memcheck-virtuoso.sh 2015-08-25T00:51 VmSize: 19057516kB 25182 [root@masala nih]# sh memcheck-virtuoso.sh 2015-08-25T00:56 VmSize: 19059076kB 25182 [root@masala nih]# sh memcheck-virtuoso.sh 2015-08-25T00:58 VmSize: 19059076kB 25182 [root@masala nih]# sh memcheck-virtuoso.sh 2015-08-25T01:00 VmSize: 19059076kB 25182 [root@masala nih]# sh memcheck-virtuoso.sh 2015-08-25T01:04 VmSize: 19059076kB 25182 [root@masala nih]# sh memcheck-virtuoso.sh 2015-08-25T01:10 VmSize: 19059076kB 25182 [root@masala nih]# Best Regards Hugh Williams Professional Services OpenLink Software, Inc. // http://www.openlinksw.com/ Weblog -- http://www.openlinksw.com/blogs/ LinkedIn -- http://www.linkedin.com/company/openlink-software/ Twitter -- http://twitter.com/OpenLink Google+ -- http://plus.google.com/100570109519069333827/ Facebook -- http://www.facebook.com/OpenLinkSoftware Universal Data Access, Integration, and Management Technology Providers > On 21 Aug 2015, at 13:42, Hugh Williams <hwilli...@openlinksw.com> wrote: > > Hi Daniel, > > Thanks, I have been able to recreate and reported to development to fix: > > VOS user report memory leak loading there datasets multiple time with the > TTLP_MT function , which I have been able to recreate as follows, > demonstrating a consistent increase of about 100MB in the memory being used > by Virtuoso: > > 1. The Datasets were downloaded from: > > ftp://ftp.nlm.nih.gov/online/mesh/void_0.9.ttl > <ftp://ftp.nlm.nih.gov/online/mesh/void_0.9.ttl> > ftp://ftp.nlm.nih.gov/online/mesh/vocabulary_0.9.ttl > <ftp://ftp.nlm.nih.gov/online/mesh/vocabulary_0.9.ttl> > ftp://ftp.nlm.nih.gov/online/mesh/mesh.nt > <ftp://ftp.nlm.nih.gov/online/mesh/mesh.nt> > > 2. Load into Virtuoso repeating the load of the mesh.nt dataset multiple > times: > > SQL> DB.DBA.TTLP_MT (file_to_string_output ('void_0.9.ttl'), '', > 'http://id.nlm.nih.gov/mesh/void' <http://id.nlm.nih.gov/mesh/void'>); > > Done. -- 13 msec. > SQL> DB.DBA.TTLP_MT (file_to_string_output ('vocabulary_0.9.ttl'), '', > 'http://id.nlm.nih.gov/mesh/vocab' <http://id.nlm.nih.gov/mesh/vocab'>); > > Done. -- 18 msec. > SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', > 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>); > > Done. -- 78536 msec. > SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', > 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>); > > Done. -- 75631 msec. > SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', > 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>); > > Done. -- 74642 msec. > SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', > 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>); > > Done. -- 74600 msec. > SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', > 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>); > > Done. -- 74507 msec. > SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', > 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>); > > Done. -- 74902 msec. > SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', > 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>); > > Done. -- 74245 msec. > SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '', > 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>); > > Done. -- 74719 msec. > SQL> SPARQL SELECT ?graph (count(?s) as ?count) WHERE { GRAPH ?graph { ?s ?p > ?o } } GROUP BY ?graph ORDER BY DESC(?count); > graph > count > LONG VARCHAR > LONG VARCHAR > _______________________________________________________________________________ > > http://id.nlm.nih.gov/mesh <http://id.nlm.nih.gov/mesh> > 11920162 > http://mor.nlm.nih.gov/mesh2014-v1 <http://mor.nlm.nih.gov/mesh2014-v1> > 26664 > http://localhost:8890/DAV/ <http://localhost:8890/DAV/> > 2994 > http://www.openlinksw.com/schemas/virtrdf# > <http://www.openlinksw.com/schemas/virtrdf#> > 2515 > http://id.nlm.nih.gov/mesh/vocab <http://id.nlm.nih.gov/mesh/vocab> > 259 > http://www.w3.org/2002/07/owl# <http://www.w3.org/2002/07/owl#> > 167 > http://id.nlm.nih.gov/mesh/void <http://id.nlm.nih.gov/mesh/void> > 34 > http://localhost:8890/sparql <http://localhost:8890/sparql> > 14 > http://www.w3.org/ns/ldp# <http://www.w3.org/ns/ldp#> > 3 > > 9 Rows. -- 857 msec. > SQL> checkpoint; > > Done. -- 4613 msec. > SQL> > SQL> status(''); > REPORT > VARCHAR > _______________________________________________________________________________ > > OpenLink Virtuoso Server > Version 07.20.3214-pthreads for Linux as of Aug 19 2015 > Started on: 2015-08-21 11:17 GMT+2 > > 3. After each load of the mesh.nt dataset monitor the memory consumption of > Virtuoso, I used the method detailed at, > http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtMonitorMemoryConsumption > > <http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtMonitorMemoryConsumption> > , where it is seen the memory consumption continually increases (even after > checkpoint is run): > > > [root@masala nih]# sh memcheck-virtuoso.sh > 2015-08-21T12:25 VmSize: 19514412kB 18714 > [root@masala nih]# sh memcheck-virtuoso.sh > 2015-08-21T12:28 VmSize: 19645484kB 18714 > [root@masala nih]# sh memcheck-virtuoso.sh > 2015-08-21T12:30 VmSize: 19711020kB 18714 > [root@masala nih]# sh memcheck-virtuoso.sh > 2015-08-21T12:31 VmSize: 19842092kB 18714 > [root@masala nih]# sh memcheck-virtuoso.sh > 2015-08-21T12:33 VmSize: 19907628kB 18714 > [root@masala nih]# sh memcheck-virtuoso.sh > 2015-08-21T12:35 VmSize: 20038700kB 18714 > [root@masala nih]# sh memcheck-virtuoso.sh > 2015-08-21T13:11 VmSize: 20038700kB 18714 > [root@masala nih]# sh memcheck-virtuoso.sh > 2015-08-21T13:18 VmSize: 20169772kB 18714 > [root@masala nih]# sh memcheck-virtuoso.sh > 2015-08-21T13:33 VmSize: 20305524kB 18714 > [root@masala nih]# sh memcheck-virtuoso.sh > 2015-08-21T13:34 VmSize: 20305524kB 18714 > > Note the datasets and test case it setup on masala:/2d2/nih > > > Best Regards > Hugh Williams > Professional Services > OpenLink Software, Inc. // http://www.openlinksw.com/ > <http://www.openlinksw.com/> > Weblog -- http://www.openlinksw.com/blogs/ > <http://www.openlinksw.com/blogs/> > LinkedIn -- http://www.linkedin.com/company/openlink-software/ > <http://www.linkedin.com/company/openlink-software/> > Twitter -- http://twitter.com/OpenLink <http://twitter.com/OpenLink> > Google+ -- http://plus.google.com/100570109519069333827/ > <http://plus.google.com/100570109519069333827/> > Facebook -- http://www.facebook.com/OpenLinkSoftware > <http://www.facebook.com/OpenLinkSoftware> > Universal Data Access, Integration, and Management Technology Providers > >> On 19 Aug 2015, at 15:07, Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov >> <mailto:daniel.da...@nih.gov>> wrote: >> >> Hugh, >> >> Thanks for trying to reproduce it. You can download the void file through >> ftp: >> ftp://ftp.nlm.nih.gov/online/mesh/void_0.9.ttl >> <ftp://ftp.nlm.nih.gov/online/mesh/void_0.9.ttl> >> and also: >> ftp://ftp.nlm.nih.gov/online/mesh/vocabulary_0.9.ttl >> <ftp://ftp.nlm.nih.gov/online/mesh/vocabulary_0.9.ttl> >> >> I imagine you got the vocabulary file through the site itself, since URLs >> likehttp://id.nlm.nih.gov/mesh/vocab#Descriptor >> <http://id.nlm.nih.gov/mesh/vocab#Descriptor> redirect to it. >> >> From: Hugh Williams [mailto:hwilli...@openlinksw.com >> <mailto:hwilli...@openlinksw.com>] >> Sent: Wednesday, August 19, 2015 9:57 AM >> To: Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov >> <mailto:daniel.da...@nih.gov>> >> Cc: virtuoso-users@lists.sourceforge.net >> <mailto:virtuoso-users@lists.sourceforge.net> >> Subject: Re: [Virtuoso-users] Memory leaks with virtuoso 7.2.0 >> >> Hi Daniel, >> >> I have downloaded “mesh.nt” and can access the vocab link but the void one >> is giving a 404 error , thus can you check it is correct please … >> >> Best Regards >> Hugh Williams >> Professional Services >> OpenLink Software, Inc. // http://www.openlinksw.com/ >> <http://www.openlinksw.com/> >> Weblog -- http://www.openlinksw.com/blogs/ >> <http://www.openlinksw.com/blogs/> >> LinkedIn -- http://www.linkedin.com/company/openlink-software/ >> <http://www.linkedin.com/company/openlink-software/> >> Twitter -- http://twitter.com/OpenLink <http://twitter.com/OpenLink> >> Google+ -- http://plus.google.com/100570109519069333827/ >> <http://plus.google.com/100570109519069333827/> >> Facebook -- http://www.facebook.com/OpenLinkSoftware >> <http://www.facebook.com/OpenLinkSoftware> >> Universal Data Access, Integration, and Management Technology Providers >> >> On 17 Aug 2015, at 15:19, Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov >> <mailto:daniel.da...@nih.gov>> wrote: >> >> Hugh Williams wrote: >> > Why were the 7.2.1 & 7.2.0 builds configured with different options, as >> > they not both being performed on the same OS, in which case I would expect >> > the configure options to the same ? >> >> I tried the same configure options as well, and then I tried different >> options because I was grasping at straws. Here is a graph showing that >> 6.1.8 does not have the problem, that the changes to configure options >> doesn’t matter for this issue. >> >> <image003.png> >> >> The basics for reproducing the problem are as follows: >> · Download ftp://ftp.nlm.nih.gov/online/mesh/mesh.nt >> <ftp://ftp.nlm.nih.gov/online/mesh/mesh.nt> and the vocabulary and void >> files from that location. >> · Load the vocabulary-0.9.ttl to graph >> http://id.nlm.nih.gov/mesh/vocab <http://id.nlm.nih.gov/mesh/vocab> >> · Load the void-0.9.ttl to graph http://id.nlm.nih.gov/mesh/void >> <http://id.nlm.nih.gov/mesh/void> >> · Load the mesh.nt file to graph http://id.nlm.nih.gov/mesh >> <http://id.nlm.nih.gov/mesh> again and again (I used 15 seconds delay) with >> DB.DBA.ttlp(). >> >> From: Hugh Williams [mailto:hwilli...@openlinksw.com >> <mailto:hwilli...@openlinksw.com>] >> Sent: Sunday, August 16, 2015 8:52 PM >> To: Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov >> <mailto:daniel.da...@nih.gov>> >> Cc: virtuoso-users@lists.sourceforge.net >> <mailto:virtuoso-users@lists.sourceforge.net> >> Subject: Re: [Virtuoso-users] Memory leaks with virtuoso 7.2.0 >> >> Hi Daniel, >> >> Why were the 7.2.1 & 7.2.0 builds configured with different options, as they >> not both being performed on the same OS, in which case I would expect the >> configure options to the same ? >> >> Best Regards >> Hugh Williams >> Professional Services >> OpenLink Software, Inc. // http://www.openlinksw.com/ >> <http://www.openlinksw.com/> >> Weblog -- http://www.openlinksw.com/blogs/ >> <http://www.openlinksw.com/blogs/> >> LinkedIn -- http://www.linkedin.com/company/openlink-software/ >> <http://www.linkedin.com/company/openlink-software/> >> Twitter -- http://twitter.com/OpenLink <http://twitter.com/OpenLink> >> Google+ -- http://plus.google.com/100570109519069333827/ >> <http://plus.google.com/100570109519069333827/> >> Facebook -- http://www.facebook.com/OpenLinkSoftware >> <http://www.facebook.com/OpenLinkSoftware> >> Universal Data Access, Integration, and Management Technology Providers >> >> On 13 Aug 2015, at 20:39, Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov >> <mailto:daniel.da...@nih.gov>> wrote: >> >> Hugh Williams wrote: >> >> >> Also, note the following 2 parameters: ThreadCleanupInterval, >> ResourcesCleanupInterval which can both be set to 1 in order to reduce >> memory leaking: >> >> >> http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ex_threadcleanupinterval >> >> <http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ex_threadcleanupinterval> >> >> although as this was not apparently occurring with 7.2.0 it may not help, >> but worth trying ... >> >> It does occur with 7.2.0 - I'd vainly hoped 7.2.1 would fix it. The XSLT >> process produces the file ftp://ftp.nlm.nih.gov/online/mesh/mesh.nt >> <ftp://ftp.nlm.nih.gov/online/mesh/mesh.nt>, which contains 11,917,672 >> triples. >> I'm also using the ontology >> ftp://ftp.nlm.nih.gov/online/mesh/vocabulary_0.9.ttl >> <ftp://ftp.nlm.nih.gov/online/mesh/vocabulary_0.9.ttl> and the void file >> ftp://ftp.nlm.nih.gov/online/mesh/void_0.9.ttl >> <ftp://ftp.nlm.nih.gov/online/mesh/void_0.9.ttl>. >> >> I cannot actually say it is a "memory leak" because I haven't pulled out >> those tools yet - VmRSS seems to grow linearly over time. >> VmSize does the same, only the sloop is not as steep. >> >> I've built Virtuoso 7.2.1 on CentOS 6.6 as follows: >> ./configure --prefix=<whatever> --with-readline >> >> Previously, with 7.2.0, my build was: >> export CFLAGS="-O2 -m64" >> ./configure --prefix=<whatever> --with-readline >> --with-iodbc=/usr --without-internal-zlib --enable-shared --disable-static >> >> I've also configured it with ThreadCleanupInterval = 1 and >> ResourcesCleanupInterval = 1. >> >> The reload I'm testing runs the following commands from an isql command-line: >> >> log_enable(2,1); >> sparql clear graph <http://id.nlm.nih.gov/mesh >> <http://id.nlm.nih.gov/mesh>>; >> sparql clear graph <http://id.nlm.nih.gov/mesh/vocab >> <http://id.nlm.nih.gov/mesh/vocab>>; >> sparql clear graph <http://id.nlm.nih.gov/mesh/void >> <http://id.nlm.nih.gov/mesh/void>>; >> DB.DBA.ttlp(file_to_string_output('path-to-files/mesh.nt'), '', >> 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>, 0); >> >> DB.DBA.ttlp(file_to_string_output('path-to-files/vocabulary_0.9.ttl'), '', >> 'http://id.nlm.nih.gov/mesh/vocab' <http://id.nlm.nih.gov/mesh/vocab'>, 0); >> rdfs_rule_set('http://id.nlm.nih.gov/mesh/vocab' >> <http://id.nlm.nih.gov/mesh/vocab'>, 'http://id.nlm.nih.gov/mesh/vocab' >> <http://id.nlm.nih.gov/mesh/vocab'>); >> DB.DBA.ttlp(file_to_string_output('path-to-files/void_0.9.ttl'), >> '', 'http://id.nlm.nih.gov/mesh/void' <http://id.nlm.nih.gov/mesh/void'>, 0); >> CHECKPOINT; >> >> I'm retrying with a shorter script, which only loads 100,000 triples: >> >> log_enable(2,1); >> sparql clear graph <http://id.nlm.nih.gov/meshtest >> <http://id.nlm.nih.gov/meshtest>>; >> DB.DBA.ttlp(file_to_string_output('path-to-files/small-mesh.nt', >> '', 'http://id.nlm.nih.gov/meshtest' <http://id.nlm.nih.gov/meshtest'>, 0); >> >> If that proves stable, I'll do the following: >> - stop clearing and reloading the vocabulary and void graphs and redefining >> the ruleset >> - let you guys know it worked. >> >> If that is not stable, I'll do one or several of the following: >> - try virtuoso-opensource-6 >> - open/comment on an issue on >> github.com/openlinksw/virtuoso-opensource/issues >> <http://github.com/openlinksw/virtuoso-opensource/issues> >> - install valgrind and valgrind-devel and build with malloc debug >> - evaluate switching to the directory scan option. >> >> Dan Davis, Systems/Applications Architect (Contractor), >> Office of Computer and Communications Systems, >> National Library of Medicine, NIH >> >> >> >> >> ------------------------------------------------------------------------------ >> _______________________________________________ >> Virtuoso-users mailing list >> Virtuoso-users@lists.sourceforge.net >> <mailto:Virtuoso-users@lists.sourceforge.net> >> https://lists.sourceforge.net/lists/listinfo/virtuoso-users >> <https://lists.sourceforge.net/lists/listinfo/virtuoso-users> >> >> ------------------------------------------------------------------------------ >> _______________________________________________ >> Virtuoso-users mailing list >> Virtuoso-users@lists.sourceforge.net >> <mailto:Virtuoso-users@lists.sourceforge.net> >> https://lists.sourceforge.net/lists/listinfo/virtuoso-users >> <https://lists.sourceforge.net/lists/listinfo/virtuoso-users> > ------------------------------------------------------------------------------ > _______________________________________________ > Virtuoso-users mailing list > Virtuoso-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/virtuoso-users
------------------------------------------------------------------------------
_______________________________________________ Virtuoso-users mailing list Virtuoso-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/virtuoso-users