Hi Daniel,
This issue has been resolved and pushed to the develop/7 git branch:
https://github.com/openlink/virtuoso-opensource/commit/342a683c8a430dff1dbe4fe976c5994b2a259181
<https://github.com/openlink/virtuoso-opensource/commit/342a683c8a430dff1dbe4fe976c5994b2a259181>
Re-testing memory consumption of repeat loads of the “mesh.nt” graph is now
constant:
SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
Type the rest of statement, end with a semicolon (;)>
'http://id.nlm.nih.gov/mesh');
Done. -- 91206 msec.
SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
'http://id.nlm.nih.gov/mesh');
Done. -- 73576 msec.
SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
'http://id.nlm.nih.gov/mesh');
Done. -- 73819 msec.
SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
'http://id.nlm.nih.gov/mesh');
Done. -- 73465 msec.
SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
'http://id.nlm.nih.gov/mesh');
Done. -- 73560 msec.
SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
'http://id.nlm.nih.gov/mesh');
Done. -- 73643 msec.
SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
'http://id.nlm.nih.gov/mesh');
Done. -- 73498 msec.
SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
'http://id.nlm.nih.gov/mesh');
Done. -- 73581 msec.
SQL> status('');
REPORT
VARCHAR
_______________________________________________________________________________
OpenLink Virtuoso Server
Version 07.20.3214-pthreads for Linux as of Aug 25 2015
Started on: 2015-08-25 00:48 GMT+2
[root@masala nih]# sh memcheck-virtuoso.sh
2015-08-25T00:50 VmSize: 17882728kB 25182
[root@masala nih]# sh memcheck-virtuoso.sh
2015-08-25T00:51 VmSize: 19057516kB 25182
[root@masala nih]# sh memcheck-virtuoso.sh
2015-08-25T00:56 VmSize: 19059076kB 25182
[root@masala nih]# sh memcheck-virtuoso.sh
2015-08-25T00:58 VmSize: 19059076kB 25182
[root@masala nih]# sh memcheck-virtuoso.sh
2015-08-25T01:00 VmSize: 19059076kB 25182
[root@masala nih]# sh memcheck-virtuoso.sh
2015-08-25T01:04 VmSize: 19059076kB 25182
[root@masala nih]# sh memcheck-virtuoso.sh
2015-08-25T01:10 VmSize: 19059076kB 25182
[root@masala nih]#
Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc. // http://www.openlinksw.com/
Weblog -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter -- http://twitter.com/OpenLink
Google+ -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers
> On 21 Aug 2015, at 13:42, Hugh Williams <hwilli...@openlinksw.com> wrote:
>
> Hi Daniel,
>
> Thanks, I have been able to recreate and reported to development to fix:
>
> VOS user report memory leak loading there datasets multiple time with the
> TTLP_MT function , which I have been able to recreate as follows,
> demonstrating a consistent increase of about 100MB in the memory being used
> by Virtuoso:
>
> 1. The Datasets were downloaded from:
>
> ftp://ftp.nlm.nih.gov/online/mesh/void_0.9.ttl
> <ftp://ftp.nlm.nih.gov/online/mesh/void_0.9.ttl>
> ftp://ftp.nlm.nih.gov/online/mesh/vocabulary_0.9.ttl
> <ftp://ftp.nlm.nih.gov/online/mesh/vocabulary_0.9.ttl>
> ftp://ftp.nlm.nih.gov/online/mesh/mesh.nt
> <ftp://ftp.nlm.nih.gov/online/mesh/mesh.nt>
>
> 2. Load into Virtuoso repeating the load of the mesh.nt dataset multiple
> times:
>
> SQL> DB.DBA.TTLP_MT (file_to_string_output ('void_0.9.ttl'), '',
> 'http://id.nlm.nih.gov/mesh/void' <http://id.nlm.nih.gov/mesh/void'>);
>
> Done. -- 13 msec.
> SQL> DB.DBA.TTLP_MT (file_to_string_output ('vocabulary_0.9.ttl'), '',
> 'http://id.nlm.nih.gov/mesh/vocab' <http://id.nlm.nih.gov/mesh/vocab'>);
>
> Done. -- 18 msec.
> SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
> 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>);
>
> Done. -- 78536 msec.
> SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
> 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>);
>
> Done. -- 75631 msec.
> SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
> 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>);
>
> Done. -- 74642 msec.
> SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
> 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>);
>
> Done. -- 74600 msec.
> SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
> 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>);
>
> Done. -- 74507 msec.
> SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
> 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>);
>
> Done. -- 74902 msec.
> SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
> 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>);
>
> Done. -- 74245 msec.
> SQL> DB.DBA.TTLP_MT (file_to_string_output ('mesh.nt'), '',
> 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>);
>
> Done. -- 74719 msec.
> SQL> SPARQL SELECT ?graph (count(?s) as ?count) WHERE { GRAPH ?graph { ?s ?p
> ?o } } GROUP BY ?graph ORDER BY DESC(?count);
> graph
> count
> LONG VARCHAR
> LONG VARCHAR
> _______________________________________________________________________________
>
> http://id.nlm.nih.gov/mesh <http://id.nlm.nih.gov/mesh>
> 11920162
> http://mor.nlm.nih.gov/mesh2014-v1 <http://mor.nlm.nih.gov/mesh2014-v1>
> 26664
> http://localhost:8890/DAV/ <http://localhost:8890/DAV/>
> 2994
> http://www.openlinksw.com/schemas/virtrdf#
> <http://www.openlinksw.com/schemas/virtrdf#>
> 2515
> http://id.nlm.nih.gov/mesh/vocab <http://id.nlm.nih.gov/mesh/vocab>
> 259
> http://www.w3.org/2002/07/owl# <http://www.w3.org/2002/07/owl#>
> 167
> http://id.nlm.nih.gov/mesh/void <http://id.nlm.nih.gov/mesh/void>
> 34
> http://localhost:8890/sparql <http://localhost:8890/sparql>
> 14
> http://www.w3.org/ns/ldp# <http://www.w3.org/ns/ldp#>
> 3
>
> 9 Rows. -- 857 msec.
> SQL> checkpoint;
>
> Done. -- 4613 msec.
> SQL>
> SQL> status('');
> REPORT
> VARCHAR
> _______________________________________________________________________________
>
> OpenLink Virtuoso Server
> Version 07.20.3214-pthreads for Linux as of Aug 19 2015
> Started on: 2015-08-21 11:17 GMT+2
>
> 3. After each load of the mesh.nt dataset monitor the memory consumption of
> Virtuoso, I used the method detailed at,
> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtMonitorMemoryConsumption
>
> <http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtMonitorMemoryConsumption>
> , where it is seen the memory consumption continually increases (even after
> checkpoint is run):
>
>
> [root@masala nih]# sh memcheck-virtuoso.sh
> 2015-08-21T12:25 VmSize: 19514412kB 18714
> [root@masala nih]# sh memcheck-virtuoso.sh
> 2015-08-21T12:28 VmSize: 19645484kB 18714
> [root@masala nih]# sh memcheck-virtuoso.sh
> 2015-08-21T12:30 VmSize: 19711020kB 18714
> [root@masala nih]# sh memcheck-virtuoso.sh
> 2015-08-21T12:31 VmSize: 19842092kB 18714
> [root@masala nih]# sh memcheck-virtuoso.sh
> 2015-08-21T12:33 VmSize: 19907628kB 18714
> [root@masala nih]# sh memcheck-virtuoso.sh
> 2015-08-21T12:35 VmSize: 20038700kB 18714
> [root@masala nih]# sh memcheck-virtuoso.sh
> 2015-08-21T13:11 VmSize: 20038700kB 18714
> [root@masala nih]# sh memcheck-virtuoso.sh
> 2015-08-21T13:18 VmSize: 20169772kB 18714
> [root@masala nih]# sh memcheck-virtuoso.sh
> 2015-08-21T13:33 VmSize: 20305524kB 18714
> [root@masala nih]# sh memcheck-virtuoso.sh
> 2015-08-21T13:34 VmSize: 20305524kB 18714
>
> Note the datasets and test case it setup on masala:/2d2/nih
>
>
> Best Regards
> Hugh Williams
> Professional Services
> OpenLink Software, Inc. // http://www.openlinksw.com/
> <http://www.openlinksw.com/>
> Weblog -- http://www.openlinksw.com/blogs/
> <http://www.openlinksw.com/blogs/>
> LinkedIn -- http://www.linkedin.com/company/openlink-software/
> <http://www.linkedin.com/company/openlink-software/>
> Twitter -- http://twitter.com/OpenLink <http://twitter.com/OpenLink>
> Google+ -- http://plus.google.com/100570109519069333827/
> <http://plus.google.com/100570109519069333827/>
> Facebook -- http://www.facebook.com/OpenLinkSoftware
> <http://www.facebook.com/OpenLinkSoftware>
> Universal Data Access, Integration, and Management Technology Providers
>
>> On 19 Aug 2015, at 15:07, Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov
>> <mailto:daniel.da...@nih.gov>> wrote:
>>
>> Hugh,
>>
>> Thanks for trying to reproduce it. You can download the void file through
>> ftp:
>> ftp://ftp.nlm.nih.gov/online/mesh/void_0.9.ttl
>> <ftp://ftp.nlm.nih.gov/online/mesh/void_0.9.ttl>
>> and also:
>> ftp://ftp.nlm.nih.gov/online/mesh/vocabulary_0.9.ttl
>> <ftp://ftp.nlm.nih.gov/online/mesh/vocabulary_0.9.ttl>
>>
>> I imagine you got the vocabulary file through the site itself, since URLs
>> likehttp://id.nlm.nih.gov/mesh/vocab#Descriptor
>> <http://id.nlm.nih.gov/mesh/vocab#Descriptor> redirect to it.
>>
>> From: Hugh Williams [mailto:hwilli...@openlinksw.com
>> <mailto:hwilli...@openlinksw.com>]
>> Sent: Wednesday, August 19, 2015 9:57 AM
>> To: Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov
>> <mailto:daniel.da...@nih.gov>>
>> Cc: virtuoso-users@lists.sourceforge.net
>> <mailto:virtuoso-users@lists.sourceforge.net>
>> Subject: Re: [Virtuoso-users] Memory leaks with virtuoso 7.2.0
>>
>> Hi Daniel,
>>
>> I have downloaded “mesh.nt” and can access the vocab link but the void one
>> is giving a 404 error , thus can you check it is correct please …
>>
>> Best Regards
>> Hugh Williams
>> Professional Services
>> OpenLink Software, Inc. // http://www.openlinksw.com/
>> <http://www.openlinksw.com/>
>> Weblog -- http://www.openlinksw.com/blogs/
>> <http://www.openlinksw.com/blogs/>
>> LinkedIn -- http://www.linkedin.com/company/openlink-software/
>> <http://www.linkedin.com/company/openlink-software/>
>> Twitter -- http://twitter.com/OpenLink <http://twitter.com/OpenLink>
>> Google+ -- http://plus.google.com/100570109519069333827/
>> <http://plus.google.com/100570109519069333827/>
>> Facebook -- http://www.facebook.com/OpenLinkSoftware
>> <http://www.facebook.com/OpenLinkSoftware>
>> Universal Data Access, Integration, and Management Technology Providers
>>
>> On 17 Aug 2015, at 15:19, Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov
>> <mailto:daniel.da...@nih.gov>> wrote:
>>
>> Hugh Williams wrote:
>> > Why were the 7.2.1 & 7.2.0 builds configured with different options, as
>> > they not both being performed on the same OS, in which case I would expect
>> > the configure options to the same ?
>>
>> I tried the same configure options as well, and then I tried different
>> options because I was grasping at straws. Here is a graph showing that
>> 6.1.8 does not have the problem, that the changes to configure options
>> doesn’t matter for this issue.
>>
>> <image003.png>
>>
>> The basics for reproducing the problem are as follows:
>> · Download ftp://ftp.nlm.nih.gov/online/mesh/mesh.nt
>> <ftp://ftp.nlm.nih.gov/online/mesh/mesh.nt> and the vocabulary and void
>> files from that location.
>> · Load the vocabulary-0.9.ttl to graph
>> http://id.nlm.nih.gov/mesh/vocab <http://id.nlm.nih.gov/mesh/vocab>
>> · Load the void-0.9.ttl to graph http://id.nlm.nih.gov/mesh/void
>> <http://id.nlm.nih.gov/mesh/void>
>> · Load the mesh.nt file to graph http://id.nlm.nih.gov/mesh
>> <http://id.nlm.nih.gov/mesh> again and again (I used 15 seconds delay) with
>> DB.DBA.ttlp().
>>
>> From: Hugh Williams [mailto:hwilli...@openlinksw.com
>> <mailto:hwilli...@openlinksw.com>]
>> Sent: Sunday, August 16, 2015 8:52 PM
>> To: Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov
>> <mailto:daniel.da...@nih.gov>>
>> Cc: virtuoso-users@lists.sourceforge.net
>> <mailto:virtuoso-users@lists.sourceforge.net>
>> Subject: Re: [Virtuoso-users] Memory leaks with virtuoso 7.2.0
>>
>> Hi Daniel,
>>
>> Why were the 7.2.1 & 7.2.0 builds configured with different options, as they
>> not both being performed on the same OS, in which case I would expect the
>> configure options to the same ?
>>
>> Best Regards
>> Hugh Williams
>> Professional Services
>> OpenLink Software, Inc. // http://www.openlinksw.com/
>> <http://www.openlinksw.com/>
>> Weblog -- http://www.openlinksw.com/blogs/
>> <http://www.openlinksw.com/blogs/>
>> LinkedIn -- http://www.linkedin.com/company/openlink-software/
>> <http://www.linkedin.com/company/openlink-software/>
>> Twitter -- http://twitter.com/OpenLink <http://twitter.com/OpenLink>
>> Google+ -- http://plus.google.com/100570109519069333827/
>> <http://plus.google.com/100570109519069333827/>
>> Facebook -- http://www.facebook.com/OpenLinkSoftware
>> <http://www.facebook.com/OpenLinkSoftware>
>> Universal Data Access, Integration, and Management Technology Providers
>>
>> On 13 Aug 2015, at 20:39, Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov
>> <mailto:daniel.da...@nih.gov>> wrote:
>>
>> Hugh Williams wrote:
>>
>>
>> Also, note the following 2 parameters: ThreadCleanupInterval,
>> ResourcesCleanupInterval which can both be set to 1 in order to reduce
>> memory leaking:
>>
>>
>> http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ex_threadcleanupinterval
>>
>> <http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ex_threadcleanupinterval>
>>
>> although as this was not apparently occurring with 7.2.0 it may not help,
>> but worth trying ...
>>
>> It does occur with 7.2.0 - I'd vainly hoped 7.2.1 would fix it. The XSLT
>> process produces the file ftp://ftp.nlm.nih.gov/online/mesh/mesh.nt
>> <ftp://ftp.nlm.nih.gov/online/mesh/mesh.nt>, which contains 11,917,672
>> triples.
>> I'm also using the ontology
>> ftp://ftp.nlm.nih.gov/online/mesh/vocabulary_0.9.ttl
>> <ftp://ftp.nlm.nih.gov/online/mesh/vocabulary_0.9.ttl> and the void file
>> ftp://ftp.nlm.nih.gov/online/mesh/void_0.9.ttl
>> <ftp://ftp.nlm.nih.gov/online/mesh/void_0.9.ttl>.
>>
>> I cannot actually say it is a "memory leak" because I haven't pulled out
>> those tools yet - VmRSS seems to grow linearly over time.
>> VmSize does the same, only the sloop is not as steep.
>>
>> I've built Virtuoso 7.2.1 on CentOS 6.6 as follows:
>> ./configure --prefix=<whatever> --with-readline
>>
>> Previously, with 7.2.0, my build was:
>> export CFLAGS="-O2 -m64"
>> ./configure --prefix=<whatever> --with-readline
>> --with-iodbc=/usr --without-internal-zlib --enable-shared --disable-static
>>
>> I've also configured it with ThreadCleanupInterval = 1 and
>> ResourcesCleanupInterval = 1.
>>
>> The reload I'm testing runs the following commands from an isql command-line:
>>
>> log_enable(2,1);
>> sparql clear graph <http://id.nlm.nih.gov/mesh
>> <http://id.nlm.nih.gov/mesh>>;
>> sparql clear graph <http://id.nlm.nih.gov/mesh/vocab
>> <http://id.nlm.nih.gov/mesh/vocab>>;
>> sparql clear graph <http://id.nlm.nih.gov/mesh/void
>> <http://id.nlm.nih.gov/mesh/void>>;
>> DB.DBA.ttlp(file_to_string_output('path-to-files/mesh.nt'), '',
>> 'http://id.nlm.nih.gov/mesh' <http://id.nlm.nih.gov/mesh'>, 0);
>>
>> DB.DBA.ttlp(file_to_string_output('path-to-files/vocabulary_0.9.ttl'), '',
>> 'http://id.nlm.nih.gov/mesh/vocab' <http://id.nlm.nih.gov/mesh/vocab'>, 0);
>> rdfs_rule_set('http://id.nlm.nih.gov/mesh/vocab'
>> <http://id.nlm.nih.gov/mesh/vocab'>, 'http://id.nlm.nih.gov/mesh/vocab'
>> <http://id.nlm.nih.gov/mesh/vocab'>);
>> DB.DBA.ttlp(file_to_string_output('path-to-files/void_0.9.ttl'),
>> '', 'http://id.nlm.nih.gov/mesh/void' <http://id.nlm.nih.gov/mesh/void'>, 0);
>> CHECKPOINT;
>>
>> I'm retrying with a shorter script, which only loads 100,000 triples:
>>
>> log_enable(2,1);
>> sparql clear graph <http://id.nlm.nih.gov/meshtest
>> <http://id.nlm.nih.gov/meshtest>>;
>> DB.DBA.ttlp(file_to_string_output('path-to-files/small-mesh.nt',
>> '', 'http://id.nlm.nih.gov/meshtest' <http://id.nlm.nih.gov/meshtest'>, 0);
>>
>> If that proves stable, I'll do the following:
>> - stop clearing and reloading the vocabulary and void graphs and redefining
>> the ruleset
>> - let you guys know it worked.
>>
>> If that is not stable, I'll do one or several of the following:
>> - try virtuoso-opensource-6
>> - open/comment on an issue on
>> github.com/openlinksw/virtuoso-opensource/issues
>> <http://github.com/openlinksw/virtuoso-opensource/issues>
>> - install valgrind and valgrind-devel and build with malloc debug
>> - evaluate switching to the directory scan option.
>>
>> Dan Davis, Systems/Applications Architect (Contractor),
>> Office of Computer and Communications Systems,
>> National Library of Medicine, NIH
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Virtuoso-users mailing list
>> Virtuoso-users@lists.sourceforge.net
>> <mailto:Virtuoso-users@lists.sourceforge.net>
>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>> <https://lists.sourceforge.net/lists/listinfo/virtuoso-users>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Virtuoso-users mailing list
>> Virtuoso-users@lists.sourceforge.net
>> <mailto:Virtuoso-users@lists.sourceforge.net>
>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>> <https://lists.sourceforge.net/lists/listinfo/virtuoso-users>
> ------------------------------------------------------------------------------
> _______________________________________________
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
------------------------------------------------------------------------------
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
