Running lilypond on a lot of files in one run, I observe that lilypond's memory usage slowly goes up with time, i.e. it seems that lilypond does not properly free all memory used for one score, before it starts with the next one.
In particular, running lilypond on all 1010 regtests the output of ps is (as time goes on): USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND reinhold 28946 127 4.7 108432 97856 pts/3 R+ 19:27 0:01 lilypond reinhold 28946 74.4 5.4 121564 110988 pts/3 R+ 19:27 0:18 lilypond reinhold 28946 72.6 5.9 137168 123036 pts/3 R+ 19:27 0:39 lilypond reinhold 28946 73.4 6.4 151544 133308 pts/3 R+ 19:27 0:59 lilypond reinhold 28946 73.8 6.7 159784 139320 pts/3 S+ 19:27 1:11 lilypond reinhold 28946 72.1 8.1 199532 166512 pts/3 R+ 19:27 1:54 lilypond reinhold 28946 72.6 11.2 271932 230260 pts/3 R+ 19:27 2:28 lilypond reinhold 28946 72.0 12.0 316108 247232 pts/3 R+ 19:27 2:56 lilypond reinhold 28946 72.0 12.6 341956 259276 pts/3 R+ 19:27 3:25 lilypond ... reinhold 28946 72.1 15.4 423400 315956 pts/3 S+ 19:27 5:35 lilypond reinhold 28946 72.4 18.8 526744 387140 pts/3 R+ 19:27 7:47 lilypond reinhold 28946 72.5 27.9 747168 572740 pts/3 R+ 19:27 9:59 lilypond Using the memcheck tool of valgrind, there are some lost memory warnings that might be leaks. Can anyone confirm that these are really leaks? However, the leaks reported by valgrind do not explain why lilypond's memory increases on average per regtest file by ~650kB VSZ and ~475kB RSS. All valgrind warnings in this mail are obtained by running valgrind on just two files. Most of the reportedly lost memory numbers go up practically linear in the number of files processed. 1) In Pango_font::text_stencil we have PangoLayout *layout = pango_layout_new (context_); but after using the layout, we never call g_object_unref (layout). ==28530== 4,080 bytes in 2 blocks are possibly lost in loss record 6,178 of 6,263 ==28530== at 0x402517B: memalign (vg_replace_malloc.c:581) ==28530== by 0x40251D8: posix_memalign (vg_replace_malloc.c:709) ==28530== by 0x43B3546: ??? (in /lib/i386-linux- gnu/libglib-2.0.so.0.2800.6) ==28530== by 0x43B4A4C: g_slice_alloc (in /lib/i386-linux- gnu/libglib-2.0.so.0.2800.6) ==28530== by 0x43B4B74: g_slice_alloc0 (in /lib/i386-linux- gnu/libglib-2.0.so.0.2800.6) ==28530== by 0x432C8C6: g_type_create_instance (in /usr/lib/i386-linux- gnu/libgobject-2.0.so.0.2800.6) ==28530== by 0x4309674: ??? (in /usr/lib/i386-linux- gnu/libgobject-2.0.so.0.2800.6) ==28530== by 0x430CCE6: g_object_newv (in /usr/lib/i386-linux- gnu/libgobject-2.0.so.0.2800.6) ==28530== by 0x430DA3F: g_object_new (in /usr/lib/i386-linux- gnu/libgobject-2.0.so.0.2800.6) ==28530== by 0x42224C5: pango_layout_new (in /usr/lib/i386-linux- gnu/libpango-1.0.so.0.2800.4) ==28530== by 0x81D02D3: Pango_font::text_stencil(Output_def*, std::string, bool) const (pango-font.cc:312) ==28530== by 0x8292E68: Text_interface::interpret_string(scm_unused_struct*, scm_unused_struct*, scm_unused_struct*) (text-interface.cc:79) ==28530== by 0x82932BE: Text_interface::interpret_markup(scm_unused_struct*, scm_unused_struct*, scm_unused_struct*) (text-interface.cc:95) 2) In includable-lexer.cc we use yy_switch_to_buffer (yy_create_buffer (file->get_istream (), YY_BUF_SIZE)); and in Source_file::~Source_file we have "delete istream_;". On the other hand, running valgrind with 2 and with 3 .ly files shows that the memory recorded as possibly lost really increases. Still, valgrind reports: ==28530== 118,852 bytes in 18 blocks are possibly lost in loss record 6,257 of 6,263 ==28530== at 0x402641D: operator new(unsigned int) (vg_replace_malloc.c:255) ==28530== by 0x44B89F7: std::string::_Rep::_S_create(unsigned int, unsigned int, std::allocator<char> const&) (in /usr/lib/i386-linux- gnu/libstdc++.so.6.0.14) ==28530== by 0x44BAC33: char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) (in /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14) ==28530== by 0x44BAD20: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, unsigned int, std::allocator<char> const&) (in /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14) ==28530== by 0x44B2B27: std::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >::basic_istringstream(std::string const&, std::_Ios_Openmode) (in /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14) ==28530== by 0x824A340: Source_file::get_istream() (source-file.cc:163) ==28530== by 0x8136886: Includable_lexer::new_input(std::string, Sources*) (includable-lexer.cc:93) ==28530== by 0x814B3A5: Lily_lexer::new_input(std::string, Sources*) (lily- lexer.cc:269) ==28530== by 0x82D26AF: Lily_lexer::yylex() (lexer.ll:305) ==28530== by 0x82E4599: yylex(YYSTYPE*, Input*, void*) (parser.yy:2784) ==28530== by 0x82D854B: yyparse(void*) (parser.cc:2486) ==28530== by 0x82E3880: Lily_parser::do_yyparse() (parser.yy:2557) ==28530== by 0x814D089: Lily_parser::parse_file(std::string, std::string, std::string) (lily-parser.cc:120) 3) There are many, many warnings about possibly lost memory in pango_layout_get_lines calls, but I don't see how they can be real memory leaks from the pango docs. Still, the numbers go up linearly in the number of files... ==28530== 5,032 (512 direct, 4,520 indirect) bytes in 2 blocks are definitely lost in loss record 6,200 of 6,263 ==28530== at 0x402695A: realloc (vg_replace_malloc.c:525) ==28530== by 0x42E4118: ??? (in /usr/lib/i386-linux- gnu/libfontconfig.so.1.4.4) [...] ==28530== by 0x421BE52: pango_itemize_with_base_dir (in /usr/lib/i386- linux-gnu/libpango-1.0.so.0.2800.4) ==28530== by 0x4224557: ??? (in /usr/lib/i386-linux- gnu/libpango-1.0.so.0.2800.4) ==28530== by 0x4226780: pango_layout_get_lines (in /usr/lib/i386-linux- gnu/libpango-1.0.so.0.2800.4) ==28530== by 0x81D0303: Pango_font::text_stencil(Output_def*, std::string, bool) const (pango-font.cc:314) ==28530== by 0x8292E68: Text_interface::interpret_string(scm_unused_struct*, scm_unused_struct*, scm_unused_struct*) (text-interface.cc:79) ==28530== by 0x82932BE: Text_interface::interpret_markup(scm_unused_struct*, scm_unused_struct*, scm_unused_struct*) (text-interface.cc:95) 4) In all-font-metrics.cc we have PangoFontMap *pfm = pango_ft2_font_map_new (); pango_ft2_fontmap_ = PANGO_FT2_FONT_MAP (pfm); And in All_font_metrics::~All_font_metrics we have: g_object_unref (pango_ft2_fontmap_); Still, valgrind reports that pango_ft2_fontmap_ is possibly lost, and the number of bytes goes up linearly in the number of files... ==30677== 32,768 bytes in 2 blocks are possibly lost in loss record 6,899 of 6,919 ==30677== at 0x4026864: malloc (vg_replace_malloc.c:236) ==30677== by 0x424C7BC: ??? (in /usr/lib/i386-linux- gnu/libfreetype.so.6.6.2) ==30677== by 0x425136A: ft_mem_qalloc (in /usr/lib/i386-linux- gnu/libfreetype.so.6.6.2) ==30677== by 0x42513C2: ft_mem_alloc (in /usr/lib/i386-linux- gnu/libfreetype.so.6.6.2) ==30677== by 0x42528C2: FT_New_Library (in /usr/lib/i386-linux- gnu/libfreetype.so.6.6.2) ==30677== by 0x424CBA8: FT_Init_FreeType (in /usr/lib/i386-linux- gnu/libfreetype.so.6.6.2) ==30677== by 0x41E82DD: pango_ft2_font_map_new (in /usr/lib/i386-linux- gnu/libpangoft2-1.0.so.0.2800.4) ==30677== by 0x8063731: All_font_metrics::All_font_metrics(std::string) (all-font-metrics.cc:48) ==30677== by 0x80661B5: ly_reset_all_fonts() (all-font-metrics- scheme.cc:29) 5) In relocate.cc (line 50) we use putenv (where we can't deallocate the memory). Is there any reason why we can't use setenv, which would create a copy? It's 50 bytes, so practically negligible, but still, we even have a comment in the source code: int retval = putenv (s); /* unfortunately, we can't portably free S here, due to various bugs in glibc prior to 2.1.1 */ Cheers, Reinhold -- ------------------------------------------------------------------ Reinhold Kainhofer, reinh...@kainhofer.com, http://reinhold.kainhofer.com/ * Financial & Actuarial Math., Vienna Univ. of Technology, Austria * http://www.fam.tuwien.ac.at/, DVR: 0005886 * LilyPond, Music typesetting, http://www.lilypond.org _______________________________________________ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel