I have backported the following fixes from master into 4.2.3: https://github.com/mono/mono/commit/b97ac0023256bf7d915552f5f24a7742b28c32b7 https://github.com/mono/mono/commit/8c52b398c5eb962bba5985e8bc01445ac5f027a5 https://github.com/mono/mono/pull/2781 https://github.com/mono/mono/pull/2783 https://github.com/mono/mono/pull/2785
This has helped tremendously. I am now down to 659 leak occurrences, of which 640 have one of the following signatures: 1) "mono_metadata_type_dup" x 327 occurrences x 12-36 bytes each ==31699== 36 bytes in 3 blocks are definitely lost in loss record 7,355 of 13,872 ==31699== at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==31699== by 0x62D1E1: monoeg_malloc (in /usr/bin/mono-sgen) ==31699== by 0x55B97F: mono_metadata_type_dup (in /usr/bin/mono-sgen) ==31699== by 0x49FD0B: get_shared_gparam (in /usr/bin/mono-sgen) ==31699== by 0x49FF30: get_shared_inst (in /usr/bin/mono-sgen) ==31699== by 0x4A07FA: mini_get_shared_method_full (in /usr/bin/mono-sgen) ==31699== by 0x414723: lookup_method (in /usr/bin/mono-sgen) ==31699== by 0x4147FA: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==31699== by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==31699== by 0x498E64: common_call_trampoline_inner (in /usr/bin/mono-sgen) 2) "mono_method_get_header" x 313 occurrences x 32-192 bytes each Note: PR 2781 brought this down from 5800 occurrences. Would https://github.com/mono/mono/pull/2705 help potentially? ==31699== 32 bytes in 1 blocks are definitely lost in loss record 7,047 of 13,872 ==31699== at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==31699== by 0x62D1E1: monoeg_malloc (in /usr/bin/mono-sgen) ==31699== by 0x62D237: monoeg_g_memdup (in /usr/bin/mono-sgen) ==31699== by 0x53CB47: mono_method_get_header (in /usr/bin/mono-sgen) ==31699== by 0x4F8EA0: mini_method_compile (in /usr/bin/mono-sgen) ==31699== by 0x4FA788: mono_jit_compile_method_inner (in /usr/bin/mono-sgen) ==31699== by 0x414A01: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==31699== by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==31699== by 0x498E64: common_call_trampoline_inner (in /usr/bin/mono-sgen) or ==31699== 192 bytes in 3 blocks are definitely lost in loss record 11,517 of 13,872 ==31699== at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==31699== by 0x62D1E1: monoeg_malloc (in /usr/bin/mono-sgen) ==31699== by 0x62D237: monoeg_g_memdup (in /usr/bin/mono-sgen) ==31699== by 0x53CB47: mono_method_get_header (in /usr/bin/mono-sgen) ==31699== by 0x430691: mono_method_to_ir (in /usr/bin/mono-sgen) ==31699== by 0x4F94F5: mini_method_compile (in /usr/bin/mono-sgen) ==31699== by 0x4FA788: mono_jit_compile_method_inner (in /usr/bin/mono-sgen) ==31699== by 0x414A01: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==31699== by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==31699== by 0x498E64: common_call_trampoline_inner (in /usr/bin/mono-sgen) Getting close! Thanks again for any help anyone can provide, Matt From: mono-devel-list-boun...@lists.ximian.com [mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Zinkevicius, Matt Sent: Tuesday, March 29, 2016 8:26 PM To: Rodrigo Kumpera <kump...@gmail.com> Cc: Straw, David (Storage) <david.st...@hpe.com>; mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] Using valgrind with Mono Unfortunately, PR 2783 did not have any noticeable effect. We still see thousands of leaks like the following: ==12142== 89,860 (89,704 direct, 156 indirect) bytes in 2,800 blocks are definitely lost in loss record 19,763 of 19,792 ==12142== at 0x4C26FEF: calloc (vg_replace_malloc.c:711) ==12142== by 0x62D269: monoeg_malloc0 (in /usr/bin/mono-sgen) ==12142== by 0x53CA32: mono_method_get_header (in /usr/bin/mono-sgen) ==12142== by 0x56CCEA: mono_basic_block_split (in /usr/bin/mono-sgen) ==12142== by 0x4323B3: mono_method_to_ir (in /usr/bin/mono-sgen) ==12142== by 0x45FC8B: inline_method (in /usr/bin/mono-sgen) ==12142== by 0x44C2F4: mono_method_to_ir (in /usr/bin/mono-sgen) ==12142== by 0x4F94A5: mini_method_compile (in /usr/bin/mono-sgen) ==12142== by 0x4FA738: mono_jit_compile_method_inner (in /usr/bin/mono-sgen) ==12142== by 0x414A01: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==12142== by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==12142== by 0x498E44: common_call_trampoline_inner (in /usr/bin/mono-sgen) Looks like https://github.com/mono/mono/pull/2781/ may address this leak? I’ll attempt to backport it and report back. Matt From: mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com> [mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Zinkevicius, Matt Sent: Tuesday, March 29, 2016 7:22 PM To: Rodrigo Kumpera <kump...@gmail.com<mailto:kump...@gmail.com>> Cc: Straw, David (Storage) <david.st...@hpe.com<mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono Thanks, Rodrigo! I’ve ported this to 4.2 to test, though these changes seem to only address AOT, and we’re seeing this leak using the normal JIT runtime. Matt From: Rodrigo Kumpera [mailto:kump...@gmail.com] Sent: Tuesday, March 29, 2016 6:15 PM To: Zinkevicius, Matt <matt.zinkevic...@hpe.com<mailto:matt.zinkevic...@hpe.com>> Cc: Straw, David (Storage) <david.st...@hpe.com<mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono This is the PR in question: https://github.com/mono/mono/pull/2783 It probably won't make into 4.2, but should definitely be in 4.4. On Tue, Mar 29, 2016 at 11:55 AM, Zinkevicius, Matt <matt.zinkevic...@hpe.com<mailto:matt.zinkevic...@hpe.com>> wrote: Hi Rodrigo, Is the following the PR that you were referring to that fixes the “mono_method_get_header” leak? https://github.com/mono/mono/pull/2705 I am eager to find it, as this particular leak seems to be the most egregious of the ones we’re experiencing, in terms of both occurrences and total bytes leaked (hundreds of MB/hour when under load). If so, I’ll try to backport it to Mono 4.2.3, though it is far from patching cleanly currently. Matt From: mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com> [mailto:mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com>] On Behalf Of Zinkevicius, Matt Sent: Monday, March 28, 2016 2:42 PM To: Rodrigo Kumpera <kump...@gmail.com<mailto:kump...@gmail.com>> Cc: Straw, David (Storage) <david.st...@hpe.com<mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono Hi Rodrigo, I am not finding a recent commit in master that mentions “mono_method_get_header”. Do you happen to have the commit ID or log message handy? Sorry for the bother, Matt From: Rodrigo Kumpera [mailto:kump...@gmail.com] Sent: Monday, March 28, 2016 1:58 PM To: Zinkevicius, Matt <matt.zinkevic...@hpe.com<mailto:matt.zinkevic...@hpe.com>> Cc: Zoltan Varga <var...@gmail.com<mailto:var...@gmail.com>>; Straw, David (Storage) <david.st...@hpe.com<mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono The memory leak on mono_method_get_header was recently fixed in master. On Mon, Mar 28, 2016 at 3:31 AM, Zinkevicius, Matt <matt.zinkevic...@hpe.com<mailto:matt.zinkevic...@hpe.com>> wrote: I can confirm that Zoltan’s fix does indeed remove any leak instances with a call stack containing “ves_icall_Type_GetNestedTypes”, but unfortunately that accounted for only 36 leak instances out of the 19,900 reported. I have filed a bug (https://bugzilla.xamarin.com/show_bug.cgi?id=39940) and attached the latest valgrind output there. Thanks again for any help you can provide, Matt Zinkevicius From: mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com> [mailto:mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com>] On Behalf Of Zinkevicius, Matt Sent: Sunday, March 27, 2016 11:06 PM To: Zoltan Varga <var...@gmail.com<mailto:var...@gmail.com>> Cc: Straw, David (Storage) <david.st...@hpe.com<mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono Hi Zoltan, Thank you for investigating my findings. I will apply the fix you linked to and report back. While I understand the other leaks are small, valgrind reported over 19,000 instances of leaked memory in under 5 minutes of our app running without load. Matt Zinkevicius From: Zoltan Varga [mailto:var...@gmail.com] Sent: Sunday, March 27, 2016 4:45 PM To: Zinkevicius, Matt <matt.zinkevic...@hpe.com<mailto:matt.zinkevic...@hpe.com>> Cc: mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com>; Straw, David (Storage) <david.st...@hpe.com<mailto:david.st...@hpe.com>> Subject: Re: [Mono-dev] Using valgrind with Mono Hi, Fixed the last one in: https://github.com/mono/mono/commit/b97ac0023256bf7d915552f5f24a7742b28c32b7 The first two are leaks, but they should be small and bounded. Will look into fixing them to decrease the noise. Zoltan On Sun, Mar 27, 2016 at 6:23 PM, Zinkevicius, Matt <matt.zinkevic...@hpe.com<mailto:matt.zinkevic...@hpe.com>> wrote: Hello, Our backend service running on Mono 4.2.2 on Linux is experiencing an unmanaged memory leak. When running our stress tests for several hours, we see the managed heap sit around 50 MB, while private memory keeps growing until the process is killed because of OOM. I am therefore attempting to use valgrind to find the culprit, but I am getting so many leaks detected that I think many must be false positives, so I thought I would ask here for some guidance about which are safe to suppress or any other valgrind + mono tricks you can share. The vast majority of leaks reported have call stacks that closely match one of the following: ==16846== 4 bytes in 1 blocks are definitely lost in loss record 74 of 19,903 ==16846== at 0x4C26FEF: calloc (vg_replace_malloc.c:711) ==16846== by 0x62D1D9: monoeg_malloc0 (in /usr/bin/mono-sgen) ==16846== by 0x4870F2: decode_exception_debug_info (in /usr/bin/mono-sgen) ==16846== by 0x488975: mono_aot_find_jit_info (in /usr/bin/mono-sgen) ==16846== by 0x53C3A7: mono_jit_info_table_find_internal (in /usr/bin/mono-sgen) ==16846== by 0x493C04: mini_jit_info_table_find_ext (in /usr/bin/mono-sgen) ==16846== by 0x4988FB: mini_add_method_trampoline (in /usr/bin/mono-sgen) ==16846== by 0x499067: common_call_trampoline_inner (in /usr/bin/mono-sgen) ==16846== by 0x403217C: ??? ==16846== by 0x10D3FB63: ??? ==16846== by 0x10D3F41B: ??? ==16846== by 0x10D3F117: ??? ==16846== 12 bytes in 1 blocks are definitely lost in loss record 1,172 of 19,903 ==16846== at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==16846== by 0x62D221: monoeg_malloc (in /usr/bin/mono-sgen) ==16846== by 0x55B8EF: mono_metadata_type_dup (in /usr/bin/mono-sgen) ==16846== by 0x49FC4B: get_shared_gparam (in /usr/bin/mono-sgen) ==16846== by 0x49FE70: get_shared_inst (in /usr/bin/mono-sgen) ==16846== by 0x4A073A: mini_get_shared_method_full (in /usr/bin/mono-sgen) ==16846== by 0x414723: lookup_method (in /usr/bin/mono-sgen) ==16846== by 0x4147FA: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==16846== by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==16846== by 0x498DA4: common_call_trampoline_inner (in /usr/bin/mono-sgen) ==16846== by 0x403405C: ??? ==16846== by 0x10D2DCA7: ??? ==16846== 10 bytes in 1 blocks are definitely lost in loss record 739 of 19,903 ==16846== at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==16846== by 0x62D221: monoeg_malloc (in /usr/bin/mono-sgen) ==16846== by 0x62BA8C: monoeg_g_utf16_to_utf8 (in /usr/bin/mono-sgen) ==16846== by 0x5A8646: mono_string_to_utf8_checked (in /usr/bin/mono-sgen) ==16846== by 0x5A885B: mono_string_to_utf8 (in /usr/bin/mono-sgen) ==16846== by 0x52DE3C: ves_icall_Type_GetNestedTypes (in /usr/bin/mono-sgen) ==16846== by 0x120D4256: ??? ==16846== by 0xE338A78: System_Type_GetMember_string_System_Reflection_BindingFlags (type.cs:806) ==16846== by 0x40C09EF: ??? ==16846== by 0x1259A6AF: ??? ==16846== by 0x73: ??? ==16846== by 0x141D191D: ??? Are these valid leaks or is valgrind confused/misconfigured? I am using the following command: valgrind --tool=memcheck -v --leak-check=full --log-file=val.txt --smc-check=all mono program.exe Thanks for any input you can offer, Matt Zinkevicius _______________________________________________ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com<mailto:Mono-devel-list@lists.ximian.com> http://lists.ximian.com/mailman/listinfo/mono-devel-list _______________________________________________ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com<mailto:Mono-devel-list@lists.ximian.com> http://lists.ximian.com/mailman/listinfo/mono-devel-list
_______________________________________________ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
