Re: [Mono-dev] Using valgrind with Mono
Hi, Fixed the last one in: https://github.com/mono/mono/commit/b97ac0023256bf7d915552f5f24a7742b28c32b7 The first two are leaks, but they should be small and bounded. Will look into fixing them to decrease the noise. Zoltan On Sun, Mar 27, 2016 at 6:23 PM, Zinkevicius, Matt wrote: > Hello, > > > > Our backend service running on Mono 4.2.2 on Linux is experiencing an > unmanaged memory leak. When running our stress tests for several hours, we > see the managed heap sit around 50 MB, while private memory keeps growing > until the process is killed because of OOM. I am therefore attempting to > use valgrind to find the culprit, but I am getting so many leaks detected > that I think many must be false positives, so I thought I would ask here > for some guidance about which are safe to suppress or any other valgrind + > mono tricks you can share. > > > > The vast majority of leaks reported have call stacks that closely match > one of the following: > > > > ==16846== 4 bytes in 1 blocks are definitely lost in loss record 74 of > 19,903 > > ==16846==at 0x4C26FEF: calloc (vg_replace_malloc.c:711) > > ==16846==by 0x62D1D9: monoeg_malloc0 (in /usr/bin/mono-sgen) > > ==16846==by 0x4870F2: decode_exception_debug_info (in > /usr/bin/mono-sgen) > > ==16846==by 0x488975: mono_aot_find_jit_info (in /usr/bin/mono-sgen) > > ==16846==by 0x53C3A7: mono_jit_info_table_find_internal (in > /usr/bin/mono-sgen) > > ==16846==by 0x493C04: mini_jit_info_table_find_ext (in > /usr/bin/mono-sgen) > > ==16846==by 0x4988FB: mini_add_method_trampoline (in > /usr/bin/mono-sgen) > > ==16846==by 0x499067: common_call_trampoline_inner (in > /usr/bin/mono-sgen) > > ==16846==by 0x403217C: ??? > > ==16846==by 0x10D3FB63: ??? > > ==16846==by 0x10D3F41B: ??? > > ==16846==by 0x10D3F117: ??? > > > > ==16846== 12 bytes in 1 blocks are definitely lost in loss record 1,172 of > 19,903 > > ==16846==at 0x4C2828A: malloc (vg_replace_malloc.c:299) > > ==16846==by 0x62D221: monoeg_malloc (in /usr/bin/mono-sgen) > > ==16846==by 0x55B8EF: mono_metadata_type_dup (in /usr/bin/mono-sgen) > > ==16846==by 0x49FC4B: get_shared_gparam (in /usr/bin/mono-sgen) > > ==16846==by 0x49FE70: get_shared_inst (in /usr/bin/mono-sgen) > > ==16846==by 0x4A073A: mini_get_shared_method_full (in > /usr/bin/mono-sgen) > > ==16846==by 0x414723: lookup_method (in /usr/bin/mono-sgen) > > ==16846==by 0x4147FA: mono_jit_compile_method_with_opt (in > /usr/bin/mono-sgen) > > ==16846==by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) > > ==16846==by 0x498DA4: common_call_trampoline_inner (in > /usr/bin/mono-sgen) > > ==16846==by 0x403405C: ??? > > ==16846==by 0x10D2DCA7: ??? > > > > ==16846== 10 bytes in 1 blocks are definitely lost in loss record 739 of > 19,903 > > ==16846==at 0x4C2828A: malloc (vg_replace_malloc.c:299) > > ==16846==by 0x62D221: monoeg_malloc (in /usr/bin/mono-sgen) > > ==16846==by 0x62BA8C: monoeg_g_utf16_to_utf8 (in /usr/bin/mono-sgen) > > ==16846==by 0x5A8646: mono_string_to_utf8_checked (in > /usr/bin/mono-sgen) > > ==16846==by 0x5A885B: mono_string_to_utf8 (in /usr/bin/mono-sgen) > > ==16846==by 0x52DE3C: ves_icall_Type_GetNestedTypes (in > /usr/bin/mono-sgen) > > ==16846==by 0x120D4256: ??? > > ==16846==by 0xE338A78: > System_Type_GetMember_string_System_Reflection_BindingFlags (type.cs:806) > > ==16846==by 0x40C09EF: ??? > > ==16846==by 0x1259A6AF: ??? > > ==16846==by 0x73: ??? > > ==16846==by 0x141D191D: ??? > > > > Are these valid leaks or is valgrind confused/misconfigured? I am using > the following command: > > valgrind --tool=memcheck -v --leak-check=full --log-file=val.txt > --smc-check=all mono program.exe > > > > Thanks for any input you can offer, > > Matt Zinkevicius > > > > > > ___ > Mono-devel-list mailing list > Mono-devel-list@lists.ximian.com > http://lists.ximian.com/mailman/listinfo/mono-devel-list > > ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Using valgrind with Mono
Hi Zoltan, Thank you for investigating my findings. I will apply the fix you linked to and report back. While I understand the other leaks are small, valgrind reported over 19,000 instances of leaked memory in under 5 minutes of our app running without load. Matt Zinkevicius From: Zoltan Varga [mailto:var...@gmail.com] Sent: Sunday, March 27, 2016 4:45 PM To: Zinkevicius, Matt Cc: mono-devel-list@lists.ximian.com; Straw, David (Storage) Subject: Re: [Mono-dev] Using valgrind with Mono Hi, Fixed the last one in: https://github.com/mono/mono/commit/b97ac0023256bf7d915552f5f24a7742b28c32b7 The first two are leaks, but they should be small and bounded. Will look into fixing them to decrease the noise. Zoltan On Sun, Mar 27, 2016 at 6:23 PM, Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> wrote: Hello, Our backend service running on Mono 4.2.2 on Linux is experiencing an unmanaged memory leak. When running our stress tests for several hours, we see the managed heap sit around 50 MB, while private memory keeps growing until the process is killed because of OOM. I am therefore attempting to use valgrind to find the culprit, but I am getting so many leaks detected that I think many must be false positives, so I thought I would ask here for some guidance about which are safe to suppress or any other valgrind + mono tricks you can share. The vast majority of leaks reported have call stacks that closely match one of the following: ==16846== 4 bytes in 1 blocks are definitely lost in loss record 74 of 19,903 ==16846==at 0x4C26FEF: calloc (vg_replace_malloc.c:711) ==16846==by 0x62D1D9: monoeg_malloc0 (in /usr/bin/mono-sgen) ==16846==by 0x4870F2: decode_exception_debug_info (in /usr/bin/mono-sgen) ==16846==by 0x488975: mono_aot_find_jit_info (in /usr/bin/mono-sgen) ==16846==by 0x53C3A7: mono_jit_info_table_find_internal (in /usr/bin/mono-sgen) ==16846==by 0x493C04: mini_jit_info_table_find_ext (in /usr/bin/mono-sgen) ==16846==by 0x4988FB: mini_add_method_trampoline (in /usr/bin/mono-sgen) ==16846==by 0x499067: common_call_trampoline_inner (in /usr/bin/mono-sgen) ==16846==by 0x403217C: ??? ==16846==by 0x10D3FB63: ??? ==16846==by 0x10D3F41B: ??? ==16846==by 0x10D3F117: ??? ==16846== 12 bytes in 1 blocks are definitely lost in loss record 1,172 of 19,903 ==16846==at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==16846==by 0x62D221: monoeg_malloc (in /usr/bin/mono-sgen) ==16846==by 0x55B8EF: mono_metadata_type_dup (in /usr/bin/mono-sgen) ==16846==by 0x49FC4B: get_shared_gparam (in /usr/bin/mono-sgen) ==16846==by 0x49FE70: get_shared_inst (in /usr/bin/mono-sgen) ==16846==by 0x4A073A: mini_get_shared_method_full (in /usr/bin/mono-sgen) ==16846==by 0x414723: lookup_method (in /usr/bin/mono-sgen) ==16846==by 0x4147FA: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==16846==by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==16846==by 0x498DA4: common_call_trampoline_inner (in /usr/bin/mono-sgen) ==16846==by 0x403405C: ??? ==16846==by 0x10D2DCA7: ??? ==16846== 10 bytes in 1 blocks are definitely lost in loss record 739 of 19,903 ==16846==at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==16846==by 0x62D221: monoeg_malloc (in /usr/bin/mono-sgen) ==16846==by 0x62BA8C: monoeg_g_utf16_to_utf8 (in /usr/bin/mono-sgen) ==16846==by 0x5A8646: mono_string_to_utf8_checked (in /usr/bin/mono-sgen) ==16846==by 0x5A885B: mono_string_to_utf8 (in /usr/bin/mono-sgen) ==16846==by 0x52DE3C: ves_icall_Type_GetNestedTypes (in /usr/bin/mono-sgen) ==16846==by 0x120D4256: ??? ==16846==by 0xE338A78: System_Type_GetMember_string_System_Reflection_BindingFlags (type.cs:806) ==16846==by 0x40C09EF: ??? ==16846==by 0x1259A6AF: ??? ==16846==by 0x73: ??? ==16846==by 0x141D191D: ??? Are these valid leaks or is valgrind confused/misconfigured? I am using the following command: valgrind --tool=memcheck -v --leak-check=full --log-file=val.txt --smc-check=all mono program.exe Thanks for any input you can offer, Matt Zinkevicius ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com<mailto:Mono-devel-list@lists.ximian.com> http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Using valgrind with Mono
I can confirm that Zoltan’s fix does indeed remove any leak instances with a call stack containing “ves_icall_Type_GetNestedTypes”, but unfortunately that accounted for only 36 leak instances out of the 19,900 reported. I have filed a bug (https://bugzilla.xamarin.com/show_bug.cgi?id=39940) and attached the latest valgrind output there. Thanks again for any help you can provide, Matt Zinkevicius From: mono-devel-list-boun...@lists.ximian.com [mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Zinkevicius, Matt Sent: Sunday, March 27, 2016 11:06 PM To: Zoltan Varga Cc: Straw, David (Storage) ; mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] Using valgrind with Mono Hi Zoltan, Thank you for investigating my findings. I will apply the fix you linked to and report back. While I understand the other leaks are small, valgrind reported over 19,000 instances of leaked memory in under 5 minutes of our app running without load. Matt Zinkevicius From: Zoltan Varga [mailto:var...@gmail.com] Sent: Sunday, March 27, 2016 4:45 PM To: Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> Cc: mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com>; Straw, David (Storage) mailto:david.st...@hpe.com>> Subject: Re: [Mono-dev] Using valgrind with Mono Hi, Fixed the last one in: https://github.com/mono/mono/commit/b97ac0023256bf7d915552f5f24a7742b28c32b7 The first two are leaks, but they should be small and bounded. Will look into fixing them to decrease the noise. Zoltan On Sun, Mar 27, 2016 at 6:23 PM, Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> wrote: Hello, Our backend service running on Mono 4.2.2 on Linux is experiencing an unmanaged memory leak. When running our stress tests for several hours, we see the managed heap sit around 50 MB, while private memory keeps growing until the process is killed because of OOM. I am therefore attempting to use valgrind to find the culprit, but I am getting so many leaks detected that I think many must be false positives, so I thought I would ask here for some guidance about which are safe to suppress or any other valgrind + mono tricks you can share. The vast majority of leaks reported have call stacks that closely match one of the following: ==16846== 4 bytes in 1 blocks are definitely lost in loss record 74 of 19,903 ==16846==at 0x4C26FEF: calloc (vg_replace_malloc.c:711) ==16846==by 0x62D1D9: monoeg_malloc0 (in /usr/bin/mono-sgen) ==16846==by 0x4870F2: decode_exception_debug_info (in /usr/bin/mono-sgen) ==16846==by 0x488975: mono_aot_find_jit_info (in /usr/bin/mono-sgen) ==16846==by 0x53C3A7: mono_jit_info_table_find_internal (in /usr/bin/mono-sgen) ==16846==by 0x493C04: mini_jit_info_table_find_ext (in /usr/bin/mono-sgen) ==16846==by 0x4988FB: mini_add_method_trampoline (in /usr/bin/mono-sgen) ==16846==by 0x499067: common_call_trampoline_inner (in /usr/bin/mono-sgen) ==16846==by 0x403217C: ??? ==16846==by 0x10D3FB63: ??? ==16846==by 0x10D3F41B: ??? ==16846==by 0x10D3F117: ??? ==16846== 12 bytes in 1 blocks are definitely lost in loss record 1,172 of 19,903 ==16846==at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==16846==by 0x62D221: monoeg_malloc (in /usr/bin/mono-sgen) ==16846==by 0x55B8EF: mono_metadata_type_dup (in /usr/bin/mono-sgen) ==16846==by 0x49FC4B: get_shared_gparam (in /usr/bin/mono-sgen) ==16846==by 0x49FE70: get_shared_inst (in /usr/bin/mono-sgen) ==16846==by 0x4A073A: mini_get_shared_method_full (in /usr/bin/mono-sgen) ==16846==by 0x414723: lookup_method (in /usr/bin/mono-sgen) ==16846==by 0x4147FA: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==16846==by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==16846==by 0x498DA4: common_call_trampoline_inner (in /usr/bin/mono-sgen) ==16846==by 0x403405C: ??? ==16846==by 0x10D2DCA7: ??? ==16846== 10 bytes in 1 blocks are definitely lost in loss record 739 of 19,903 ==16846==at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==16846==by 0x62D221: monoeg_malloc (in /usr/bin/mono-sgen) ==16846==by 0x62BA8C: monoeg_g_utf16_to_utf8 (in /usr/bin/mono-sgen) ==16846==by 0x5A8646: mono_string_to_utf8_checked (in /usr/bin/mono-sgen) ==16846==by 0x5A885B: mono_string_to_utf8 (in /usr/bin/mono-sgen) ==16846==by 0x52DE3C: ves_icall_Type_GetNestedTypes (in /usr/bin/mono-sgen) ==16846==by 0x120D4256: ??? ==16846==by 0xE338A78: System_Type_GetMember_string_System_Reflection_BindingFlags (type.cs:806) ==16846==by 0x40C09EF: ??? ==16846==by 0x1259A6AF: ??? ==16846==by 0x73: ??? ==16846==by 0x141D191D: ??? Are these valid leaks or is valgrind confused/misconfigured? I am using the following command: valgrind --tool=memcheck -v --leak-check=full --log-file=val.txt --smc-check=all mono program.exe Thanks for
Re: [Mono-dev] Using valgrind with Mono
The memory leak on mono_method_get_header was recently fixed in master. On Mon, Mar 28, 2016 at 3:31 AM, Zinkevicius, Matt wrote: > I can confirm that Zoltan’s fix does indeed remove any leak instances with > a call stack containing “ves_icall_Type_GetNestedTypes”, but unfortunately > that accounted for only 36 leak instances out of the 19,900 reported. > > > > I have filed a bug (https://bugzilla.xamarin.com/show_bug.cgi?id=39940) > and attached the latest valgrind output there. > > > > Thanks again for any help you can provide, > > Matt Zinkevicius > > > > *From:* mono-devel-list-boun...@lists.ximian.com [mailto: > mono-devel-list-boun...@lists.ximian.com] *On Behalf Of *Zinkevicius, Matt > *Sent:* Sunday, March 27, 2016 11:06 PM > *To:* Zoltan Varga > *Cc:* Straw, David (Storage) ; > mono-devel-list@lists.ximian.com > > *Subject:* Re: [Mono-dev] Using valgrind with Mono > > > > Hi Zoltan, > > > > Thank you for investigating my findings. I will apply the fix you linked > to and report back. While I understand the other leaks are small, valgrind > reported over 19,000 instances of leaked memory in under 5 minutes of our > app running without load. > > > > Matt Zinkevicius > > > > *From:* Zoltan Varga [mailto:var...@gmail.com ] > *Sent:* Sunday, March 27, 2016 4:45 PM > *To:* Zinkevicius, Matt > *Cc:* mono-devel-list@lists.ximian.com; Straw, David (Storage) < > david.st...@hpe.com> > *Subject:* Re: [Mono-dev] Using valgrind with Mono > > > > Hi, > > > > Fixed the last one in: > > > https://github.com/mono/mono/commit/b97ac0023256bf7d915552f5f24a7742b28c32b7 > > > > The first two are leaks, but they should be small and bounded. Will look > into fixing them to decrease the noise. > > > > Zoltan > > > > On Sun, Mar 27, 2016 at 6:23 PM, Zinkevicius, Matt < > matt.zinkevic...@hpe.com> wrote: > > Hello, > > > > Our backend service running on Mono 4.2.2 on Linux is experiencing an > unmanaged memory leak. When running our stress tests for several hours, we > see the managed heap sit around 50 MB, while private memory keeps growing > until the process is killed because of OOM. I am therefore attempting to > use valgrind to find the culprit, but I am getting so many leaks detected > that I think many must be false positives, so I thought I would ask here > for some guidance about which are safe to suppress or any other valgrind + > mono tricks you can share. > > > > The vast majority of leaks reported have call stacks that closely match > one of the following: > > > > ==16846== 4 bytes in 1 blocks are definitely lost in loss record 74 of > 19,903 > > ==16846==at 0x4C26FEF: calloc (vg_replace_malloc.c:711) > > ==16846==by 0x62D1D9: monoeg_malloc0 (in /usr/bin/mono-sgen) > > ==16846==by 0x4870F2: decode_exception_debug_info (in > /usr/bin/mono-sgen) > > ==16846==by 0x488975: mono_aot_find_jit_info (in /usr/bin/mono-sgen) > > ==16846==by 0x53C3A7: mono_jit_info_table_find_internal (in > /usr/bin/mono-sgen) > > ==16846==by 0x493C04: mini_jit_info_table_find_ext (in > /usr/bin/mono-sgen) > > ==16846==by 0x4988FB: mini_add_method_trampoline (in > /usr/bin/mono-sgen) > > ==16846==by 0x499067: common_call_trampoline_inner (in > /usr/bin/mono-sgen) > > ==16846==by 0x403217C: ??? > > ==16846==by 0x10D3FB63: ??? > > ==16846==by 0x10D3F41B: ??? > > ==16846==by 0x10D3F117: ??? > > > > ==16846== 12 bytes in 1 blocks are definitely lost in loss record 1,172 of > 19,903 > > ==16846==at 0x4C2828A: malloc (vg_replace_malloc.c:299) > > ==16846==by 0x62D221: monoeg_malloc (in /usr/bin/mono-sgen) > > ==16846==by 0x55B8EF: mono_metadata_type_dup (in /usr/bin/mono-sgen) > > ==16846==by 0x49FC4B: get_shared_gparam (in /usr/bin/mono-sgen) > > ==16846==by 0x49FE70: get_shared_inst (in /usr/bin/mono-sgen) > > ==16846==by 0x4A073A: mini_get_shared_method_full (in > /usr/bin/mono-sgen) > > ==16846==by 0x414723: lookup_method (in /usr/bin/mono-sgen) > > ==16846==by 0x4147FA: mono_jit_compile_method_with_opt (in > /usr/bin/mono-sgen) > > ==16846==by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) > > ==16846==by 0x498DA4: common_call_trampoline_inner (in > /usr/bin/mono-sgen) > > ==16846==by 0x403405C: ??? > > ==16846==by 0x10D2DCA7: ??? > > > > ==16846== 10 bytes in 1 blocks are definitely lost in loss record 739 of > 19,903 > > ==16846==at 0x4C2828A: malloc (vg_replace_malloc.c:299) > > ==16846==by 0x62D221:
Re: [Mono-dev] Using valgrind with Mono
Hi Rodrigo, I am not finding a recent commit in master that mentions “mono_method_get_header”. Do you happen to have the commit ID or log message handy? Sorry for the bother, Matt From: Rodrigo Kumpera [mailto:kump...@gmail.com] Sent: Monday, March 28, 2016 1:58 PM To: Zinkevicius, Matt Cc: Zoltan Varga ; Straw, David (Storage) ; mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] Using valgrind with Mono The memory leak on mono_method_get_header was recently fixed in master. On Mon, Mar 28, 2016 at 3:31 AM, Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> wrote: I can confirm that Zoltan’s fix does indeed remove any leak instances with a call stack containing “ves_icall_Type_GetNestedTypes”, but unfortunately that accounted for only 36 leak instances out of the 19,900 reported. I have filed a bug (https://bugzilla.xamarin.com/show_bug.cgi?id=39940) and attached the latest valgrind output there. Thanks again for any help you can provide, Matt Zinkevicius From: mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com> [mailto:mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com>] On Behalf Of Zinkevicius, Matt Sent: Sunday, March 27, 2016 11:06 PM To: Zoltan Varga mailto:var...@gmail.com>> Cc: Straw, David (Storage) mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono Hi Zoltan, Thank you for investigating my findings. I will apply the fix you linked to and report back. While I understand the other leaks are small, valgrind reported over 19,000 instances of leaked memory in under 5 minutes of our app running without load. Matt Zinkevicius From: Zoltan Varga [mailto:var...@gmail.com] Sent: Sunday, March 27, 2016 4:45 PM To: Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> Cc: mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com>; Straw, David (Storage) mailto:david.st...@hpe.com>> Subject: Re: [Mono-dev] Using valgrind with Mono Hi, Fixed the last one in: https://github.com/mono/mono/commit/b97ac0023256bf7d915552f5f24a7742b28c32b7 The first two are leaks, but they should be small and bounded. Will look into fixing them to decrease the noise. Zoltan On Sun, Mar 27, 2016 at 6:23 PM, Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> wrote: Hello, Our backend service running on Mono 4.2.2 on Linux is experiencing an unmanaged memory leak. When running our stress tests for several hours, we see the managed heap sit around 50 MB, while private memory keeps growing until the process is killed because of OOM. I am therefore attempting to use valgrind to find the culprit, but I am getting so many leaks detected that I think many must be false positives, so I thought I would ask here for some guidance about which are safe to suppress or any other valgrind + mono tricks you can share. The vast majority of leaks reported have call stacks that closely match one of the following: ==16846== 4 bytes in 1 blocks are definitely lost in loss record 74 of 19,903 ==16846==at 0x4C26FEF: calloc (vg_replace_malloc.c:711) ==16846==by 0x62D1D9: monoeg_malloc0 (in /usr/bin/mono-sgen) ==16846==by 0x4870F2: decode_exception_debug_info (in /usr/bin/mono-sgen) ==16846==by 0x488975: mono_aot_find_jit_info (in /usr/bin/mono-sgen) ==16846==by 0x53C3A7: mono_jit_info_table_find_internal (in /usr/bin/mono-sgen) ==16846==by 0x493C04: mini_jit_info_table_find_ext (in /usr/bin/mono-sgen) ==16846==by 0x4988FB: mini_add_method_trampoline (in /usr/bin/mono-sgen) ==16846==by 0x499067: common_call_trampoline_inner (in /usr/bin/mono-sgen) ==16846==by 0x403217C: ??? ==16846==by 0x10D3FB63: ??? ==16846==by 0x10D3F41B: ??? ==16846==by 0x10D3F117: ??? ==16846== 12 bytes in 1 blocks are definitely lost in loss record 1,172 of 19,903 ==16846==at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==16846==by 0x62D221: monoeg_malloc (in /usr/bin/mono-sgen) ==16846==by 0x55B8EF: mono_metadata_type_dup (in /usr/bin/mono-sgen) ==16846==by 0x49FC4B: get_shared_gparam (in /usr/bin/mono-sgen) ==16846==by 0x49FE70: get_shared_inst (in /usr/bin/mono-sgen) ==16846==by 0x4A073A: mini_get_shared_method_full (in /usr/bin/mono-sgen) ==16846==by 0x414723: lookup_method (in /usr/bin/mono-sgen) ==16846==by 0x4147FA: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==16846==by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==16846==by 0x498DA4: common_call_trampoline_inner (in /usr/bin/mono-sgen) ==16846==by 0x403405C: ??? ==16846==by 0x10D2DCA7: ??? ==16846== 10 bytes in 1 blocks are definitely lost in loss record 739 of 19,903 ==16846==at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==16846==by 0x62
Re: [Mono-dev] Using valgrind with Mono
Hi Rodrigo, Is the following the PR that you were referring to that fixes the “mono_method_get_header” leak? https://github.com/mono/mono/pull/2705 I am eager to find it, as this particular leak seems to be the most egregious of the ones we’re experiencing, in terms of both occurrences and total bytes leaked (hundreds of MB/hour when under load). If so, I’ll try to backport it to Mono 4.2.3, though it is far from patching cleanly currently. Matt From: mono-devel-list-boun...@lists.ximian.com [mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Zinkevicius, Matt Sent: Monday, March 28, 2016 2:42 PM To: Rodrigo Kumpera Cc: Straw, David (Storage) ; mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] Using valgrind with Mono Hi Rodrigo, I am not finding a recent commit in master that mentions “mono_method_get_header”. Do you happen to have the commit ID or log message handy? Sorry for the bother, Matt From: Rodrigo Kumpera [mailto:kump...@gmail.com] Sent: Monday, March 28, 2016 1:58 PM To: Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> Cc: Zoltan Varga mailto:var...@gmail.com>>; Straw, David (Storage) mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono The memory leak on mono_method_get_header was recently fixed in master. On Mon, Mar 28, 2016 at 3:31 AM, Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> wrote: I can confirm that Zoltan’s fix does indeed remove any leak instances with a call stack containing “ves_icall_Type_GetNestedTypes”, but unfortunately that accounted for only 36 leak instances out of the 19,900 reported. I have filed a bug (https://bugzilla.xamarin.com/show_bug.cgi?id=39940) and attached the latest valgrind output there. Thanks again for any help you can provide, Matt Zinkevicius From: mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com> [mailto:mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com>] On Behalf Of Zinkevicius, Matt Sent: Sunday, March 27, 2016 11:06 PM To: Zoltan Varga mailto:var...@gmail.com>> Cc: Straw, David (Storage) mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono Hi Zoltan, Thank you for investigating my findings. I will apply the fix you linked to and report back. While I understand the other leaks are small, valgrind reported over 19,000 instances of leaked memory in under 5 minutes of our app running without load. Matt Zinkevicius From: Zoltan Varga [mailto:var...@gmail.com] Sent: Sunday, March 27, 2016 4:45 PM To: Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> Cc: mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com>; Straw, David (Storage) mailto:david.st...@hpe.com>> Subject: Re: [Mono-dev] Using valgrind with Mono Hi, Fixed the last one in: https://github.com/mono/mono/commit/b97ac0023256bf7d915552f5f24a7742b28c32b7 The first two are leaks, but they should be small and bounded. Will look into fixing them to decrease the noise. Zoltan On Sun, Mar 27, 2016 at 6:23 PM, Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> wrote: Hello, Our backend service running on Mono 4.2.2 on Linux is experiencing an unmanaged memory leak. When running our stress tests for several hours, we see the managed heap sit around 50 MB, while private memory keeps growing until the process is killed because of OOM. I am therefore attempting to use valgrind to find the culprit, but I am getting so many leaks detected that I think many must be false positives, so I thought I would ask here for some guidance about which are safe to suppress or any other valgrind + mono tricks you can share. The vast majority of leaks reported have call stacks that closely match one of the following: ==16846== 4 bytes in 1 blocks are definitely lost in loss record 74 of 19,903 ==16846==at 0x4C26FEF: calloc (vg_replace_malloc.c:711) ==16846==by 0x62D1D9: monoeg_malloc0 (in /usr/bin/mono-sgen) ==16846==by 0x4870F2: decode_exception_debug_info (in /usr/bin/mono-sgen) ==16846==by 0x488975: mono_aot_find_jit_info (in /usr/bin/mono-sgen) ==16846==by 0x53C3A7: mono_jit_info_table_find_internal (in /usr/bin/mono-sgen) ==16846==by 0x493C04: mini_jit_info_table_find_ext (in /usr/bin/mono-sgen) ==16846==by 0x4988FB: mini_add_method_trampoline (in /usr/bin/mono-sgen) ==16846==by 0x499067: common_call_trampoline_inner (in /usr/bin/mono-sgen) ==16846==by 0x403217C: ??? ==16846==by 0x10D3FB63: ??? ==16846==by 0x10D3F41B: ??? ==16846==by 0x10D3F117: ??? ==16846== 12 bytes in 1 blocks are definitely lost in loss record 1,172 of 19,903 ==16846==at 0x4C2828A: malloc (vg_r
Re: [Mono-dev] Using valgrind with Mono
This is the PR in question: https://github.com/mono/mono/pull/2783 It probably won't make into 4.2, but should definitely be in 4.4. On Tue, Mar 29, 2016 at 11:55 AM, Zinkevicius, Matt < matt.zinkevic...@hpe.com> wrote: > Hi Rodrigo, > > > > Is the following the PR that you were referring to that fixes the > “mono_method_get_header” leak? > > > > https://github.com/mono/mono/pull/2705 > > > > I am eager to find it, as this particular leak seems to be the most > egregious of the ones we’re experiencing, in terms of both occurrences and > total bytes leaked (hundreds of MB/hour when under load). > > > > If so, I’ll try to backport it to Mono 4.2.3, though it is far from > patching cleanly currently. > > > > Matt > > > > *From:* mono-devel-list-boun...@lists.ximian.com [mailto: > mono-devel-list-boun...@lists.ximian.com] *On Behalf Of *Zinkevicius, Matt > *Sent:* Monday, March 28, 2016 2:42 PM > *To:* Rodrigo Kumpera > > *Cc:* Straw, David (Storage) ; > mono-devel-list@lists.ximian.com > *Subject:* Re: [Mono-dev] Using valgrind with Mono > > > > Hi Rodrigo, > > > > I am not finding a recent commit in master that mentions > “mono_method_get_header”. Do you happen to have the commit ID or log > message handy? > > > > Sorry for the bother, > > Matt > > > > *From:* Rodrigo Kumpera [mailto:kump...@gmail.com ] > *Sent:* Monday, March 28, 2016 1:58 PM > *To:* Zinkevicius, Matt > *Cc:* Zoltan Varga ; Straw, David (Storage) < > david.st...@hpe.com>; mono-devel-list@lists.ximian.com > *Subject:* Re: [Mono-dev] Using valgrind with Mono > > > > The memory leak on mono_method_get_header was recently fixed in master. > > > > > > > > On Mon, Mar 28, 2016 at 3:31 AM, Zinkevicius, Matt < > matt.zinkevic...@hpe.com> wrote: > > I can confirm that Zoltan’s fix does indeed remove any leak instances with > a call stack containing “ves_icall_Type_GetNestedTypes”, but unfortunately > that accounted for only 36 leak instances out of the 19,900 reported. > > > > I have filed a bug (https://bugzilla.xamarin.com/show_bug.cgi?id=39940) > and attached the latest valgrind output there. > > > > Thanks again for any help you can provide, > > Matt Zinkevicius > > > > *From:* mono-devel-list-boun...@lists.ximian.com [mailto: > mono-devel-list-boun...@lists.ximian.com] *On Behalf Of *Zinkevicius, Matt > *Sent:* Sunday, March 27, 2016 11:06 PM > *To:* Zoltan Varga > *Cc:* Straw, David (Storage) ; > mono-devel-list@lists.ximian.com > > > *Subject:* Re: [Mono-dev] Using valgrind with Mono > > > > Hi Zoltan, > > > > Thank you for investigating my findings. I will apply the fix you linked > to and report back. While I understand the other leaks are small, valgrind > reported over 19,000 instances of leaked memory in under 5 minutes of our > app running without load. > > > > Matt Zinkevicius > > > > *From:* Zoltan Varga [mailto:var...@gmail.com ] > *Sent:* Sunday, March 27, 2016 4:45 PM > *To:* Zinkevicius, Matt > *Cc:* mono-devel-list@lists.ximian.com; Straw, David (Storage) < > david.st...@hpe.com> > *Subject:* Re: [Mono-dev] Using valgrind with Mono > > > > Hi, > > > > Fixed the last one in: > > > https://github.com/mono/mono/commit/b97ac0023256bf7d915552f5f24a7742b28c32b7 > > > > The first two are leaks, but they should be small and bounded. Will look > into fixing them to decrease the noise. > > > > Zoltan > > > > On Sun, Mar 27, 2016 at 6:23 PM, Zinkevicius, Matt < > matt.zinkevic...@hpe.com> wrote: > > Hello, > > > > Our backend service running on Mono 4.2.2 on Linux is experiencing an > unmanaged memory leak. When running our stress tests for several hours, we > see the managed heap sit around 50 MB, while private memory keeps growing > until the process is killed because of OOM. I am therefore attempting to > use valgrind to find the culprit, but I am getting so many leaks detected > that I think many must be false positives, so I thought I would ask here > for some guidance about which are safe to suppress or any other valgrind + > mono tricks you can share. > > > > The vast majority of leaks reported have call stacks that closely match > one of the following: > > > > ==16846== 4 bytes in 1 blocks are definitely lost in loss record 74 of > 19,903 > > ==16846==at 0x4C26FEF: calloc (vg_replace_malloc.c:711) > > ==16846==by 0x62D1D9: monoeg_malloc0 (in /usr/bin/mono-sgen) > > ==16846==by 0x4870F2: decode_exception_debug_info (in > /u
Re: [Mono-dev] Using valgrind with Mono
Thanks, Rodrigo! I’ve ported this to 4.2 to test, though these changes seem to only address AOT, and we’re seeing this leak using the normal JIT runtime. Matt From: Rodrigo Kumpera [mailto:kump...@gmail.com] Sent: Tuesday, March 29, 2016 6:15 PM To: Zinkevicius, Matt Cc: Straw, David (Storage) ; mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] Using valgrind with Mono This is the PR in question: https://github.com/mono/mono/pull/2783 It probably won't make into 4.2, but should definitely be in 4.4. On Tue, Mar 29, 2016 at 11:55 AM, Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> wrote: Hi Rodrigo, Is the following the PR that you were referring to that fixes the “mono_method_get_header” leak? https://github.com/mono/mono/pull/2705 I am eager to find it, as this particular leak seems to be the most egregious of the ones we’re experiencing, in terms of both occurrences and total bytes leaked (hundreds of MB/hour when under load). If so, I’ll try to backport it to Mono 4.2.3, though it is far from patching cleanly currently. Matt From: mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com> [mailto:mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com>] On Behalf Of Zinkevicius, Matt Sent: Monday, March 28, 2016 2:42 PM To: Rodrigo Kumpera mailto:kump...@gmail.com>> Cc: Straw, David (Storage) mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono Hi Rodrigo, I am not finding a recent commit in master that mentions “mono_method_get_header”. Do you happen to have the commit ID or log message handy? Sorry for the bother, Matt From: Rodrigo Kumpera [mailto:kump...@gmail.com] Sent: Monday, March 28, 2016 1:58 PM To: Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> Cc: Zoltan Varga mailto:var...@gmail.com>>; Straw, David (Storage) mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono The memory leak on mono_method_get_header was recently fixed in master. On Mon, Mar 28, 2016 at 3:31 AM, Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> wrote: I can confirm that Zoltan’s fix does indeed remove any leak instances with a call stack containing “ves_icall_Type_GetNestedTypes”, but unfortunately that accounted for only 36 leak instances out of the 19,900 reported. I have filed a bug (https://bugzilla.xamarin.com/show_bug.cgi?id=39940) and attached the latest valgrind output there. Thanks again for any help you can provide, Matt Zinkevicius From: mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com> [mailto:mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com>] On Behalf Of Zinkevicius, Matt Sent: Sunday, March 27, 2016 11:06 PM To: Zoltan Varga mailto:var...@gmail.com>> Cc: Straw, David (Storage) mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono Hi Zoltan, Thank you for investigating my findings. I will apply the fix you linked to and report back. While I understand the other leaks are small, valgrind reported over 19,000 instances of leaked memory in under 5 minutes of our app running without load. Matt Zinkevicius From: Zoltan Varga [mailto:var...@gmail.com] Sent: Sunday, March 27, 2016 4:45 PM To: Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> Cc: mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com>; Straw, David (Storage) mailto:david.st...@hpe.com>> Subject: Re: [Mono-dev] Using valgrind with Mono Hi, Fixed the last one in: https://github.com/mono/mono/commit/b97ac0023256bf7d915552f5f24a7742b28c32b7 The first two are leaks, but they should be small and bounded. Will look into fixing them to decrease the noise. Zoltan On Sun, Mar 27, 2016 at 6:23 PM, Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> wrote: Hello, Our backend service running on Mono 4.2.2 on Linux is experiencing an unmanaged memory leak. When running our stress tests for several hours, we see the managed heap sit around 50 MB, while private memory keeps growing until the process is killed because of OOM. I am therefore attempting to use valgrind to find the culprit, but I am getting so many leaks detected that I think many must be false positives, so I thought I would ask here for some guidance about which are safe to suppress or any other valgrind + mono tricks you can share. The vast majority of leaks reported have call stacks that closely match one of the following: ==16846== 4 bytes in 1 blocks are definitely lost in loss record 74 of 19,903
Re: [Mono-dev] Using valgrind with Mono
Unfortunately, PR 2783 did not have any noticeable effect. We still see thousands of leaks like the following: ==12142== 89,860 (89,704 direct, 156 indirect) bytes in 2,800 blocks are definitely lost in loss record 19,763 of 19,792 ==12142==at 0x4C26FEF: calloc (vg_replace_malloc.c:711) ==12142==by 0x62D269: monoeg_malloc0 (in /usr/bin/mono-sgen) ==12142==by 0x53CA32: mono_method_get_header (in /usr/bin/mono-sgen) ==12142==by 0x56CCEA: mono_basic_block_split (in /usr/bin/mono-sgen) ==12142==by 0x4323B3: mono_method_to_ir (in /usr/bin/mono-sgen) ==12142==by 0x45FC8B: inline_method (in /usr/bin/mono-sgen) ==12142==by 0x44C2F4: mono_method_to_ir (in /usr/bin/mono-sgen) ==12142==by 0x4F94A5: mini_method_compile (in /usr/bin/mono-sgen) ==12142==by 0x4FA738: mono_jit_compile_method_inner (in /usr/bin/mono-sgen) ==12142==by 0x414A01: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==12142==by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==12142==by 0x498E44: common_call_trampoline_inner (in /usr/bin/mono-sgen) Looks like https://github.com/mono/mono/pull/2781/ may address this leak? I’ll attempt to backport it and report back. Matt From: mono-devel-list-boun...@lists.ximian.com [mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Zinkevicius, Matt Sent: Tuesday, March 29, 2016 7:22 PM To: Rodrigo Kumpera Cc: Straw, David (Storage) ; mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] Using valgrind with Mono Thanks, Rodrigo! I’ve ported this to 4.2 to test, though these changes seem to only address AOT, and we’re seeing this leak using the normal JIT runtime. Matt From: Rodrigo Kumpera [mailto:kump...@gmail.com] Sent: Tuesday, March 29, 2016 6:15 PM To: Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> Cc: Straw, David (Storage) mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono This is the PR in question: https://github.com/mono/mono/pull/2783 It probably won't make into 4.2, but should definitely be in 4.4. On Tue, Mar 29, 2016 at 11:55 AM, Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> wrote: Hi Rodrigo, Is the following the PR that you were referring to that fixes the “mono_method_get_header” leak? https://github.com/mono/mono/pull/2705 I am eager to find it, as this particular leak seems to be the most egregious of the ones we’re experiencing, in terms of both occurrences and total bytes leaked (hundreds of MB/hour when under load). If so, I’ll try to backport it to Mono 4.2.3, though it is far from patching cleanly currently. Matt From: mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com> [mailto:mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com>] On Behalf Of Zinkevicius, Matt Sent: Monday, March 28, 2016 2:42 PM To: Rodrigo Kumpera mailto:kump...@gmail.com>> Cc: Straw, David (Storage) mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono Hi Rodrigo, I am not finding a recent commit in master that mentions “mono_method_get_header”. Do you happen to have the commit ID or log message handy? Sorry for the bother, Matt From: Rodrigo Kumpera [mailto:kump...@gmail.com] Sent: Monday, March 28, 2016 1:58 PM To: Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> Cc: Zoltan Varga mailto:var...@gmail.com>>; Straw, David (Storage) mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono The memory leak on mono_method_get_header was recently fixed in master. On Mon, Mar 28, 2016 at 3:31 AM, Zinkevicius, Matt mailto:matt.zinkevic...@hpe.com>> wrote: I can confirm that Zoltan’s fix does indeed remove any leak instances with a call stack containing “ves_icall_Type_GetNestedTypes”, but unfortunately that accounted for only 36 leak instances out of the 19,900 reported. I have filed a bug (https://bugzilla.xamarin.com/show_bug.cgi?id=39940) and attached the latest valgrind output there. Thanks again for any help you can provide, Matt Zinkevicius From: mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com> [mailto:mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com>] On Behalf Of Zinkevicius, Matt Sent: Sunday, March 27, 2016 11:06 PM To: Zoltan Varga mailto:var...@gmail.com>> Cc: Straw, David (Storage) mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono Hi Zoltan, Thank you for investigating my findings. I will app
Re: [Mono-dev] Using valgrind with Mono
I have backported the following fixes from master into 4.2.3: https://github.com/mono/mono/commit/b97ac0023256bf7d915552f5f24a7742b28c32b7 https://github.com/mono/mono/commit/8c52b398c5eb962bba5985e8bc01445ac5f027a5 https://github.com/mono/mono/pull/2781 https://github.com/mono/mono/pull/2783 https://github.com/mono/mono/pull/2785 This has helped tremendously. I am now down to 659 leak occurrences, of which 640 have one of the following signatures: 1) "mono_metadata_type_dup" x 327 occurrences x 12-36 bytes each ==31699== 36 bytes in 3 blocks are definitely lost in loss record 7,355 of 13,872 ==31699==at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==31699==by 0x62D1E1: monoeg_malloc (in /usr/bin/mono-sgen) ==31699==by 0x55B97F: mono_metadata_type_dup (in /usr/bin/mono-sgen) ==31699==by 0x49FD0B: get_shared_gparam (in /usr/bin/mono-sgen) ==31699==by 0x49FF30: get_shared_inst (in /usr/bin/mono-sgen) ==31699==by 0x4A07FA: mini_get_shared_method_full (in /usr/bin/mono-sgen) ==31699==by 0x414723: lookup_method (in /usr/bin/mono-sgen) ==31699==by 0x4147FA: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==31699==by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==31699==by 0x498E64: common_call_trampoline_inner (in /usr/bin/mono-sgen) 2) "mono_method_get_header" x 313 occurrences x 32-192 bytes each Note: PR 2781 brought this down from 5800 occurrences. Would https://github.com/mono/mono/pull/2705 help potentially? ==31699== 32 bytes in 1 blocks are definitely lost in loss record 7,047 of 13,872 ==31699==at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==31699==by 0x62D1E1: monoeg_malloc (in /usr/bin/mono-sgen) ==31699==by 0x62D237: monoeg_g_memdup (in /usr/bin/mono-sgen) ==31699==by 0x53CB47: mono_method_get_header (in /usr/bin/mono-sgen) ==31699==by 0x4F8EA0: mini_method_compile (in /usr/bin/mono-sgen) ==31699==by 0x4FA788: mono_jit_compile_method_inner (in /usr/bin/mono-sgen) ==31699==by 0x414A01: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==31699==by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==31699==by 0x498E64: common_call_trampoline_inner (in /usr/bin/mono-sgen) or ==31699== 192 bytes in 3 blocks are definitely lost in loss record 11,517 of 13,872 ==31699==at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==31699==by 0x62D1E1: monoeg_malloc (in /usr/bin/mono-sgen) ==31699==by 0x62D237: monoeg_g_memdup (in /usr/bin/mono-sgen) ==31699==by 0x53CB47: mono_method_get_header (in /usr/bin/mono-sgen) ==31699==by 0x430691: mono_method_to_ir (in /usr/bin/mono-sgen) ==31699==by 0x4F94F5: mini_method_compile (in /usr/bin/mono-sgen) ==31699==by 0x4FA788: mono_jit_compile_method_inner (in /usr/bin/mono-sgen) ==31699==by 0x414A01: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==31699==by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==31699==by 0x498E64: common_call_trampoline_inner (in /usr/bin/mono-sgen) Getting close! Thanks again for any help anyone can provide, Matt From: mono-devel-list-boun...@lists.ximian.com [mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Zinkevicius, Matt Sent: Tuesday, March 29, 2016 8:26 PM To: Rodrigo Kumpera Cc: Straw, David (Storage) ; mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] Using valgrind with Mono Unfortunately, PR 2783 did not have any noticeable effect. We still see thousands of leaks like the following: ==12142== 89,860 (89,704 direct, 156 indirect) bytes in 2,800 blocks are definitely lost in loss record 19,763 of 19,792 ==12142==at 0x4C26FEF: calloc (vg_replace_malloc.c:711) ==12142==by 0x62D269: monoeg_malloc0 (in /usr/bin/mono-sgen) ==12142==by 0x53CA32: mono_method_get_header (in /usr/bin/mono-sgen) ==12142==by 0x56CCEA: mono_basic_block_split (in /usr/bin/mono-sgen) ==12142==by 0x4323B3: mono_method_to_ir (in /usr/bin/mono-sgen) ==12142==by 0x45FC8B: inline_method (in /usr/bin/mono-sgen) ==12142==by 0x44C2F4: mono_method_to_ir (in /usr/bin/mono-sgen) ==12142==by 0x4F94A5: mini_method_compile (in /usr/bin/mono-sgen) ==12142==by 0x4FA738: mono_jit_compile_method_inner (in /usr/bin/mono-sgen) ==12142==by 0x414A01: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==12142==by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==12142==by 0x498E44: common_call_trampoline_inner (in /usr/bin/mono-sgen) Looks like https://github.com/mono/mono/pull/2781/ may address this leak? I’ll attempt to backport it and report back. Matt From: mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com> [mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Zinkevicius, Matt Sent: Tuesday, March 29, 2016 7:22 PM To: Rodrigo Kumpera mailto:kump...@gmail.com>> Cc: Straw, David (Storage) mailto:david.st...@hpe.com>>; mono-d
Re: [Mono-dev] Using valgrind with Mono
To close on this, most of the valgrind leaks reported have now been plugged in the Mono 4.4 branch. Unfortunately, these leaks turned out to be very small portion of the overall leak we’re experiencing. It turns out that Mono is leaking almost 100 bytes every time a System.Reflection.Emit.DynamicMethod is created, or any method that uses DynamicMethod (Like Expression.DynamicInvoke). Dynamic methods are used at a high-frequency in libraries like NHibernate, and so our service runs out of memory in short order. https://bugzilla.xamarin.com/show_bug.cgi?id=40691 I would be very grateful for any assistance fixing this issue, and am willing to provide any additional info required. --Matt From: mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com> [mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Zinkevicius, Matt Sent: Wednesday, March 30, 2016 2:06 AM To: Rodrigo Kumpera mailto:kump...@gmail.com>> Cc: Straw, David (Storage) mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono I have backported the following fixes from master into 4.2.3: https://github.com/mono/mono/commit/b97ac0023256bf7d915552f5f24a7742b28c32b7 https://github.com/mono/mono/commit/8c52b398c5eb962bba5985e8bc01445ac5f027a5 https://github.com/mono/mono/pull/2781 https://github.com/mono/mono/pull/2783 https://github.com/mono/mono/pull/2785 This has helped tremendously. I am now down to 659 leak occurrences, of which 640 have one of the following signatures: 1) "mono_metadata_type_dup" x 327 occurrences x 12-36 bytes each ==31699== 36 bytes in 3 blocks are definitely lost in loss record 7,355 of 13,872 ==31699==at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==31699==by 0x62D1E1: monoeg_malloc (in /usr/bin/mono-sgen) ==31699==by 0x55B97F: mono_metadata_type_dup (in /usr/bin/mono-sgen) ==31699==by 0x49FD0B: get_shared_gparam (in /usr/bin/mono-sgen) ==31699==by 0x49FF30: get_shared_inst (in /usr/bin/mono-sgen) ==31699==by 0x4A07FA: mini_get_shared_method_full (in /usr/bin/mono-sgen) ==31699==by 0x414723: lookup_method (in /usr/bin/mono-sgen) ==31699==by 0x4147FA: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==31699==by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==31699==by 0x498E64: common_call_trampoline_inner (in /usr/bin/mono-sgen) 2) "mono_method_get_header" x 313 occurrences x 32-192 bytes each Note: PR 2781 brought this down from 5800 occurrences. Would https://github.com/mono/mono/pull/2705 help potentially? ==31699== 32 bytes in 1 blocks are definitely lost in loss record 7,047 of 13,872 ==31699==at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==31699==by 0x62D1E1: monoeg_malloc (in /usr/bin/mono-sgen) ==31699==by 0x62D237: monoeg_g_memdup (in /usr/bin/mono-sgen) ==31699==by 0x53CB47: mono_method_get_header (in /usr/bin/mono-sgen) ==31699==by 0x4F8EA0: mini_method_compile (in /usr/bin/mono-sgen) ==31699==by 0x4FA788: mono_jit_compile_method_inner (in /usr/bin/mono-sgen) ==31699==by 0x414A01: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==31699==by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==31699==by 0x498E64: common_call_trampoline_inner (in /usr/bin/mono-sgen) or ==31699== 192 bytes in 3 blocks are definitely lost in loss record 11,517 of 13,872 ==31699==at 0x4C2828A: malloc (vg_replace_malloc.c:299) ==31699==by 0x62D1E1: monoeg_malloc (in /usr/bin/mono-sgen) ==31699==by 0x62D237: monoeg_g_memdup (in /usr/bin/mono-sgen) ==31699==by 0x53CB47: mono_method_get_header (in /usr/bin/mono-sgen) ==31699==by 0x430691: mono_method_to_ir (in /usr/bin/mono-sgen) ==31699==by 0x4F94F5: mini_method_compile (in /usr/bin/mono-sgen) ==31699==by 0x4FA788: mono_jit_compile_method_inner (in /usr/bin/mono-sgen) ==31699==by 0x414A01: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen) ==31699==by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen) ==31699==by 0x498E64: common_call_trampoline_inner (in /usr/bin/mono-sgen) Getting close! Thanks again for any help anyone can provide, Matt From: mono-devel-list-boun...@lists.ximian.com<mailto:mono-devel-list-boun...@lists.ximian.com> [mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Zinkevicius, Matt Sent: Tuesday, March 29, 2016 8:26 PM To: Rodrigo Kumpera mailto:kump...@gmail.com>> Cc: Straw, David (Storage) mailto:david.st...@hpe.com>>; mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com> Subject: Re: [Mono-dev] Using valgrind with Mono Unfortunately, PR 2783 did not have any noticeable effect. We still see thousands of leaks like the following: ==12142== 89,860 (89,704 direct, 156 indirect) bytes in 2,800 blocks are definitely lo
