[sr-dev] Re: [SR-Users] pkg memory leak when acc module cdr_enabled
Hello, quick remark regarding the multi-thread lock/mutex situation in glibc, in recent version this can be now tuned as well with the MALLOC_ARENA_MAX environment variable. This way glibc will also maintain multiple memory pool, but apparently at the cost of a higher memory consumption during run-time. Cheers, Henning -- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.com -Original Message- From: Daniel-Constantin Mierla Sent: Thursday, January 5, 2023 9:06 PM To: Kamailio (SER) - Development Mailing List ; Alex Balashov ; Henning Westerholt Cc: Kamailio (SER) - Users Mailing List Subject: Re: [sr-dev] Re: [SR-Users] pkg memory leak when acc module cdr_enabled Not the original developer of the memory managers, but, iirc, one of the reasons for private memory manager was to avoid a multi-threading related lock/mutex that is done by system malloc()/free(), which is not necessary for pure multi-process application. I did not analyzed myself, but I guess this is still there if it was a valid assertion in the past. On the other hand, I am not sure of its relevant impact, a lot of modules link with external libs that use anyhow the system allocator. Another thing was (faster?) indexing of free fragments up to 16kb that would enable quick selection of a such fragment that would usually fit in a (classic) SIP request/reply or attributes related to them. But I think this is no much relevant these days, because of the auto-join (merge) of free fragments -- without the latter, there were fragmentation problems. Split of large fragments is quite fast. Then, the ability to add own troubleshooting mechanism, with special structures at the beginning and the end of allocated chunks to track the place from where the chunk was allocated/freed and detect buffer overflows by checking the overwriting of special guard values. This is something that I kind of find still pretty useful for troubleshooting and usage statistics. But this can be implemented as a new memory manager for pkg-only that uses behind directly system malloc()/free(), it is actually in my to-do list for long time, but it didn't materialized yet due to lack of spare time (or from another perspective: lack of funding :-) ). At this moment, if you install from sources, just do: make MEMPKG=sys include_modules="..." ... cfg make all make install and you get kamailio compiled to use system malloc()/free() instead of the custom pkg qm/fm/... manager. I do it when I perform static code analysis, because it enables pkg memory leak detections by those tools. So it should be fully functional, on the other hand, I don't think I run such instance in production at this moment -- there are no pkg usage stats available. For shm, the custom manager has to be kept, it is still complex to work directly with shared memory, even now most of deployments are on linux, no longer common to have sunos, solaris, vax vms, aix, ... like in early 2000s. Cheers, Daniel On 05.01.23 16:57, Alex Balashov wrote: > Thanks, that's educational. > > I assumed the original argument was performance-related. I just wasn't > sure if that was still (sufficiently to be important) true. After all, > lots of SER/OpenSER design decisions in the early 2000s made the most > sense then, like inventing own imperative scripting language. :-) > > -- Alex > >> On Jan 5, 2023, at 10:16 AM, Henning Westerholt wrote: >> >> (adding sr-dev) >> >> Hi Alex, >> >> the memory allocator of glibc was not really efficient regarding the >> particular needs of a SIP server (allocation of many small string objects). >> That has probably improved in the last years, also system performance just >> got much faster. >> >> Other programs/tools also use their own allocators, e.g. Firefox, Rust [1] >> for jemalloc. >> >> There is some (not really well tested) support in the core for using the >> system memory allocator by the #define SYS_MALLOC. >> >> It probably needs to be set in Makefile.defs, also deactivating the other >> PKG memory managers, but did not looked into it right now. >> >> Cheers, >> >> Henning >> >> [1] https://news.ycombinator.com/item?id=8612645 >> >> -- >> Henning Westerholt - https://skalatan.de/blog/ Kamailio services - >> https://gilawa.com >> >> -Original Message- >> From: Alex Balashov >> Sent: Thursday, January 5, 2023 3:59 PM >> To: Henning Westerholt >> Cc: Kamailio (SER) - Users Mailing List >> Subject: Re: [SR-Users] pkg memory leak when acc module cdr_enabled >> >> >>> On Jan 5, 2023, at 9:40 AM, Henning Westerholt wrote: >>> >>> Hello Alex, >>> >>> there might be some performance
[sr-dev] Re: [SR-Users] pkg memory leak when acc module cdr_enabled
Not the original developer of the memory managers, but, iirc, one of the reasons for private memory manager was to avoid a multi-threading related lock/mutex that is done by system malloc()/free(), which is not necessary for pure multi-process application. I did not analyzed myself, but I guess this is still there if it was a valid assertion in the past. On the other hand, I am not sure of its relevant impact, a lot of modules link with external libs that use anyhow the system allocator. Another thing was (faster?) indexing of free fragments up to 16kb that would enable quick selection of a such fragment that would usually fit in a (classic) SIP request/reply or attributes related to them. But I think this is no much relevant these days, because of the auto-join (merge) of free fragments -- without the latter, there were fragmentation problems. Split of large fragments is quite fast. Then, the ability to add own troubleshooting mechanism, with special structures at the beginning and the end of allocated chunks to track the place from where the chunk was allocated/freed and detect buffer overflows by checking the overwriting of special guard values. This is something that I kind of find still pretty useful for troubleshooting and usage statistics. But this can be implemented as a new memory manager for pkg-only that uses behind directly system malloc()/free(), it is actually in my to-do list for long time, but it didn't materialized yet due to lack of spare time (or from another perspective: lack of funding :-) ). At this moment, if you install from sources, just do: make MEMPKG=sys include_modules="..." ... cfg make all make install and you get kamailio compiled to use system malloc()/free() instead of the custom pkg qm/fm/... manager. I do it when I perform static code analysis, because it enables pkg memory leak detections by those tools. So it should be fully functional, on the other hand, I don't think I run such instance in production at this moment -- there are no pkg usage stats available. For shm, the custom manager has to be kept, it is still complex to work directly with shared memory, even now most of deployments are on linux, no longer common to have sunos, solaris, vax vms, aix, ... like in early 2000s. Cheers, Daniel On 05.01.23 16:57, Alex Balashov wrote: > Thanks, that's educational. > > I assumed the original argument was performance-related. I just wasn't sure > if that was still (sufficiently to be important) true. After all, lots of > SER/OpenSER design decisions in the early 2000s made the most sense then, > like inventing own imperative scripting language. :-) > > -- Alex > >> On Jan 5, 2023, at 10:16 AM, Henning Westerholt wrote: >> >> (adding sr-dev) >> >> Hi Alex, >> >> the memory allocator of glibc was not really efficient regarding the >> particular needs of a SIP server (allocation of many small string objects). >> That has probably improved in the last years, also system performance just >> got much faster. >> >> Other programs/tools also use their own allocators, e.g. Firefox, Rust [1] >> for jemalloc. >> >> There is some (not really well tested) support in the core for using the >> system memory allocator by the #define SYS_MALLOC. >> >> It probably needs to be set in Makefile.defs, also deactivating the other >> PKG memory managers, but did not looked into it right now. >> >> Cheers, >> >> Henning >> >> [1] https://news.ycombinator.com/item?id=8612645 >> >> -- >> Henning Westerholt - https://skalatan.de/blog/ >> Kamailio services - https://gilawa.com >> >> -Original Message- >> From: Alex Balashov >> Sent: Thursday, January 5, 2023 3:59 PM >> To: Henning Westerholt >> Cc: Kamailio (SER) - Users Mailing List >> Subject: Re: [SR-Users] pkg memory leak when acc module cdr_enabled >> >> >>> On Jan 5, 2023, at 9:40 AM, Henning Westerholt wrote: >>> >>> Hello Alex, >>> >>> there might be some performance implications by switching to system malloc. >>> There is also easier debugging by internal Kamailio memory manager support. >>> >>> In this particular example with the leak, Kamailio would use in the end all >>> of the system memory, and the machine out of memory killer will then >>> randomly processes. So the limited memory pool also helps to protect the >>> system against this kind of leaks. >> I am in no position to assess the relative efficiencies of various memory >> allocators. But it seems a bit extraordinary to suppose that a custom >> allocator is more efficient than the general-purpose libc allocator, >> although it's obviously possible; some application-specific optimised >> allocators clearly make this argument (i.e. Redis + jemalloc). >> >> Also, I wonder if the answer to this has changed over 20 years. >> >> Unbounded allocation from leaks can certainly be a problem. But rendering a >> process useless by running out of (much more limited) package memory (much >> more quickly) can also be a problem. :-) >> >> -- Alex >>
[sr-dev] Re: [SR-Users] pkg memory leak when acc module cdr_enabled
Thanks, that's educational. I assumed the original argument was performance-related. I just wasn't sure if that was still (sufficiently to be important) true. After all, lots of SER/OpenSER design decisions in the early 2000s made the most sense then, like inventing own imperative scripting language. :-) -- Alex > On Jan 5, 2023, at 10:16 AM, Henning Westerholt wrote: > > (adding sr-dev) > > Hi Alex, > > the memory allocator of glibc was not really efficient regarding the > particular needs of a SIP server (allocation of many small string objects). > That has probably improved in the last years, also system performance just > got much faster. > > Other programs/tools also use their own allocators, e.g. Firefox, Rust [1] > for jemalloc. > > There is some (not really well tested) support in the core for using the > system memory allocator by the #define SYS_MALLOC. > > It probably needs to be set in Makefile.defs, also deactivating the other PKG > memory managers, but did not looked into it right now. > > Cheers, > > Henning > > [1] https://news.ycombinator.com/item?id=8612645 > > -- > Henning Westerholt - https://skalatan.de/blog/ > Kamailio services - https://gilawa.com > > -Original Message- > From: Alex Balashov > Sent: Thursday, January 5, 2023 3:59 PM > To: Henning Westerholt > Cc: Kamailio (SER) - Users Mailing List > Subject: Re: [SR-Users] pkg memory leak when acc module cdr_enabled > > >> On Jan 5, 2023, at 9:40 AM, Henning Westerholt wrote: >> >> Hello Alex, >> >> there might be some performance implications by switching to system malloc. >> There is also easier debugging by internal Kamailio memory manager support. >> >> In this particular example with the leak, Kamailio would use in the end all >> of the system memory, and the machine out of memory killer will then >> randomly processes. So the limited memory pool also helps to protect the >> system against this kind of leaks. > > I am in no position to assess the relative efficiencies of various memory > allocators. But it seems a bit extraordinary to suppose that a custom > allocator is more efficient than the general-purpose libc allocator, although > it's obviously possible; some application-specific optimised allocators > clearly make this argument (i.e. Redis + jemalloc). > > Also, I wonder if the answer to this has changed over 20 years. > > Unbounded allocation from leaks can certainly be a problem. But rendering a > process useless by running out of (much more limited) package memory (much > more quickly) can also be a problem. :-) > > -- Alex > > -- > Alex Balashov > Principal Consultant > Evariste Systems LLC > Web: https://evaristesys.com > Tel: +1-706-510-6800 > -- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://evaristesys.com Tel: +1-706-510-6800 ___ Kamailio (SER) - Development Mailing List To unsubscribe send an email to sr-dev-le...@lists.kamailio.org
[sr-dev] Re: [SR-Users] pkg memory leak when acc module cdr_enabled
Just to add, this paper gives a lot of benchmark results for different allocators and two glibc versions: https://adms-conf.org/2019-camera-ready/durner_adms19.pdf It seems that even in 2019 using a special allocator can have some benefits for certain workloads. Cheers, Henning -- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://gilawa.com -Original Message- From: Henning Westerholt Sent: Thursday, January 5, 2023 4:17 PM To: Alex Balashov Cc: Kamailio (SER) - Users Mailing List ; sr-dev@lists.kamailio.org Subject: [sr-dev] Re: [SR-Users] pkg memory leak when acc module cdr_enabled (adding sr-dev) Hi Alex, the memory allocator of glibc was not really efficient regarding the particular needs of a SIP server (allocation of many small string objects). That has probably improved in the last years, also system performance just got much faster. Other programs/tools also use their own allocators, e.g. Firefox, Rust [1] for jemalloc. There is some (not really well tested) support in the core for using the system memory allocator by the #define SYS_MALLOC. It probably needs to be set in Makefile.defs, also deactivating the other PKG memory managers, but did not looked into it right now. Cheers, Henning [1] https://news.ycombinator.com/item?id=8612645 -- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://gilawa.com -Original Message- From: Alex Balashov Sent: Thursday, January 5, 2023 3:59 PM To: Henning Westerholt Cc: Kamailio (SER) - Users Mailing List Subject: Re: [SR-Users] pkg memory leak when acc module cdr_enabled > On Jan 5, 2023, at 9:40 AM, Henning Westerholt wrote: > > Hello Alex, > > there might be some performance implications by switching to system malloc. > There is also easier debugging by internal Kamailio memory manager support. > > In this particular example with the leak, Kamailio would use in the end all > of the system memory, and the machine out of memory killer will then randomly > processes. So the limited memory pool also helps to protect the system > against this kind of leaks. I am in no position to assess the relative efficiencies of various memory allocators. But it seems a bit extraordinary to suppose that a custom allocator is more efficient than the general-purpose libc allocator, although it's obviously possible; some application-specific optimised allocators clearly make this argument (i.e. Redis + jemalloc). Also, I wonder if the answer to this has changed over 20 years. Unbounded allocation from leaks can certainly be a problem. But rendering a process useless by running out of (much more limited) package memory (much more quickly) can also be a problem. :-) -- Alex -- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://evaristesys.com Tel: +1-706-510-6800 ___ Kamailio (SER) - Development Mailing List To unsubscribe send an email to sr-dev-le...@lists.kamailio.org ___ Kamailio (SER) - Development Mailing List To unsubscribe send an email to sr-dev-le...@lists.kamailio.org
[sr-dev] Re: [SR-Users] pkg memory leak when acc module cdr_enabled
(adding sr-dev) Hi Alex, the memory allocator of glibc was not really efficient regarding the particular needs of a SIP server (allocation of many small string objects). That has probably improved in the last years, also system performance just got much faster. Other programs/tools also use their own allocators, e.g. Firefox, Rust [1] for jemalloc. There is some (not really well tested) support in the core for using the system memory allocator by the #define SYS_MALLOC. It probably needs to be set in Makefile.defs, also deactivating the other PKG memory managers, but did not looked into it right now. Cheers, Henning [1] https://news.ycombinator.com/item?id=8612645 -- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://gilawa.com -Original Message- From: Alex Balashov Sent: Thursday, January 5, 2023 3:59 PM To: Henning Westerholt Cc: Kamailio (SER) - Users Mailing List Subject: Re: [SR-Users] pkg memory leak when acc module cdr_enabled > On Jan 5, 2023, at 9:40 AM, Henning Westerholt wrote: > > Hello Alex, > > there might be some performance implications by switching to system malloc. > There is also easier debugging by internal Kamailio memory manager support. > > In this particular example with the leak, Kamailio would use in the end all > of the system memory, and the machine out of memory killer will then randomly > processes. So the limited memory pool also helps to protect the system > against this kind of leaks. I am in no position to assess the relative efficiencies of various memory allocators. But it seems a bit extraordinary to suppose that a custom allocator is more efficient than the general-purpose libc allocator, although it's obviously possible; some application-specific optimised allocators clearly make this argument (i.e. Redis + jemalloc). Also, I wonder if the answer to this has changed over 20 years. Unbounded allocation from leaks can certainly be a problem. But rendering a process useless by running out of (much more limited) package memory (much more quickly) can also be a problem. :-) -- Alex -- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://evaristesys.com Tel: +1-706-510-6800 ___ Kamailio (SER) - Development Mailing List To unsubscribe send an email to sr-dev-le...@lists.kamailio.org
