Hello, that's the way it was done for older versions of kamailio.
In master and 4.4 the memory debugging is turned on and it is reflected by the presence of DBG_SR_MEMORY in the output of 'kamailio -v'. Anyhow, what you reported is not a leak inside kamailio memory manager, but a leak of using system memory, so it is not affected by DBG_SR_MEMORY and cannot be troubleshooted using the mechanisms for pkg and shm managers. Cheers, Daniel On 07/10/16 15:45, Jurijs Ivolga wrote: > Hi Daniel, > > I tried to compile with memory manager in debugging mode like this: > > MEMDBG=1 make cfg 'include_modules= textopsx db_mysql jansson json > snmpstats tls utils uuid' > make all > /etc/init.d/kamailioa stop > make install > /etc/init.d/kamailioa start > > But still I'm getting nothing on: > > kamailio -I | grep DBG_QM_MALLOC > > Any hints? > > With kind regards, > > Jurijs > > On Fri, Oct 7, 2016 at 10:39 AM, Daniel-Constantin Mierla > <mico...@gmail.com <mailto:mico...@gmail.com>> wrote: > > Hello, > > There are some results when searching for memory leak on openssl > 1.0.1, but might not be valid for this case. > > Would you be able to run kamailio through valgrind? It may slow > down a bit the processing, but may be the fastest way to catch the > system memory leak. Maybe you have an instance will less traffic > and you can run through valgrind for a while, you don't have to > wait until you get the OOM. > > Cheers, > Daniel > > > On 07/10/16 09:27, Jurijs Ivolga wrote: >> Hi Daniel, >> >> openssl.x86_64 >> >> 1.0.1e-48.el6_8.1 >> CentOS release 6.8 >> >> With kind regards, >> >> Jurijs >> >> On Fri, Oct 7, 2016 at 10:20 AM, Daniel-Constantin Mierla >> <mico...@gmail.com <mailto:mico...@gmail.com>> wrote: >> >> Hello, >> >> the tls module in kamailio is using shm memory, but can be >> something internal for libssl. What operating system do you >> use and what is the libssl version? >> >> Cheers, >> Daniel >> >> >> On 06/10/16 17:43, Jurijs Ivolga wrote: >>> Hi Daniel, >>> >>> I do not use puke.top rpc command. Maybe this issue related >>> to TLS? Servers what has this problem are utilizing TLS >>> heavily, servers which do not has this problem use UDP. >>> >>> With kind regards, >>> >>> Jurijs >>> >>> On Thu, Oct 6, 2016 at 6:05 PM, Daniel-Constantin Mierla >>> <mico...@gmail.com <mailto:mico...@gmail.com>> wrote: >>> >>> Hello, >>> >>> are you using pike.top rpc command? I noticed in the >>> code that it uses system malloc, but I haven't >>> investigated further yet, first to see if this would be >>> a possibility ... >>> >>> Cheers, >>> Daniel >>> >>> >>> On 06/10/16 16:33, Jurijs Ivolga wrote: >>>> Hi Daniel, >>>> >>>> We do not do any external operations. >>>> >>>> We are using janson 2.7 everywhere. I will try to >>>> update to latest janson version tomorrow. >>>> All json operation is pretty much same, we are using >>>> only jansson_get. >>>> >>>> In attachment you can see memory consumption. On the >>>> right 2 servers which are faced internet on the left >>>> which don't face internet. As you can see memory >>>> consumption is pretty dramatic. >>>> >>>> Thank you for your help! >>>> >>>> With kind regards, >>>> >>>> Jurijs >>>> >>>> On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin >>>> Mierla <mico...@gmail.com <mailto:mico...@gmail.com>> >>>> wrote: >>>> >>>> Hello, >>>> >>>> are you doing different external operations than on >>>> the other instances, like mi/rpc commands. >>>> >>>> From the list of the modules you exposed, I think >>>> jansson has the higher probability to work with >>>> system memory. Are you doing different json >>>> operations in config that in the other instances of >>>> kamailio? Are you using same version of libjansson >>>> everywhere? >>>> >>>> Cheers, >>>> Daniel >>>> >>>> On 06/10/16 13:46, Jurijs Ivolga wrote: >>>>> Hi Daniel, >>>>> >>>>> This modules what we are using: >>>>> >>>>> loadmodule "mi_fifo.so" >>>>> loadmodule "kex.so" >>>>> loadmodule "corex.so" >>>>> loadmodule "tm.so" >>>>> loadmodule "tmx.so" >>>>> loadmodule "sl.so" >>>>> loadmodule "rr.so" >>>>> loadmodule "pv.so" >>>>> loadmodule "maxfwd.so" >>>>> loadmodule "textops.so" >>>>> loadmodule "siputils.so" >>>>> loadmodule "xlog.so" >>>>> loadmodule "sanity.so" >>>>> loadmodule "ctl.so" >>>>> loadmodule "cfg_rpc.so" >>>>> loadmodule "mi_rpc.so" >>>>> loadmodule "dispatcher.so" >>>>> loadmodule "utils.so" >>>>> loadmodule "path.so" >>>>> loadmodule "ipops.so" >>>>> loadmodule "jansson.so" >>>>> loadmodule "auth.so" >>>>> loadmodule "nathelper.so" >>>>> loadmodule "tls.so" >>>>> loadmodule "htable.so" >>>>> loadmodule "pike.so" >>>>> >>>>> We have several other Kamailio instances but they >>>>> are not faced to internet and they do not have >>>>> such memory issue. That other Kamailio instances >>>>> have same modules, except modules listed below. >>>>> So if you think that issue is inside external >>>>> library, probably we need to check first modules >>>>> from list below. >>>>> >>>>> loadmodule "ipops.so" >>>>> loadmodule "auth.so" >>>>> loadmodule "nathelper.so" >>>>> loadmodule "pike.so" >>>>> >>>>> But maybe this other Kamailio instances do not >>>>> have this memory issue, just because they did not >>>>> face to internet and did not have same load as >>>>> instances with memory issue. >>>>> >>>>> kamailio -v >>>>> version: kamailio 4.4.3 (x86_64/linux) 5a2195 >>>>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, >>>>> TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, >>>>> USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, >>>>> PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, >>>>> DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, >>>>> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, >>>>> USE_DST_BLACKLIST, HAVE_RESOLV_RES >>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE >>>>> 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE >>>>> 65535, DEFAULT PKG_SIZE 8MB >>>>> poll method support: poll, epoll_lt, epoll_et, >>>>> sigio_rt, select. >>>>> id: 5a2195 >>>>> compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7 >>>>> >>>>> >>>>> With kind regards, >>>>> >>>>> >>>>> Jurijs >>>>> >>>>> On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin >>>>> Mierla <mico...@gmail.com >>>>> <mailto:mico...@gmail.com>> wrote: >>>>> >>>>> Hello, >>>>> >>>>> it looks like a leak from the system memory, >>>>> not from kamailio's pkg or shm memory. This >>>>> can be due to an improper use of an external >>>>> library (e.g., libxml2) by a kamailio module >>>>> or because of a problem in the library. >>>>> >>>>> Can you list the modules used in your config >>>>> (the loadmodule lines)? I will try to guess >>>>> from the list which one relying on external >>>>> libs with higher risk of leak issues. >>>>> >>>>> Also, provide the version of kamailio you are >>>>> using (kamailio -v). >>>>> >>>>> Cheers, >>>>> Daniel >>>>> >>>>> >>>>> On 04/10/16 15:42, Jurijs Ivolga wrote: >>>>>> Hi, >>>>>> >>>>>> Our Kamailio server is crashing once per >>>>>> week, with following error: >>>>>> >>>>>> Oct 1 06:25:06 kamailio kernel: >>>>>> [26982632.803789] Out of memory in UB 210: >>>>>> OOM killed process 12261 (kamailio) score 0 >>>>>> vm:1614768kB, rss:280200kB, swap:131408kB >>>>>> >>>>>> Core dump was never created, probably it is >>>>>> because of my environment, but I will try to >>>>>> get it. >>>>>> >>>>>> Server constantly eats memory, maybe some >>>>>> kind of memory leak? >>>>>> >>>>>> Any help is highly appreciated! >>>>>> >>>>>> Jurijs >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - >>>>>> sr-users mailing list >>>>>> sr-users@lists.sip-router.org >>>>>> <mailto:sr-users@lists.sip-router.org> >>>>>> >>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>>>>> >>>>>> <http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users> >>>>> >>>>> -- >>>>> Daniel-Constantin Mierla >>>>> http://twitter.com/#!/miconda >>>>> <http://twitter.com/#%21/miconda> - >>>>> http://www.linkedin.com/in/miconda >>>>> <http://www.linkedin.com/in/miconda> >>>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - >>>>> http://www.asipto.com >>>>> >>>>> _______________________________________________ >>>>> SIP Express Router (SER) and Kamailio >>>>> (OpenSER) - sr-users mailing list >>>>> sr-users@lists.sip-router.org >>>>> <mailto:sr-users@lists.sip-router.org> >>>>> >>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>>>> >>>>> <http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users> >>>>> >>>>> >>>> -- >>>> Daniel-Constantin Mierla >>>> http://twitter.com/#!/miconda >>>> <http://twitter.com/#%21/miconda> - >>>> http://www.linkedin.com/in/miconda >>>> <http://www.linkedin.com/in/miconda> >>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - >>>> http://www.asipto.com >>>> >>> -- >>> Daniel-Constantin Mierla >>> http://twitter.com/#!/miconda >>> <http://twitter.com/#%21/miconda> - >>> http://www.linkedin.com/in/miconda >>> <http://www.linkedin.com/in/miconda> >>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - >>> http://www.asipto.com >>> >> -- >> Daniel-Constantin Mierla >> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - >> http://www.linkedin.com/in/miconda >> <http://www.linkedin.com/in/miconda> >> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - >> http://www.asipto.com >> > -- > Daniel-Constantin Mierla > http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - > http://www.linkedin.com/in/miconda > <http://www.linkedin.com/in/miconda> > Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - > http://www.asipto.com > -- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
