Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[pali]

On Debian stable I'm periodically getting following error:
```
/usr/sbin/kamailio[13996]: ERROR: tls [tls_server.c:192]: tls_complete_init(): 
tls: ssl bug #1491 workaround: not enough memory for safe operation: 
shm=7494944 threshold1=7864320
```

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-676542999___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Daniel-Constantin Mierla]

This is likely related to libmysqlclient changes for ubuntu, like reported by 
#2394. On Debian stable I was not able to reproduce on a couple of heavy duty 
deployments, so might be something different.

I am closing this one, if you get it on Debian stable again, open a new issue 
and attach the the backtrace, ideally for a kamailio 5.3 or 5.3

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-671853514___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Daniel-Constantin Mierla]

Closed #1895.

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#event-3642402457___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[pali]

It seems that kamailio in Debian stable (5.2.1-1) is affected by this bug too.

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-620484819___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Brandon Armstead]

Are there any other versions affected anyone knows of ?

On Fri, Nov 15, 2019 at 5:10 AM Jay J  wrote:

> @henningw  we have stuck to db_mode 0, so
> the issue no longer resurfaces. It is still present, however, with db_mode
> 1.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> ,
> or unsubscribe
> 
> .
> ___
> Kamailio (SER) - Development Mailing List
> sr-dev@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
>
___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

@henningw we have stuck to db_mode 0, so the issue no longer resurfaces. It is 
still present, however, with db_mode 1.

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-554352735___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[nomadcalendar]

@henningw we have stuck to db_mode 0, so the issue no longer resurfaces. It is 
still present, however, with db_mode 1.

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-554321486___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Henning Westerholt]

@Jaykah - any update on this one?

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-554320631___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Daniel-Constantin Mierla]

As I looked again over the issue, I see in the backtrace (I attached) that the 
mysql client library is using yassl not libssl (which I expected).

@Jaykah - do you know if the library is installed from the default ubuntu 
repository or is something customized for RDS? Also, for you backtrace, install 
debugging symbols to see where the code get stuck in libmysqlclient library. 
There is another report on sr-dev with a blocking happening in libmysqlclient, 
maybe the backtrace with debug symbols will show the same like there.

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-532694977___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

Hi @henningw - yes, I do. I saw a similar problem on the mailing list and we 
are also using RDS - perhaps that's the issue (deadlock?). It looks like this 
happens on a record delete for the dialog - looks like db_mode 0 resolves this 
- so far instances with that option enabled have not experienced the issue. 
However, we are looking for a more permanent fix that would allow us to use 
db_mode 1

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-529679669___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Henning Westerholt]

@Jaykah - do you still see this error, maybe after updating to a new kamailio 
version?

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-529655813___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

(gdb) bt n
#0  0x7fab34880860 in __errno_location@plt () from 
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#1  0x7fab348bb589 in ?? () from 
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#2  0x7fab348bb849 in ?? () from 
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#3  0x7fab34895bc0 in ?? () from 
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#4  0x7fab34896c2c in ?? () from 
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#5  0x7fab3488aedc in ?? () from 
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#6  0x7fab3488bfa8 in ?? () from 
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#7  0x7fab3488dd61 in mysql_real_query () from 
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#8  0x7fab34e902ae in db_mysql_submit_query (_h=0x7fab35237ee0, 
_s=0x7fab346502d0 ) at km_dbase.c:116
#9  0x7fab3443ff6d in db_do_submit_query (_h=0x7fab35237ee0, 
_query=0x7fab346502d0 , submit_query=0x7fab34e8fb36 
) at db_query.c:53
#10 0x7fab3444310d in db_do_delete (_h=0x7fab35237ee0, _k=0x7ffdad2ea400, 
_o=0x0, _v=0x7ffdad2ea420, _n=2, val2str=0x7fab34e8e136 ,
submit_query=0x7fab34e8fb36 ) at db_query.c:291
#11 0x7fab34e94d64 in db_mysql_delete (_h=0x7fab35237ee0, 
_k=0x7ffdad2ea400, _o=0x0, _v=0x7ffdad2ea420, _n=2) at km_dbase.c:515
#12 0x7fab3285f714 in remove_dialog_from_db (cell=0x7faaae3e6fd0) at 
dlg_db_handler.c:714
#13 0x7fab3283b375 in destroy_dlg (dlg=0x7faaae3e6fd0) at dlg_hash.c:371
#14 0x7fab32842ca0 in dlg_unref_helper (dlg=0x7faaae3e6fd0, cnt=2, 
fname=0x7fab328d353f "dlg_handlers.c", fline=410) at dlg_hash.c:961
#15 0x7fab32867fb5 in dlg_ontdestroy (t=0x7faaae56ea00, type=131072, 
param=0x7ffdad2ea710) at dlg_handlers.c:410
#16 0x7fab32bd5b05 in run_trans_callbacks_internal (cb_lst=0x7faaae56ea78, 
type=131072, trans=0x7faaae56ea00, params=0x7ffdad2ea710) at t_hooks.c:260
#17 0x7fab32bd5c2c in run_trans_callbacks (type=131072, 
trans=0x7faaae56ea00, req=0x0, rpl=0x0, code=0) at t_hooks.c:287
#18 0x7fab32b0bf45 in free_cell_helper (dead_cell=0x7faaae56ea00, silent=0, 
fname=0x7fab32c1e882 "timer.c", fline=689) at h_table.c:166
#19 0x7fab32bc3bf6 in wait_handler (ti=1826413493, wait_tl=0x7faaae56ea88, 
data=0x7faaae56ea00) at timer.c:689
#20 0x556a3d1b32c5 in timer_list_expire (t=1826413493, h=0x7faaad841200, 
slow_l=0x7faaad8418b8, slow_mark=72) at core/timer.c:874
#21 0x556a3d1b376d in timer_handler () at core/timer.c:939
#22 0x556a3d1b3c36 in timer_main () at core/timer.c:978
#23 0x556a3d105322 in main_loop () at main.c:1727
#24 0x556a3d10ca1b in main (argc=7, argv=0x7ffdad2ead98) at main.c:2696


-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-507458520___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

 strings /usr/lib/x86_64-linux-gnu/libcrypto.so | grep "^OpenSSL"
OpenSSLDie
OpenSSL_add_all_ciphers
OpenSSL_add_all_digests
OpenSSL 1.0.2n  7 Dec 2017
OpenSSL HMAC method
OpenSSL EC algorithm
OpenSSL RSA method
OpenSSL DSA method
OpenSSL ECDSA method
OpenSSL DH Method
OpenSSL X9.42 DH method
OpenSSL PKCS#3 DH method
OpenSSL ECDH method
OpenSSL 'dlfcn' shared library method
OpenSSL default
OpenSSL default user interface
OpenSSL CMAC method


Yet the issue is still reproducible



-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-507456247___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[E. Schmidbauer]

FYI- we have to build our own debian9 pkgs due to the openssl mem leak
issues
not sure why this isn't in the kamailio's debian build system yet because
im pretty sure there is an issue on GH related to it
this is from our `build.sh` container:

# fix for libssl bug
apt-get -y purge libssl-dev && apt-get -y -qq install libssl1.0-dev

...

sed -ie 's/libssl-dev/libssl1.0-dev/g' pkg/kamailio/deb/debian/control


On Wed, Jun 12, 2019 at 2:33 AM Daniel-Constantin Mierla <
notificati...@github.com> wrote:

> Adding also that the mysql timeout might not work always, as presented in
> manual:
>
>- https://dev.mysql.com/doc/refman/5.5/en/mysql-options.html
>
> Citing:
>
> MYSQL_OPT_READ_TIMEOUT (argument type: unsigned int *)
>
>
>
> The timeout in seconds for each attempt to read from the server. There are 
> retries if necessary, so the total effective timeout value is three times the 
> option value. You can set the value so that a lost connection can be detected 
> earlier than the TCP/IP Close_Wait_Timeout value of 10 minutes.
>
>
>
> Implementation of this timeout uses mechanisms that may not be available on 
> all platforms. On such a platform, a client that issues a read call might 
> under certain circumstances wait without timing out. For example, a client 
> might not time out if the server is not responding because it is waiting for 
> a “disk full” condition to clear.
>
>
> Therefore you should also check if there are problems with mysql server.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> ,
> or mute the thread
> 
> .
> ___
> Kamailio (SER) - Development Mailing List
> sr-dev@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
>
___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Daniel-Constantin Mierla]

Adding also that the mysql timeout might not work always, as presented in 
manual:

  * https://dev.mysql.com/doc/refman/5.5/en/mysql-options.html

Citing:

```
MYSQL_OPT_READ_TIMEOUT (argument type: unsigned int *)

The timeout in seconds for each attempt to read from the server. There are 
retries if necessary, so the total effective timeout value is three times the 
option value. You can set the value so that a lost connection can be detected 
earlier than the TCP/IP Close_Wait_Timeout value of 10 minutes.

Implementation of this timeout uses mechanisms that may not be available on all 
platforms. On such a platform, a client that issues a read call might under 
certain circumstances wait without timing out. For example, a client might not 
time out if the server is not responding because it is waiting for a “disk 
full” condition to clear.
```
Therefore you should also check if there are problems with mysql server.

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-501139692___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Daniel-Constantin Mierla]

Update after I got access to a system exposing the issue.

The timer process of kamailio was using 100% CPU, being blocked, and by that 
not releasing/removing active/terminated transactions, leading to filling the 
memory - top output:

```
  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
13582 root  20   0 2274068  18012  12856 R  99.7  0.4   6179:53 kamailio
```

The kamclt ps:

```
{
IDX: 14
PID: 13582
DSC: timer
}
```

Attaching with gdb revealed the following backtrace:

```
#0  0x7fec0b6ea7d8 in yaSSL::States::What (this=this@entry=0x55b99e2fbf70) 
at ./extra/yassl/src/yassl_int.cpp:208
#1  0x7fec0b6eab31 in yaSSL::SSL::GetError (this=this@entry=0x55b99e2fb600) 
at ./extra/yassl/src/yassl_int.cpp:1520
#2  0x7fec0b6f89fb in yaSSL::receiveData (ssl=..., data=..., 
peek=peek@entry=false)
at ./extra/yassl/src/handshake.cpp:1047
#3  0x7fec0b6de65d in yaSSL::yaSSL_read (ssl=ssl@entry=0x55b99e2fb600, 
buffer=buffer@entry=0x55b99e2f91a0,
sz=sz@entry=4) at ./extra/yassl/src/ssl.cpp:363
#4  0x7fec0b6c6797 in vio_ssl_read (vio=0x55b99e2d83b0, buf=0x55b99e2f91a0 
"@", size=)
at ./vio/viossl.c:186
#5  0x7fec0b69fbae in net_read_raw_loop (count=4, net=0x7fec0c0378f0) at 
./sql/net_serv.cc:672
#6  net_read_packet_header (net=0x7fec0c0378f0) at ./sql/net_serv.cc:762
#7  net_read_packet (net=0x7fec0c0378f0, complen=0x7ffe188c6ff0) at 
./sql/net_serv.cc:822
#8  0x7fec0b6a0c2c in my_net_read (net=net@entry=0x7fec0c0378f0) at 
./sql/net_serv.cc:899
#9  0x7fec0b694edc in cli_safe_read_with_ok 
(mysql=mysql@entry=0x7fec0c0378f0, parse_ok=parse_ok@entry=0 '\000',
is_data_packet=is_data_packet@entry=0x0) at ./sql-common/client.c:1061
#10 0x7fec0b69518f in cli_safe_read (mysql=mysql@entry=0x7fec0c0378f0, 
is_data_packet=is_data_packet@entry=0x0)
at ./sql-common/client.c:1194
#11 0x7fec0b695fa8 in cli_read_query_result (mysql=0x7fec0c0378f0) at 
./sql-common/client.c:5245
#12 0x7fec0b697d61 in mysql_real_query (mysql=0x7fec0c0378f0, 
query=, length=)
at ./sql-common/client.c:5341
#13 0x7fec0bc9a2ae in db_mysql_submit_query (_h=0x7fec0c1a4808, 
_s=0x7fec0b45a2d0 ) at km_dbase.c:116
#14 0x7fec0b249f6d in db_do_submit_query (_h=0x7fec0c1a4808, 
_query=0x7fec0b45a2d0 ,
submit_query=0x7fec0bc99b36 ) at db_query.c:53
#15 0x7fec0b24d10d in db_do_delete (_h=0x7fec0c1a4808, _k=0x7ffe188c7310, 
_o=0x0, _v=0x7ffe188c7330, _n=2,
val2str=0x7fec0bc98136 , submit_query=0x7fec0bc99b36 
) at db_query.c:291
#16 0x7fec0bc9ed64 in db_mysql_delete (_h=0x7fec0c1a4808, 
_k=0x7ffe188c7310, _o=0x0, _v=0x7ffe188c7330, _n=2)
at km_dbase.c:515
#17 0x7fec09669714 in remove_dialog_from_db (cell=0x7feb84c98a98) at 
dlg_db_handler.c:714
#18 0x7fec09645375 in destroy_dlg (dlg=0x7feb84c98a98) at dlg_hash.c:371
#19 0x7fec0964cca0 in dlg_unref_helper (dlg=0x7feb84c98a98, cnt=2, 
fname=0x7fec096dd53f "dlg_handlers.c", fline=410)
at dlg_hash.c:961
#20 0x7fec09671fb5 in dlg_ontdestroy (t=0x7feb84dbf640, type=131072, 
param=0x7ffe188c7620) at dlg_handlers.c:410
#21 0x7fec099dfb05 in run_trans_callbacks_internal (cb_lst=0x7feb84dbf6b8, 
type=131072, trans=0x7feb84dbf640,
params=0x7ffe188c7620) at t_hooks.c:260
#22 0x7fec099dfc2c in run_trans_callbacks (type=131072, 
trans=0x7feb84dbf640, req=0x0, rpl=0x0, code=0)
at t_hooks.c:287
---Type  to continue, or q  to quit---
#23 0x7fec09915f45 in free_cell_helper (dead_cell=0x7feb84dbf640, silent=0, 
fname=0x7fec09a28882 "timer.c",
fline=689) at h_table.c:166
#24 0x7fec099cdbf6 in wait_handler (ti=1667374146, wait_tl=0x7feb84dbf6c8, 
data=0x7feb84dbf640) at timer.c:689
#25 0x55b99c9f02c5 in timer_list_expire (t=1667374146, h=0x7feb8461a050, 
slow_l=0x7feb8461d988, slow_mark=1904)
at core/timer.c:874
#26 0x55b99c9f076d in timer_handler () at core/timer.c:939
#27 0x55b99c9f0c36 in timer_main () at core/timer.c:978
#28 0x55b99c942322 in main_loop () at main.c:1727
#29 0x55b99c949a1b in main (argc=7, argv=0x7ffe188c7ca8) at main.c:2696
```

Showing that the processes executing code from inside libmysqlclient, related 
to tcp/tls connection. I noticed in config that tls module is used and the 
libssl version is 1.1, so this can be a side effect of the issue related to the 
new locking API in libssl/libcrypto.

The solutions to try would be either to compile against libssl 1.0.x or use the 
pre-loaded library workaround:

  * 
https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/openssl_mutex_shared

Run it with one of these two and let's see if it works fine or the issue 
appears again.

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-500895303___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.k

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Daniel-Constantin Mierla]

You can write to miconda [at] gmail.com

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-493864863___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

@miconda yes, I do have a machine where the issue is reproduced, where can I 
email the information? If you could reply to that email with your public key 
I'll add that right away as well.

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-491821601___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Daniel-Constantin Mierla]

Is there a way to get access to the system exposing the issue and do some 
troubleshooting there (it may require sudo privileges)? The leak is identified 
by the number of allocated memory chunks by uac, with side effects to 
transactions, but the reason for it is yet unknown. With Kamailio World 2019 
being finished, I expect to have more time for troubleshooting here.

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-491729808___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

@miconda just wanted to follow up on this - dumping those records should show 
us what the issue might be, how can that be inspected?

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-491592213___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

@miconda what would be the best way to dump the kamailio uac memory segments to 
see those records and compare with actual registrations?

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-489063870___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

@henningw will do that instead, thanks!

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-487394400___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Henning Westerholt]

@Jaykah - bug bounties are not really common in the Kamailio project so far. 
Commercial support options can be found on e.g. in the directory on 
kamailio.org. I will write you also an email with some more information.

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-487320096___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

Upping bounty to $250

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-487283380___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

@miconda We got rid of the trunks/accounts causing 500 errors and auth failure, 
yet are still seeing the leak in the kamailio process.

Any other way to troubleshoot this? 

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-486602112___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

@miconda ~2000 matches for "authentication failed for"
 ~2800 matches for "got sip response 500 while registering"

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-485386533___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Daniel-Constantin Mierla]

@Jaykah - if they repeat, is important to see how many times -- can you count 
how many error messages that contain the next texts are from start to out of 
memory logs?

```
authentication failed for
got sip response 500 while registering
```

Something like:

```
grep "authentication failed for" /path/to/logfile | wc -l
```
Just be sure that the logfile has only the logs from startup to out of memory, 
not older logs.


-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-484990798___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

@kamailio-sync could you please clarify? Not sure I understand 

-- 
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-484844338___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[David Villasmil]

Have you tried moving to db and see what happens?

On Thu, 18 Apr 2019 at 18:26, Jay J  wrote:

> also doubling the bounty for this, hoping the community would help us get
> a stable 5.1.7+
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> ,
> or mute the thread
> 
> .
> ___
> Kamailio (SER) - Development Mailing List
> sr-dev@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
>
-- 
Regards,

David Villasmil
email: david.villasmil.w...@gmail.com
phone: +34669448337
___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

also doubling the bounty for this, hoping the community would help us get a 
stable 5.1.7+

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-484601643___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

@miconda what I meant is the non-expiring registration records stored in memory

As to the error messages, those are about it, everything else is repetitive and 
the remaining logs consist of registrations, test calls, etc.

Perhaps there's a good way to export all shm data related to uac?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-484118498___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Daniel-Constantin Mierla]

Which hash table cleanup?

Are these all the error messages you get from starting kamailio till the out of 
memory errors?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-483983192___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

@miconda the number of records in uacreg is significantly less than that, that 
makes me wonder if the hash table cleanup is to blame

```
Apr 16 00:05:37 ip-10-0-0-0 kamailio[12057]: ERROR: uac [uac_reg.c:913]: 
uac_reg_tm_callback(): authentication failed for 
Apr 16 00:05:37 ip-10-0-0-0 kamailio[12040]: ERROR: uac [uac_reg.c:1032]: 
uac_reg_tm_callback(): got sip response 500 while registering [test]
Apr 16 01:39:03 ip-10-0-0-0 kamailio[12054]: ERROR: tls [tls_server.c:192]: 
tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe 
operation: shm=17097536 threshold1=17301504
Apr 16 01:39:03 ip-10-0-0-0 kamailio[12054]: ERROR:  
[core/tcp_read.c:1485]: tcp_read_req(): ERROR: tcp_read_req: error reading - c: 
0x7f7ace43eb68 r: 0x7f7ace43ebe8
```

In regards to the logs, that's something that we use for tests in the lab


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-483732724___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Daniel-Constantin Mierla]

Allocated size for tm is indeed very high and it also looks like being induced 
from uac module, because the uac_reg_update() allocates size of UUID (less than 
100) for each registration update, overall being 11472688, means over 100k 
remote registrations. Do you have so many records in you uacreg table?

After starting kamailio, but before the first error log message containing `no 
more shm memory`, do you have other error messages in the logs (especially from 
tm or uac modules)?

Any reason you are running kamailio printing logs to stderror (terminal) and 
not to syslog?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-483654305___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Daniel-Constantin Mierla]

With busy traveling during March, this issue got out of my sight. It looks like 
a problem with TM UAC, the allocated size there is big. Could be related to 
uac_reg (uac remote registrations), I will dig into it a bit.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-483632787___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

Posting a $100 bounty on this issue if anyone wants to take a look.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-481678999___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

We also have a test system with the issue reproduced if a closer look is needed 
@miconda 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-480599816___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

@miconda any other info I can gather that would help locate the issue here? The 
issue is recurring.

Thank you

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-479549863___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

Another occurence:

```
11(8342) ERROR: usrloc [ucontact.c:100]: new_ucontact(): no more shm memory
11(8342) ERROR: usrloc [urecord.c:149]: mem_insert_ucontact(): failed to create 
new contact
11(8342) ERROR: usrloc [urecord.c:573]: insert_ucontact(): failed to insert 
contact
11(8342) ERROR: registrar [save.c:522]: insert_contacts(): failed to insert 
contact
11(8342) ERROR: sl [sl_funcs.c:362]: sl_reply_error(): stateless error reply 
used: I'm terribly sorry, server error occurred (1/SL)
31(8362) ERROR:  [core/tcp_main.c:973]: tcpconn_new(): mem. allocation 
failure
31(8362) ERROR:  [core/tcp_main.c:4042]: handle_new_connect(): 
tcpconn_new failed, closing socket
31(8362) ERROR:  [core/tcp_main.c:973]: tcpconn_new(): mem. allocation 
failure
31(8362) ERROR:  [core/tcp_main.c:4042]: handle_new_connect(): 
tcpconn_new failed, closing socket
31(8362) ERROR:  [core/tcp_main.c:973]: tcpconn_new(): mem. allocation 
failure
```

```
 kamcmd mod.stats all shm
Module: core
{
dns_cache_mk_bad_entry(881): 7864
dns_cache_mk_rd_entry(1230): 13800
msg_lump_cloner(986): 501728
sip_msg_shm_clone(496): 2862448
tcpconn_new(971): 36272
build_req_buf_from_sip_req(2182): 814696
counters_prefork_init(207): 39168
create_avp(175): 57912
cfg_clone_str(130): 256
cfg_shmize(217): 1088
init_pt(113): 8
init_pt(108): 8
init_pt(107): 4760
cfg_parse_str(906): 192
register_timer(1012): 560
cfg_register_ctx(47): 96
init_tcp(4700): 8192
init_tcp(4694): 32768
init_tcp(4686): 8
init_tcp(4679): 8
init_tcp(4672): 8
init_tcp(4666): 8
init_tcp(4654): 8
init_avps(90): 8
init_avps(89): 8
init_dst_blacklist(437): 16384
init_dst_blacklist(430): 8
timer_alloc(515): 96
init_dns_cache(369): 8
init_dns_cache(361): 16384
init_dns_cache(354): 16
init_dns_cache(345): 8
init_timer(284): 8
init_timer(283): 16384
init_timer(282): 8
init_timer(281): 8
init_timer(270): 8
init_timer(238): 8
init_timer(221): 278544
init_timer(220): 8
init_timer(207): 8
cfg_child_cb_new(828): 64
sr_cfg_init(360): 8
sr_cfg_init(353): 8
sr_cfg_init(346): 8
sr_cfg_init(334): 8
sr_cfg_init(322): 8
qm_shm_lock_init(1195): 8
Total: 4709872
}

Module: ndb_redis
{
Total: 0
}

Module: pv
{
Total: 0
}

Module: sqlops
{
Total: 0
}

Module: textops
{
Total: 0
}

Module: xhttp_rpc
{
Total: 0
}

Module: db_mysql
{
Total: 0
}

Module: auth
{
Total: 0
}

Module: htable
{
ht_init_tables(386): 24576
ht_add_table(281): 936
Total: 25512
}

Module: xlog
{
Total: 0
}

Module: cnxcc
{
__alloc_new_credit_data(1217): 1080
__alloc_new_credit_data(1207): 480
__get_or_create_credit_data_entry(1177): 288
__add_call_by_cid(1534): 4464
__alloc_new_call_by_money(1278): 16512
__alloc_new_credit_data(1233): 48
shm_str_dup(723): 10728
__shm_str_hash_alloc(1090): 21984
__mod_init(286): 16
__mod_init(276): 16
__mod_init(275): 16
__mod_init(273): 16
__mod_init(272): 16
__mod_init(270): 16
__mod_init(269): 16
Total: 55696
}

Module: rtpengine
{
shm_str_dup(723): 3336
rtpp_function_call(2281): 3592
rtpengine_hash_table_init(85): 1024
rtpengine_hash_table_init(76): 2048
rtpengine_hash_table_init(67): 2048
get_rtpp_set(746): 200
add_rtpengine_socks(841): 88
get_rtpp_set(755): 8
rtpengine_hash_table_init(58): 72
add_rtpengine_socks(860): 24
mod_init(1448): 72
mod_init(1441): 24
mod_init(1428): 88
mod_init(1421): 8
rtpengine_hash_table_init(111): 14400
rtpengine_hash_table_init(96): 2048
Total: 29080
}

Module: mangler
{
Total: 0
}

Module: acc
{
Total: 0
}

Module: rr
{
Total: 0
}

Module: dialog
{
build_new_dlg(456): 151088
dlg_set_leg_info(541): 24712
dlg_get_iuid_shm_clone(287): 8328
dlg_set_toroute(1157): 8128
dlg_set_leg_info(532): 8136
dlg_set_leg_info(524): 36240
init_dlg_callback(55): 16
register_dlgcb(124): 40
init_dlg_table(295): 131088
init_dlg_table(287): 8
init_dlg_table(280): 8
init_dlg_table(275): 8
init_dlg_timer(57): 8
init_dlg_timer(48): 32
new_dlg_profile(188): 312
Total: 368152
}

Module: uac
{
reg_ht_add(530): 41784
reg_ht_add_byuser(479): 1760
reg_ht_add_byuuid(448): 2016
uac_reg_update(1093): 11472688
uac_reg_init_ht(237): 512
uac_reg_init_ht(223): 24
uac_reg_init_

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

@miconda -m 2048 -M 16

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-474684038___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Daniel-Constantin Mierla]

It is the shared memory that runs out of space. As I can see it, you provided 
the stats for pkg. You have to take the ones for shm.

What are the values you give for `-m` and `-M` command line parameters for 
kamailio?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-474398317___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

[sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

[Jay J]

### Description

Kamailio runs out of memory and stops processing requests.

```
ERROR:  [core/tcp_main.c:973]: tcpconn_new(): mem. allocation failure
ERROR:  [core/tcp_main.c:4042]: handle_new_connect(): tcpconn_new failed, 
closing socket
ERROR: uac [uac_reg.c:533]: reg_ht_add(): no more shm
ERROR: uac [uac_reg.c:1347]: uac_reg_load_db(): Error adding reg to htable
```

### Troubleshooting

Restarting the Kamailio process resolves the issue

 Reproduction

The issue occurs every 48 hours or so on the Kamailio instance

 Debugging Data


```
#kamcmd mod.stats all pkg
Module: core
{
init_io_wait(475): 6480
init_dst_set(83): 32208
init_io_wait(524): 3576
yyparse(1782): 8
mk_new_net(52): 48
fix_param(1196): 128
route_new_list(200): 352
fix_expr(541): 616
exp_optimize_right(411): 40
fixup_regexp_null(213): 216
fix_param(1257): 352
yyparse(2729): 4872
mk_rval_expr1(2616): 68256
mk_elem(90): 976
mk_rval_expr2(2674): 321160
mk_rval_expr_v(2548): 768
mk_rval_expr_v(2537): 808136
mk_action(118): 196384
parse_params2(585): 560
set_mod_param_regex(121): 848
set_mod_param_regex(110): 264
tr_table_add(1875): 400
sr_cmd_exports_convert(235): 34160
register_module(280): 6192
load_module(496): 2072
new_sock_info(235): 176
new_sock_info(230): 3640
yyparse(642): 352
yyparse(1732): 288
add_alias(93): 80
add_alias(91): 128
yyparse(1446): 8
yyparse(2262): 296
yyparse(661): 416
subst_str(514): 416
subst_parser(301): 264
subst_parser(291): 2384
subst_parser(274): 960
pp_subst_add(57): 312
pp_define(1766): 1096
addstr(1399): 261168
str_hash_alloc(59): 512
pv_table_add(236): 22616
parse_pvname_list(1515): 2496
tr_new(1646): 1512
fix_match_rve(3028): 2152
pv_cache_add(347): 20664
fix_param(1162): 18936
init_modules(1026): 16
cnt_hash_add(383): 2048
db_do_init2(299): 336
dupl_string_name(72): 32
dupl_string(48): 144
new_db_id(272): 288
get_abs_pathname(262): 80
rpc_hash_add(146): 2048
pv_parse_format(1150): 40592
get_hdr_field(116): 232
parse_headers(320): 64
register_select_table(458): 104
rpc_hash_add(100): 26264
add_callback(59): 288
sr_wtimer_init(308): 136
cfg_new_group(79): 1160
cfg_declare(50): 4120
fix_socket_list(1509): 96
fix_sock_str(420): 912
fix_hostname(1301): 160
grp_hash_add(234): 952
pv_init_buffer(2055): 163840
pv_init_buffer(2045): 320
init_nonsip_hooks(43): 24
init_rlist(146): 40
route_add(124): 1784
str_hash_alloc(59): 768
rval_get_str(1256): 33648
init_counters(122): 128
cnt_hash_add(332): 26352
str_hash_alloc(59): 1280
Total: 2138200
}

Module: ndb_redis
{
redis_parse_index(627): 88
pv_parse_redisc_name(704): 16496
redisc_add_server(257): 24064
Total: 40648
}

Module: pv
{
tr_parse_string(2340): 64
tr_parse_string(2310): 192
tr_parse_string(2300): 192
tr_parse_string(2360): 32
tr_parse_string(2264): 544
add_var(65): 512
add_var(58): 2080
tr_parse_string(2275): 496
tr_parse_paramlist(2668): 32
Total: 4144
}

Module: sqlops
{
sql_get_result(200): 64
pv_parse_dbr_name(134): 864
sql_init_con(83): 240
sqlops_tr_buffer_init(46): 2048
Total: 3216
}

Module: textops
{
tr_txt_parse_re(212): 96
hname_fixup(2819): 616
fixup_method(2889): 1256
Total: 1968
}

Module: xhttp_rpc
{
child_init(789): 832
Total: 832
}

Module: db_mysql
{
db_mysql_new_connection(75): 4088
db_mysql_new_connection(65): 96
Total: 4184
}

Module: auth
{
generate_random_secret(239): 32
generate_random_secret(238): 32
Total: 64
}

Module: htable
{
pv_parse_ht_name(158): 160
fixup_ht_key(339): 88
Total: 248
}

Module: xlog
{
xlog_fixup_helper(535): 17472
mod_init(214): 4104
xdbg_fixup_helper(500): 3352
Total: 24928
}

Module: cnxcc
{
__alloc_redis(149): 24
__alloc_redis(146): 40
Total: 64
}

Module: rtpengine
{
build_rtpp_socks(1611): 8
child_init(1768): 240
Total: 248
}

Module: mangler
{
compile_expresions(557): 64
compile_expresions(541): 64
Total: 128
}

Module: acc
{
parse_acc_extra(116): 672
Total: 672
}

Module: rr
{
register_rrcb(61): 32
Total: 32
}

Module: dialog
{
dlg_bridge_init_hdrs(66): 80