from:"Andrey Deykunov"

[sr-dev] Build kamailio 5.3.1 db_mysql module with percona.

2021-07-06 Thread Andrey Deykunov

Hi,

We have kamailio server v. 5.3.1 with db_mysql.so built with our old mysql
library 5.1.41. Now I'm trying to build it with libperconaclient20. I've
tried many variants of build configuration, but every time I got errors
when kamailio stops or starts, such:





*2021-07-06T06:01:20.943809-07:00 devhpbx005-1
/var/lib/ums/sbin/kamailio[25711]: ERROR: db_mysql [km_my_con.c:139]:
db_mysql_new_connection(): driver error: Lost connection to MySQL server at
'reading authorization packet', system error:
112021-07-06T06:01:20.946591-07:00 devhpbx005-1
/var/lib/ums/sbin/kamailio[25711]: ERROR:  [db.c:319]: db_do_init2():
could not add connection to the pool2021-07-06T06:01:20.947224-07:00
devhpbx005-1 /var/lib/ums/sbin/kamailio[25711]: ERROR: auth_db
[auth_db_mod.c:159]: child_init(): unable to connect to the
database2021-07-06T06:01:20.947592-07:00 devhpbx005-1
/var/lib/ums/sbin/kamailio[25711]: ERROR:  [core/sr_module.c:806]:
init_mod_child(): error while initializing module auth_db
(/var/lib/ums/lib64/kamailio/modules/auth_db.so) (idx: 5 rank: 5 desc:
[udpreceiver child=4 sock=64.78.52.86:5060 ])*

or



*2021-07-06T06:18:43.159744-07:00 devhpbx005-1
/var/lib/ums/sbin/kamailio[30954]: CRITICAL: 
[core/mem/q_malloc.c:150]: qm_debug_check_frag(): BUG: qm: prev. fragm.
tail overwritten(0, 0)[0x7fd5c6c85bc8:0x7fd5c6c85c00]! Memory allocator was
calledfrom core: db_res.c:114. Fragment marked by db_mysql: km_my_con.c:75.
Exec from core/mem/q_malloc.c:391.*

Is there some specific way to build kamailio with percona?

Thanks,
Andrey
___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segmentation fault occured in rpc_mod_print function in runtime. (#2460)

2020-10-12 Thread Andrey Deykunov

Thanks, I will take into account your comments.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2460#issuecomment-70719___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segmentation fault occured in rpc_mod_print function in runtime. (#2460)

2020-10-12 Thread Andrey Deykunov

Hi Daniel,

I've performed some load tests. The loading was about 400 calls per second on 
peak. In parallel, I started a script calling commands below with a frequency 
of 200 milliseconds:

```
/var/lib/ums/sbin/kamcmd -s udp:localhost:2048 stats.get_statistics 
websocket:
/var/lib/ums/sbin/kamcmd -s udp:localhost:2048 stats.get_statistics tcp:
/var/lib/ums/sbin/kamcmd -s udp:localhost:2048 stats.get_statistics shmem:
/var/lib/ums/sbin/kamcmd -s udp:localhost:2048 mod.stats tls pkg
/var/lib/ums/sbin/kamcmd -s udp:localhost:2048 mod.stats tls shm
/var/lib/ums/sbin/kamcmd -s udp:localhost:2048 mod.stats websocket pkg
/var/lib/ums/sbin/kamcmd -s udp:localhost:2048 mod.stats websocket shm
/var/lib/ums/sbin/kamcmd -s udp:localhost:2048 mod.stats core shm

```

This is a graph of loading:

![load_400_max](https://user-images.githubusercontent.com/9265562/95750045-3dbf0c80-0ca5-11eb-9dfa-e503526ecd62.png)

The tests have finished successfully, there were no any fails ocurred.

Andrey








-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2460#issuecomment-707118377___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segmentation fault occured in rpc_mod_print function in runtime. (#2460)

2020-09-09 Thread Andrey Deykunov

Thanks Daniel, I'll be testing it as soon as I deal with my current tasks.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2460#issuecomment-689740277___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

2020-09-01 Thread Andrey Deykunov

Created issue #2460

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2433#issuecomment-685018354___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

[sr-dev] [kamailio/kamailio] Segmentation fault occured in rpc_mod_print function in runtime. (#2460)

2020-09-01 Thread Andrey Deykunov



### Description

We have segfault in Kamailio v5.3.1 installed on Debain 9.x 64 bit occured 
while kamailio was shutting down while our script tryed to get metrics using 
kamcmd utility at the same time.



### Troubleshooting

No troubleshooting was done, since it happened on a production server. We 
simply restarted the server.

 Reproduction

The problem periodically happens on production servers in runtime. Kamailio 
crashes when one of our scripts tried getting statistics about websocket and 
tls modules using kamcmd. As I see in core dump, shared memory was already 
freed when rpc_mod_print called in the child process.

 Debugging Data

```
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/lib/ums/sbin/kamailio...done.
[New LWP 17075]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/var/lib/ums/sbin/kamailio -m 2048 -M 12 -P 
/var/run/kamailio/kamailio.pid -f /'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strcmp_sse2_unaligned ()
at ../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S:32
#0  __strcmp_sse2_unaligned () at 
../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S:32
#1  0x7fcae1ef62f5 in rpc_mod_print (rpc=0x7fcae180b540 , 
ctx=0x7ffe7a6f7808, mname=0x1edd0c9 "websocket", stats=0x1ef8310, flag=2) at 
mod_stats.c:117
#2  0x7fcae1ef60eb in rpc_mod_print_one (rpc=0x7fcae180b540 
, ctx=0x7ffe7a6f7808, mname=0x1edd0c9 "websocket", 
pkg_stats=0x1ef69d0, shm_stats=0x1ef8310, flag=2) at mod_stats.c:159
#3  0x7fcae1ef5ee1 in rpc_mod_mem_stats_mode (rpc=0x7fcae180b540 
, ctx=0x7ffe7a6f7808, fmode=0) at mod_stats.c:239
#4  0x7fcae1ef584f in rpc_mod_mem_stats (rpc=0x7fcae180b540 
, ctx=0x7ffe7a6f7808) at mod_stats.c:251
#5  0x7fcae15dac80 in process_rpc_req (buf=0x1edd0b4 
"\241\003\035\020\333Qg\221\nmod.stats", size=36, bytes_needed=0x7ffe7a6f7c50, 
sh=0x7ffe7a6f7bc0, saved_state=0x1eed0b8) at binrpc_run.c:678
#6  0x7fcae15c872f in handle_stream_read (s_c=0x1edd080, idx=-1) at 
io_listener.c:511
#7  0x7fcae15c4121 in handle_io (fm=0x7fcb65600cb0, events=1, idx=-1) at 
io_listener.c:706
#8  0x7fcae15c293a in io_wait_loop_epoll (h=0x7fcae180b348 , t=10, 
repeat=0) at ./../../core/io_wait.h:1062
#9  0x7fcae15b662c in io_listen_loop (fd_no=2, cs_lst=0x1df1940) at 
io_listener.c:281
#10 0x7fcae15ec72c in mod_child (rank=0) at ctl.c:338
#11 0x00638c14 in init_mod_child (m=0x7fcb6547f4b0, rank=0) at 
core/sr_module.c:780
#12 0x0063862d in init_mod_child (m=0x7fcb6547fb78, rank=0) at 
core/sr_module.c:776
#13 0x0063862d in init_mod_child (m=0x7fcb65480018, rank=0) at 
core/sr_module.c:776
#14 0x0063862d in init_mod_child (m=0x7fcb65480528, rank=0) at 
core/sr_module.c:776
#15 0x0063862d in init_mod_child (m=0x7fcb654809c8, rank=0) at 
core/sr_module.c:776
#16 0x0063862d in init_mod_child (m=0x7fcb65481140, rank=0) at 
core/sr_module.c:776
#17 0x0063862d in init_mod_child (m=0x7fcb654817b0, rank=0) at 
core/sr_module.c:776
#18 0x0063862d in init_mod_child (m=0x7fcb65481c38, rank=0) at 
core/sr_module.c:776
#19 0x006385b2 in init_child (rank=0) at core/sr_module.c:825
#20 0x0043140c in main_loop () at main.c:1753
#21 0x0043df6f in main (argc=9, argv=0x7ffe7a6fbf88) at main.c:2802
```

 Log Messages

No any useful logs available.

 SIP Traffic

No SIP traffic available.

### Possible Solutions



### Additional Information

* **This is a sequence of commands that python script runs every 10 seconds:**

```
/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 stats.get_statistics websocket:
/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 stats.get_statistics tcp:
/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 stats.get_statistics shmem:
/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 mod.stats tls pkg
/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 mod.stats tls shm
/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 mod.stats websocket pkg
/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 mod.stats websocket shm
/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 mod.stats core shm
```

  * **Kamailio Version** - output of `kamailio -v`

```
version: kamailio 5.3.1

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

2020-09-01 Thread Andrey Deykunov

The problem is not reproducible on the hosts where we disabled metrics 
gathering on kamailio.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2433#issuecomment-684831361___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

2020-09-01 Thread Andrey Deykunov

I understand, but we've got a lot of such crashes during normal kamailio 
operation, NOT during shutdown.  This is the issue I'm concerned about.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2433#issuecomment-684827111___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

2020-09-01 Thread Andrey Deykunov

Bump! Any updates?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2433#issuecomment-684758225___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

2020-08-21 Thread Andrey Deykunov

This is a sequence of commands that python script runs every 10 seconds:

/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 stats.get_statistics websocket:
/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 stats.get_statistics tcp:
/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 stats.get_statistics shmem:
/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 mod.stats tls pkg
/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 mod.stats tls shm
/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 mod.stats websocket pkg
/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 mod.stats websocket shm
/var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 mod.stats core shm

This script parses output of every command and gets needed metrics. I see at 
least one problem here - the script should use udp instead of tcp.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2433#issuecomment-678234199___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

2020-08-20 Thread Andrey Deykunov

> Was this a case of running the rpc command many times at the same time?

I'll ask our LSA about it tomorow. Do you think it could be a problem? I 
thought that RPC handlers have thread safe (interprocess, in this case) access 
to shared memory, is it?




-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2433#issuecomment-677828367___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

2020-08-20 Thread Andrey Deykunov

**Updates:**

A few days ago we've got the same segfault on production server, but in this 
case it happened without shut down. I think there is a race when calling 
process_rpc_req function and it does not relate to server cooldown. 

**Core dump:**

GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/lib/ums/sbin/kamailio...done.
[New LWP 6226]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/var/lib/ums/sbin/kamailio -m 2048 -M 12 -P 
/var/run/kamailio/kamailio.pid -f /'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strcmp_sse2_unaligned ()
at ../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S:32
#0  __strcmp_sse2_unaligned () at 
../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S:32
#1  0x7f327f55c2f5 in rpc_mod_print (rpc=0x7f327ee71540 , 
ctx=0x7ffe1418a9b8, mname=0x1dca0c8 "tls", stats=0x1de5310, flag=2) at 
mod_stats.c:117
#2  0x7f327f55c0eb in rpc_mod_print_one (rpc=0x7f327ee71540 
, ctx=0x7ffe1418a9b8, mname=0x1dca0c8 "tls", 
pkg_stats=0x1de39d0, shm_stats=0x1de5310, flag=2) at mod_stats.c:159
#3  0x7f327f55bee1 in rpc_mod_mem_stats_mode (rpc=0x7f327ee71540 
, ctx=0x7ffe1418a9b8, fmode=0) at mod_stats.c:239
#4  0x7f327f55b84f in rpc_mod_mem_stats (rpc=0x7f327ee71540 
, ctx=0x7ffe1418a9b8) at mod_stats.c:251
#5  0x7f327ec40c80 in process_rpc_req (buf=0x1dca0b4 
"\241\003\026F\270\067\r\221\nmod.stats", size=29, bytes_needed=0x7ffe1418ae00, 
sh=0x7ffe1418ad70, saved_state=0x1dda0b8) at binrpc_run.c:678
#6  0x7f327ec2e72f in handle_stream_read (s_c=0x1dca080, idx=-1) at 
io_listener.c:511
#7  0x7f327ec2a121 in handle_io (fm=0x7f3302c66cc0, events=1, idx=-1) at 
io_listener.c:706
#8  0x7f327ec2893a in io_wait_loop_epoll (h=0x7f327ee71348 , t=10, 
repeat=0) at ./../../core/io_wait.h:1062
#9  0x7f327ec1c62c in io_listen_loop (fd_no=2, cs_lst=0x1cde940) at 
io_listener.c:281
#10 0x7f327ec5272c in mod_child (rank=0) at ctl.c:338
#11 0x00638c14 in init_mod_child (m=0x7f3302ae54b0, rank=0) at 
core/sr_module.c:780
#12 0x0063862d in init_mod_child (m=0x7f3302ae5b78, rank=0) at 
core/sr_module.c:776
#13 0x0063862d in init_mod_child (m=0x7f3302ae6018, rank=0) at 
core/sr_module.c:776
#14 0x0063862d in init_mod_child (m=0x7f3302ae6528, rank=0) at 
core/sr_module.c:776
#15 0x0063862d in init_mod_child (m=0x7f3302ae69c8, rank=0) at 
core/sr_module.c:776
#16 0x0063862d in init_mod_child (m=0x7f3302ae7140, rank=0) at 
core/sr_module.c:776
#17 0x0063862d in init_mod_child (m=0x7f3302ae77b0, rank=0) at 
core/sr_module.c:776
#18 0x0063862d in init_mod_child (m=0x7f3302ae7c38, rank=0) at 
core/sr_module.c:776
#19 0x006385b2 in init_child (rank=0) at core/sr_module.c:825
#20 0x0043140c in main_loop () at main.c:1753
#21 0x0043df6f in main (argc=9, argv=0x7ffe1418f138) at main.c:2802


**Logs:**

2020-08-18T12:36:03.755718-07:00 hpbx031-1.va /var/lib/ums/sbin/kamailio[6204]: 
INFO:

Re: [sr-dev] [kamailio/kamailio] core: keep a global flag in shm for shut down phase (6da946e)

2020-08-11 Thread Andrey Deykunov

if(_ksr_shutdown_phase) { 

instance of _ksr_shutdown_phase cannot be initialized here.

it supposed to be: if(!_ksr_shutdown_phase) { 



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/commit/6da946e4a3d6d6949dd6babe9cf3919edc0b4519#commitcomment-41371247___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

2020-08-11 Thread Andrey Deykunov

Thanks Daniel. I'll add this fix as a patch to our kamailio build.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2433#issuecomment-671840917___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

[sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

2020-08-10 Thread Andrey Deykunov



### Description

We have segfault in Kamailio v5.3.1 installed on Debain 9.x 64 bit occured 
while kamailio was shutting down while our script tryed to get metric using 
kamcmd utility at the same time.

### Troubleshooting

No troubleshooting was done, since it happened on a production server. We 
simply restarted the server.

 Reproduction

The problem periodically happens on production servers during restart. Kamailio 
crashes when one of our scripts tried getting statistics about websocket and 
tls modules using kamcmd during server restart. As I see in core dump, shared 
memory was already freed when rpc_mod_print called in the child process.

 Debugging Data



```
core.kamailio.17684

/var/lib/ums/sbin/kamailio


GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/lib/ums/sbin/kamailio...done.
[New LWP 17684]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/var/lib/ums/sbin/kamailio -m 2048 -M 12 -P 
/var/run/kamailio/kamailio.pid -f /'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strcmp_ssse3 () at ../sysdeps/x86_64/multiarch/../strcmp.S:1197
#0  __strcmp_ssse3 () at ../sysdeps/x86_64/multiarch/../strcmp.S:1197
#1  0x7ff4d6ddb2f5 in rpc_mod_print (rpc=0x7ff4d66f0540 , 
ctx=0x7fff11cd5658, mname=0x1f140b8 "tls", stats=0x1f2f180, flag=2) at 
mod_stats.c:117
#2  0x7ff4d6ddb0eb in rpc_mod_print_one (rpc=0x7ff4d66f0540 
, ctx=0x7fff11cd5658, mname=0x1f140b8 "tls", 
pkg_stats=0x1f2d9c0, shm_stats=0x1f2f180, flag=2) at mod_stats.c:159
#3  0x7ff4d6ddaee1 in rpc_mod_mem_stats_mode (rpc=0x7ff4d66f0540 
, ctx=0x7fff11cd5658, fmode=0) at mod_stats.c:239
#4  0x7ff4d6dda84f in rpc_mod_mem_stats (rpc=0x7ff4d66f0540 
, ctx=0x7fff11cd5658) at mod_stats.c:251
#5  0x7ff4d64bfc80 in process_rpc_req (buf=0x1f140a4 
"\241\003\026t\256\214=\221\nmod.stats", size=29, bytes_needed=0x7fff11cd5aa0, 
sh=0x7fff11cd5a10, saved_state=0x1f240a8) at binrpc_run.c:678
#6  0x7ff4d64ad72f in handle_stream_read (s_c=0x1f14070, idx=-1) at 
io_listener.c:511
#7  0x7ff4d64a9121 in handle_io (fm=0x7ff55a4e4ae0, events=1, idx=-1) at 
io_listener.c:706
#8  0x7ff4d64a793a in io_wait_loop_epoll (h=0x7ff4d66f0348 , t=10, 
repeat=0) at ./../../core/io_wait.h:1062
#9  0x7ff4d649b62c in io_listen_loop (fd_no=2, cs_lst=0x1e28940) at 
io_listener.c:281
#10 0x7ff4d64d172c in mod_child (rank=0) at ctl.c:338
#11 0x00638c14 in init_mod_child (m=0x7ff55a3634b0, rank=0) at 
core/sr_module.c:780
#12 0x0063862d in init_mod_child (m=0x7ff55a363b78, rank=0) at 
core/sr_module.c:776
#13 0x0063862d in init_mod_child (m=0x7ff55a364018, rank=0) at 
core/sr_module.c:776
#14 0x0063862d in init_mod_child (m=0x7ff55a364528, rank=0) at 
core/sr_module.c:776
#15 0x0063862d in init_mod_child (m=0x7ff55a3649c8, rank=0) at 
core/sr_module.c:776
#16 0x0063862d in init_mod_child (m=0x7ff55a365140, rank=0) at 
core/sr_module.c:776
#17 0x0063862d in init_mod_child (m=0x7ff55a3657b0, rank=0) at 
core/sr_module.c:776
#18 0x0063862d in init_mod_child (m=0x7ff55a365c38, rank=0) at 
core/sr_module.c:776
#19 0x006385b2 in init_child (rank=0) at core/sr_module.c:825
#20 0x0043140c in main_loop () at main.c:1753
#21 0x0043df6f in main (argc=9, argv=0x7fff11cd9dd8) at main.c:2802
```

 Log Messages

No any useful logs available.

 SIP Traffic

No SIP traffic available.

### Possible Solutions



### Additional Information

  * **Kamailio Version** - output of `kamailio -v`

```
version: kamailio 5.3.1 (x86_64/linux) 283e46
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, 
USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, 
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, 
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, 
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 283e46
compiled on 14:23:37 Jul 28 2020 with clang 9.0

```

* **Operating System**:



```
Linux devhpbx005-1.vx 4.9.0-12-amd64 #1 SMP Debian

Re: [sr-dev] Removing Record-Route header

2020-06-08 Thread Andrey Deykunov

Hi Henning,

Here is an example where I call set_advertised_address() and
set_advertised_port() before relaying to the B2BUA:


route
{
...

...

t_on_reply("REPLY_FROM_B2BUA");

if (!add_contact_alias()) {
xlog("L_ERR", "sip_call_id=$ci; Failed aliasing contact $ct
");
send_reply("400", "Bad request");
exit;
}

rewritehostport("<%= @ipaddr %>:1024");

set_advertised_address("<%= @ipaddr %>");
set_advertised_port(5060);

if (!t_relay_to_udp("<%= @ipaddr %>", "1024")) {
xlog("L_ERR", "sip_call_id=$ci; Failed relaying $rm to b2bua");
sl_reply_error();
}
}




The SIP message sent from kamailio to B2BUA after that looks like:

IP (tos 0x10, ttl 64, id 53886, offset 0, flags [none], proto UDP (17),
length 1509)
10.99.8.107.5060 > 10.99.8.107.1024: [bad udp cksum 0x2b7e -> 0xf867!]
UDP, length 1481
E~..@.|.
c.k
c.k..+~INVITE sip:301@10.99.8.107:1024 SIP/2.0
Record-Route: 
Via: SIP/2.0/UDP 10.99.8.107:5060
;branch=z9hG4bKa09f.30a09da6e92bdfa999ccb517ecba6f97.0
Via: SIP/2.0/UDP 10.9.18.77:50342
;received=37.202.8.10;rport=50342;branch=z9hG4bKPj888cd8a8501547fc851e5724ef3f7b1a
Max-Forwards: 69
From: ;tag=51c5ba2b8447498798a4e630dca6c17a
To: 
Contact: 
Call-ID: f92450e6fad14d59bfea70100358bd05
CSeq: 232 INVITE
Allow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY,
REFER, MESSAGE, OPTIONS
Supported: replaces, 100rel, timer, norefersub
Session-Expires: 1800
Min-SE: 90
User-Agent: MicroSIP/3.19.28
Authorization: Digest username="74807310", realm="
hpbx200-1.euc1.telecomsvc.com", nonce="Xt3p017d6KfFSpRngRjuDCInFYIAVnPB",
uri="sip:301@18.156.22.8", response="5e8f6a54064aba0d703c62dcbc746807"
Content-Type: application/sdp
Content-Length:   386
P-Debug-header: advertized local


where  18.156.22.8 in the *'Record-Route'* header is an external IP set by
the '*advertise*' setting  in the '*listen*' block:


listen=udp:<%= @ipaddr %>:5060 advertise <%= node['hpbx'][
'nat_public_ipaddr'] %>:5060


and 10.99.8.107:5060 in the *'Via'* header is an internal IP:port replaced
by  *set_advertised_address*() and *set_advertised_port*() functions before
relaying to B2BUA. So, I need '*Record-Route*' to be having internal IP
just as in the '*Via*' header.

The functions such enable_double_rr() and record_route_advertised_address()
did not help here.
Any ideas?


Thanks,
Andrey





вс, 7 июн. 2020 г. в 12:35, Henning Westerholt :

> Hello,
>
>
>
> not know the details of your cfg, but did you tried already to deactivate
> this parameter:
>
>
>
>
> https://kamailio.org/docs/modules/devel/modules/rr.html#rr.p.enable_double_rr
>
>
>
> Another option could be not not use record_route() but this function
> instead:
>
>
>
>
> https://kamailio.org/docs/modules/devel/modules/rr.html#rr.f.record_route_adv_addr
>
>
>
> Cheers,
>
>
>
> Henning
>
>
>
> --
>
> Henning Westerholt – https://skalatan.de/blog/
>
> Kamailio services – https://gilawa.com
>
>
>
> *From:* sr-dev  *On Behalf Of *Andrey
> Deykunov
> *Sent:* Friday, June 5, 2020 2:33 PM
> *To:* Kamailio (SER) - Development Mailing List  >
> *Subject:* [sr-dev] Removing Record-Route header
>
>
>
> Hi guys,
>
>
>
> I'm using Kamailio 5.3.1 in the AWS cloud. For this purpose I'm setting up
> a 'nat_public_ipaddr' in the 'advertise' feature:
>
>
>
>
>
> listen=tls:<%= @ipaddr %>:5061 advertise <%= node['hpbx'][
> 'nat_public_ipaddr'] %>:5061
>
> listen=tcp:<%= @ipaddr %>:5060 advertise <%= node['hpbx'][
> 'nat_public_ipaddr'] %>:5060
>
> listen=udp:<%= @ipaddr %>:5060 advertise <%= node['hpbx'][
> 'nat_public_ipaddr'] %>:5060
>
>
>
>
>
>
>
> When I need relaying an incoming message to one of our internal services
> (for example B2BUA, VMAIL etc) I'm setting up an internal address
> '@ipaddr'  in the Via header instead of 'nat_public_ipaddr' :
>
>
>
>
>
> set_advertised_address("<%= @ipaddr %>");
>
> set_advertised_port(5060);
>
>
>
>
>
>
>
> All works fine, but Kamailio adds a 'Record-Route' header containing
> 'nat_public_ipaddr' automatically into such messages. How can I remove this
> header or insert my internal address instead before relaying the message?
> Is it possible?
>
>
>
> Thanks,
>
> Andrey
>
>
>
>
>
>
>
___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

[sr-dev] Removing Record-Route header

2020-06-05 Thread Andrey Deykunov

Hi guys,

I'm using Kamailio 5.3.1 in the AWS cloud. For this purpose I'm setting up
a 'nat_public_ipaddr' in the 'advertise' feature:


listen=tls:<%= @ipaddr %>:5061 advertise <%= node['hpbx'][
'nat_public_ipaddr'] %>:5061
listen=tcp:<%= @ipaddr %>:5060 advertise <%= node['hpbx'][
'nat_public_ipaddr'] %>:5060
listen=udp:<%= @ipaddr %>:5060 advertise <%= node['hpbx'][
'nat_public_ipaddr'] %>:5060



When I need relaying an incoming message to one of our internal services
(for example B2BUA, VMAIL etc) I'm setting up an internal address
'@ipaddr'  in the Via header instead of 'nat_public_ipaddr' :


set_advertised_address("<%= @ipaddr %>");
set_advertised_port(5060);



All works fine, but Kamailio adds a 'Record-Route' header containing
'nat_public_ipaddr' automatically into such messages. How can I remove this
header or insert my internal address instead before relaying the message?
Is it possible?

Thanks,
Andrey
___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

[sr-dev] Fwd: RPC command to close all WS connections.

2020-05-18 Thread Andrey Deykunov

-- Forwarded message -
От: Andrey Deykunov 
Date: пн, 18 мая 2020 г. в 12:24
Subject: RPC command to close all WS connections.
To: Daniel-Constantin Mierla 

Hi Daniel,

We're using two nodes (active and passive) of our PBX server in production.
When an active node becomes passive, we should forcibly close all WS
connections, established by clients on this node. So, I've added
'ws.close_all' command to websocket module to let our failover service be
able closing WS connections remotely.

I've added the following code to ws_frame.c:

void ws_rpc_close_all(rpc_t *rpc, void *ctx)
{
ws_connection_t **list = NULL, **list_head = NULL;
ws_connection_t *wsc = NULL;
int ret;

list_head = wsconn_get_list();
if(!list_head)
return;

list = list_head;
wsc = *list_head;
while(wsc) {
LM_WARN("Closing connection\n");
ret = close_connection(, LOCAL_CLOSE, 1000
, str_status_normal_closure);
wsc = *(++list);
}
wsconn_put_list(list_head);
}

but I think this code may be unsafe and could corrupt shared
memory, because I've got some segfaults during failovers after adding this
command.

What do you think? Is it possible to close connections properly
and safety for shared memory?

Thanks,
Andrey
___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

[sr-dev] Kamailio stucks at __read_nocancel during stop.

2020-04-09 Thread Andrey Deykunov

Hi,

We're getting periodically core dumps when stopping Kamailio:


GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/lib/ums/sbin/kamailio...done.
[New LWP 9832]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/var/lib/ums/sbin/kamailio -m 2048 -M 12 -P
/var/run/kamailio/kamailio.pid -f /'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x7f79c24e842a in __GI_abort () at abort.c:89
#2  0x0043f92a in sig_alarm_abort (signo=14) at main.c:679
#3  
#4  0x7f79c258f910 in __read_nocancel () at
../sysdeps/unix/syscall-template.S:84
#5  0x7f793cbd7421 in vio_read_buff () from
/var/lib/ums/lib/mysql/libmysqlclient.so.16
#6  0x7f793cbd8aa0 in my_real_read () from
/var/lib/ums/lib/mysql/libmysqlclient.so.16
#7  0x7f793cbd87d6 in my_net_read () from
/var/lib/ums/lib/mysql/libmysqlclient.so.16
#8  0x7f793cbd085d in cli_safe_read () from
/var/lib/ums/lib/mysql/libmysqlclient.so.16
#9  0x7f793cbd539a in cli_read_query_result () from
/var/lib/ums/lib/mysql/libmysqlclient.so.16
#10 0x7f793cbd436d in mysql_real_query () from
/var/lib/ums/lib/mysql/libmysqlclient.so.16
#11 0x7f793ce80c13 in db_mysql_submit_query (_h=0x7f79c1786890,
_s=0x7f793f379250 ) at km_dbase.c:111
#12 0x7f793f16baba in db_do_submit_query (_h=0x7f79c1786890,
_query=0x7f793f379250 , submit_query=0x7f793ce801a0
) at db_query.c:58
#13 0x7f793f16e114 in db_do_delete (_h=0x7f79c1786890,
_k=0x7ffeb3ca8df8, _o=0x0, _v=0x7ffeb3ca8dd0, _n=1, val2str=0x7f793cea8990
, submit_query=0x7f793ce801a0 ) at
db_query.c:300
#14 0x7f793ce86c17 in db_mysql_delete (_h=0x7f79c1786890,
_k=0x7ffeb3ca8df8, _o=0x0, _v=0x7ffeb3ca8dd0, _n=1) at km_dbase.c:510
#15 0x7f793f3cb841 in db_delete_ucontact_ruid (_c=0x7f794186b3c0) at
ucontact.c:1552
#16 0x7f793f3cd096 in db_delete_ucontact (_c=0x7f794186b3c0) at
ucontact.c:1570
#17 0x7f793f3ad0c0 in wb_timer (_r=0x7f79411d7330) at urecord.c:401
#18 0x7f793f3ab451 in timer_urecord (_r=0x7f79411d7330) at urecord.c:463
#19 0x7f793f3978aa in mem_timer_udomain (_d=0x7f7940d9baa0, istart=0,
istep=1) at udomain.c:1224
#20 0x7f793f3d4608 in synchronize_all_udomains (istart=0, istep=1) at
dlist.c:756
#21 0x7f793f3a2768 in destroy () at usrloc_mod.c:464
#22 0x00637e2c in destroy_modules () at core/sr_module.c:746
#23 0x0041c4d8 in cleanup (show_status=1) at main.c:555
#24 0x00423af7 in shutdown_children (sig=15, show_status=1) at
main.c:696
#25 0x0041f081 in handle_sigs () at main.c:727
#26 0x00432a01 in main_loop () at main.c:1806
#27 0x0043df6f in main (argc=9, argv=0x7ffeb3cac348) at main.c:2802



Looks like libmysql stucks at __read_nocancel() syscall. The exit_timeout
parameter is set by default (60  sec), Kamailio version 5.3.1.

Any ideas?

Thanks,
Andrey
___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] High memory consumption during load tests

2020-04-09 Thread Andrey Deykunov

Thank you Henning.  I theorized it and you proved my point.
We're using 5.3.1.

Andrey

чт, 9 апр. 2020 г. в 09:59, Henning Westerholt :

> Hello,
>
>
>
> Please also give the Kamailio version that you are using, just to sort out
> eventual already fixed memory leaks.
>
>
>
> About the question regarding the 2GB memory mapping – according to your
> configuration this should be shared memory (not assuming a memory leak in
> system memory because of a library etc..). So, it should be not 2GB per
> process, but 2GB in total.
>
>
>
> And the memory is also not completely used, only 855MB of it:
>
>
>
> shmem:real_used_size = 855.574.648
>
>
>
> Maximum private per process memory is according to your cfg 12 MB.
>
>
>
> Cheers,
>
>
>
> Henning
>
>
>
> --
>
> Henning Westerholt – https://skalatan.de/blog/
>
> Kamailio services – https://gilawa.com
>
>
>
>
>
> *From:* sr-dev  *On Behalf Of *Andrey
> Deykunov
> *Sent:* Wednesday, April 8, 2020 12:19 PM
> *To:* Kamailio (SER) - Development Mailing List  >
> *Subject:* [sr-dev] High memory consumption during load tests
>
>
>
> Hi,
>
>
>
> I've noticed more than 14 Gb RSS memory consumption during load test (8000
> WSS clients send registrations):
>
>
>
>
>
> […]
>
>
>
>
>
> # ps aux | grep kamailio
>
> hpbx 15948  0.0  0.1 2232328 22920 ?   S01:40   0:00
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15949  1.8  3.8 2232460 789212 ?  S01:40   1:23
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15950  1.8  3.8 2232460 790164 ?  S01:40   1:22
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15951  1.7  3.8 2232460 790656 ?  S01:40   1:22
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15952  1.8  3.8 2232460 790180 ?  S01:40   1:22
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15953  1.7  3.8 2232460 788192 ?  S01:40   1:22
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15954  1.8  3.8 2232460 789484 ?  S01:40   1:22
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15955  1.7  3.8 2232460 789640 ?  S01:40   1:22
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15959  1.8  3.8 2232460 788104 ?  S01:40   1:22
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15960  0.0  0.0 2232328 8272 ?S01:40   0:00
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15961  0.0  0.0 2232328 8272 ?S01:40   0:00
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15963  0.0  0.0 2232328 8272 ?S01:40   0:00
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15964  0.0  0.0 2232328 8272 ?S01:40   0:00
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15966  0.0  0.0 2232328 8272 ?S01:40   0:00
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15968  0.0  0.0 2232328 8272 ?S01:40   0:00
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15969  0.0  0.0 2232328 8272 ?S01:40   0:00
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15971  0.0  0.0 2232328 8272 ?S01:40   0:00
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
> hpbx 15972  0.0  0.0 2232328 8152 ?S01:40   0:00
> /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P
> /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
>

Re: [sr-dev] Many CLOSE_WAIT connections

2020-01-21 Thread Andrey Deykunov

Hi Daniel,

We build ver. 5.3.1 from sources. So, probably we should enable this flag
in our build script. How to do that?

BTW. Also, we need some rpc command that closes all WS connections. I'm
suggesting something like this:

void ws_rpc_close_all(rpc_t *rpc, void *ctx)
{
int hash, ret;
ws_connection_t *wsc;

WSCONN_LOCK;
wsc = wsconn_used_list->head;
while(wsc) {
ret = close_connection(, LOCAL_CLOSE, 1000, str_status_normal_closure);
}
WSCONN_UNLOCK;
}

but I think the code above is not so quite safety and may causes deadlock.
How to close the connections properly?


Thanks,
Andrey



вт, 21 янв. 2020 г. в 15:37, Daniel-Constantin Mierla :

> Hello,
>
> if you installed from deb packages provided via deb.kamailio.org, upgrade
> to latest 5.3.x (now 5.3.2), the fix to use process shared mutex in libssl
> 1.1 was not enabled before due to a packaging mismatching make condition.
>
> Cheers,
> Daniel
> On 20.01.20 12:07, Andrey Deykunov wrote:
>
> Hi guys,
>
> Got the issue on 5.3.1 with openssl1.1, debian9.
> After 3 working days of tests (about ~30-50 WSS clients), suddenly we've
> got
> a lot of connections stucked in CLOSE_WAIT state. Kamailio called
> sig_alarm_abort() when we try to reboot.
>
> Thanks,
> Andrey
>
> ___
> Kamailio (SER) - Development Mailing 
> Listsr-dev@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
>
> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- 
> www.linkedin.com/in/miconda
> Kamailio Advanced Training - March 9-11, 2020, Berlin - www.asipto.com
> Kamailio World Conference - April 27-29, 2020, in Berlin -- 
> www.kamailioworld.com
>
>
___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

[sr-dev] Many CLOSE_WAIT connections

2020-01-20 Thread Andrey Deykunov

Hi guys,

Got the issue on 5.3.1 with openssl1.1, debian9.
After 3 working days of tests (about ~30-50 WSS clients), suddenly we've got
a lot of connections stucked in CLOSE_WAIT state. Kamailio called
sig_alarm_abort() when we try to reboot.

Thanks,
Andrey
___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Memory leaks in websocket module on adding tcp connection check (#1892)

2019-05-20 Thread Andrey Deykunov

Hi Daniel! I was on PTO, will look into it asap.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1892#issuecomment-493991962___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

[sr-dev] [kamailio/kamailio] Memory leaks in websocket module (#1892)

2019-03-15 Thread Andrey Deykunov

### Description

We had Kamailio 5.1.4 with websocket module. Unfortunately, our clients don't 
support websocket keepalive mechanism at all, so I used TCP keepalive instead 
with the following parameters:

```
tcp_keepalive=yes
tcp_keepcnt=6
tcp_keepidle=60
tcp_keepintvl=10
```


and set up KEEPALIVE_MECHANISM_NONE:

`modparam("websocket", "keepalive_mechanism", 0)`


During load testing and debugging, when 8k clients sent registrations, it was 
found out that shared memory was not freed after closing connections 
(_ws_connection_list_t *wsconn_used_list_ variable in ws_conn.c ).


### Possible Solutions

I've decided to add new keepalive mechanism that periodically checks TCP 
connection related to websocket:

```
enum
{
KEEPALIVE_MECHANISM_NONE = 0,
KEEPALIVE_MECHANISM_PING = 1,
KEEPALIVE_MECHANISM_PONG = 2,
KEEPALIVE_MECHANISM_TCP_CONN_CHECK = 3
};  
```


and added the line to config:

```
# Enable custom tcp-connection-health based keepalive mechanism (3)
# KEEPALIVE_MECHANISM_NONE = 0,
# KEEPALIVE_MECHANISM_PING = 1,
# KEEPALIVE_MECHANISM_PONG = 2
# KEEPALIVE_MECHANISM_TCP_CONN_CHECK = 3
modparam("websocket", "keepalive_mechanism", 3)
```


Also, I've implemented the mechanism in ws_keepalive function:

```
void ws_keepalive(unsigned int ticks, void *param)
{
int check_time =
(int)time(NULL) - cfg_get(websocket, ws_cfg, keepalive_timeout);

ws_connection_t **list = NULL, **list_head = NULL;
ws_connection_t *wsc = NULL;

/* get an array of pointer to all ws connection */
list_head = wsconn_get_list();
if(!list_head)
return;

list = list_head;
wsc = *list_head;
while(wsc && wsc->last_used < check_time) {
if (ws_keepalive_mechanism == KEEPALIVE_MECHANISM_TCP_CONN_CHECK) {
struct tcp_connection *con = tcpconn_get(wsc->id, 0, 0, 0, 0);
if(!con) {
LM_INFO("tcp connection has been lost\n");
wsc->state = WS_S_CLOSING;
}
}

if(wsc->state == WS_S_CLOSING || wsc->awaiting_pong) {
LM_INFO("forcibly closing connection\n");
wsconn_close_now(wsc);
} else {
int opcode = (ws_keepalive_mechanism == KEEPALIVE_MECHANISM_PING)
? OPCODE_PING
: OPCODE_PONG;
ping_pong(wsc, opcode);
}

wsc = *(++list);
}

wsconn_put_list(list_head);
}

```

and changed memory allocation method in wsconn_get_list and wsconn_put_list 
methods from pkg to shm, because, as it turned out during load testing, using 
pkg_malloc (the C malloc) in this functions may cousing fails under serious 
loads.

These modifications solved the problem. But about a week ago we've started 
switching to ver. 5.2.1 and found a lot of changes in the websocket module. So, 
I've added my changes in this commit 
https://github.com/korizza/kamailio/commit/b3e03d03574ff4ff076005bb8a01d7461af2f8f5
 . Please take a look.



### Additional Information

Adding ws_conn_put_id in this commit 
https://github.com/kamailio/kamailio/commit/a975bca1702ea2f3db47f834f7e4da2786ced201#diff-59c50f19ab1ccf4afe10617cdc346bc2
 did not solve problem with ref counter increasing.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1892___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

[sr-dev] Memory leaks in websocket module

2019-03-15 Thread Andrey Deykunov

Hi guys,

We had Kamailio 5.1.4 with websocket module. Unfortunately, our clients
don't support websocket keepalive mechanism at all, so I used TCP keepalive
instead with the following parameters:

tcp_keepalive=yes
tcp_keepcnt=6
tcp_keepidle=60
tcp_keepintvl=10


and set up KEEPALIVE_MECHANISM_NONE:

modparam("websocket", "keepalive_mechanism", 0)


During load testing and debugging, when 8k clients sent registrations, it
was found out that shared memory was not freed after closing connections.
So I've decided to add new keepalive mechanism that periodically checks TCP
connection related to websocket:

enum
{
KEEPALIVE_MECHANISM_NONE = 0,
KEEPALIVE_MECHANISM_PING = 1,
KEEPALIVE_MECHANISM_PONG = 2,
KEEPALIVE_MECHANISM_TCP_CONN_CHECK = 3
};


and added the line to config:

# Enable custom tcp-connection-health based keepalive mechanism (3)
# KEEPALIVE_MECHANISM_NONE = 0,
# KEEPALIVE_MECHANISM_PING = 1,
# KEEPALIVE_MECHANISM_PONG = 2
# KEEPALIVE_MECHANISM_TCP_CONN_CHECK = 3
modparam("websocket", "keepalive_mechanism", 3)


Also, I've implemented the mechanism in ws_keepalive function:

void ws_keepalive(unsigned int ticks, void *param)
{
int check_time =
(int)time(NULL) - cfg_get(websocket, ws_cfg, keepalive_timeout);

ws_connection_t **list = NULL, **list_head = NULL;
ws_connection_t *wsc = NULL;

/* get an array of pointer to all ws connection */
list_head = wsconn_get_list();
if(!list_head)
return;

list = list_head;
wsc = *list_head;
while(wsc && wsc->last_used < check_time) {
if (ws_keepalive_mechanism == KEEPALIVE_MECHANISM_TCP_CONN_CHECK) {
struct tcp_connection *con = tcpconn_get(wsc->id, 0, 0, 0, 0);
if(!con) {
LM_INFO("tcp connection has been lost\n");
wsc->state = WS_S_CLOSING;
}
}

if(wsc->state == WS_S_CLOSING || wsc->awaiting_pong) {
LM_INFO("forcibly closing connection\n");
wsconn_close_now(wsc);
} else {
int opcode = (ws_keepalive_mechanism ==
KEEPALIVE_MECHANISM_PING)
? OPCODE_PING
: OPCODE_PONG;
ping_pong(wsc, opcode);
}

wsc = *(++list);
}

wsconn_put_list(list_head);
}


and changed memory allocation method in wsconn_get_list and wsconn_put_list
methods from pkg to shm, because, as it turned out during load testing,
pkg_malloc (the C malloc) may cousing fails under huge loads.

These modifications solved the problem. But about a week ago we've started
switching to ver. 5.2.1 and found a lot of changes in the websocket module.
So, I've added my changes in this commit
https://github.com/korizza/kamailio/commit/b3e03d03574ff4ff076005bb8a01d7461af2f8f5
.
Please take a look.

BTW: adding ws_conn_put_id in this commit
https://github.com/kamailio/kamailio/commit/a975bca1702ea2f3db47f834f7e4da2786ced201#diff-59c50f19ab1ccf4afe10617cdc346bc2
did
not solve problem with ref counter increasing.


Thanks,
Andrey Deykunov
___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

[sr-dev] Build kamailio 5.3.1 db_mysql module with percona.

Re: [sr-dev] [kamailio/kamailio] Segmentation fault occured in rpc_mod_print function in runtime. (#2460)

Re: [sr-dev] [kamailio/kamailio] Segmentation fault occured in rpc_mod_print function in runtime. (#2460)

Re: [sr-dev] [kamailio/kamailio] Segmentation fault occured in rpc_mod_print function in runtime. (#2460)

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

[sr-dev] [kamailio/kamailio] Segmentation fault occured in rpc_mod_print function in runtime. (#2460)

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

Re: [sr-dev] [kamailio/kamailio] core: keep a global flag in shm for shut down phase (6da946e)

Re: [sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

[sr-dev] [kamailio/kamailio] Segmentation fault in rpc_mod_print function while shutting down server. (#2433)

Re: [sr-dev] Removing Record-Route header

[sr-dev] Removing Record-Route header

[sr-dev] Fwd: RPC command to close all WS connections.

[sr-dev] Kamailio stucks at __read_nocancel during stop.

Re: [sr-dev] High memory consumption during load tests

Re: [sr-dev] Many CLOSE_WAIT connections

[sr-dev] Many CLOSE_WAIT connections

Re: [sr-dev] [kamailio/kamailio] Memory leaks in websocket module on adding tcp connection check (#1892)

[sr-dev] [kamailio/kamailio] Memory leaks in websocket module (#1892)

[sr-dev] Memory leaks in websocket module

25 matches

Site Navigation

Mail list logo

Footer information