[SR-Users] R: R: Kamailio connection issue
Hi All, thanks for your replies. As suggested I did the following steps: 1) Checked our backend and now, after creating some indexes, is running properly with the right query time 2) Increased the connection limit to 64K in kamailio.cfg file 3) Set tcp_connection_lifetime to 5s for new connection and once a sip client gets registered it’s increased to 3605s 4) Set tcp_connect_timeout=2 and tcp_send_timeout = 2 5) Added tcp_keepalive as suggested but the issue still persists. I've also checked our kernel settings which are the same as described in doc/tutorials/tcp_tunning.txt. I haven't used async module which can be used for sending push notification yet because this issue happens also when the system is in idle with no calls in progress. Kamailio processes (tcp main and receivers) are not running with high CPU (1-2% of CPU load) but sometimes tcp_main process gets our app connection after some secondsit causes delayed app registration which lead to have some missed calls, because calls are closed before app gets registered. In the kamailio log file there are the same warning messages described below, in which tcp_main is not able to find free tcp receivers. Mar 4 15:40:23 server-xip-01 /usr/local/sbin/kamailio[29417]: DEBUG: [tcp_main.c:3816]: send2child(): WARNING: no free tcp receiver, connection passed to the least busy one (20) Moreover the tcp_main thread, which should be the thread that handles the incoming connection, is always busy to release the connections got from its children processes Mar 6 07:53:30 server-xip-01 /usr/local/sbin/kamailio[1845]: DEBUG: [io_wait.h:380]: io_watch_add(): DBG: io_watch_add(0xa20dc0, 3975, 2, 0x7f19e6d6d2b8), fd_no=3922 Mar 6 07:53:30 server-xip-01 /usr/local/sbin/kamailio[1845]: DEBUG: [tcp_main.c:3378]: handle_tcp_child(): CONN_RELEASE 0x7f19e6d6d2b8 refcnt= 1 Mar 6 07:53:30 server-xip-01 /usr/local/sbin/kamailio[1845]: DEBUG: [io_wait.h:602]: io_watch_del(): DBG: io_watch_del (0xa20dc0, 2272, -1, 0x0) fd_no=3923 called Any ideas what could cause this behaviour? Many thanks T. Tomas Zanet R Software Designer tza...@came.com CAME S.p.A. Da: sr-users [mailto:sr-users-boun...@lists.kamailio.org] Per conto di M S Inviato: mercoledì 4 marzo 2020 20:10 A: Kamailio (SER) - Users Mailing List Oggetto: Re: [SR-Users] R: Kamailio connection issue You have many problems, 1. You do not seem to have TCP keepalives setup. So idle TCP connections may be disrupted, closed or reset by any router on the internet, forcing SIP client to re-register. As a result many more tcp connections are created by a single SIP end point then expected. You said you have 5K active users with 8K connection limit, which is way too low. The limit should be at least more then twice the active users (12.5K in your case). I usually just set it to maximum i.e. 64K. 2. TCP connection life time of 3605 seconds is way too high. There will be many many hack attempts on your server, some users will enter wrong password, some users change their network (and thus need to re-register), so allowing 3605 seconds life time for ALL of them is a HUGE mistake, and you will run out of connection limit pretty quick. So keep the life time as small as possible 5 or 10 seconds (approx. max time the server takes to process sip register x 20 e.g. 250ms x 20 = 5000 ms = 5 seconds). Once the SIP client successfully registers, you can increase the TCP life time using, https://kamailio.org/docs/modules/4.4.x/modules/tcpops.html#tcpops.f.tcp_set_connection_lifetime 3. Similarly setup TCP keepalives according to SIP Register success, https://kamailio.org/docs/modules/4.4.x/modules/tcpops.html#tcpops.f.tcp_keepalive_enable https://kamailio.org/docs/modules/4.4.x/modules/tcpops.html#tcpops.f.tcp_keepalive_disable 4. If average SIP packet processing time is too high e.g. due to slow db connection for authentication etc. push notification setup, web queries etc. then use ASYNC for all time consuming tasks. So TCP listener processes get freed quickly and can process more new connections. 5. It's good to monitor and analyze TCP connection states and events, especially when TCP connections closes and why? See the TCPOPS module for more help. Hope this helps. On Wed, Mar 4, 2020 at 5:30 PM Tomas Zanet wrote: As you suggested I increased the debug level from -1 to 3 and I got this Kamailio log: Mar 4 15:40:21 server-xip-01 /usr/local/sbin/kamailio[29417]: DEBUG: [io_wait.h:602]: io_watch_del(): DBG: io_watch_del (0xa20dc0, 3433, -1, 0x0) fd_no=2922 called Mar 4 15:40:21 server-xip-01 /usr/local/sbin/kamailio[29417]: DEBUG: [tcp_main.c:4139]: handle_tcpconn_ev(): sending to child, events 1 Mar 4 15:40:21 server-xip-01 /usr/local/sbin/kamailio[29417]: DEBUG: [tcp_main.c:3816]: send2child(): WARNING: no free tcp receiver, connection passed to the least busy one (30) Mar 4 15:40:21 server-xip-01 /usr/local/sbin/kamailio[29417]: DEBUG: [tcp_main.c:3821
[SR-Users] R: Kamailio connection issue
on passed to the least busy one (18) Mar 4 15:40:24 server-xip-01 /usr/local/sbin/kamailio[29417]: DEBUG: [tcp_main.c:3816]: send2child(): WARNING: no free tcp receiver, connection passed to the least busy one (18) Mar 4 15:40:24 server-xip-01 /usr/local/sbin/kamailio[29417]: DEBUG: [tcp_main.c:3816]: send2child(): WARNING: no free tcp receiver, connection passed to the least busy one (18) ….. …. As far as I know, I should increase the tcp_children parameter, right?. Now, it’s set to 16 but it’s pretty clear which is not enough. I increased it step by step till 48 but that’s still not enough. If I set it to 64 then Kamailio failed to start. What would you recommend for a server which has more or less 5K users? Many thanks. T. Tomas Zanet R Software Designer tza...@came.com <http://www.came.com>[cid:LogoNew_f0706ef0-76e6-499c-97ed-10496392d3f5.png] CAME S.p.A. Sede legale e operativa Via Martiri della Libertà 15 31030 Dosson di Casier (TV) / Italy T +39 0422 4940 F +39 0422 4941 P.IVA 03481280265 Cod. dest. fatturazione elettronica: UHWULCH Sede operativa Via Cornia 1/b,c 33079 Sesto al Reghena (PN) / Italy www.came.com<http://www.came.com> Follow us on: Facebook<https://www.facebook.com/CAME-Italia-1604586846431438/> / Twitter<https://twitter.com/CameGlobal> / Youtube<https://www.youtube.com/user/CameChannel> / Linkedin<https://www.linkedin.com/company/came> Le informazioni contenute in questa e-mail e negli eventuali allegati sono riservate e confidenziali e ne è vietata la diffusione e distribuzione in qualunque modo. Qualora non ne fosse il legittimo destinatario, è invitato a non diffonderle e ad eliminarle, dandone gentilmente informazione al mittente. The information included in this e-mail and any attachments are confidential and can not be disclosed, duplicated or redistributed by any means. If you are not the correct recipient, you are kindly requested to cancel it and notify the sender immediately Da: sr-users [mailto:sr-users-boun...@lists.kamailio.org] Per conto di Denys Pozniak Inviato: mercoledì 4 marzo 2020 15:00 A: Kamailio (SER) - Users Mailing List Oggetto: Re: [SR-Users] Kamailio connection issue Hello! You can try to increase the debug printing level (like debug=2 or even debug=3) and check syslog when Kamailio gets stuck. ср, 4 мар. 2020 г. в 13:12, Tomas Zanet mailto:tza...@came.com>>: Hi All, In these days I'm facing an issue or a misconfiguration, I don't know, with our remote apps (SIP clients) and our cloud server in which there's a Kamailio version 4.4.4. Sometimes our app takes some seconds (about 10-12sec) to connect to the Kamailio socket (sips on 5061) and our app is stuck on connect sytem call. To debug this behaviour I put a rule in the iptable like this: sudo iptables -I INPUT -p tcp -m tcp --dport 5061 -m state --state NEW -j LOG --log-level 1 --log-prefix "New Connection " when everything goes well, there is one row which describes the new incoming connection Mar 4 07:08:40 server-xip-01 kernel: New Connection IN=eth0 OUT= MAC=bc:76:4e:20:1f:fb:e8:ed:f3:3a:52:41:08:00 SRC=X.X.X.X DST=Y.Y.Y.Y LEN=64 TOS=0x00 PREC=0x00 TTL=45 ID=0 DF PROTO=TCP SPT=15819 DPT=5061 WINDOW=65535 RES=0x00 CWR ECE SYN URGP=0 when our app gets stuck in the connect, there are many attempts and seems that Kamailio is not ready to accept this connection: Mar 4 07:11:54 server-xip-01 kernel: New Connection IN=eth0 OUT= MAC=bc:76:4e:20:1f:fb:e8:ed:f3:3a:52:41:08:00 SRC=X.X.X.X DST=Y.Y.Y.Y LEN=64 TOS=0x00 PREC=0x00 TTL=45 ID=0 DF PROTO=TCP SPT=52900 DPT=5061 WINDOW=65535 RES=0x00 CWR ECE SYN URGP=0 Mar 4 07:11:54 server-xip-01 kernel: New Connection IN=eth0 OUT= MAC=bc:76:4e:20:1f:fb:e8:ed:f3:3a:52:41:08:00 SRC=X.X.X.X DST=Y.Y.Y.Y LEN=64 TOS=0x00 PREC=0x00 TTL=45 ID=0 DF PROTO=TCP SPT=52900 DPT=5061 WINDOW=65535 RES=0x00 SYN URGP=0 Mar 4 07:11:55 server-xip-01 kernel: New Connection IN=eth0 OUT= MAC=bc:76:4e:20:1f:fb:e8:ed:f3:3a:52:41:08:00 SRC=X.X.X.X DST=Y.Y.Y.Y LEN=64 TOS=0x00 PREC=0x00 TTL=45 ID=0 DF PROTO=TCP SPT=52900 DPT=5061 WINDOW=65535 RES=0x00 SYN URGP=0 Mar 4 07:11:55 server-xip-01 kernel: New Connection IN=eth0 OUT= MAC=bc:76:4e:20:1f:fb:e8:ed:f3:3a:52:41:08:00 SRC=X.X.X.X DST=Y.Y.Y.Y LEN=64 TOS=0x00 PREC=0x00 TTL=45 ID=0 DF PROTO=TCP SPT=52900 DPT=5061 WINDOW=65535 RES=0x00 SYN URGP=0 Mar 4 07:11:55 server-xip-01 kernel: New Connection IN=eth0 OUT= MAC=bc:76:4e:20:1f:fb:e8:ed:f3:3a:52:41:08:00 SRC=X.X.X.X DST=Y.Y.Y.Y LEN=64 TOS=0x00 PREC=0x00 TTL=45 ID=0 DF PROTO=TCP SPT=52900 DPT=5061 WINDOW=65535 RES=0x00 SYN URGP=0 Mar 4 07:11:56 server-xip-01 kernel: New Connection IN=eth0 OUT= MAC=bc:76:4e:20:1f:fb:e8:ed:f3:3a:52:41:08:00 SRC=X.X.X.X DST=Y.Y.Y.Y LEN=64 TOS=0x00 PREC=0x00 TTL=45 ID=0 DF PROTO=TCP SPT=52900 DPT=5061 WINDOW=65535 RES=0x00 SYN URGP=0 Mar 4 07:11:56 server-xip-01 kernel: New Connection IN=eth0 OUT= MAC=bc:76:4e:20:1f:fb:e8:ed:f3:3a:52:41:08:00 SRC
[SR-Users] Kamailio connection issue
don't see anything wrong, I don't see messages like "[tcp_main.c:3913]: handle_new_connect(): maximum number of tls connections exceeded". Here is the output of kamctl ps command Process:: ID=0 PID=11725 Type=main process - attendant Process:: ID=1 PID=11727 Type=udp receiver child=0 sock=10.208.194.26:5061 Process:: ID=2 PID=11728 Type=udp receiver child=1 sock=10.208.194.26:5061 Process:: ID=3 PID=11729 Type=udp receiver child=2 sock=10.208.194.26:5061 Process:: ID=4 PID=11730 Type=udp receiver child=3 sock=10.208.194.26:5061 Process:: ID=5 PID=11731 Type=udp receiver child=4 sock=10.208.194.26:5061 Process:: ID=6 PID=11732 Type=udp receiver child=5 sock=10.208.194.26:5061 Process:: ID=7 PID=11733 Type=udp receiver child=6 sock=10.208.194.26:5061 Process:: ID=8 PID=11734 Type=udp receiver child=7 sock=10.208.194.26:5061 Process:: ID=9 PID=11735 Type=udp receiver child=8 sock=10.208.194.26:5061 Process:: ID=10 PID=11736 Type=udp receiver child=9 sock=10.208.194.26:5061 Process:: ID=11 PID=11737 Type=udp receiver child=10 sock=10.208.194.26:5061 Process:: ID=12 PID=11738 Type=udp receiver child=11 sock=10.208.194.26:5061 Process:: ID=13 PID=11739 Type=udp receiver child=12 sock=10.208.194.26:5061 Process:: ID=14 PID=11740 Type=udp receiver child=13 sock=10.208.194.26:5061 Process:: ID=15 PID=11741 Type=udp receiver child=14 sock=10.208.194.26:5061 Process:: ID=16 PID=11742 Type=udp receiver child=15 sock=10.208.194.26:5061 Process:: ID=17 PID=11743 Type=slow timer Process:: ID=18 PID=11744 Type=timer Process:: ID=19 PID=11745 Type=secondary timer Process:: ID=20 PID=11746 Type=MI FIFO Process:: ID=21 PID=11747 Type=USRLOC Timer Process:: ID=22 PID=11748 Type=USRLOC Timer Process:: ID=23 PID=11749 Type=USRLOC Timer Process:: ID=24 PID=11750 Type=USRLOC Timer Process:: ID=25 PID=11751 Type=ctl handler Process:: ID=26 PID=11752 Type=RTIMER SEC EXEC Process:: ID=27 PID=11753 Type=TIMER NH Process:: ID=28 PID=11754 Type=tcp receiver (generic) child=0 Process:: ID=29 PID=11755 Type=tcp receiver (generic) child=1 Process:: ID=30 PID=11756 Type=tcp receiver (generic) child=2 Process:: ID=31 PID=11757 Type=tcp receiver (generic) child=3 Process:: ID=32 PID=11758 Type=tcp receiver (generic) child=4 Process:: ID=33 PID=11759 Type=tcp receiver (generic) child=5 Process:: ID=34 PID=11760 Type=tcp receiver (generic) child=6 Process:: ID=35 PID=11761 Type=tcp receiver (generic) child=7 Process:: ID=36 PID=11762 Type=tcp receiver (generic) child=8 Process:: ID=37 PID=11763 Type=tcp receiver (generic) child=9 Process:: ID=38 PID=11764 Type=tcp receiver (generic) child=10 Process:: ID=39 PID=11765 Type=tcp receiver (generic) child=11 Process:: ID=40 PID=11766 Type=tcp receiver (generic) child=12 Process:: ID=41 PID=11767 Type=tcp receiver (generic) child=13 Process:: ID=42 PID=11768 Type=tcp receiver (generic) child=14 Process:: ID=43 PID=11769 Type=tcp receiver (generic) child=15 Process:: ID=44 PID=11770 Type=tcp main process Can anyone address me to find out this happens, please? Please let me know if you need further information... Thanks in advance, Best Regards Tomas Tomas Zanet R Software Designer tza...@came.com CAME S.p.A. ___ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
[SR-Users] R: 4.4.4 CPU 100%
Hello, I found the issue which was due to a misconfiguration of kamalio. I forgot to set open_files_limit value which was set to default values 1024, not enough for our installation. Increasing it, now everything works fine. Maybe this could help anyone who use TLS connections. BR. T. Tomas Zanet R Software designer tza...@came.com CAME S.p.A. -Messaggio originale- Da: sr-users [mailto:sr-users-boun...@lists.kamailio.org] Per conto di Tomas Zanet Inviato: venerdì 15 dicembre 2017 15:11 A: Kamailio (SER) - Users Mailing List <sr-us...@lists.sip-router.org> Oggetto: [SR-Users] 4.4.4 CPU 100% Hi all, I’ve been using Kamailio 4.4.4 for 8 months without any problems but in recent days, I’m having some issues with our server and, when it happens, it cannot get new registrations. Using top I saw a kamailio process is eating all CPU. PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 1556 kamailio 20 0 1010884 27624 24520 R 98.5 1.4 150:07.59 kamailio I got backtrace attaching to Kamailio with gdb. Here is the backtrace: #0 0x7fb996992d6d in accept () at ../sysdeps/unix/syscall-template.S:81 #1 0x005f3615 in handle_new_connect (si=0x7fb9960b4fa8) at tcp_main.c:3894 #2 0x005f7029 in handle_io (fm=0x7fb9961da520, ev=1, idx=-1) at tcp_main.c:4199 #3 0x005c68b2 in io_wait_loop_epoll (h=0xa1fdc0 , t=5, repeat=0) at io_wait.h:1065 #4 0x005fa04a in tcp_main_loop () at tcp_main.c:4490 #5 0x004b48ba in main_loop () at main.c:1712 #6 0x004bad12 in main (argc=13, argv=0x7ffd14247c18) at main.c:2631 (gdb) bt At the moment I found the issue, there were 862 sip client registered correctly. Our server is configured to use only TLS and mysql as backend. /usr/local/sbin/kamailio -v version: kamailio 4.4.4 (x86_64/linux) 852c47 flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 852c47 compiled on 15:56:18 Dec 15 2016 with gcc 4.8.4 Can anyone address me to find out why this happens? Many thanks, T. Tomas Zanet R Software designer tza...@came.com CAME S.p.A. ___ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users ___ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
