Hi Bogdan,
Please find the values. Any common folder you have access for the outside world
to put files?
If yes, share me the details. I can upload the core file there.
(gdb) print r_buf
$1 = (struct retr_buf *) 0x7fded2484098
(gdb) print *r_buf
$2 = {activ_type = 0, buffer = {
s = 0x7fded0ca79f0 "INVITE sip:++12001000004@10.0.0.1:5060
SIP/2.0\r\nRecord-Route: <sip:96.119.240.35;lr>\r\nTo: sut
<sip:+12452123456@10.0.0.1:5060;user=phone>\r\nFrom: sipp
<sip:sipp@10.0.0.2:5060;user=phone"..., len = 968}, dst = {proto = 1,
proto_reserved1 = 0, to = {s = {sa_family = 2, sa_data =
"\023\304`w\347\375\000\000\000\000\000\000\000"}, sin = {
sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 4259805024},
sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {
sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 4259805024,
sin6_addr = {__in6_u = {
__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0,
0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
sin6_scope_id = 0}}, send_sock = 0x7fded841dcb0}, retr_timer = {next_tl
= 0x7fded1af3518, prev_tl = 0x7fded15c8ae0,
ld_tl = 0x0, time_out = 169500000, timer_list = 0x7fded0acbc48, deleted =
1, set = 0}, fr_timer = {next_tl = 0x0, prev_tl = 0x0,
ld_tl = 0x0, time_out = 165, timer_list = 0x7fded8005b00 <detached_timer>,
deleted = 0, set = 0}, retr_list = RT_T2, my_T = 0x0,
branch = 0}
Regards,
Agalya
From: Bogdan-Andrei Iancu [mailto:bog...@opensips.org]
Sent: Tuesday, October 04, 2016 11:52 AM
To: Ramachandran, Agalya (Contractor) <agalya_ramachand...@comcast.com>;
OpenSIPS users mailling list <users@lists.opensips.org>
Subject: Re: opensips crash when doing load test
Could you also print r_buf and *r_buf ?
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
On 04.10.2016 16:08, Ramachandran, Agalya (Contractor) wrote:
Hi Bogdan,
I have printed the "t" variable value. It says "$1 = (struct cell *) 0x0".
Is there any link available to upload/share the core files? Because size of the
core file 45KB and couldn't attach in e-mail.
(gdb) frame 0
#0 final_response_handler (fr_tl=0x7fded2484110) at timer.c:383
383 if (is_invite(t) && should_cancel_branch(t, r_buf->branch) ) {
(gdb) list +
378 current_processing_ctx = my_ctx;
379 /* set the T context too */
380 set_t( t );
381
382 /* out-of-lock do the cancel I/O */
383 if (is_invite(t) && should_cancel_branch(t, r_buf->branch) ) {
384 set_cancel_extra_hdrs( CANCEL_REASON_SIP_480,
sizeof(CANCEL_REASON_SIP_480)-1);
385 cancel_branch(t, r_buf->branch );
386 set_cancel_extra_hdrs( NULL, 0);
387 }
(gdb) p t
$1 = (struct cell *) 0x0
Regards,
Agalya
From: Bogdan-Andrei Iancu [mailto:bog...@opensips.org]
Sent: Tuesday, October 04, 2016 6:44 AM
To: Ramachandran, Agalya (Contractor)
<agalya_ramachand...@comcast.com><mailto:agalya_ramachand...@comcast.com>;
OpenSIPS users mailling list
<users@lists.opensips.org><mailto:users@lists.opensips.org>
Subject: Re: opensips crash when doing load test
Hello Agalya,
In frame 0, could you print the "t" variable ? BTW, is there any way to get
access to the core file ?
Best regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
On 30.09.2016 16:48, Ramachandran, Agalya (Contractor) wrote:
Hi Bogdan,
One more dump
#0 final_response_handler (fr_tl=0x7fded2484110) at timer.c:383
383 if (is_invite(t) && should_cancel_branch(t, r_buf->branch) ) {
Missing separate debuginfos, use: debuginfo-install
cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 glibc-2.17-106.el7_2.4.x86_64
keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.13.2-12.el7_2.x86_64
libcom_err-1.42.9-7.el7.x86_64 libcurl-7.29.0-25.el7.centos.x86_64
libidn-1.28-4.el7.x86_64 libselinux-2.2.2-6.el7.x86_64
libssh2-1.4.3-10.el7_2.1.x86_64 nspr-4.10.8-2.el7_1.x86_64
nss-3.19.1-19.el7_2.x86_64 nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64
nss-util-3.19.1-9.el7_2.x86_64 openldap-2.4.40-9.el7_2.x86_64
openssl-libs-1.0.1e-51.el7_2.4.x86_64 pcre-8.32-15.el7.x86_64
xz-libs-5.1.2-12alpha.el7.x86_64 zlib-1.2.7-15.el7.x86_64
(gdb) bt
#0 final_response_handler (fr_tl=0x7fded2484110) at timer.c:383
#1 timer_routine (ticks=<optimized out>, set=<optimized out>) at timer.c:1066
#2 0x000000000049d3cd in handle_timer_job () at timer.c:597
#3 0x000000000059223e in handle_io (fm=<optimized out>, fm=<optimized out>,
fm=<optimized out>, idx=<optimized out>, event_type=2)
at net/net_udp.c:265
#4 io_wait_loop_epoll (h=<optimized out>, t=<optimized out>, repeat=<optimized
out>) at net/../io_wait_loop.h:225
#5 udp_rcv_loop (si=si@entry=0x7fded841dcb0) at net/net_udp.c:308
#6 0x0000000000593e05 in udp_start_processes (chd_rank=chd_rank@entry=0x84c570
<chd_rank.10940>, startup_done=startup_done@entry=0x0)
at net/net_udp.c:448
#7 0x0000000000419b73 in main_loop () at main.c:736
#8 main (argc=<optimized out>, argv=<optimized out>) at main.c:1282
From: Ramachandran, Agalya (Contractor)
Sent: Friday, September 30, 2016 9:44 AM
To: 'Bogdan-Andrei Iancu' <bog...@opensips.org><mailto:bog...@opensips.org>;
OpenSIPS users mailling list
<users@lists.opensips.org><mailto:users@lists.opensips.org>
Subject: opensips crash when doing load test
Hi Bogdan,
I have taken 2.1.4 code base, added the below fixes on top of it
1. Fix potential starvation for the timer tasks.
66c337cc89a2b5c3e1bbd78c289371efcda21886
2. rest_client: Improve polling logic during TCP connect -
c40f2b057209b8078520abc95e5d1bf5e5a28db6
3. rest_client: Add "connect_poll_interval" module parameter -
f5abfb3dee649a14ae39ad74d7f04454f815748f
4. rest_client: Fix bogus condition
-3ba7be6692c6987faecb9c4c904035d5e7c0f916
When I try to perform load test on that code base, randomly am seeing OpenSIPS
is crashing.
Here is the full trace of the core file. Let me know if you need any additional
information or logs.
Am seeing crash is occurring at start_timer_extra_processes().
[root@poc-siplb-cmc-e-004 corefiles]# gdb /usr/local/sbin/opensips
core.opensips.sig11.3977
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/sbin/opensips...done.
[New LWP 3977]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/opensips -P /var/run/opensips.pid'.
Program terminated with signal 11, Segmentation fault.
#0 final_response_handler (fr_tl=0x7f55fcd82948) at timer.c:383
383 if (is_invite(t) && should_cancel_branch(t, r_buf->branch) ) {
Missing separate debuginfos, use: debuginfo-install
cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 glibc-2.17-106.el7_2.4.x86_64
keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.13.2-12.el7_2.x86_64
libcom_err-1.42.9-7.el7.x86_64 libcurl-7.29.0-25.el7.centos.x86_64
libidn-1.28-4.el7.x86_64 libselinux-2.2.2-6.el7.x86_64
libssh2-1.4.3-10.el7_2.1.x86_64 nspr-4.10.8-2.el7_1.x86_64
nss-3.19.1-19.el7_2.x86_64 nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64
nss-util-3.19.1-9.el7_2.x86_64 openldap-2.4.40-9.el7_2.x86_64
openssl-libs-1.0.1e-51.el7_2.4.x86_64 pcre-8.32-15.el7.x86_64
xz-libs-5.1.2-12alpha.el7.x86_64 zlib-1.2.7-15.el7.x86_64
(gdb) bt
#0 final_response_handler (fr_tl=0x7f55fcd82948) at timer.c:383
#1 timer_routine (ticks=<optimized out>, set=<optimized out>) at timer.c:1066
#2 0x000000000049d03a in handle_timer_job () at timer.c:597
#3 start_timer_extra_processes (chd_rank=chd_rank@entry=0x84c570
<chd_rank.10940>) at timer.c:557
#4 0x0000000000419b9a in main_loop () at main.c:748
#5 main (argc=<optimized out>, argv=<optimized out>) at main.c:1282
Regards,
Agalya
_______________________________________________
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
