[OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying...
Hi, this is what happened tonight on trunk version of opensips. Any ideas? This is from log, I'm including backtrace also: Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_do_prepared_query: doing BIND_PARAM in... Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:has_stmt_ctx: ctx found for location Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_do_prepared_query: conn=0x791860 (tail=7936152) MC=0x798138 Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_do_prepared_query: set values for the statement run Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_val2bind: added val (0): len=11; type=254; is_null=0 Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_val2bind: added val (1): len=32; type=254; is_null=0 Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_val2bind: added val (2): len=40; type=12; is_null=0 Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_val2bind: added val (3): len=8; type=5; is_null=0 Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_val2bind: added val (4): len=25; type=254; is_null=0 Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_val2bind: added val (5): len=4; type=3; is_null=0 Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_val2bind: added val (6): len=4; type=3; is_null=0 Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_val2bind: added val (7): len=4; type=3; is_null=0 Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_val2bind: added val (8): len=10; type=254; is_null=0 Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_val2bind: added val (11): len=21; type=254; is_null=0 Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_val2bind: added val (13): len=40; type=12; is_null=0 Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_val2bind: added val (14): len=12; type=254; is_null=0 Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: DBG:db_mysql:db_mysql_do_prepared_query: doing BIND_PARAM in... Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying... Backtrace from gdb: bt: #0 0x7f7d671b3ed5 in raise () from /lib/libc.so.6 #1 0x7f7d671b53f3 in abort () from /lib/libc.so.6 #2 0x004280e0 in sig_alarm_abort (signo=) at main.c:426 #3 #4 0x7f7d67243970 in read () from /lib/libc.so.6 #5 0x7f7d66be3008 in vio_read () from /usr/lib/libmysqlclient.so.15 #6 0x7f7d66be3067 in vio_read_buff () from /usr/lib/libmysqlclient.so.15 #7 0x7f7d66be41f9 in ?? () from /usr/lib/libmysqlclient.so.15 #8 0x7f7d66be459d in my_net_read () from /usr/lib/libmysqlclient.so.15 #9 0x7f7d66bdeb72 in cli_safe_read () from /usr/lib/libmysqlclient.so.15 #10 0x7f7d66bdf2e9 in ?? () from /usr/lib/libmysqlclient.so.15 #11 0x7f7d66bb354c in ?? () from /usr/lib/libmysqlclient.so.15 #12 0x7f7d66bb61aa in cli_stmt_execute () from /usr/lib/libmysqlclient.so.15 #13 0x7f7d66bb5c1d in mysql_stmt_execute () from /usr/lib/libmysqlclient.so.15 #14 0x7f7d66f691c6 in db_mysql_do_prepared_query (conn=0x791860, query=0x7f7d671816d0, v=0x79c7bdd0, n=15, uv=0x0, un=0) at dbase.c:479 #15 0x7f7d66f6b235 in db_mysql_insert (_h=0x791860, _k=0x79c7bfb0, _v=0x79c7bdd0, _n=15) at dbase.c:864 #16 0x7f7d653c233e in db_insert_ucontact (_c=) at ucontact.c:513 #17 0x7f7d653cf456 in timer_urecord (_r=0x7f7d607edb88) at urecord.c:326 #18 0x7f7d653c80f4 in mem_timer_udomain (_d=0x7f7d5b781d98) at udomain.c:695 #19 0x7f7d653bd78c in synchronize_all_udomains () at dlist.c:589 #20 0x7f7d653cda90 in destroy () at ul_mod.c:373 #21 0x00487e6c in destroy_modules () at sr_module.c:370 #22 0x00427c4c in cleanup (show_status=1) at main.c:336 ---Type to continue, or q to quit--- #23 0x004287b2 in handle_sigs () at main.c:533 #24 0x0042b78d in main (argc=, argv=0x79c7c3b8) at main.c:913 bt full: #0 0x7f7d671b3ed5 in raise () from /lib/libc.so.6 No symbol table info available. #1 0x7f7d671b53f3 in abort () from /lib/libc.so.6 No symbol table info available. #2 0x004280e0 in sig_alarm_abort (signo=) at main.c:426 __FUNCTION__ = "sig_alarm_abort" #3 No symbol table info available. #4 0x7f7d67243970 in read () from /lib/libc.so.6 No symbol table info available. #5 0x7f7d66be3008 in vio_read () from /usr/lib/libmysqlclient.so.15 No symbol table info available. #6 0x7f7d66be3067 in vio_read_buff () from /usr/lib/libmysqlclient.so.15 No symbol table info available. #7 0x7f7d66be41f9 in ?? () from /usr/lib/libmysqlclient.so.15 No symbol t
Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying...
Hi Josip, A key question - how many records do you have in usrloc? I'm asking because opensips is flushing the usrloc at shutdown and if you have too many records, this will take some time. Also, the shutdown time is control by an alarm (couple of seconds), so if the shutdown takes too long, the alarm will simply kill opensips. Regards, Bogdan Josip Djuricic wrote: > Hi, > > this is what happened tonight on trunk version of opensips. Any ideas? > > This is from log, I'm including backtrace also: > > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_do_prepared_query: doing BIND_PARAM in... > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:has_stmt_ctx: ctx found for location > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_do_prepared_query: conn=0x791860 (tail=7936152) > MC=0x798138 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_do_prepared_query: set values for the statement run > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (0): len=11; type=254; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (1): len=32; type=254; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (2): len=40; type=12; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (3): len=8; type=5; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (4): len=25; type=254; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (5): len=4; type=3; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (6): len=4; type=3; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (7): len=4; type=3; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (8): len=10; type=254; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (11): len=21; type=254; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (13): len=40; type=12; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (14): len=12; type=254; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_do_prepared_query: doing BIND_PARAM in... > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying... > > Backtrace from gdb: > bt: > #0 0x7f7d671b3ed5 in raise () from /lib/libc.so.6 > #1 0x7f7d671b53f3 in abort () from /lib/libc.so.6 > #2 0x004280e0 in sig_alarm_abort (signo=) > at main.c:426 > #3 > #4 0x7f7d67243970 in read () from /lib/libc.so.6 > #5 0x7f7d66be3008 in vio_read () from /usr/lib/libmysqlclient.so.15 > #6 0x7f7d66be3067 in vio_read_buff () from > /usr/lib/libmysqlclient.so.15 > #7 0x7f7d66be41f9 in ?? () from /usr/lib/libmysqlclient.so.15 > #8 0x7f7d66be459d in my_net_read () from /usr/lib/libmysqlclient.so.15 > #9 0x7f7d66bdeb72 in cli_safe_read () from > /usr/lib/libmysqlclient.so.15 > #10 0x7f7d66bdf2e9 in ?? () from /usr/lib/libmysqlclient.so.15 > #11 0x7f7d66bb354c in ?? () from /usr/lib/libmysqlclient.so.15 > #12 0x7f7d66bb61aa in cli_stmt_execute () from > /usr/lib/libmysqlclient.so.15 > #13 0x7f7d66bb5c1d in mysql_stmt_execute () from > /usr/lib/libmysqlclient.so.15 > #14 0x7f7d66f691c6 in db_mysql_do_prepared_query (conn=0x791860, > query=0x7f7d671816d0, v=0x79c7bdd0, n=15, uv=0x0, un=0) at dbase.c:479 > #15 0x7f7d66f6b235 in db_mysql_insert (_h=0x791860, > _k=0x79c7bfb0, _v=0x79c7bdd0, _n=15) at dbase.c:864 > #16 0x7f7d653c233e in db_insert_ucontact (_c=) > at ucontact.c:513 > #17 0x7f7d653cf456 in timer_urecord (_r=0x7f7d607edb88) at urecord.c:326 > #18 0x7f7d653c80f4 in mem_timer_udomain (_d=0x7f7d5b781d98) at > udomain.c:695 > #19 0x7f7d653bd78c in synchronize_all_udomains () at dlist.c:589 > #20 0x7f7d653cda90 in destroy () at ul_mod.c:373 > #21 0x00487e6c in destroy_modules () at sr_module.c:370 > #22 0x00427c4c in cleanup (show_status=1) at main.c:336 > ---Type to continue, or q to quit--- > #23 0x004287b2 in handle_sigs () at main.c:533 > #24 0x0042b78d in main (argc=, > argv=0x79c7c3b8) at main.c:913 > > > bt full: > #0 0x7f7d671b3ed5 in raise () from /lib/libc.so.6 > No symbol table info available. > #1 0x7f7d671b53f3 in abort () from /lib/libc.so.6 > No sy
Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying...
Hi Bogdan, I've had to swithch to v1.6 stable, so It's working now :) What I notice is that on trunk version I had this UsrLoc Stats: usrloc:registered_users = 387432 usrloc:location-users = 387432 usrloc:location-contacts = 12005 usrloc:location-expires = 375427 but on stable 1.6 I have this: UsrLoc Stats: usrloc:registered_users = 12005 usrloc:location-users = 12005 usrloc:location-contacts = 12005 usrloc:location-expires = 375427 And I can confirm that memory is now stable, I think it seg faulted because at that ime it has gone 10 times trough 10users registration, what means usrloc:registered_users had more than 1 000 000 users, that could explain what happened. Somehow I think it was not clearing registered users no matter they expired and was deleted from db. Perhaps you can confirm that you can reproduce this problem? Also is there a possibility to get list of known limitations or perhaps bugs on v1.6 that I should be aware of (concerning stability issues before puttying the system in production use)? I know you mentioned release 1.6.1, so what should be important fixes you mentioned in that mail? Once again sorry for lot of questions. Thanks, Josip -Original Message- From: users-boun...@lists.opensips.org [mailto:users-boun...@lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu Sent: Friday, December 18, 2009 1:26 PM To: OpenSIPS users mailling list Subject: Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying... Hi Josip, A key question - how many records do you have in usrloc? I'm asking because opensips is flushing the usrloc at shutdown and if you have too many records, this will take some time. Also, the shutdown time is control by an alarm (couple of seconds), so if the shutdown takes too long, the alarm will simply kill opensips. Regards, Bogdan Josip Djuricic wrote: > Hi, > > this is what happened tonight on trunk version of opensips. Any ideas? > > This is from log, I'm including backtrace also: > > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_do_prepared_query: doing BIND_PARAM in... > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:has_stmt_ctx: ctx found for location > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_do_prepared_query: conn=0x791860 (tail=7936152) > MC=0x798138 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_do_prepared_query: set values for the statement run > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (0): len=11; type=254; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (1): len=32; type=254; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (2): len=40; type=12; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (3): len=8; type=5; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (4): len=25; type=254; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (5): len=4; type=3; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (6): len=4; type=3; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (7): len=4; type=3; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (8): len=10; type=254; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (11): len=21; type=254; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (13): len=40; type=12; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_val2bind: added val (14): len=12; type=254; is_null=0 > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > DBG:db_mysql:db_mysql_do_prepared_query: doing BIND_PARAM in... > Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: > CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying... > > Backtrace from gdb: > bt: > #0 0x7f7d671b3ed5 in raise () from /lib/libc.so.6 > #1 0x7f7d671b53f3 in abort () from /lib/libc.so.6 > #2 0x004280e0 in sig_alarm_abort (signo=) > at main.c:426 > #3 > #4 0x7f7d67243970 in read () from /lib/libc.so.6 > #5 0x7f7d66be3008 in vio_read () from /usr/lib/libmysqlclient.so.15 > #6 0x7f7d66be3067 i
Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying...
Hi Josip, So, what you are saying is that trunk has a problem (versus 1.6) - some records do not expire . Could you detail a bit the way I could reproduce this scenario ? Known bugs are listed on the tracker - http://www.opensips.org/Development/Tracker Regards, Bogdan Josip Djuricic wrote: > Hi Bogdan, > > I've had to swithch to v1.6 stable, so It's working now :) > > What I notice is that on trunk version I had this > UsrLoc Stats: > usrloc:registered_users = 387432 > usrloc:location-users = 387432 > usrloc:location-contacts = 12005 > usrloc:location-expires = 375427 > > but on stable 1.6 I have this: > UsrLoc Stats: > usrloc:registered_users = 12005 > usrloc:location-users = 12005 > usrloc:location-contacts = 12005 > usrloc:location-expires = 375427 > > And I can confirm that memory is now stable, I think it seg faulted because > at that ime it has gone 10 times trough 10users registration, what means > usrloc:registered_users had more than 1 000 000 users, that could explain > what happened. Somehow I think it was not clearing registered users no matter > they expired and was deleted from db. > > Perhaps you can confirm that you can reproduce this problem? > > Also is there a possibility to get list of known limitations or perhaps bugs > on v1.6 that I should be aware of (concerning stability issues before > puttying the system in production use)? I know you mentioned release 1.6.1, > so what should be important fixes you mentioned in that mail? > > Once again sorry for lot of questions. > > Thanks, > > Josip > > > -Original Message- > From: users-boun...@lists.opensips.org > [mailto:users-boun...@lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu > Sent: Friday, December 18, 2009 1:26 PM > To: OpenSIPS users mailling list > Subject: Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown > timeout triggered, dying... > > Hi Josip, > > A key question - how many records do you have in usrloc? > > I'm asking because opensips is flushing the usrloc at shutdown and if > you have too many records, this will take some time. Also, the shutdown > time is control by an alarm (couple of seconds), so if the shutdown > takes too long, the alarm will simply kill opensips. > > Regards, > Bogdan > > Josip Djuricic wrote: > >> Hi, >> >> this is what happened tonight on trunk version of opensips. Any ideas? >> >> This is from log, I'm including backtrace also: >> >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_do_prepared_query: doing BIND_PARAM in... >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:has_stmt_ctx: ctx found for location >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_do_prepared_query: conn=0x791860 (tail=7936152) >> MC=0x798138 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_do_prepared_query: set values for the statement run >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_val2bind: added val (0): len=11; type=254; is_null=0 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_val2bind: added val (1): len=32; type=254; is_null=0 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_val2bind: added val (2): len=40; type=12; is_null=0 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_val2bind: added val (3): len=8; type=5; is_null=0 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_val2bind: added val (4): len=25; type=254; is_null=0 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_val2bind: added val (5): len=4; type=3; is_null=0 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_val2bind: added val (6): len=4; type=3; is_null=0 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_val2bind: added val (7): len=4; type=3; is_null=0 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_val2bind: added val (8): len=10; type=254; is_null=0 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_val2bind: added val (11): len=21; type=254; is_null=0 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_val2bind: added val (13): len=40; type=12; is_null=0 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_val
Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying...
Hi Bogdan, yes, I have noticed in the trunk I've tested that even though registrations expires, and opensips removes it, and opensipsctl moni shows them under: usrloc:location-expires, and usrloc:location-contacts keeps getting lower by those usrloc:location-expires numbers. But usrloc:registered_users and usrloc:location_users keep rising constantly. And the memory usage also keeps rising constantly until I get out of shared memory. This only seams to happen when contact's expires, if re register comes in before timeout expires the memory stops rising. I've tested this with sipp and 100 000 users registering with 20 minutes expiration time, and 40-60 registrations per second. Going on and on...after few ours, no more shared memory available. I can send you any other detail you require? Also I can send you config and sipp scenario file? Best regards, Josip -Original Message- From: users-boun...@lists.opensips.org [mailto:users-boun...@lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu Sent: Tuesday, December 22, 2009 10:28 AM To: OpenSIPS users mailling list Subject: Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying... Hi Josip, So, what you are saying is that trunk has a problem (versus 1.6) - some records do not expire . Could you detail a bit the way I could reproduce this scenario ? Known bugs are listed on the tracker - http://www.opensips.org/Development/Tracker Regards, Bogdan Josip Djuricic wrote: > Hi Bogdan, > > I've had to swithch to v1.6 stable, so It's working now :) > > What I notice is that on trunk version I had this > UsrLoc Stats: > usrloc:registered_users = 387432 > usrloc:location-users = 387432 > usrloc:location-contacts = 12005 > usrloc:location-expires = 375427 > > but on stable 1.6 I have this: > UsrLoc Stats: > usrloc:registered_users = 12005 > usrloc:location-users = 12005 > usrloc:location-contacts = 12005 > usrloc:location-expires = 375427 > > And I can confirm that memory is now stable, I think it seg faulted because > at that ime it has gone 10 times trough 10users registration, what means > usrloc:registered_users had more than 1 000 000 users, that could explain > what happened. Somehow I think it was not clearing registered users no matter > they expired and was deleted from db. > > Perhaps you can confirm that you can reproduce this problem? > > Also is there a possibility to get list of known limitations or perhaps bugs > on v1.6 that I should be aware of (concerning stability issues before > puttying the system in production use)? I know you mentioned release 1.6.1, > so what should be important fixes you mentioned in that mail? > > Once again sorry for lot of questions. > > Thanks, > > Josip > > > -Original Message- > From: users-boun...@lists.opensips.org > [mailto:users-boun...@lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu > Sent: Friday, December 18, 2009 1:26 PM > To: OpenSIPS users mailling list > Subject: Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown > timeout triggered, dying... > > Hi Josip, > > A key question - how many records do you have in usrloc? > > I'm asking because opensips is flushing the usrloc at shutdown and if > you have too many records, this will take some time. Also, the shutdown > time is control by an alarm (couple of seconds), so if the shutdown > takes too long, the alarm will simply kill opensips. > > Regards, > Bogdan > > Josip Djuricic wrote: > >> Hi, >> >> this is what happened tonight on trunk version of opensips. Any ideas? >> >> This is from log, I'm including backtrace also: >> >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_do_prepared_query: doing BIND_PARAM in... >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:has_stmt_ctx: ctx found for location >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_do_prepared_query: conn=0x791860 (tail=7936152) >> MC=0x798138 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_do_prepared_query: set values for the statement run >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_val2bind: added val (0): len=11; type=254; is_null=0 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_val2bind: added val (1): len=32; type=254; is_null=0 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]: >> DBG:db_mysql:db_mysql_val2bind: added val (2): len=40; type=12; is_null=0 >> Dec 18 03:40:34 test03 /usr/local/sbin/opensips[14946]:
Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying...
Hi Josip, Please send me the sipp scripts and how exactly to use it (I know how to use sipp) in order to get your scenario. Regards, Bogdan Josip Djuricic wrote: > Hi Bogdan, > > yes, I have noticed in the trunk I've tested that even though registrations > expires, and opensips removes it, and opensipsctl moni shows them under: > usrloc:location-expires, and usrloc:location-contacts keeps getting lower by > those usrloc:location-expires numbers. But usrloc:registered_users and > usrloc:location_users keep rising constantly. And the memory usage also keeps > rising constantly until I get out of shared memory. > > This only seams to happen when contact's expires, if re register comes in > before timeout expires the memory stops rising. > > I've tested this with sipp and 100 000 users registering with 20 minutes > expiration time, and 40-60 registrations per second. Going on and on...after > few ours, no more shared memory available. > > I can send you any other detail you require? Also I can send you config and > sipp scenario file? > > Best regards, > > Josip > > > > -Original Message- > From: users-boun...@lists.opensips.org > [mailto:users-boun...@lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu > Sent: Tuesday, December 22, 2009 10:28 AM > To: OpenSIPS users mailling list > Subject: Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown > timeout triggered, dying... > > Hi Josip, > > So, what you are saying is that trunk has a problem (versus 1.6) - some > records do not expire . > > Could you detail a bit the way I could reproduce this scenario ? > > Known bugs are listed on the tracker - > http://www.opensips.org/Development/Tracker > > Regards, > Bogdan > > Josip Djuricic wrote: > >> Hi Bogdan, >> >> I've had to swithch to v1.6 stable, so It's working now :) >> >> What I notice is that on trunk version I had this >> UsrLoc Stats: >> usrloc:registered_users = 387432 >> usrloc:location-users = 387432 >> usrloc:location-contacts = 12005 >> usrloc:location-expires = 375427 >> >> but on stable 1.6 I have this: >> UsrLoc Stats: >> usrloc:registered_users = 12005 >> usrloc:location-users = 12005 >> usrloc:location-contacts = 12005 >> usrloc:location-expires = 375427 >> >> And I can confirm that memory is now stable, I think it seg faulted because >> at that ime it has gone 10 times trough 10users registration, what means >> usrloc:registered_users had more than 1 000 000 users, that could explain >> what happened. Somehow I think it was not clearing registered users no >> matter they expired and was deleted from db. >> >> Perhaps you can confirm that you can reproduce this problem? >> >> Also is there a possibility to get list of known limitations or perhaps bugs >> on v1.6 that I should be aware of (concerning stability issues before >> puttying the system in production use)? I know you mentioned release 1.6.1, >> so what should be important fixes you mentioned in that mail? >> >> Once again sorry for lot of questions. >> >> Thanks, >> >> Josip >> >> >> -Original Message- >> From: users-boun...@lists.opensips.org >> [mailto:users-boun...@lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu >> Sent: Friday, December 18, 2009 1:26 PM >> To: OpenSIPS users mailling list >> Subject: Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown >> timeout triggered, dying... >> >> Hi Josip, >> >> A key question - how many records do you have in usrloc? >> >> I'm asking because opensips is flushing the usrloc at shutdown and if >> you have too many records, this will take some time. Also, the shutdown >> time is control by an alarm (couple of seconds), so if the shutdown >> takes too long, the alarm will simply kill opensips. >> >> Regards, >> Bogdan >> >> Josip Djuricic wrote: >> >> >>> Hi, >>> >>> this is what happened tonight on trunk version of opensips. Any ideas? >>> >>> This is from log, I'm including backtrace also: >>> ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying...
I've sent you all the files on private e-mail. Hope that is not a problem. Thanks, Josip -Original Message- From: users-boun...@lists.opensips.org [mailto:users-boun...@lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu Sent: Tuesday, December 22, 2009 12:04 PM To: OpenSIPS users mailling list Subject: Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying... Hi Josip, Please send me the sipp scripts and how exactly to use it (I know how to use sipp) in order to get your scenario. Regards, Bogdan Josip Djuricic wrote: > Hi Bogdan, > > yes, I have noticed in the trunk I've tested that even though registrations expires, and opensips removes it, and opensipsctl moni shows them under: usrloc:location-expires, and usrloc:location-contacts keeps getting lower by those usrloc:location-expires numbers. But usrloc:registered_users and usrloc:location_users keep rising constantly. And the memory usage also keeps rising constantly until I get out of shared memory. > > This only seams to happen when contact's expires, if re register comes in before timeout expires the memory stops rising. > > I've tested this with sipp and 100 000 users registering with 20 minutes expiration time, and 40-60 registrations per second. Going on and on...after few ours, no more shared memory available. > > I can send you any other detail you require? Also I can send you config and sipp scenario file? > > Best regards, > > Josip > > > > -Original Message- > From: users-boun...@lists.opensips.org [mailto:users-boun...@lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu > Sent: Tuesday, December 22, 2009 10:28 AM > To: OpenSIPS users mailling list > Subject: Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying... > > Hi Josip, > > So, what you are saying is that trunk has a problem (versus 1.6) - some > records do not expire . > > Could you detail a bit the way I could reproduce this scenario ? > > Known bugs are listed on the tracker - > http://www.opensips.org/Development/Tracker > > Regards, > Bogdan > > Josip Djuricic wrote: > >> Hi Bogdan, >> >> I've had to swithch to v1.6 stable, so It's working now :) >> >> What I notice is that on trunk version I had this >> UsrLoc Stats: >> usrloc:registered_users = 387432 >> usrloc:location-users = 387432 >> usrloc:location-contacts = 12005 >> usrloc:location-expires = 375427 >> >> but on stable 1.6 I have this: >> UsrLoc Stats: >> usrloc:registered_users = 12005 >> usrloc:location-users = 12005 >> usrloc:location-contacts = 12005 >> usrloc:location-expires = 375427 >> >> And I can confirm that memory is now stable, I think it seg faulted because at that ime it has gone 10 times trough 10users registration, what means usrloc:registered_users had more than 1 000 000 users, that could explain what happened. Somehow I think it was not clearing registered users no matter they expired and was deleted from db. >> >> Perhaps you can confirm that you can reproduce this problem? >> >> Also is there a possibility to get list of known limitations or perhaps bugs on v1.6 that I should be aware of (concerning stability issues before puttying the system in production use)? I know you mentioned release 1.6.1, so what should be important fixes you mentioned in that mail? >> >> Once again sorry for lot of questions. >> >> Thanks, >> >> Josip >> >> >> -Original Message- >> From: users-boun...@lists.opensips.org [mailto:users-boun...@lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu >> Sent: Friday, December 18, 2009 1:26 PM >> To: OpenSIPS users mailling list >> Subject: Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying... >> >> Hi Josip, >> >> A key question - how many records do you have in usrloc? >> >> I'm asking because opensips is flushing the usrloc at shutdown and if >> you have too many records, this will take some time. Also, the shutdown >> time is control by an alarm (couple of seconds), so if the shutdown >> takes too long, the alarm will simply kill opensips. >> >> Regards, >> Bogdan >> >> Josip Djuricic wrote: >> >> >>> Hi, >>> >>> this is what happened tonight on trunk version of opensips. Any ideas? >>> >>> This is from log, I'm including backtrace also: >>> ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying...
Hi Josip, Got your email, thank you - I will put it to work once I'm back into the office ...most probably in 2010 ;) Regards, Bogdan Josip Djuricic wrote: > I've sent you all the files on private e-mail. Hope that is not a problem. > > Thanks, > > Josip > > > > -Original Message- > From: users-boun...@lists.opensips.org > [mailto:users-boun...@lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu > Sent: Tuesday, December 22, 2009 12:04 PM > To: OpenSIPS users mailling list > Subject: Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - shutdown > timeout triggered, dying... > > Hi Josip, > > Please send me the sipp scripts and how exactly to use it (I know how to > use sipp) in order to get your scenario. > > Regards, > Bogdan > > Josip Djuricic wrote: > >> Hi Bogdan, >> >> yes, I have noticed in the trunk I've tested that even though >> > registrations expires, and opensips removes it, and opensipsctl moni shows > them under: usrloc:location-expires, and usrloc:location-contacts keeps > getting lower by those usrloc:location-expires numbers. But > usrloc:registered_users and usrloc:location_users keep rising constantly. > And the memory usage also keeps rising constantly until I get out of shared > memory. > >> This only seams to happen when contact's expires, if re register comes in >> > before timeout expires the memory stops rising. > >> I've tested this with sipp and 100 000 users registering with 20 minutes >> > expiration time, and 40-60 registrations per second. Going on and on...after > few ours, no more shared memory available. > >> I can send you any other detail you require? Also I can send you config >> > and sipp scenario file? > >> Best regards, >> >> Josip >> >> >> >> -Original Message----- >> From: users-boun...@lists.opensips.org >> > [mailto:users-boun...@lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu > >> Sent: Tuesday, December 22, 2009 10:28 AM >> To: OpenSIPS users mailling list >> Subject: Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - >> > shutdown timeout triggered, dying... > >> Hi Josip, >> >> So, what you are saying is that trunk has a problem (versus 1.6) - some >> records do not expire . >> >> Could you detail a bit the way I could reproduce this scenario ? >> >> Known bugs are listed on the tracker - >> http://www.opensips.org/Development/Tracker >> >> Regards, >> Bogdan >> >> Josip Djuricic wrote: >> >> >>> Hi Bogdan, >>> >>> I've had to swithch to v1.6 stable, so It's working now :) >>> >>> What I notice is that on trunk version I had this >>> UsrLoc Stats: >>> usrloc:registered_users = 387432 >>> usrloc:location-users = 387432 >>> usrloc:location-contacts = 12005 >>> usrloc:location-expires = 375427 >>> >>> but on stable 1.6 I have this: >>> UsrLoc Stats: >>> usrloc:registered_users = 12005 >>> usrloc:location-users = 12005 >>> usrloc:location-contacts = 12005 >>> usrloc:location-expires = 375427 >>> >>> And I can confirm that memory is now stable, I think it seg faulted >>> > because at that ime it has gone 10 times trough 10users registration, > what means usrloc:registered_users had more than 1 000 000 users, that could > explain what happened. Somehow I think it was not clearing registered users > no matter they expired and was deleted from db. > >>> Perhaps you can confirm that you can reproduce this problem? >>> >>> Also is there a possibility to get list of known limitations or perhaps >>> > bugs on v1.6 that I should be aware of (concerning stability issues before > puttying the system in production use)? I know you mentioned release 1.6.1, > so what should be important fixes you mentioned in that mail? > >>> Once again sorry for lot of questions. >>> >>> Thanks, >>> >>> Josip >>> >>> >>> -Original Message- >>> From: users-boun...@lists.opensips.org >>> > [mailto:users-boun...@lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu > >>> Sent: Friday, December 18, 2009 1:26 PM >>> To: OpenSIPS users mailling list >>> Subject: Re: [OpenSIPS-Users] CRITICAL:core:sig_alarm_abort: BUG - >>> > shutdown timeout triggered, dying... > &g
