Hello, due to nature of parallel processing and using locks/mutexes, a kill of kamailio can happen inside a locked semaphore and can result in a deadlock while trying to sync data to backend. Rare, but can happen, for such cases you can tune the exit_timeout parameter value.
I assume you are using ims usrloc modules, because I haven't seen this case happening lately for the standard usrloc module. The later does a lock-free sync to database at shutdown, not sure if the ims-usrloc module take care of doing similar approach. Also, be sure you have enabled dumping of core file per process/pid, it can be another process that is the source of blocking at shutdown, but its core file was overwritten. Cheers, Daniel On 09.04.20 12:58, Andrey Deykunov wrote: > > Hi, > > We're getting periodically core dumps when stopping Kamailio: > > > GNU gdb (Debian 7.12-6) 7.12.0.20161007-git > Copyright (C) 2016 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-linux-gnu". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > <http://www.gnu.org/software/gdb/bugs/>. > Find the GDB manual and other documentation resources online at: > <http://www.gnu.org/software/gdb/documentation/>. > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from /var/lib/ums/sbin/kamailio...done. > [New LWP 9832] > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > Core was generated by `/var/lib/ums/sbin/kamailio -m 2048 -M 12 -P > /var/run/kamailio/kamailio.pid -f /'. > Program terminated with signal SIGABRT, Aborted. > #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 > #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 > #1 0x00007f79c24e842a in __GI_abort () at abort.c:89 > #2 0x000000000043f92a in sig_alarm_abort (signo=14) at main.c:679 > #3 <signal handler called> > #4 0x00007f79c258f910 in __read_nocancel () at > ../sysdeps/unix/syscall-template.S:84 > #5 0x00007f793cbd7421 in vio_read_buff () from > /var/lib/ums/lib/mysql/libmysqlclient.so.16 > #6 0x00007f793cbd8aa0 in my_real_read () from > /var/lib/ums/lib/mysql/libmysqlclient.so.16 > #7 0x00007f793cbd87d6 in my_net_read () from > /var/lib/ums/lib/mysql/libmysqlclient.so.16 > #8 0x00007f793cbd085d in cli_safe_read () from > /var/lib/ums/lib/mysql/libmysqlclient.so.16 > #9 0x00007f793cbd539a in cli_read_query_result () from > /var/lib/ums/lib/mysql/libmysqlclient.so.16 > #10 0x00007f793cbd436d in mysql_real_query () from > /var/lib/ums/lib/mysql/libmysqlclient.so.16 > #11 0x00007f793ce80c13 in db_mysql_submit_query (_h=0x7f79c1786890, > _s=0x7f793f379250 <sql_str>) at km_dbase.c:111 > #12 0x00007f793f16baba in db_do_submit_query (_h=0x7f79c1786890, > _query=0x7f793f379250 <sql_str>, submit_query=0x7f793ce801a0 > <db_mysql_submit_query>) at db_query.c:58 > #13 0x00007f793f16e114 in db_do_delete (_h=0x7f79c1786890, > _k=0x7ffeb3ca8df8, _o=0x0, _v=0x7ffeb3ca8dd0, _n=1, > val2str=0x7f793cea8990 <db_mysql_val2str>, submit_query=0x7f793ce801a0 > <db_mysql_submit_query>) at db_query.c:300 > #14 0x00007f793ce86c17 in db_mysql_delete (_h=0x7f79c1786890, > _k=0x7ffeb3ca8df8, _o=0x0, _v=0x7ffeb3ca8dd0, _n=1) at km_dbase.c:510 > #15 0x00007f793f3cb841 in db_delete_ucontact_ruid (_c=0x7f794186b3c0) > at ucontact.c:1552 > #16 0x00007f793f3cd096 in db_delete_ucontact (_c=0x7f794186b3c0) at > ucontact.c:1570 > #17 0x00007f793f3ad0c0 in wb_timer (_r=0x7f79411d7330) at urecord.c:401 > #18 0x00007f793f3ab451 in timer_urecord (_r=0x7f79411d7330) at > urecord.c:463 > #19 0x00007f793f3978aa in mem_timer_udomain (_d=0x7f7940d9baa0, > istart=0, istep=1) at udomain.c:1224 > #20 0x00007f793f3d4608 in synchronize_all_udomains (istart=0, istep=1) > at dlist.c:756 > #21 0x00007f793f3a2768 in destroy () at usrloc_mod.c:464 > #22 0x0000000000637e2c in destroy_modules () at core/sr_module.c:746 > #23 0x000000000041c4d8 in cleanup (show_status=1) at main.c:555 > #24 0x0000000000423af7 in shutdown_children (sig=15, show_status=1) at > main.c:696 > #25 0x000000000041f081 in handle_sigs () at main.c:727 > #26 0x0000000000432a01 in main_loop () at main.c:1806 > #27 0x000000000043df6f in main (argc=9, argv=0x7ffeb3cac348) at > main.c:2802 > > > Looks like libmysql stucks at __read_nocancel() syscall. The > exit_timeout parameter is set by default (60 sec), Kamailio version > 5.3.1. > > Any ideas? > > Thanks, > Andrey > > > > > > _______________________________________________ > Kamailio (SER) - Development Mailing List > sr-dev@lists.kamailio.org > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev -- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda
_______________________________________________ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev