Thanks for your reply, Daniel. I'm not sure I understood you and Henning correctly.
At the beginning I indicated that I was observing a problem during a stop > I encountered a problem stopping Kamailio with FIPS OpenSSL I stop service using systemd, so it is expected to see a shutdown_children call in the backtrace, as I understand it. Also I have pid-specific core pattern (kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e) and unlimited core file size. вт, 14 мая 2024 г. в 22:21, Daniel-Constantin Mierla <mico...@gmail.com>: > The backtrace is from shutdown cleanup as pointed before, so it is not the > one that caused the crash. > > To get more than one core file, so it is not going to have core file > overwritten, you have to enable one core file per pid/process (or set core > file name pattern), some hints at: > > - > https://www.kamailio.org/wikidocs/tutorials/troubleshooting/coredumpfile/ > > Searching on the web should reveal other tutorials about it. > > Then you should get more than one core file on a crash and you should grab > the backtrace from all of them. > > You should also install the debugging symbols for libssl and libcrypto, > there could be useful details shown in the backtraces. > > Cheers, > Daniel > > > On 14.05.24 19:38, Marat Gareev via sr-users wrote: > > Henning, > I can't find anything else. But I caught one more segfault in the same > scenario (stopping service)... > > Program terminated with signal SIGSEGV, Segmentation fault. > #0 0x00007f3614f3c609 in init_thread_deregister.isra () from > /lib64/libcrypto.so.3 > Missing separate debuginfos, use: dnf debuginfo-install > kamailio-5.7.5-4817.x86_64 > (gdb) bt > #0 0x00007f3614f3c609 in init_thread_deregister.isra () from > /lib64/libcrypto.so.3 > #1 0x00007f3614e69daa in ossl_provider_free.part () from > /lib64/libcrypto.so.3 > #2 0x00007f3614ea81a0 in OPENSSL_sk_pop_free () from /lib64/libcrypto.so.3 > #3 0x00007f3614e68878 in prov_conf_ossl_ctx_free () from > /lib64/libcrypto.so.3 > #4 0x00007f3614e5d405 in CRYPTO_free_ex_data () from /lib64/libcrypto.so.3 > #5 0x00007f3614e5d59f in context_deinit.part () from /lib64/libcrypto.so.3 > #6 0x00007f3614e600b2 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 > #7 0x00007f36151b921e in ?? () > #8 0x000000010000a298 in ?? () > #9 0x00007f3604cc66c8 in ?? () > #10 0x00007ffead90c920 in ?? () > #11 0x000000000071e0a0 in futex_release (lock=0x7f3615b7c930 <syslog>) at > core/mem/../mem/../futexlock.h:134 > #12 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75 > #13 0x000000000041f278 in cleanup (show_status=1) at main.c:595 > #14 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at > main.c:722 > #15 0x0000000000421717 in handle_sigs () at main.c:753 > #16 0x0000000000430c88 in main_loop () at main.c:1989 > #17 0x0000000000439d13 in main (argc=14, argv=0x7ffead90d2f8) at main.c:3213 > > + unexpected message in log > > INFO kernel: [26983.427997] traps: kamailio[88753] general protection fault > ip:7f5a83585609 sp:7ffc9b2b4400 error:0 in > libcrypto.so.3.0.7[7f5a8339d000+25c000] > > > Richard, > I start service with the following parameters > /usr/local/sbin/kamailio --atexit=no -m 256 -P /var/run/ser/kamailio.pid > -u ser -g ser -f /usr/local/etc/kamailio/kamailio.cfg -w /usr/local > > вт, 14 мая 2024 г. в 18:57, Richard Chan via sr-users < > sr-users@lists.kamailio.org>: > >> Can you try with >> >> kamailio ... --atexit=no .... >> >> >> >> On Tue, 14 May 2024, 13:13 Marat Gareev via sr-users, < >> sr-users@lists.kamailio.org> wrote: >> >>> Hello again, >>> >>> I've updated Kamailio to 5.7.5, set tls_threads_mode=2 and got another >>> segfault: >>> >>> Program terminated with signal SIGSEGV, Segmentation fault. >>> #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 >>> Missing separate debuginfos, use: dnf debuginfo-install >>> kamailio-5.7.5-4817.x86_64 >>> (gdb) bt >>> #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 >>> #1 0x00007f26bb31a278 in __vfprintf_internal () from /lib64/libc.so.6 >>> #2 0x00007f26bb3dd4ea in __vsyslog_internal () from /lib64/libc.so.6 >>> #3 0x00007f26bb3dd9ca in syslog () from /lib64/libc.so.6 >>> #4 0x000000000071e574 in qm_debug_check_frag (qm=0x7f26aa4ee000, >>> f=0x7f26aa638388, file=0x7f26baa5b0b6 "tls: tls_init.c", line=399, >>> efile=0x8abb39 "core/mem/q_malloc.c", eline=526) at core/mem/q_malloc.c:126 >>> #5 0x00000000007227c3 in qm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, >>> file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", >>> line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:526 >>> #6 0x000000000072d2c9 in qm_shm_free (qmp=0x7f26aa4ee000, >>> p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", >>> func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") >>> at core/mem/q_malloc.c:1364 >>> #7 0x00007f26baa12ea9 in ?? () >>> #8 0x00007f26aa6383c0 in ?? () >>> #9 0x000001b3ba70914b in ?? () >>> #10 0x00007f26ba853e4b in ?? () from /lib64/libcrypto.so.3 >>> #11 0x00007f26aa6383c0 in ?? () >>> #12 0x00007f26aa6383c0 in ?? () >>> #13 0x00007f26ba61cfc5 in conf_modules_finish_int () from >>> /lib64/libcrypto.so.3 >>> #14 0x00007f26ba61d694 in CONF_modules_unload () from /lib64/libcrypto.so.3 >>> #15 0x00007f26ba6c0ff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 >>> #16 0x00007f26baa1a21e in ?? () >>> #17 0x00000001000623b0 in ?? () >>> #18 0x00007f26aa5276c8 in ?? () >>> #19 0x00007ffd66587330 in ?? () >>> #20 0x000000000071e0a0 in futex_release (lock=0x7f26bb3dd930 <syslog>) at >>> core/mem/../mem/../futexlock.h:134 >>> #21 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75 >>> #22 0x000000000041f278 in cleanup (show_status=1) at main.c:595 >>> #23 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at >>> main.c:722 >>> #24 0x0000000000421717 in handle_sigs () at main.c:753 >>> #25 0x0000000000430c88 in main_loop () at main.c:1989 >>> #26 0x0000000000439d13 in main (argc=14, argv=0x7ffd66587d08) at main.c:3213 >>> (gdb) >>> >>> >>> >>> And yes, the problem is definitely related to FIPS, because I did not >>> see any errors with regular OpenSSL 3.x. >>> >>> пн, 13 мая 2024 г. в 13:39, Marat Gareev <maratki...@gmail.com>: >>> >>>> Hello Henning, >>>> >>>> yes, I use this major version >>>> >>>> $ openssl version >>>> OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022) >>>> >>>> Thanks, I'll try updating Kamailio and report the results. >>>> >>>> >>>> пн, 13 мая 2024 г. в 13:19, Henning Westerholt <h...@gilawa.com>: >>>> >>>>> Hello, >>>>> >>>>> >>>>> >>>>> are you on openssl 3.x by any chance? If yes, please upgrade to >>>>> kamailio 5.7.5 or 5.8.1 and set tls_thread_mode=2 in the kamailio.cfg, as >>>>> it fixes certain memory corruption issues on this openssl version. >>>>> >>>>> If you are still getting crashes after the upgrade and setting, please >>>>> let us know, it might be something related to the FIPS mode. >>>>> >>>>> >>>>> >>>>> Cheers, >>>>> >>>>> >>>>> >>>>> Henning >>>>> >>>>> >>>>> >>>>> *From:* Marat Gareev via sr-users <sr-users@lists.kamailio.org> >>>>> *Sent:* Montag, 13. Mai 2024 09:19 >>>>> *To:* Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org >>>>> > >>>>> *Cc:* Marat Gareev <maratki...@gmail.com> >>>>> *Subject:* [SR-Users] TLS module crashes with FIPS OpenSSL >>>>> >>>>> >>>>> >>>>> Hello, >>>>> >>>>> >>>>> >>>>> I encountered a problem stopping Kamailio with FIPS OpenSSL: >>>>> >>>>> >>>>> >>>>> Program terminated with signal SIGSEGV, Segmentation fault. >>>>> >>>>> #0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3 >>>>> >>>>> Missing separate debuginfos, use: dnf debuginfo-install >>>>> kamailio-5.7.3-4816.x86_64 >>>>> >>>>> (gdb) bt >>>>> >>>>> #0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3 >>>>> >>>>> #1 0x00007ff72914bf5b in conf_modules_finish_int () from >>>>> /lib64/libcrypto.so.3 >>>>> >>>>> #2 0x00007ff72914c694 in CONF_modules_unload () from >>>>> /lib64/libcrypto.so.3 >>>>> >>>>> #3 0x00007ff7291efff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 >>>>> >>>>> #4 0x00007ff72954702b in ?? () >>>>> >>>>> #5 0x0000000100061c08 in ?? () >>>>> >>>>> #6 0x00007ff7190566c8 in ?? () >>>>> >>>>> #7 0x00007ffccf196a20 in ?? () >>>>> >>>>> #8 0x000000000071da8a in futex_release (lock=0x7ff729f08b50 <syslog>) at >>>>> core/mem/../mem/../futexlock.h:134 >>>>> >>>>> #9 0x00000000006e9448 in destroy_tls () at core/tls_hooks.c:75 >>>>> >>>>> #10 0x000000000041f278 in cleanup (show_status=1) at main.c:594 >>>>> >>>>> #11 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at >>>>> main.c:721 >>>>> >>>>> #12 0x0000000000421717 in handle_sigs () at main.c:752 >>>>> >>>>> #13 0x0000000000430c88 in main_loop () at main.c:1988 >>>>> >>>>> #14 0x0000000000439d13 in main (argc=14, argv=0x7ffccf1973f8) at >>>>> main.c:3212 >>>>> >>>>> (gdb) >>>>> >>>>> >>>>> >>>>> Environment: >>>>> >>>>> Oracle Linux Server 9.3 >>>>> >>>>> Kamailio 5.7.3 >>>>> >>>>> yum list --installed | grep ssl >>>>> >>>>> openssl.x86_64 10:3.0.7-24.0.3.el9_fips >>>>> @tools >>>>> >>>>> openssl-libs.x86_64 10:3.0.7-24.0.3.el9_fips >>>>> @tools >>>>> >>>>> openssl-pkcs11.x86_64 0.4.11-7.el9 >>>>> @anaconda >>>>> >>>>> xmlsec1-openssl.x86_64 1.2.29-9.el9 >>>>> @AppStream >>>>> >>>>> >>>>> >>>>> What can I do for further investigation? >>>>> >>>>> >>>>> >>>>> Thanks >>>>> >>>> __________________________________________________________ >>> Kamailio - Users Mailing List - Non Commercial Discussions >>> To unsubscribe send an email to sr-users-le...@lists.kamailio.org >>> Important: keep the mailing list in the recipients, do not reply only to >>> the sender! >>> Edit mailing list options or unsubscribe: >>> >> __________________________________________________________ >> Kamailio - Users Mailing List - Non Commercial Discussions >> To unsubscribe send an email to sr-users-le...@lists.kamailio.org >> Important: keep the mailing list in the recipients, do not reply only to >> the sender! >> Edit mailing list options or unsubscribe: >> > > __________________________________________________________ > Kamailio - Users Mailing List - Non Commercial Discussions > To unsubscribe send an email to sr-users-le...@lists.kamailio.org > Important: keep the mailing list in the recipients, do not reply only to the > sender! > Edit mailing list options or unsubscribe: > > > -- > Daniel-Constantin Mierla (@ asipto.com)twitter.com/miconda -- > linkedin.com/in/miconda > Kamailio Consultancy, Training and Development Services -- asipto.com > >
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-le...@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
