[SR-Users] Re: TLS module crashes with FIPS OpenSSL

[Henning Westerholt via sr-users]

Hello,

ok – so we probably had a slight misunderstanding here. I at least thought you 
are having a problem (crash) in the openssl FIPS part that causes Kamailio to 
stop.

If its only causing a crash at stopping time, this is of course less critical, 
nevertheless it should be fixed.

Maybe you can provide more details on how this problem can be reproduced, or 
create an github issue with a small cfg to reproduce it.

Cheers,

Henning

From: Marat Gareev via sr-users 
Sent: Dienstag, 14. Mai 2024 23:47
To: mico...@gmail.com
Cc: Kamailio (SER) - Users Mailing List ; Marat 
Gareev 
Subject: [SR-Users] Re: TLS module crashes with FIPS OpenSSL

Thanks for your reply, Daniel.

I'm not sure I understood you and Henning correctly.

At the beginning I indicated that I was observing a problem during a stop
> I encountered a problem stopping Kamailio with FIPS OpenSSL

I stop service using systemd, so it is expected to see a shutdown_children call 
in the backtrace, as I understand it.

Also I have pid-specific core pattern (kernel.core_pattern = 
|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e) and unlimited core 
file size.

вт, 14 мая 2024 г. в 22:21, Daniel-Constantin Mierla 
mailto:mico...@gmail.com>>:
The backtrace is from shutdown cleanup as pointed before, so it is not the one 
that caused the crash.

To get more than one core file, so it is not going to have core file 
overwritten, you have to enable one core file per pid/process (or set core file 
name pattern), some hints at:

 - https://www.kamailio.org/wikidocs/tutorials/troubleshooting/coredumpfile/

Searching on the web should reveal other tutorials about it.

Then you should get more than one core file on a crash and you should grab the 
backtrace from all of them.

You should also install the debugging symbols for libssl and libcrypto, there 
could be useful details shown in the backtraces.

Cheers,
Daniel


On 14.05.24 19:38, Marat Gareev via sr-users wrote:
Henning,
I can't find anything else. But I caught one more segfault in the same scenario 
(stopping service)...


Program terminated with signal SIGSEGV, Segmentation fault.

#0  0x7f3614f3c609 in init_thread_deregister.isra () from 
/lib64/libcrypto.so.3

Missing separate debuginfos, use: dnf debuginfo-install 
kamailio-5.7.5-4817.x86_64

(gdb) bt

#0  0x7f3614f3c609 in init_thread_deregister.isra () from 
/lib64/libcrypto.so.3

#1  0x7f3614e69daa in ossl_provider_free.part () from /lib64/libcrypto.so.3

#2  0x7f3614ea81a0 in OPENSSL_sk_pop_free () from /lib64/libcrypto.so.3

#3  0x7f3614e68878 in prov_conf_ossl_ctx_free () from /lib64/libcrypto.so.3

#4  0x7f3614e5d405 in CRYPTO_free_ex_data () from /lib64/libcrypto.so.3

#5  0x7f3614e5d59f in context_deinit.part () from /lib64/libcrypto.so.3

#6  0x7f3614e600b2 in OPENSSL_cleanup () from /lib64/libcrypto.so.3

#7  0x7f36151b921e in ?? ()

#8  0x0001a298 in ?? ()

#9  0x7f3604cc66c8 in ?? ()

#10 0x7ffead90c920 in ?? ()

#11 0x0071e0a0 in futex_release (lock=0x7f3615b7c930 ) at 
core/mem/../mem/../futexlock.h:134

#12 0x006e993e in destroy_tls () at core/tls_hooks.c:75

#13 0x0041f278 in cleanup (show_status=1) at main.c:595

#14 0x00420af1 in shutdown_children (sig=15, show_status=1) at 
main.c:722

#15 0x00421717 in handle_sigs () at main.c:753

#16 0x00430c88 in main_loop () at main.c:1989

#17 0x00439d13 in main (argc=14, argv=0x7ffead90d2f8) at main.c:3213
+ unexpected message in log

INFO kernel: [26983.427997] traps: kamailio[88753] general protection fault 
ip:7f5a83585609 sp:7ffc9b2b4400 error:0 in 
libcrypto.so.3.0.7[7f5a8339d000+25c000]

Richard,
I start service with the following parameters
/usr/local/sbin/kamailio --atexit=no -m 256 -P /var/run/ser/kamailio.pid -u ser 
-g ser -f /usr/local/etc/kamailio/kamailio.cfg -w /usr/local

вт, 14 мая 2024 г. в 18:57, Richard Chan via sr-users 
mailto:sr-users@lists.kamailio.org>>:
Can you try with

kamailio ... --atexit=no 



On Tue, 14 May 2024, 13:13 Marat Gareev via sr-users, 
mailto:sr-users@lists.kamailio.org>> wrote:
Hello again,

I've updated Kamailio to 5.7.5, set tls_threads_mode=2 and got another segfault:


Program terminated with signal SIGSEGV, Segmentation fault.

#0  0x7f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6

Missing separate debuginfos, use: dnf debuginfo-install 
kamailio-5.7.5-4817.x86_64

(gdb) bt

#0  0x7f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6

#1  0x7f26bb31a278 in __vfprintf_internal () from /lib64/libc.so.6

#2  0x7f26bb3dd4ea in __vsyslog_internal () from /lib64/libc.so.6

#3  0x7f26bb3dd9ca in syslog () from /lib64/libc.so.6

#4  0x0071e574 in qm_debug_check_frag (qm=0x7f26aa4ee000, 
f=0x7f26aa638388, file=0x7f26baa5b0b6 "tls: tls_init.c", line=399, 
efile=0x8abb39 "core/mem/q_malloc.c", eline=526) at core/mem/q_malloc.c:126

#5  0x007227c3 in qm_free (qmp=0x7f26aa4ee000, p=0x7f26

[SR-Users] Re: q_malloc.c:520 WARNINGs

[Ben Kaufman via sr-users]

The issue is trivial to reproduce.  I didn’t test every dialplan module 
function, but it occurs with both dp_transalate() and dp_match().

I’ve opened an issue in GitHub:  
https://github.com/kamailio/kamailio/issues/3851

Regards,
Kaufman

From: Ben Kaufman via sr-users 
Sent: Monday, May 13, 2024 5:05 PM
To: Kamailio (SER) - Users Mailing List 
Cc: Ben Kaufman 
Subject: [SR-Users] Re: q_malloc.c:520 WARNINGs


CAUTION: This email originated from outside the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.

I’ve just noticed a similar behavior in kamailio 5.8.1 after upgrading from 
5.6.3.  I haven’t had time to poke around on it much yet.  Wondering if anyone 
found the root cause?  In my case I’m calling just `dp_match("1", "$rU");`

I’ll try to update with a simplified scenario, etc.


From: Joel Serrano via sr-users 
mailto:sr-users@lists.kamailio.org>>
Sent: Saturday, March 9, 2024 9:30 AM
To: Kamailio (SER) - Users Mailing List 
mailto:sr-users@lists.kamailio.org>>
Cc: Joel Serrano mailto:j...@textplus.com>>
Subject: [SR-Users] Re: q_malloc.c:520 WARNINGs


CAUTION: This email originated from outside the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.

I've tried setting corelog=L_INFO but I can still see the WARNINGS.

Also, one correction in my previous msg, I don't have 2 "debug" lines, I have 
them defined within a condition:

#!ifdef WITH_DEBUG
debug=L_DBG
#!else
debug=L_NOTICE
#!endif
...
...


Joel.




On Fri, Mar 8, 2024 at 9:54 PM Joel Serrano 
mailto:j...@textplus.com>> wrote:
Hi,

Does anyone know what this WARNING means or how to fix it?

2024-03-09T00:39:15.406674-05:00 csbc03 csbc[33024]: WARNING: {1 1 INVITE 
CALL-ID}  [core/mem/q_malloc.c:520]: qm_free(): WARNING: free(0) called 
from dialplan: dp_db.c: pcre2_free(206)
2024-03-09T00:39:15.406823-05:00 csbc03 csbc[33024]: WARNING: {1 1 INVITE 
CALL-ID}  [core/mem/q_malloc.c:520]: qm_free(): WARNING: free(0) called 
from dialplan: dp_db.c: pcre2_free(206)
2024-03-09T00:39:15.406973-05:00 csbc03 csbc[33024]: WARNING: {1 1 INVITE 
CALL-ID}  [core/mem/q_malloc.c:520]: qm_free(): WARNING: free(0) called 
from dialplan: dp_db.c: pcre2_free(206)
2024-03-09T00:39:15.407089-05:00 csbc03 csbc[33024]: WARNING: {1 1 INVITE 
CALL-ID}  [core/mem/q_malloc.c:520]: qm_free(): WARNING: free(0) called 
from dialplan: dp_db.c: pcre2_free(206)
2024-03-09T00:39:15.407178-05:00 csbc03 csbc[33024]: WARNING: {1 1 INVITE 
CALL-ID}  [core/mem/q_malloc.c:520]: qm_free(): WARNING: free(0) called 
from dialplan: dp_db.c: pcre2_free(206)
2024-03-09T00:39:15.407278-05:00 csbc03 csbc[33024]: WARNING: {1 1 INVITE 
CALL-ID}  [core/mem/q_malloc.c:520]: qm_free(): WARNING: free(0) called 
from dialplan: dp_db.c: pcre2_free(206)
2024-03-09T00:39:15.407375-05:00 csbc03 csbc[33024]: WARNING: {1 1 INVITE 
CALL-ID}  [core/mem/q_malloc.c:520]: qm_free(): WARNING: free(0) called 
from dialplan: dp_db.c: pcre2_free(206)
It happens after running dp_translate().

These are my settings:

debug=L_DBG
debug=L_NOTICE
memdbg=L_INFO
memlog=L_INFO
corelog=L_ERR
mem_summary=12
log_stderror=no
log_facility=LOG_LOCAL0
log_name="csbc"
log_prefix="{$mt $hdr(CSeq) $ci} "
mem_join=1


Any ideas?

Thanks,
Joel.


__
Kamailio - Users Mailing List - Non Commercial Discussions
To unsubscribe send an email to sr-users-le...@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:

[SR-Users] Re: Failover with dialog dmq while dialog state 2

[Henning Westerholt via sr-users]

Hello Björn,

Thanks for the update. This sounds indeed a like a somewhat ugly hack. 😉
It should be probably fixed in the code instead of manually creating KDMQ 
messages to update the dialog module state in the cfg.

Cheers,

Henning

From: Björn Klasen via sr-users 
Sent: Freitag, 10. Mai 2024 13:27
To: sr-users@lists.kamailio.org
Cc: Björn Klasen 
Subject: [SR-Users] Re: Failover with dialog dmq while dialog state 2

Hi Henning,

thank you for your reply. I did a lot testings during the past days and I think 
I found a solution that is working, although it's very dirty...

I found out it is possible to send a manual KDMQ-Message to all Proxy-Nodes 
including the one that it sending die KDMQ-Message. So I create a dialog state 
4 message on ACK and indeed the Dialog on all Peers are Updated correctly to 
state 4. I also tried this for state 3 on 200 OK, but this does not work. I 
haven't had a look into the code yet but my experiments shows, that there is 
never is state 3 update via DMQ, so I think it's not implemented.

Interestingly the TM-Timer still times out after 300 seconds but in normal 
circumstances this doesn't bother, because either the failed Kamailio will not 
trigger this, because the Software or the Server crashed, or it tries to send a 
CANCEL that is denied because we are in dialog state 4.

The only thing is your really have to be careful about your hash tables. You 
have to delete them at the correct point.

I really need to test a lot at this point, but till now, billing seems to be OK 
even on failover scenarios.

This is why I really like Kamailio :)

BR, Björn
Am 09.05.24 um 20:57 schrieb Henning Westerholt:

Hello,



thanks for the detailed e-mail. As also indicated in the module documentation, 
the dialog module DMQ replication will not replicate everything, its main 
use-case is for profile data sharing. 
https://kamailio.org/docs/modules/5.8.x/modules/dialog.html#dialog.p.enable_dmq



In the past months there have been some other discussions on the users lists 
about similar scenarios (I think related to billing/accounting) and dialog with 
DMQ, which might be interesting for you in this regard.



If you find issues where the DMQ synchronisation is lacking some functionality 
in the dialog module, you can create a feature request in our issue tracker. 
There is of course no guarantee that this limitation is also timely addressed.



Regarding the INVITE and CANCEL scenario, this is usually not related to dialog 
but to the tm module. As you also mentioned, there is no replication of 
transaction state in tm.



Cheers,



Henning


--
Björn Klasen, Senior Specialist (VoIP)
TNG Stadtnetz GmbH, TNG-Technik
Gerhard-Fröhler-Straße 12
24106 Kiel・Deutschland

T +49 431 7097-10
F +49 431 7097-555
bkla...@tng.de
https://www.tng.de

Executive board (Geschäftsführer):
Dr. Sven Willert (CEO/Vorsitz),
Gunnar Peter, Sven Schade,
Carsten Tolkmit, Bernd Sontheimer

Amtsgericht Kiel HRB 6002 KI
USt-ID: DE225201428
Die Information über die Verarbeitung Ihrer Daten
gemäß Artikel 12 DSGVO können Sie unter https://www.tng.de/datenschutz/ abrufen.
__
__
Kamailio - Users Mailing List - Non Commercial Discussions
To unsubscribe send an email to sr-users-le...@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:

[SR-Users] Re: TLS module crashes with FIPS OpenSSL

[Marat Gareev via sr-users]

Hello,
I have thoughts about the root cause...

Sometimes I don't get a segfault, but I see critical errors in the log:

 0(987) NOTICE:  [main.c:750]: handle_sigs(): Thank you for
flying kamailio!!!
 7(998) INFO:  [main.c:874]: sig_usr(): signal 15 received
 3(991) INFO:  [main.c:874]: sig_usr(): signal 15 received
 4(994) INFO:  [main.c:874]: sig_usr(): signal 15 received
 8(999) INFO:  [main.c:874]: sig_usr(): signal 15 received
 1(988) INFO:  [main.c:874]: sig_usr(): signal 15 received
 9(1000) INFO:  [main.c:874]: sig_usr(): signal 15 received
 0(987) CRITICAL:  [core/mem/q_malloc.c:535]: qm_free(): BUG:
freeing already freed pointer (0x7fdaac135800), called from tls:
tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) -
ignoring
 0(987) CRITICAL:  [core/mem/q_malloc.c:535]: qm_free(): BUG:
freeing already freed pointer (0x7fdaac164850), called from tls:
tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) -
ignoring
 0(987) CRITICAL:  [core/mem/q_malloc.c:535]: qm_free(): BUG:
freeing already freed pointer (0x7fdaac166878), called from tls:
tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) -
ignoring
 0(987) CRITICAL:  [core/mem/q_malloc.c:535]: qm_free(): BUG:
freeing already freed pointer (0x7fdaac135a80), called from tls:
tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) -
ignoring
 0(987) CRITICAL:  [core/mem/q_malloc.c:535]: qm_free(): BUG:
freeing already freed pointer (0x7fdaac166900), called from tls:
tls_init.c: ser_free(412), first free tls: tls_init.c: ser_malloc(367)
- ignoring
 0(987) CRITICAL:  [core/mem/q_malloc.c:535]: qm_free(): BUG:
freeing already freed pointer (0x7fdaac126bd0), called from tls:
tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) -
ignoring
 0(987) CRITICAL:  [core/mem/q_malloc.c:535]: qm_free(): BUG:
freeing already freed pointer (0x7fdaac126c70), called from tls:
tls_init.c: ser_free(412), first free tls: tls_init.c: ser_malloc(367)
- ignoring
 0(987) CRITICAL:  [core/mem/q_malloc.c:535]: qm_free(): BUG:
freeing already freed pointer (0x7fdaac40fd50), called from tls:
tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) -
ignoring
 0(987) CRITICAL:  [core/mem/q_malloc.c:535]: qm_free(): BUG:
freeing already freed pointer (0x7fdaac40fdf0), called from tls:
tls_init.c: ser_free(412), first free tls: tls_init.c: ser_malloc(367)
- ignoring
 0(987) CRITICAL:  [core/mem/q_malloc.c:535]: qm_free(): BUG:
freeing already freed pointer (0x7fdaac17e920), called from tls:
tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) -
ignoring
 0(987) CRITICAL:  [core/mem/q_malloc.c:535]: qm_free(): BUG:
freeing already freed pointer (0x7fdaac17e898), called from tls:
tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) -
ignoring
 0(987) CRITICAL:  [core/mem/q_malloc.c:126]:
qm_debug_check_frag(): BUG: qm: fragm. 0x7fdaac169690 (address
0x7fdaac1696c8) beginning overwritten (7fdaac165948)! Memory allocator
was called from tls: tls_init.c:412. Fragment marked by
:140574279598083. Exec from core/mem/q_malloc.c:526.


As I understand it, the program does not crash due to luck - the error is
ignored due to the memory safety control option.

How can I investigate and interpret this error?

Thanks.


ср, 15 мая 2024 г. в 09:56, Henning Westerholt :

> Hello,
>
>
>
> ok – so we probably had a slight misunderstanding here. I at least thought
> you are having a problem (crash) in the openssl FIPS part that causes
> Kamailio to stop.
>
>
>
> If its only causing a crash at stopping time, this is of course less
> critical, nevertheless it should be fixed.
>
>
>
> Maybe you can provide more details on how this problem can be reproduced,
> or create an github issue with a small cfg to reproduce it.
>
>
>
> Cheers,
>
>
>
> Henning
>
>
>
> *From:* Marat Gareev via sr-users 
> *Sent:* Dienstag, 14. Mai 2024 23:47
> *To:* mico...@gmail.com
> *Cc:* Kamailio (SER) - Users Mailing List ;
> Marat Gareev 
> *Subject:* [SR-Users] Re: TLS module crashes with FIPS OpenSSL
>
>
>
> Thanks for your reply, Daniel.
>
>
>
> I'm not sure I understood you and Henning correctly.
>
>
>
> At the beginning I indicated that I was observing a problem during a stop
>
> > I encountered a problem stopping Kamailio with FIPS OpenSSL
>
>
>
> I stop service using systemd, so it is expected to see a shutdown_children
> call in the backtrace, as I understand it.
>
>
>
> Also I have pid-specific core pattern (kernel.core_pattern =
> |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e) and unlimited core
> file size.
>
>
>
> вт, 14 мая 2024 г. в 22:21, Daniel-Constantin Mierla :
>
> The backtrace is from shutdown cleanup as pointed before, so it is not the
> one that caused the crash.
>
>
>
> To get more than one core file, so it is not going to have core file
> overwritten, you have to enable one core file per pid/process (or set core
> file name pattern), some hints at:
>
>
>
>  -
> https://w

[SR-Users] Kamailio IPSec module

[H Yavari via sr-users]

 Hi all,
I was reviewing the `ims_ipsec_pcscf` code and noticed that this module creates 
a pool of sockets using different ports (ipsec_max_connections). I'm unclear on 
the necessity of this approach. Can't we simply create one server listener and 
one client listener to handle all UE connections? If this is feasible, is there 
still a need to create the pool at startup?
I'm curious if this is due to an architectural limitation or if the IPSec 
module can be modified to replace the current implementation with a more 
efficient one.
Thank you for your insights.
Best regards,Hossein__
Kamailio - Users Mailing List - Non Commercial Discussions
To unsubscribe send an email to sr-users-le...@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe: