Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

Hi @sagarmalam , do you mean `delete_counter` parameter? I tried and seems to work for me (The truth is that if I set it to 3 I get 4 attempts but then stops). BTW id you set that parameter to "0" means no limit. Not sure if that is the case. Anyway, I can't find the relation with changes don

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

@NGSegovia , Thanks for all the efforts in fix this issue. I think i have found regression issue. KA timeout parameter is not working anymore with latest changes. It keeps on sending KA even after failing for number of times defined in ka_timeout mod param. -- You are receiving this because yo

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

Closed #2448. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2448#event-3794004692___ Kamailio (SER) - Development Mailing List sr-dev@l

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

@sagarmalam merged PR fixing this problem. Re-open if there are still issues. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2448#issuecomment-696811732__

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

@NGSegovia , While testing latest fixes we have found that Keepalive module is leaking memory. We are testing with 7K destinations with time interval of 20 seconds This log file has output of : kamcmd mod.stats keepalive shm for every sec : [keepalive_memory_consumption.log](https://github.com/k

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

Hello Please ignore above comment.Fix is working.I picked up wrong branch while testing it. Sorry for confusion. Thanks -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2448#issu

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

Hello Again, I was able to reproduce again. It crashed from the same point and it seems that frequency of this issue has reduced but it is not fixed yet. As per my observation, it happens when entry of destination from KA list is removed during an ongoing KA transaction. I am removing destinat

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

ok. Thanks. I will share my results. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2448#issuecomment-68471___ Kamailio (SER) - Deve

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

No idea @sagarmalam -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2448#issuecomment-686612763___ Kamailio (SER) - Development Mailing

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

@NGSegovia , It is working fine now. Thanks a lot. Question : As a developer do you think this module can handle load of sending keepalives to 50 K destinations every 20 seconds ? or you suggest to use ursloc module instead for such a load ? -- You are receiving this because you are subsc

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

@sagarmalam I have updated the PR, please check again. Thanks! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2448#issuecomment-685867040__

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

@NGSegovia , I tested latest changes under https://github.com/NGSegovia/kamailio/commit/aaded162a6523642873703f6b52e191c86d5aee6 Now it is crashing again but from different place : ``` (gdb) bt full #0 0x7f3153cd82cc in __strncpy_ssse3 () from /lib64/libc.so.6 No symbol table info available

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

Not a problem @miconda , updated. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2448#issuecomment-685140654___ Kamailio (SER) - Develop

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

@NGSegovia - yes, that kind of problem. The approach in the fix is correct, just wondering if worth adding dependency on external libuuid library instead of using the internal core/utils/sruid.{c,h}, like it is done for other modules. I planned to extend the internal sruid code to be able to us

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

@NGSegovia , The changes you made has fixed the issue. Kamailio service is running since last 2 hours and I am not able to reproduce issue. Thanks so much. @miconda , I understand you are busy with Kamailio world, Is there any chances for this fix to be available in next stable release ? --

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

@miconda the problem you pointed is the same pointed in this message, right? https://github.com/kamailio/kamailio/blob/master/src/modules/keepalive/keepalive_core.c#L102 BTW I did some code, hopefully fixing this, but I didn't cleaned-up and test properly yet, but you can take a look. https://g

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

I preferred this module over usrloc module because i dont store registrations on kamailio. It is stored on application server behind Kamailio. So it was easy for me to use keepalive module to send KA instead of usrloc. -- You are receiving this because you are subscribed to this thread. Reply t

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

@NGSegovia - not related to the issue itself, but is there any specific reason to do this complex logic for keepalives? If it is for registrations, the usrloc module in kamailio v5.4.x can do keepalive (storing also the round trip time). Otherwise, the issue seems to be related to keeping the po

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

@miconda , Yes i am adding and removing on the fly. I am adding(using ka_add_destination) to KA list on successful registration and removing(using ka_del_destination) when registration expires. ``` #KEEPALIVE### modparam("keepalive", "ping_

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

The timer for keepalive is stopped, but here it seems to come a pointer which was stored inside sip transaction structure and used when the transaction completes (on 408 timeout or 200ok response). If between the options and final response the ka destination structure is destroyed, then the tm c

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

In principle, there is a lock used for destination list: https://github.com/kamailio/kamailio/blob/master/src/modules/keepalive/keepalive_api.c#L178 and when freeing the destination the timer is being stopped, https://github.com/kamailio/kamailio/blob/master/src/modules/keepalive/keepalive_api.c#

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

@NGSegovia - it is what I asked in my first comments, based on what I saw in the code. But the last backtrace shows also URI field being invalid, so maybe the ka structure is compromised/overwritten -- like already freed/destroyed while the keepalive was ongoing, and another process got the mem

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

I'd say the reporter is adding the destination from another module, and not from .cfg. `statechanged_clb` should be null (and not run) when the destination is added through cfg. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHu

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

Can you make a minimal kamailio config to reproduce this issue? The uri field is compromised, maybe due to a free/destroy of the keepalive structure. Are you adding/removing ka destinations on the fly/during runtime? -- You are receiving this because you are subscribed to this thread. Reply to

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

I have generated fresh core dump : ``` GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is N

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

@NGSegovia FYI -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2448#issuecomment-683634852___ Kamailio (SER) - Development Mailing List s

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

In the first backtrace, the frame 1 was different: ``` #1 0x7f931dc69c7e in ka_options_callback (t=0x7f91e58f05c0, type=1024, ps=0x7ffda0977bd0) at keepalive_core.c:126 ``` Did you get another core file? I need the output of `p *ka_dest` in the frame showing execution of `ka_options_callba

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

Hi, (gdb) frame 1 #1 0x7ff5bbb2ddd8 in run_trans_callbacks_internal (cb_lst=0x7ff47c9f86a8, type=1024, trans=0x7ff47c9f8630, params=0x7ffdb5b643e0) at t_hooks.c:258 258 cbp->callback( trans, type, params ); (gdb) p * ka_dest No symbol "ka_dest" in current contex

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

Supposing you still have the core file, can you get the output of next commands executed inside gdb: ``` frame 1 p * ka_dest ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/24

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

@miconda , Thanks i understand. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2448#issuecomment-677599029___ Kamailio (SER) - Developm

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

I am not the developer of the module, I just looked quickly in the code and it seemed that it should not execute the callback that caused the crash if the function was used in the config. That's why I asked if you use it inside a custom module. If I get some spare time, I will look again, but I

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

Hi, @miconda So it a quick fix or complex one ? Can i expect it to get fixed in next stable version ? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2448#issuecomment-677542203

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

I have used ka_add_destination in config file. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2448#issuecomment-676616900___ Kamailio (S

Re: [sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

Do you use `ka_add_destination()` in kamailio configuration file? Or have you developed a module that uses internally this function? Because looking at the code, when using the function in config, is not setting any state change callback function, which seems to be the one missing at the top of

[sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)

### Description I am using keepalive module to send keepalives to all the destinations registered with an application server which sits behind kamailio. When Kamailio recieves 200 Ok. I take URI contact from 200 OK and pass it as a param to ka_add_destination function.Most of the time it w