Re: [SR-Users] Log levels severely impacts performance

[Alex Balashov]

> On Sep 15, 2022, at 10:54 PM, Amit  wrote:
> 
> I’ve already added the “-“ in front of the log file and that didn’t seem to 
> make any recognizable difference.

That’s the only place that I know.

Have you tested your storage throughput? Even something as simple as `hdparm -t 
/dev/sda`[1]? 

What about monitoring background I/O demand, i.e. `iostat -x 1`? Look at the 
%util on the drive being written to.

I know you said your log isn’t being written to the local disk, but other 
storage activity could still be pushing up your I/O base load.

— Alex

[1] Replace `/dev/sda` with your actual storage device.

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/


__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Log levels severely impacts performance

[Amit]

Hi Alex,
Can you share specifically where in the configs I would be able to confirm that 
asynchronous logging is enabled?
I’ve already added the “-“ in front of the log file and that didn’t seem to 
make any recognizable difference.
Thanks in advance.
Amit


From: sr-users  on behalf of Alex Balashov 

Date: Thursday, September 15, 2022 at 5:21 PM
To: Kamailio (SER) - Users Mailing List 
Subject: Re: [SR-Users] Log levels severely impacts performance
Agree, have had some very high-volume systems with prodigious logging output 
for every call and no problems.

Have also had these types of load problems on other systems with much less 
logging. The problem was either that asynchronous logging was not turned on, or 
the I/O subsystem was abnormally slow for other reasons, such as other heavy 
demand on storage, or just intrinsically slow.

— Alex

> On Sep 15, 2022, at 5:18 PM, Henning Westerholt  wrote:
>
> Hello,
>
> Kamailio is using the system interface for syslog, this is also a code which 
> is quite stable and has not seen a lot of changes in the last years.
> People are using Kamailio with a lot of logging in high load situation 
> usually without problems, there has not been a lot of reports about that in 
> the past.
>
> Maybe it something related to a specific distribution version? I checked, the 
> rsyslog is using asynchronous writing as default since a long time. What 
> logging engine do you use?
>
> Cheers,
>
> Henning
>
> --
> Henning Westerholt - https://skalatan.de/blog/
> Kamailio services - https://gilawa.com
>
> -Original Message-
> From: sr-users  On Behalf Of Jeremy 
> Kister
> Sent: Thursday, September 15, 2022 9:59 PM
> To: sr-users@lists.kamailio.org
> Subject: Re: [SR-Users] Log levels severely impacts performance
>
> On 9/13/2022 10:18 PM, Amit wrote:
>> Does anyone have any experience with this or have any suggestions? We
>> are finding it difficult to scale as a result of this issue.
>
> I was just dealing with this behavior. I was logging to a local rsyslog and 
> rsyslog was just writing to a file on a local ssd (unsynced).
>
> It's a ton of log.  But meh.
>
> I disabled syslog entirely and enabled log_stderror = yes
>
> Then I just captured stdout+stderr via multilog and write it to a file on the 
> disk.  The problem went away.
>
> I think there is some huge overhead in the way kam packages up the log and 
> sends it out to syslog.
>
> --
>
> Jeremy Kister
> https://jeremy.kister.net./
>
> __
> Kamailio - Users Mailing List - Non Commercial Discussions
>  * sr-users@lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to the 
> sender!
> Edit mailing list options or unsubscribe:
>  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> __
> Kamailio - Users Mailing List - Non Commercial Discussions
>  * sr-users@lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to the 
> sender!
> Edit mailing list options or unsubscribe:
>  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

--
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/


__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Log levels severely impacts performance

[Amit]

Jeremy,
Thank you for sharing your experience.
I did a little digging on multitool and only found the manpage, but not very 
much on how to install it. It doesn’t seem to be available via the yum package 
manager
Would you mind sharing some info on installation and how you run it to capture 
stderr and stdout?
Thank you and much appreciated.
Amit


From: sr-users  on behalf of Jeremy Kister 

Date: Thursday, September 15, 2022 at 4:00 PM
To: sr-users@lists.kamailio.org 
Subject: Re: [SR-Users] Log levels severely impacts performance
On 9/13/2022 10:18 PM, Amit wrote:
> Does anyone have any experience with this or have any suggestions? We
> are finding it difficult to scale as a result of this issue.

I was just dealing with this behavior. I was logging to a local rsyslog
and rsyslog was just writing to a file on a local ssd (unsynced).

It's a ton of log.  But meh.

I disabled syslog entirely and enabled log_stderror = yes

Then I just captured stdout+stderr via multilog and write it to a file
on the disk.  The problem went away.

I think there is some huge overhead in the way kam packages up the log
and sends it out to syslog.

--

Jeremy Kister
https://jeremy.kister.net./

__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Log levels severely impacts performance

[Alex Balashov]

Agree, have had some very high-volume systems with prodigious logging output 
for every call and no problems. 

Have also had these types of load problems on other systems with much less 
logging. The problem was either that asynchronous logging was not turned on, or 
the I/O subsystem was abnormally slow for other reasons, such as other heavy 
demand on storage, or just intrinsically slow.

— Alex

> On Sep 15, 2022, at 5:18 PM, Henning Westerholt  wrote:
> 
> Hello,
> 
> Kamailio is using the system interface for syslog, this is also a code which 
> is quite stable and has not seen a lot of changes in the last years.
> People are using Kamailio with a lot of logging in high load situation 
> usually without problems, there has not been a lot of reports about that in 
> the past.
> 
> Maybe it something related to a specific distribution version? I checked, the 
> rsyslog is using asynchronous writing as default since a long time. What 
> logging engine do you use?
> 
> Cheers,
> 
> Henning
> 
> -- 
> Henning Westerholt - https://skalatan.de/blog/
> Kamailio services - https://gilawa.com
> 
> -Original Message-
> From: sr-users  On Behalf Of Jeremy 
> Kister
> Sent: Thursday, September 15, 2022 9:59 PM
> To: sr-users@lists.kamailio.org
> Subject: Re: [SR-Users] Log levels severely impacts performance
> 
> On 9/13/2022 10:18 PM, Amit wrote:
>> Does anyone have any experience with this or have any suggestions? We 
>> are finding it difficult to scale as a result of this issue.
> 
> I was just dealing with this behavior. I was logging to a local rsyslog and 
> rsyslog was just writing to a file on a local ssd (unsynced).
> 
> It's a ton of log.  But meh.
> 
> I disabled syslog entirely and enabled log_stderror = yes
> 
> Then I just captured stdout+stderr via multilog and write it to a file on the 
> disk.  The problem went away.
> 
> I think there is some huge overhead in the way kam packages up the log and 
> sends it out to syslog.
> 
> -- 
> 
> Jeremy Kister
> https://jeremy.kister.net./
> 
> __
> Kamailio - Users Mailing List - Non Commercial Discussions
>  * sr-users@lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to the 
> sender!
> Edit mailing list options or unsubscribe:
>  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> 
> __
> Kamailio - Users Mailing List - Non Commercial Discussions
>  * sr-users@lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to the 
> sender!
> Edit mailing list options or unsubscribe:
>  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/


__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Log levels severely impacts performance

[Henning Westerholt]

Hello,

Kamailio is using the system interface for syslog, this is also a code which is 
quite stable and has not seen a lot of changes in the last years.
People are using Kamailio with a lot of logging in high load situation usually 
without problems, there has not been a lot of reports about that in the past.

Maybe it something related to a specific distribution version? I checked, the 
rsyslog is using asynchronous writing as default since a long time. What 
logging engine do you use?

Cheers,

Henning

-- 
Henning Westerholt - https://skalatan.de/blog/
Kamailio services - https://gilawa.com

-Original Message-
From: sr-users  On Behalf Of Jeremy Kister
Sent: Thursday, September 15, 2022 9:59 PM
To: sr-users@lists.kamailio.org
Subject: Re: [SR-Users] Log levels severely impacts performance

On 9/13/2022 10:18 PM, Amit wrote:
> Does anyone have any experience with this or have any suggestions? We 
> are finding it difficult to scale as a result of this issue.

I was just dealing with this behavior. I was logging to a local rsyslog and 
rsyslog was just writing to a file on a local ssd (unsynced).

It's a ton of log.  But meh.

I disabled syslog entirely and enabled log_stderror = yes

Then I just captured stdout+stderr via multilog and write it to a file on the 
disk.  The problem went away.

I think there is some huge overhead in the way kam packages up the log and 
sends it out to syslog.

-- 

Jeremy Kister
https://jeremy.kister.net./

__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Log levels severely impacts performance

[Jeremy Kister]

On 9/13/2022 10:18 PM, Amit wrote:
Does anyone have any experience with this or have any suggestions? We 
are finding it difficult to scale as a result of this issue.


I was just dealing with this behavior. I was logging to a local rsyslog 
and rsyslog was just writing to a file on a local ssd (unsynced).


It's a ton of log.  But meh.

I disabled syslog entirely and enabled log_stderror = yes

Then I just captured stdout+stderr via multilog and write it to a file 
on the disk.  The problem went away.


I think there is some huge overhead in the way kam packages up the log 
and sends it out to syslog.


--

Jeremy Kister
https://jeremy.kister.net./

__
Kamailio - Users Mailing List - Non Commercial Discussions
 * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
 * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Kamailio stops processing requests mid script

[Joel Serrano]

In sngrep I do see the the INVITEs coming in, I’ll check more on the OS
side and see what I can find.

Again, thanks for checking! I’ll follow up with what I find.

On Thu, Sep 15, 2022 at 09:16 Daniel-Constantin Mierla 
wrote:

> Hello,
>
> if the traffic is over UDP, the Kamailio workers were blocked in
> recvfrom(), meaning that nothing is passed from the network layer to them.
> If you see UDP packets coming to Kamailio via ngrep/sngrep/..., then is the
> OS that drops them via firewall or some other app that controls the network
> traffic.
>
> Cheers,
> Daniel
> On 15.09.22 07:37, Joel Serrano wrote:
>
> Hi Daniel,
>
> I've tried with apparmor disabled unfortunately the same issue happens.
>
> I've sent you privately the output of the kamctl trap. I personally don't
> think it's Kamailio's fault per se, this is on a standard debian 11. I'm
> just lost and don't really understand what the hell is going on.
>
> Thanks, I appreciate your help with this.
> Joel.
>
>
>
> On Wed, Sep 14, 2022 at 9:50 AM Joel Serrano  wrote:
>
>> Hi Daniel,
>>
>> I've followed your suggestions and compared this "bad" server with the
>> two "good" ones.
>>
>> - Pike:
>>
>> In all cases we have:
>>
>> if (src_ip!=myself && !ds_is_from_list()) {
>> if($sht(ipban=>$si)!=$null) {
>> xlog("L_ALERT","ALERT: blocked by pike R=$ru from $fu
>> (IP:$si:$sp)\n");
>> exit;
>> }
>> if (!pike_check_req()) {
>> xlog("L_ALERT","ALERT: pike blocking R=$ru from $fu
>> (IP:$si:$sp)\n");
>> $sht(ipban=>$si) = 1;
>> exit;
>> }
>> }
>>
>> And we are not seeing any logs, therefore I'm discarding pike.
>>
>> - Firewall:
>>
>> I checked all 3 servers, and none of them have -local- firewall policies.
>>
>> - conntrack:
>>
>> All 3 servers have nf_conntrack loaded in kernel.
>>
>> - selinux/etc:
>>
>> The two good servers have "AppArmor" disabled.
>> The bad server has "AppArmor" enabled. !! <-- I'm
>> hoping this could be the cause and I'm going to test tonight without it.
>>
>>
>> Thanks for checking this, I was so lost I actually went ahead and did
>> "kamctl trap" last night too just in case. When I run it, it didn't stop by
>> itself (I stopped it with CTRL+C after some time) but it did generate a
>> file with a lot of output. Can I send you it privately? I'm not sure how to
>> interpret it.
>>
>> Before anything I'm going to test tonight:
>>
>> 1- disabling apparmor
>> 2- unloading nf_conntrack
>>
>> I'll report back with the resutls.
>>
>>
>>
>>
>>
>> On Wed, Sep 14, 2022 at 5:27 AM Daniel-Constantin Mierla <
>> mico...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> be sure you do not hit some limits set by Kamailio (e.g., pike) or the
>>> system/firewall (e.g., selinux, conntrack).
>>>
>>> You should install gdb and run "kamctl trap" when it stops processing
>>> and inspect the written file to see where each kamailio process is in the
>>> execution stack.
>>>
>>> Cheers,
>>> Daniel
>>> On 14.09.22 10:20, Joel Serrano wrote:
>>>
>>> Bumping this! Any comments? Or suggestions on what to check? I'm feeling
>>> it has to be something stupid but I can't see it :(
>>>
>>>
>>> On Sun, Sep 11, 2022 at 12:56 AM Joel Serrano  wrote:
>>>
 Hello,

 I have a cluster of 2 kamailios v5.5 on debian 9 working flawlessly.

 We have added a third node, also on v5.5 but debian 11. Kamailio
 doesn't work correctly for some reason that I'm not seeing.

 The symptoms are:

 1- Kamailio receives INVITEs and starts to process them per routing
 script.
 2- There is a specific place where something happens and the calls are
 dropped (Kamailio is not even replying to the source). Note that the config
 is exactly the same on all 3 servers, only one of the three is having the
 issue.

 I enabled debug logs and I could see:

 [...]
 Sep 10 12:30:48 sbc03 sbc[956340]: exec: {1 102 INVITE
 5111a7c71ec485e443099844491ef...@sip02.example.com} ***
 cfgtrace:dbg_cfg_trace(): request_route=[GET_OUTBOUND_API_DATA]
 c=[/etc/kamailio/sbc/api.cfg] l=61 a=5 n=route
 Sep 10 12:30:48 sbc03 sbc[956340]: exec: {1 102 INVITE
 5111a7c71ec485e443099844491ef...@sip02.example.com} ***
 cfgtrace:dbg_cfg_trace(): request_route=[APPLY_REWRITE_RULE]
 c=[/etc/kamailio/sbc/extras.cfg] l=211 a=26 n=xlog
 Sep 10 12:30:48 sbc03 sbc[956340]: INFO: {1 102 INVITE
 5111a7c71ec485e443099844491ef...@sip02.vozelia.com.pa} 

Re: [SR-Users] ndb_redis value substitution or escaping spaces

[Alex Balashov]

I see what you mean: if postprocessing the CSV is required, that sounds like a 
pain. Ideally there’s a way to unencode the column values as they’re streamed 
into pg_bulkupload somehow, even if it requires an additional programmatic shim.

> On Sep 15, 2022, at 12:47 PM, Brooks Bridges  wrote:
> 
> I'm more concerned about the time and load involved in having to loop through 
> a massive csv and run that against specific columns of each line.  There may 
> be an option to avoid having to do that though, I'll experiment with it a 
> bit.  Thanks for the suggestion!
> 
> -Original Message-
> From: sr-users  On Behalf Of Alex 
> Balashov
> Sent: Thursday, September 15, 2022 09:39
> To: Kamailio (SER) - Users Mailing List 
> Subject: Re: [SR-Users] ndb_redis value substitution or escaping spaces
> 
> base64 is a pretty trivial algorithm. You’d be surprised.
> 
>> On Sep 15, 2022, at 12:37 PM, Brooks Bridges  wrote:
>> 
>> Unfortunately due to the volume of records, I don't think that's going to be 
>> really feasible as I'm dumping these out directly to a csv to be archived 
>> and imported into a database using pg_bulkload, and I don't think the 
>> additional overhead of having to process every record (of which there will 
>> be at least 1M per minute, likely more in the future) is feasible without 
>> impacting the performance of the system.
>> 
>> -Original Message-
>> From: sr-users  On Behalf Of Alex 
>> Balashov
>> Sent: Thursday, September 15, 2022 09:11
>> To: Kamailio (SER) - Users Mailing List 
>> Subject: Re: [SR-Users] ndb_redis value substitution or escaping spaces
>> 
>> A common approach to avoid delimiter issues in general is to store 
>> base64-encoded strings in Redis rather than the original strings. If 
>> whatever is reading from Redis can un-encode them, that pretty well solves 
>> the problem.
>> 
>>> On Sep 15, 2022, at 12:09 PM, Brooks Bridges  wrote:
>>> 
>>> I am aware of, and have used sucessfully, the %s substitution option in 
>>> redis_cmd, however when trying to do 4 values I'm getting a parse error 
>>> about too many arguments.  A review of the module's source appears to 
>>> indicate that there is a hard limit of 3 substitution values when using 
>>> this method.
>>> 
>>> Is there a suitable workaround anyone has for this to enable insertion of 
>>> values with spaces in them that won't get interpreted by Kamailio itself?  
>>> Escaping the value, even by using $_s to eval a dynamic string, ends up 
>>> with Kamailio itself apparently trying to parse the escape and breaking 
>>> things further.
>>> 
>>> Thanks!
>>> Confidentiality Notice: This e-mail, and any attachment to it, contains 
>>> privileged and confidential information intended only for the use of the 
>>> individual(s) or entity named on the e-mail. If the reader of this e-mail 
>>> is not the intended recipient, or the employee or agent responsible for 
>>> delivering it to the intended recipient, you are hereby notified that 
>>> reading this e-mail is strictly prohibited. If you have received this 
>>> e-mail in error, please immediately return it to the sender and delete it 
>>> from your system.
>>> 
>>> __
>>> Kamailio - Users Mailing List - Non Commercial Discussions
>>> * sr-users@lists.kamailio.org
>>> Important: keep the mailing list in the recipients, do not reply only to 
>>> the sender!
>>> Edit mailing list options or unsubscribe:
>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>> 
>> --
>> Alex Balashov | Principal | Evariste Systems LLC
>> 
>> Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
>> Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
>> 
>> 
>> __
>> Kamailio - Users Mailing List - Non Commercial Discussions
>> * sr-users@lists.kamailio.org
>> Important: keep the mailing list in the recipients, do not reply only to the 
>> sender!
>> Edit mailing list options or unsubscribe:
>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>> Confidentiality Notice: This e-mail, and any attachment to it, contains 
>> privileged and confidential information intended only for the use of the 
>> individual(s) or entity named on the e-mail. If the reader of this e-mail is 
>> not the intended recipient, or the employee or agent responsible for 
>> delivering it to the intended recipient, you are hereby notified that 
>> reading this e-mail is strictly prohibited. If you have received this e-mail 
>> in error, please immediately return it to the sender and delete it from your 
>> system.
>> 
>> __
>> Kamailio - Users Mailing List - Non Commercial Discussions
>> * sr-users@lists.kamailio.org
>> Important: keep the mailing list in the recipients, do not reply only to the 
>> sender!
>> Edit mailing list options or unsubscribe:
>> * 

Re: [SR-Users] ndb_redis value substitution or escaping spaces

[Brooks Bridges]

I'm more concerned about the time and load involved in having to loop through a 
massive csv and run that against specific columns of each line.  There may be 
an option to avoid having to do that though, I'll experiment with it a bit.  
Thanks for the suggestion!

-Original Message-
From: sr-users  On Behalf Of Alex Balashov
Sent: Thursday, September 15, 2022 09:39
To: Kamailio (SER) - Users Mailing List 
Subject: Re: [SR-Users] ndb_redis value substitution or escaping spaces

base64 is a pretty trivial algorithm. You’d be surprised.

> On Sep 15, 2022, at 12:37 PM, Brooks Bridges  wrote:
>
> Unfortunately due to the volume of records, I don't think that's going to be 
> really feasible as I'm dumping these out directly to a csv to be archived and 
> imported into a database using pg_bulkload, and I don't think the additional 
> overhead of having to process every record (of which there will be at least 
> 1M per minute, likely more in the future) is feasible without impacting the 
> performance of the system.
>
> -Original Message-
> From: sr-users  On Behalf Of Alex 
> Balashov
> Sent: Thursday, September 15, 2022 09:11
> To: Kamailio (SER) - Users Mailing List 
> Subject: Re: [SR-Users] ndb_redis value substitution or escaping spaces
>
> A common approach to avoid delimiter issues in general is to store 
> base64-encoded strings in Redis rather than the original strings. If whatever 
> is reading from Redis can un-encode them, that pretty well solves the problem.
>
>> On Sep 15, 2022, at 12:09 PM, Brooks Bridges  wrote:
>>
>> I am aware of, and have used sucessfully, the %s substitution option in 
>> redis_cmd, however when trying to do 4 values I'm getting a parse error 
>> about too many arguments.  A review of the module's source appears to 
>> indicate that there is a hard limit of 3 substitution values when using this 
>> method.
>>
>> Is there a suitable workaround anyone has for this to enable insertion of 
>> values with spaces in them that won't get interpreted by Kamailio itself?  
>> Escaping the value, even by using $_s to eval a dynamic string, ends up with 
>> Kamailio itself apparently trying to parse the escape and breaking things 
>> further.
>>
>> Thanks!
>> Confidentiality Notice: This e-mail, and any attachment to it, contains 
>> privileged and confidential information intended only for the use of the 
>> individual(s) or entity named on the e-mail. If the reader of this e-mail is 
>> not the intended recipient, or the employee or agent responsible for 
>> delivering it to the intended recipient, you are hereby notified that 
>> reading this e-mail is strictly prohibited. If you have received this e-mail 
>> in error, please immediately return it to the sender and delete it from your 
>> system.
>>
>> __
>> Kamailio - Users Mailing List - Non Commercial Discussions
>> * sr-users@lists.kamailio.org
>> Important: keep the mailing list in the recipients, do not reply only to the 
>> sender!
>> Edit mailing list options or unsubscribe:
>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> --
> Alex Balashov | Principal | Evariste Systems LLC
>
> Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
> Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
>
>
> __
> Kamailio - Users Mailing List - Non Commercial Discussions
>  * sr-users@lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to the 
> sender!
> Edit mailing list options or unsubscribe:
>  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> Confidentiality Notice: This e-mail, and any attachment to it, contains 
> privileged and confidential information intended only for the use of the 
> individual(s) or entity named on the e-mail. If the reader of this e-mail is 
> not the intended recipient, or the employee or agent responsible for 
> delivering it to the intended recipient, you are hereby notified that reading 
> this e-mail is strictly prohibited. If you have received this e-mail in 
> error, please immediately return it to the sender and delete it from your 
> system.
>
> __
> Kamailio - Users Mailing List - Non Commercial Discussions
>  * sr-users@lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to the 
> sender!
> Edit mailing list options or unsubscribe:
>  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

--
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/


__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit 

Re: [SR-Users] ndb_redis value substitution or escaping spaces

[Alex Balashov]

base64 is a pretty trivial algorithm. You’d be surprised.

> On Sep 15, 2022, at 12:37 PM, Brooks Bridges  wrote:
> 
> Unfortunately due to the volume of records, I don't think that's going to be 
> really feasible as I'm dumping these out directly to a csv to be archived and 
> imported into a database using pg_bulkload, and I don't think the additional 
> overhead of having to process every record (of which there will be at least 
> 1M per minute, likely more in the future) is feasible without impacting the 
> performance of the system.
> 
> -Original Message-
> From: sr-users  On Behalf Of Alex 
> Balashov
> Sent: Thursday, September 15, 2022 09:11
> To: Kamailio (SER) - Users Mailing List 
> Subject: Re: [SR-Users] ndb_redis value substitution or escaping spaces
> 
> A common approach to avoid delimiter issues in general is to store 
> base64-encoded strings in Redis rather than the original strings. If whatever 
> is reading from Redis can un-encode them, that pretty well solves the problem.
> 
>> On Sep 15, 2022, at 12:09 PM, Brooks Bridges  wrote:
>> 
>> I am aware of, and have used sucessfully, the %s substitution option in 
>> redis_cmd, however when trying to do 4 values I'm getting a parse error 
>> about too many arguments.  A review of the module's source appears to 
>> indicate that there is a hard limit of 3 substitution values when using this 
>> method.
>> 
>> Is there a suitable workaround anyone has for this to enable insertion of 
>> values with spaces in them that won't get interpreted by Kamailio itself?  
>> Escaping the value, even by using $_s to eval a dynamic string, ends up with 
>> Kamailio itself apparently trying to parse the escape and breaking things 
>> further.
>> 
>> Thanks!
>> Confidentiality Notice: This e-mail, and any attachment to it, contains 
>> privileged and confidential information intended only for the use of the 
>> individual(s) or entity named on the e-mail. If the reader of this e-mail is 
>> not the intended recipient, or the employee or agent responsible for 
>> delivering it to the intended recipient, you are hereby notified that 
>> reading this e-mail is strictly prohibited. If you have received this e-mail 
>> in error, please immediately return it to the sender and delete it from your 
>> system.
>> 
>> __
>> Kamailio - Users Mailing List - Non Commercial Discussions
>> * sr-users@lists.kamailio.org
>> Important: keep the mailing list in the recipients, do not reply only to the 
>> sender!
>> Edit mailing list options or unsubscribe:
>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> 
> --
> Alex Balashov | Principal | Evariste Systems LLC
> 
> Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
> Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
> 
> 
> __
> Kamailio - Users Mailing List - Non Commercial Discussions
>  * sr-users@lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to the 
> sender!
> Edit mailing list options or unsubscribe:
>  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> Confidentiality Notice: This e-mail, and any attachment to it, contains 
> privileged and confidential information intended only for the use of the 
> individual(s) or entity named on the e-mail. If the reader of this e-mail is 
> not the intended recipient, or the employee or agent responsible for 
> delivering it to the intended recipient, you are hereby notified that reading 
> this e-mail is strictly prohibited. If you have received this e-mail in 
> error, please immediately return it to the sender and delete it from your 
> system.
> 
> __
> Kamailio - Users Mailing List - Non Commercial Discussions
>  * sr-users@lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to the 
> sender!
> Edit mailing list options or unsubscribe:
>  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/


__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] ndb_redis value substitution or escaping spaces

[Brooks Bridges]

Unfortunately due to the volume of records, I don't think that's going to be 
really feasible as I'm dumping these out directly to a csv to be archived and 
imported into a database using pg_bulkload, and I don't think the additional 
overhead of having to process every record (of which there will be at least 1M 
per minute, likely more in the future) is feasible without impacting the 
performance of the system.

-Original Message-
From: sr-users  On Behalf Of Alex Balashov
Sent: Thursday, September 15, 2022 09:11
To: Kamailio (SER) - Users Mailing List 
Subject: Re: [SR-Users] ndb_redis value substitution or escaping spaces

A common approach to avoid delimiter issues in general is to store 
base64-encoded strings in Redis rather than the original strings. If whatever 
is reading from Redis can un-encode them, that pretty well solves the problem.

> On Sep 15, 2022, at 12:09 PM, Brooks Bridges  wrote:
>
> I am aware of, and have used sucessfully, the %s substitution option in 
> redis_cmd, however when trying to do 4 values I'm getting a parse error about 
> too many arguments.  A review of the module's source appears to indicate that 
> there is a hard limit of 3 substitution values when using this method.
>
> Is there a suitable workaround anyone has for this to enable insertion of 
> values with spaces in them that won't get interpreted by Kamailio itself?  
> Escaping the value, even by using $_s to eval a dynamic string, ends up with 
> Kamailio itself apparently trying to parse the escape and breaking things 
> further.
>
> Thanks!
> Confidentiality Notice: This e-mail, and any attachment to it, contains 
> privileged and confidential information intended only for the use of the 
> individual(s) or entity named on the e-mail. If the reader of this e-mail is 
> not the intended recipient, or the employee or agent responsible for 
> delivering it to the intended recipient, you are hereby notified that reading 
> this e-mail is strictly prohibited. If you have received this e-mail in 
> error, please immediately return it to the sender and delete it from your 
> system.
>
> __
> Kamailio - Users Mailing List - Non Commercial Discussions
>  * sr-users@lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to the 
> sender!
> Edit mailing list options or unsubscribe:
>  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

--
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/


__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Confidentiality Notice: This e-mail, and any attachment to it, contains 
privileged and confidential information intended only for the use of the 
individual(s) or entity named on the e-mail. If the reader of this e-mail is 
not the intended recipient, or the employee or agent responsible for delivering 
it to the intended recipient, you are hereby notified that reading this e-mail 
is strictly prohibited. If you have received this e-mail in error, please 
immediately return it to the sender and delete it from your system.

__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Log levels severely impacts performance

[Ovidiu Sas]

Then rework the logs.
Do you have more than one log per registration? If yes, use only one.
If you already have a single log per registration, then try reducing the
size.
Experiment to see where the system breaks. Try new hardware. Decrease the
number of workers to see if it has an impact.

-ovidiu

On Thu, Sep 15, 2022 at 11:30 Amit  wrote:

> Hi Henning,
>
> No we are using a local graylog server, not cloud.
>
> I suppose if the bottleneck is log generation, it doesn’t matter how or
> where they are handled/sent.
>
> Amit
>
>
>
> *From: *Henning Westerholt 
> *Date: *Thursday, September 15, 2022 at 9:53 AM
> *To: *Amit 
> *Cc: *Kamailio (SER) - Users Mailing List 
> *Subject: *RE: [SR-Users] Log levels severely impacts performance
>
> Hello Amit,
>
>
>
> this sounds strange. If you send them remotely it should not really affect
> the operation anymore.
>
> Maybe it’s a special limitation of the cloud server, if you use a public
> cloud service provider, but just guessing here.
>
>
>
> Cheers,
>
>
>
> Henning
>
>
>
> --
>
> Henning Westerholt – https://skalatan.de/blog/
>
> Kamailio services – https://gilawa.com
>
>
>
> *From:* Amit 
> *Sent:* Thursday, September 15, 2022 3:16 PM
> *To:* Henning Westerholt 
> *Cc:* Kamailio (SER) - Users Mailing List 
> *Subject:* Re: [SR-Users] Log levels severely impacts performance
>
>
>
> Hi Henning,
>
>
>
> We are actually sending the logs directly to our local graylog server via
> the $programname property in rsyslog, and we’ve disabled writing Kamailio
> logs to disk (i.e. only in memory by setting Storage=volatile in
> rsyslog.conf) but that didn’t seem to help with the application socket
> receive buffer backlog.
>
>
>
> Keep in mind that for testing purposes, I am blasting Kamailio with a
> fixed 2000 simultaneous registrations, so at log level 2 it really
> struggles and the socket buffer quickly fills up. Reducing the log level to
> 1 allows Kamailio to rip through the requests and the buffer stays at 0.
>
>
>
> 
>
> Amit Nir, MBA
>
> Office: 305.999.0911
>
> www.BryteCall.com
>
>
>
>
>
> *From: *Henning Westerholt 
> *Date: *Thursday, September 15, 2022 at 9:01 AM
> *To: *Amit 
> *Cc: *Kamailio (SER) - Users Mailing List 
> *Subject: *RE: [SR-Users] Log levels severely impacts performance
>
> Hello,
>
>
>
> how much are you actually logging? As already suggested, maybe just need
> to decrease the amount of log messages.
>
>
>
> If you need this much logging, you can of course forward it to a dedicated
> log server over the network.
>
>
>
> Cheers,
>
>
>
> Henning
>
>
>
> --
>
> Henning Westerholt – https://skalatan.de/blog/
>
> Kamailio services – https://gilawa.com
>
>
>
> *From:* Amit 
> *Sent:* Thursday, September 15, 2022 1:33 PM
> *To:* Henning Westerholt 
> *Cc:* Kamailio (SER) - Users Mailing List 
> *Subject:* Re: [SR-Users] Log levels severely impacts performance
>
>
>
> Hi Henning,
>
> Thanks for the suggestion.
>
> Disk I/O was our initial assumption on another Kamilio instance we are
> running.
>
> To test this theory we installed Kamailio on separate hardware and
> configurations with faster underlying SSD disks, however we are seeing the
> same crippled performance and congestion in the applications socket receive
> buffer when LOG level is set to INFO (2).
>
>
>
> 
>
> Amit Nir, MBA
>
> Office: 305.999.0911
>
> www.BryteCall.com
>
>
>
>
> On Sep 15, 2022, at 6:14 AM, Henning Westerholt  wrote:
>
> 
>
> Hello,
>
>
>
> additionally, try to see if you have some general I/O issues, e.g., by
> observing io-wait CPU, looking to I/O transaction times with monitoring
> tools etc..
>
> Maybe there are other services also causing excessive I/O.
>
>
>
> Cheers,
>
>
>
> Henning
>
>
>
> --
>
> Henning Westerholt – https://skalatan.de/blog/
>
> Kamailio services – https://gilawa.com
>
>
>
> *From:* sr-users  *On Behalf Of 
> *Daniel-Constantin
> Mierla
> *Sent:* Wednesday, September 14, 2022 2:25 PM
> *To:* Kamailio (SER) - Users Mailing List ;
> Amit 
> *Subject:* Re: [SR-Users] Log levels severely impacts performance
>
>
>
> Hello,
>
> excessive logging is known to impact the performances. If you have many
> log messages printed from config, try to reduce them.
>
> Performance could be also a matter of how many child processes you
> created, via the children global parameter.
>
> Cheers,
> Daniel
>
> On 14.09.22 04:18, Amit wrote:
>
> Hello,
>
>
>
> We’ve been running Kamailio 5.5.0 for a while now and recently ran into an
> issue where the application buffer would get full and Kamailio has a lot of
> difficulty keeping up with REG requests. This severely impacted Kamailio’s
> ability to perform and resulted in many REG timeouts.
>
>
>
> Running ‘watch netstat -ulpn’ revealed the buffer issue.
>
>
>
> We usually run the production system at log level INFO (2) and had not ran
> into any issues in the past.
>
>
>
> When the buffer issue occurred and after some testing, we discovered that
> changing the log level to 

Re: [SR-Users] Kamailio stops processing requests mid script

[Daniel-Constantin Mierla]

Hello,

if the traffic is over UDP, the Kamailio workers were blocked in
recvfrom(), meaning that nothing is passed from the network layer to
them. If you see UDP packets coming to Kamailio via ngrep/sngrep/...,
then is the OS that drops them via firewall or some other app that
controls the network traffic.

Cheers,
Daniel

On 15.09.22 07:37, Joel Serrano wrote:
> Hi Daniel, 
>
> I've tried with apparmor disabled unfortunately the same issue happens. 
>
> I've sent you privately the output of the kamctl trap. I personally
> don't think it's Kamailio's fault per se, this is on a standard debian
> 11. I'm just lost and don't really understand what the hell is going on.
>
> Thanks, I appreciate your help with this.
> Joel.
>
>
>
> On Wed, Sep 14, 2022 at 9:50 AM Joel Serrano  wrote:
>
> Hi Daniel, 
>
> I've followed your suggestions and compared this "bad" server with
> the two "good" ones.
>
> - Pike:
>
> In all cases we have:
>
>     if (src_ip!=myself && !ds_is_from_list()) {
>         if($sht(ipban=>$si)!=$null) {
>             xlog("L_ALERT","ALERT: blocked by pike R=$ru from $fu
> (IP:$si:$sp)\n");
>             exit;
>         }
>         if (!pike_check_req()) {
>             xlog("L_ALERT","ALERT: pike blocking R=$ru from $fu
> (IP:$si:$sp)\n");
>             $sht(ipban=>$si) = 1;
>             exit;
>         }
>     }
>
> And we are not seeing any logs, therefore I'm discarding pike.
>
> - Firewall:
>
> I checked all 3 servers, and none of them have -local- firewall
> policies.
>
> - conntrack:
>
> All 3 servers have nf_conntrack loaded in kernel.
>
> - selinux/etc:
>
> The two good servers have "AppArmor" disabled.
> The bad server has "AppArmor" enabled. !! <--
> I'm hoping this could be the cause and I'm going to test tonight
> without it.
>
>
> Thanks for checking this, I was so lost I actually went ahead and
> did "kamctl trap" last night too just in case. When I run it, it
> didn't stop by itself (I stopped it with CTRL+C after some time)
> but it did generate a file with a lot of output. Can I send you
> it privately? I'm not sure how to interpret it.
>
> Before anything I'm going to test tonight:
>
> 1- disabling apparmor
> 2- unloading nf_conntrack
>
> I'll report back with the resutls.
>
>
>
>
>
> On Wed, Sep 14, 2022 at 5:27 AM Daniel-Constantin Mierla
>  wrote:
>
> Hello,
>
> be sure you do not hit some limits set by Kamailio (e.g.,
> pike) or the system/firewall (e.g., selinux, conntrack).
>
> You should install gdb and run "kamctl trap" when it stops
> processing and inspect the written file to see where each
> kamailio process is in the execution stack.
>
> Cheers,
> Daniel
>
> On 14.09.22 10:20, Joel Serrano wrote:
>> Bumping this! Any comments? Or suggestions on what to check?
>> I'm feeling it has to be something stupid but I can't see it :(
>>
>>
>> On Sun, Sep 11, 2022 at 12:56 AM Joel Serrano
>>  wrote:
>>
>> Hello,
>>
>> I have a cluster of 2 kamailios v5.5 on debian 9 working
>> flawlessly.
>>
>> We have added a third node, also on v5.5 but debian 11.
>> Kamailio doesn't work correctly for some reason that I'm
>> not seeing.
>>
>> The symptoms are:
>>
>> 1- Kamailio receives INVITEs and starts to process them
>> per routing script.
>> 2- There is a specific place where something happens and
>> the calls are dropped (Kamailio is not even replying to
>> the source). Note that the config is exactly the same on
>> all 3 servers, only one of the three is having the issue.
>>
>> I enabled debug logs and I could see:
>>
>> [...]
>> Sep 10 12:30:48 sbc03 sbc[956340]: exec: {1 102 INVITE
>> 5111a7c71ec485e443099844491ef...@sip02.example.com} ***
>> cfgtrace:dbg_cfg_trace():
>> request_route=[GET_OUTBOUND_API_DATA]
>> c=[/etc/kamailio/sbc/api.cfg] l=61 a=5 n=route
>> Sep 10 12:30:48 sbc03 sbc[956340]: exec: {1 102 INVITE
>> 5111a7c71ec485e443099844491ef...@sip02.example.com} ***
>> cfgtrace:dbg_cfg_trace():
>> request_route=[APPLY_REWRITE_RULE]
>> c=[/etc/kamailio/sbc/extras.cfg] l=211 a=26 n=xlog
>> Sep 10 12:30:48 sbc03 sbc[956340]: INFO: {1 102 INVITE
>> 5111a7c71ec485e443099844491ef...@sip02.vozelia.com.pa}
>> 

Re: [SR-Users] ndb_redis value substitution or escaping spaces

[Alex Balashov]

A common approach to avoid delimiter issues in general is to store 
base64-encoded strings in Redis rather than the original strings. If whatever 
is reading from Redis can un-encode them, that pretty well solves the problem.

> On Sep 15, 2022, at 12:09 PM, Brooks Bridges  wrote:
> 
> I am aware of, and have used sucessfully, the %s substitution option in 
> redis_cmd, however when trying to do 4 values I'm getting a parse error about 
> too many arguments.  A review of the module's source appears to indicate that 
> there is a hard limit of 3 substitution values when using this method.
> 
> Is there a suitable workaround anyone has for this to enable insertion of 
> values with spaces in them that won't get interpreted by Kamailio itself?  
> Escaping the value, even by using $_s to eval a dynamic string, ends up with 
> Kamailio itself apparently trying to parse the escape and breaking things 
> further.
> 
> Thanks!
> Confidentiality Notice: This e-mail, and any attachment to it, contains 
> privileged and confidential information intended only for the use of the 
> individual(s) or entity named on the e-mail. If the reader of this e-mail is 
> not the intended recipient, or the employee or agent responsible for 
> delivering it to the intended recipient, you are hereby notified that reading 
> this e-mail is strictly prohibited. If you have received this e-mail in 
> error, please immediately return it to the sender and delete it from your 
> system.
> 
> __
> Kamailio - Users Mailing List - Non Commercial Discussions
>  * sr-users@lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to the 
> sender!
> Edit mailing list options or unsubscribe:
>  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/


__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

[SR-Users] ndb_redis value substitution or escaping spaces

[Brooks Bridges]

I am aware of, and have used sucessfully, the %s substitution option in 
redis_cmd, however when trying to do 4 values I'm getting a parse error about 
too many arguments.  A review of the module's source appears to indicate that 
there is a hard limit of 3 substitution values when using this method.

Is there a suitable workaround anyone has for this to enable insertion of 
values with spaces in them that won't get interpreted by Kamailio itself?  
Escaping the value, even by using $_s to eval a dynamic string, ends up with 
Kamailio itself apparently trying to parse the escape and breaking things 
further.

Thanks!
Confidentiality Notice: This e-mail, and any attachment to it, contains 
privileged and confidential information intended only for the use of the 
individual(s) or entity named on the e-mail. If the reader of this e-mail is 
not the intended recipient, or the employee or agent responsible for delivering 
it to the intended recipient, you are hereby notified that reading this e-mail 
is strictly prohibited. If you have received this e-mail in error, please 
immediately return it to the sender and delete it from your system.

__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Log levels severely impacts performance

[Amit]

Hi Henning,
No we are using a local graylog server, not cloud.
I suppose if the bottleneck is log generation, it doesn’t matter how or where 
they are handled/sent.
Amit

From: Henning Westerholt 
Date: Thursday, September 15, 2022 at 9:53 AM
To: Amit 
Cc: Kamailio (SER) - Users Mailing List 
Subject: RE: [SR-Users] Log levels severely impacts performance
Hello Amit,

this sounds strange. If you send them remotely it should not really affect the 
operation anymore.
Maybe it’s a special limitation of the cloud server, if you use a public cloud 
service provider, but just guessing here.

Cheers,

Henning

--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com

From: Amit 
Sent: Thursday, September 15, 2022 3:16 PM
To: Henning Westerholt 
Cc: Kamailio (SER) - Users Mailing List 
Subject: Re: [SR-Users] Log levels severely impacts performance

Hi Henning,

We are actually sending the logs directly to our local graylog server via the 
$programname property in rsyslog, and we’ve disabled writing Kamailio logs to 
disk (i.e. only in memory by setting Storage=volatile in rsyslog.conf) but that 
didn’t seem to help with the application socket receive buffer backlog.

Keep in mind that for testing purposes, I am blasting Kamailio with a fixed 
2000 simultaneous registrations, so at log level 2 it really struggles and the 
socket buffer quickly fills up. Reducing the log level to 1 allows Kamailio to 
rip through the requests and the buffer stays at 0.


Amit Nir, MBA
Office: 305.999.0911
www.BryteCall.com


From: Henning Westerholt mailto:h...@gilawa.com>>
Date: Thursday, September 15, 2022 at 9:01 AM
To: Amit mailto:a...@brytecall.com>>
Cc: Kamailio (SER) - Users Mailing List 
mailto:sr-users@lists.kamailio.org>>
Subject: RE: [SR-Users] Log levels severely impacts performance
Hello,

how much are you actually logging? As already suggested, maybe just need to 
decrease the amount of log messages.

If you need this much logging, you can of course forward it to a dedicated log 
server over the network.

Cheers,

Henning

--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com

From: Amit mailto:a...@brytecall.com>>
Sent: Thursday, September 15, 2022 1:33 PM
To: Henning Westerholt mailto:h...@gilawa.com>>
Cc: Kamailio (SER) - Users Mailing List 
mailto:sr-users@lists.kamailio.org>>
Subject: Re: [SR-Users] Log levels severely impacts performance

Hi Henning,
Thanks for the suggestion.
Disk I/O was our initial assumption on another Kamilio instance we are running.
To test this theory we installed Kamailio on separate hardware and 
configurations with faster underlying SSD disks, however we are seeing the same 
crippled performance and congestion in the applications socket receive buffer 
when LOG level is set to INFO (2).


Amit Nir, MBA
Office: 305.999.0911
www.BryteCall.com



On Sep 15, 2022, at 6:14 AM, Henning Westerholt 
mailto:h...@gilawa.com>> wrote:

Hello,

additionally, try to see if you have some general I/O issues, e.g., by 
observing io-wait CPU, looking to I/O transaction times with monitoring tools 
etc..
Maybe there are other services also causing excessive I/O.

Cheers,

Henning

--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com

From: sr-users 
mailto:sr-users-boun...@lists.kamailio.org>>
 On Behalf Of Daniel-Constantin Mierla
Sent: Wednesday, September 14, 2022 2:25 PM
To: Kamailio (SER) - Users Mailing List 
mailto:sr-users@lists.kamailio.org>>; Amit 
mailto:a...@brytecall.com>>
Subject: Re: [SR-Users] Log levels severely impacts performance


Hello,

excessive logging is known to impact the performances. If you have many log 
messages printed from config, try to reduce them.

Performance could be also a matter of how many child processes you created, via 
the children global parameter.

Cheers,
Daniel
On 14.09.22 04:18, Amit wrote:
Hello,

We’ve been running Kamailio 5.5.0 for a while now and recently ran into an 
issue where the application buffer would get full and Kamailio has a lot of 
difficulty keeping up with REG requests. This severely impacted Kamailio’s 
ability to perform and resulted in many REG timeouts.

Running ‘watch netstat -ulpn’ revealed the buffer issue.

We usually run the production system at log level INFO (2) and had not ran into 
any issues in the past.

When the buffer issue occurred and after some testing, we discovered that 
changing the log level to NOTICE (1) was sufficient to bring the system back 
into operational status.

For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while at 
log level NOTICE (1) we reach ~500-700 REG/s.

I did find this article in the wiki and tried using the “-“ in front of the log 
path but this didn’t seem to help.

Re: [SR-Users] Kill child process created by app_python module

[Marat Gareev]

Hello Daniel,

Probably, the process is not killed, because the child processes of the
python module are not included in the kamailio children list.

How can I register an event route in KEMI script?
Doc

says:

> event route callback - the name of the Python function to be executed
> instead of module specific event_route blocks is provided via
> event_callback parameter of that module

but evrexec
 doesn't
have an event_callback parameter.

I tried to set up exec parameter (_update is Python function)
modparam("evrexec", "exec", "name=_update;workers=1;")
and got warning
WARNING: evrexec [evrexec_mod.c:180]: evrexec_process(): empty event route
block [_update]


ср, 14 сент. 2022 г. в 16:21, Daniel-Constantin Mierla :

> Hello,
>
> not sure I got right what is the issue and what would be a solution for
> it, but maybe you can leverage evrexec to create a child process and by
> that keep it in kamailio children list.
>
> Cheers,
> Daniel
> On 13.09.22 19:15, Marat Gareev wrote:
>
> Hello!
>
> I'm trying to use app_python module to perform periodic actions.
>
> This is a simple python code
> def _update():
> while True:
> _action()
> time.sleep(60)
> class noop_handler:
> def child_init(self, rank):
> return 0
> def ksr_request_route(self, msg):
> return 1
> def mod_init():
> # signal.signal(signal.SIGTERM, signal.default_int_handler)
> proc = multiprocessing.Process(target=_update)
> # proc.daemon = True
> proc.start()
> ksr.notice('Forked process {} to update\n'.format(proc.pid))
> return noop_handler()
>
> After killing kamailio processes with SIGTERM signal (killall kamailio) I
> see zombie process.
> And if I try to process the signal in daemon mode (see the comments in the
> snippet above), I get critical errors in the logs and still see processes:
> 0(37001) NOTICE:  [core/kemi.c:124]: sr_kemi_core_notice(): Forked
> process 37002 to update
> 1(37003) ERROR:  [core/udp_server.c:464]: udp_rcv_loop(): recvfrom:[
> 4] Interrupted system call
> Process Process-1:
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in
> _bootstrap
> self.run()
> File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
> self._target(*self._args, **self._kwargs)
> File "/tmp/script.py", line 155, in _update
> time.sleep(period)
> KeyboardInterrupt
> 0(37001) ALERT:  [main.c:774]: handle_sigs(): child process 37002
> exited normally, status=1
> 0(37001) INFO:  [main.c:802]: handle_sigs(): terminating due to
> SIGCHLD
> 1(37003) ERROR:  [core/udp_server.c:464]: udp_rcv_loop(): recvfrom:[
> 4] Interrupted system call
> 8(37014) CRITICAL:  [core/pass_fd.c:277]: receive_fd(): EOF on 4
> 1(37003) ERROR:  [core/udp_server.c:464]: udp_rcv_loop(): recvfrom:[
> 4] Interrupted system call
>
> So, how I can kill child process by SIGTERM signal?
>
> __
> Kamailio - Users Mailing List - Non Commercial Discussions
>   * sr-users@lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to the 
> sender!
> Edit mailing list options or unsubscribe:
>   * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- 
> www.linkedin.com/in/miconda
>
>
__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Log levels severely impacts performance

[Amit]

Hi Henning,

We are actually sending the logs directly to our local graylog server via the 
$programname property in rsyslog, and we’ve disabled writing Kamailio logs to 
disk (i.e. only in memory by setting Storage=volatile in rsyslog.conf) but that 
didn’t seem to help with the application socket receive buffer backlog.

Keep in mind that for testing purposes, I am blasting Kamailio with a fixed 
2000 simultaneous registrations, so at log level 2 it really struggles and the 
socket buffer quickly fills up. Reducing the log level to 1 allows Kamailio to 
rip through the requests and the buffer stays at 0.


Amit Nir, MBA
Office: 305.999.0911
www.BryteCall.com


From: Henning Westerholt 
Date: Thursday, September 15, 2022 at 9:01 AM
To: Amit 
Cc: Kamailio (SER) - Users Mailing List 
Subject: RE: [SR-Users] Log levels severely impacts performance
Hello,

how much are you actually logging? As already suggested, maybe just need to 
decrease the amount of log messages.

If you need this much logging, you can of course forward it to a dedicated log 
server over the network.

Cheers,

Henning

--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com

From: Amit 
Sent: Thursday, September 15, 2022 1:33 PM
To: Henning Westerholt 
Cc: Kamailio (SER) - Users Mailing List 
Subject: Re: [SR-Users] Log levels severely impacts performance

Hi Henning,
Thanks for the suggestion.
Disk I/O was our initial assumption on another Kamilio instance we are running.
To test this theory we installed Kamailio on separate hardware and 
configurations with faster underlying SSD disks, however we are seeing the same 
crippled performance and congestion in the applications socket receive buffer 
when LOG level is set to INFO (2).


Amit Nir, MBA
Office: 305.999.0911
www.BryteCall.com



On Sep 15, 2022, at 6:14 AM, Henning Westerholt 
mailto:h...@gilawa.com>> wrote:

Hello,

additionally, try to see if you have some general I/O issues, e.g., by 
observing io-wait CPU, looking to I/O transaction times with monitoring tools 
etc..
Maybe there are other services also causing excessive I/O.

Cheers,

Henning

--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com

From: sr-users 
mailto:sr-users-boun...@lists.kamailio.org>>
 On Behalf Of Daniel-Constantin Mierla
Sent: Wednesday, September 14, 2022 2:25 PM
To: Kamailio (SER) - Users Mailing List 
mailto:sr-users@lists.kamailio.org>>; Amit 
mailto:a...@brytecall.com>>
Subject: Re: [SR-Users] Log levels severely impacts performance


Hello,

excessive logging is known to impact the performances. If you have many log 
messages printed from config, try to reduce them.

Performance could be also a matter of how many child processes you created, via 
the children global parameter.

Cheers,
Daniel
On 14.09.22 04:18, Amit wrote:
Hello,

We’ve been running Kamailio 5.5.0 for a while now and recently ran into an 
issue where the application buffer would get full and Kamailio has a lot of 
difficulty keeping up with REG requests. This severely impacted Kamailio’s 
ability to perform and resulted in many REG timeouts.

Running ‘watch netstat -ulpn’ revealed the buffer issue.

We usually run the production system at log level INFO (2) and had not ran into 
any issues in the past.

When the buffer issue occurred and after some testing, we discovered that 
changing the log level to NOTICE (1) was sufficient to bring the system back 
into operational status.

For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while at 
log level NOTICE (1) we reach ~500-700 REG/s.

I did find this article in the wiki and tried using the “-“ in front of the log 
path but this didn’t seem to help.
http://www.kamailio.org/wiki/tutorials/3.2.x/syslog

Even writing the logs directly to our external log server and preventing them 
from being written to the local disk doesn’t seem to alleviate the issue when 
the log level is set to INFO (2).

Does anyone have any experience with this or have any suggestions? We are 
finding it difficult to scale as a result of this issue.

Thank you in advance.
Amit






__

Kamailio - Users Mailing List - Non Commercial Discussions

  * sr-users@lists.kamailio.org

Important: keep the mailing list in the recipients, do not reply only to the 
sender!

Edit mailing list options or unsubscribe:

  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

--

Daniel-Constantin Mierla -- www.asipto.com

www.twitter.com/miconda -- 
www.linkedin.com/in/miconda
__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep 

Re: [SR-Users] Log levels severely impacts performance

[Henning Westerholt]

Hello,

how much are you actually logging? As already suggested, maybe just need to 
decrease the amount of log messages.

If you need this much logging, you can of course forward it to a dedicated log 
server over the network.

Cheers,

Henning

--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com

From: Amit 
Sent: Thursday, September 15, 2022 1:33 PM
To: Henning Westerholt 
Cc: Kamailio (SER) - Users Mailing List 
Subject: Re: [SR-Users] Log levels severely impacts performance

Hi Henning,
Thanks for the suggestion.
Disk I/O was our initial assumption on another Kamilio instance we are running.
To test this theory we installed Kamailio on separate hardware and 
configurations with faster underlying SSD disks, however we are seeing the same 
crippled performance and congestion in the applications socket receive buffer 
when LOG level is set to INFO (2).


Amit Nir, MBA
Office: 305.999.0911
www.BryteCall.com


On Sep 15, 2022, at 6:14 AM, Henning Westerholt 
mailto:h...@gilawa.com>> wrote:

Hello,

additionally, try to see if you have some general I/O issues, e.g., by 
observing io-wait CPU, looking to I/O transaction times with monitoring tools 
etc..
Maybe there are other services also causing excessive I/O.

Cheers,

Henning

--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com

From: sr-users 
mailto:sr-users-boun...@lists.kamailio.org>>
 On Behalf Of Daniel-Constantin Mierla
Sent: Wednesday, September 14, 2022 2:25 PM
To: Kamailio (SER) - Users Mailing List 
mailto:sr-users@lists.kamailio.org>>; Amit 
mailto:a...@brytecall.com>>
Subject: Re: [SR-Users] Log levels severely impacts performance


Hello,

excessive logging is known to impact the performances. If you have many log 
messages printed from config, try to reduce them.

Performance could be also a matter of how many child processes you created, via 
the children global parameter.

Cheers,
Daniel
On 14.09.22 04:18, Amit wrote:
Hello,

We’ve been running Kamailio 5.5.0 for a while now and recently ran into an 
issue where the application buffer would get full and Kamailio has a lot of 
difficulty keeping up with REG requests. This severely impacted Kamailio’s 
ability to perform and resulted in many REG timeouts.

Running ‘watch netstat -ulpn’ revealed the buffer issue.

We usually run the production system at log level INFO (2) and had not ran into 
any issues in the past.

When the buffer issue occurred and after some testing, we discovered that 
changing the log level to NOTICE (1) was sufficient to bring the system back 
into operational status.

For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while at 
log level NOTICE (1) we reach ~500-700 REG/s.

I did find this article in the wiki and tried using the “-“ in front of the log 
path but this didn’t seem to help.
http://www.kamailio.org/wiki/tutorials/3.2.x/syslog

Even writing the logs directly to our external log server and preventing them 
from being written to the local disk doesn’t seem to alleviate the issue when 
the log level is set to INFO (2).

Does anyone have any experience with this or have any suggestions? We are 
finding it difficult to scale as a result of this issue.

Thank you in advance.
Amit





__

Kamailio - Users Mailing List - Non Commercial Discussions

  * sr-users@lists.kamailio.org

Important: keep the mailing list in the recipients, do not reply only to the 
sender!

Edit mailing list options or unsubscribe:

  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

--

Daniel-Constantin Mierla -- www.asipto.com

www.twitter.com/miconda -- 
www.linkedin.com/in/miconda
__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Log levels severely impacts performance

[Amit]

Hi Henning,
Thanks for the suggestion.
Disk I/O was our initial assumption on another Kamilio instance we are running.
To test this theory we installed Kamailio on separate hardware and 
configurations with faster underlying SSD disks, however we are seeing the same 
crippled performance and congestion in the applications socket receive buffer 
when LOG level is set to INFO (2).


Amit Nir, MBA
Office: 305.999.0911
www.BryteCall.com

On Sep 15, 2022, at 6:14 AM, Henning Westerholt  wrote:


Hello,

additionally, try to see if you have some general I/O issues, e.g., by 
observing io-wait CPU, looking to I/O transaction times with monitoring tools 
etc..
Maybe there are other services also causing excessive I/O.

Cheers,

Henning

--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com

From: sr-users  On Behalf Of 
Daniel-Constantin Mierla
Sent: Wednesday, September 14, 2022 2:25 PM
To: Kamailio (SER) - Users Mailing List ; Amit 

Subject: Re: [SR-Users] Log levels severely impacts performance


Hello,

excessive logging is known to impact the performances. If you have many log 
messages printed from config, try to reduce them.

Performance could be also a matter of how many child processes you created, via 
the children global parameter.

Cheers,
Daniel
On 14.09.22 04:18, Amit wrote:
Hello,

We’ve been running Kamailio 5.5.0 for a while now and recently ran into an 
issue where the application buffer would get full and Kamailio has a lot of 
difficulty keeping up with REG requests. This severely impacted Kamailio’s 
ability to perform and resulted in many REG timeouts.

Running ‘watch netstat -ulpn’ revealed the buffer issue.

We usually run the production system at log level INFO (2) and had not ran into 
any issues in the past.

When the buffer issue occurred and after some testing, we discovered that 
changing the log level to NOTICE (1) was sufficient to bring the system back 
into operational status.

For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while at 
log level NOTICE (1) we reach ~500-700 REG/s.

I did find this article in the wiki and tried using the “-“ in front of the log 
path but this didn’t seem to help.
http://www.kamailio.org/wiki/tutorials/3.2.x/syslog

Even writing the logs directly to our external log server and preventing them 
from being written to the local disk doesn’t seem to alleviate the issue when 
the log level is set to INFO (2).

Does anyone have any experience with this or have any suggestions? We are 
finding it difficult to scale as a result of this issue.

Thank you in advance.
Amit




__

Kamailio - Users Mailing List - Non Commercial Discussions

  * sr-users@lists.kamailio.org

Important: keep the mailing list in the recipients, do not reply only to the 
sender!

Edit mailing list options or unsubscribe:

  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

--

Daniel-Constantin Mierla -- www.asipto.com

www.twitter.com/miconda -- 
www.linkedin.com/in/miconda
__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

[SR-Users] SIPp uas(SIPp)->client reINVITE late offer

[David Villasmil]

Hello all,

Anyone got a SIPp scenario for reINVITE with late offer? I.e.:

UAT--->INVITE (sdp)--->SIPp
UAT<--- 100 <---SIPp
UAT<--- 183 <---SIPp
UAT<--- 200 OK <---SIPp
UAT> ACK --->SIPp
UAT<--- INVITE (NO sdp) <---SIPp
UAT---> 100 --->SIPp
UAT---> 183 --->SIPp
UAT---> 200 OK (sdp) -->SIPp
UAT<--- ACK (sdp) <---SIPp
UAT---> BYE -->SIPp
UAT<--- ACK <--SIPp

I can’t get past

UAT<--- INVITE (NO sdp) <---SIPp


Kamailio responds with “not here”


Appreciate the help!




-- 
Regards,

David Villasmil
email: david.villasmil.w...@gmail.com
phone: +34669448337
__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

[SR-Users] New feature: #!ifexp - conditional preprocessor blocks

[Daniel-Constantin Mierla]

Hello,

a new feature has landed to Kamailio devel version, bringing in more
flexibility for preprocessing of the kamailio.cfg file.

It is about #!ifexp, which allows to evaluate an expression created with
defined IDs, string and number values. Based on evaluation result, being
true or false, parts of the config file can be enabled or disabled.

An extensive number of operators can be used in expressions, such as: +,
-, *, /, ==, !=, <, >, <=, >=, ||, &&, etc. The evaluation is done using
snexpr (https://github.com/miconda/snexpr), which was embedded inside
Kamailio code.

Here are some simple kamailio.cfg examples:

#!ifexp KAMAILIO_VERSION >= 5006000
loadmodule "tlsa.so"
#!else
loadmodule "tls.so"
#!endif

#!ifexp MOD_xlog && (OS_NAME == "darwin")
xlog("running on MacOS\n");
#!endif

Documentation and more examples can be found on the wiki portal at:

  - https://www.kamailio.org/wikidocs/cookbooks/devel/core/#ifexp

Testing would be appreciated, feedback can be addressed to sr-users
mailing list!

Cheers,
Daniel

-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda


__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

[SR-Users] kamailio.org short unavailability

[Daniel-Constantin Mierla]

Hello,

kamailio.org system needs a reboot and it will become unavailable for a
short time frame about 08:00UTC (20 min from now). Website, mailing
list, wiki, and a few other web services will be affected.

Cheers,
Daniel

-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda


__
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the 
sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Kamailio stops processing requests mid script

[David Villasmil]

Is this metal or cloud? If aws try stopping/starting so it starts in a
different underlying hardware. If metal try with a different metal… could
be the hardware.

On Thu, 15 Sep 2022 at 07:44, Joel Serrano  wrote:

> Hi Daniel,
>
> I've tried with apparmor disabled unfortunately the same issue happens.
>
> I've sent you privately the output of the kamctl trap. I personally don't
> think it's Kamailio's fault per se, this is on a standard debian 11. I'm
> just lost and don't really understand what the hell is going on.
>
> Thanks, I appreciate your help with this.
> Joel.
>
>
>
> On Wed, Sep 14, 2022 at 9:50 AM Joel Serrano  wrote:
>
>> Hi Daniel,
>>
>> I've followed your suggestions and compared this "bad" server with the
>> two "good" ones.
>>
>> - Pike:
>>
>> In all cases we have:
>>
>> if (src_ip!=myself && !ds_is_from_list()) {
>> if($sht(ipban=>$si)!=$null) {
>> xlog("L_ALERT","ALERT: blocked by pike R=$ru from $fu
>> (IP:$si:$sp)\n");
>> exit;
>> }
>> if (!pike_check_req()) {
>> xlog("L_ALERT","ALERT: pike blocking R=$ru from $fu
>> (IP:$si:$sp)\n");
>> $sht(ipban=>$si) = 1;
>> exit;
>> }
>> }
>>
>> And we are not seeing any logs, therefore I'm discarding pike.
>>
>> - Firewall:
>>
>> I checked all 3 servers, and none of them have -local- firewall policies.
>>
>> - conntrack:
>>
>> All 3 servers have nf_conntrack loaded in kernel.
>>
>> - selinux/etc:
>>
>> The two good servers have "AppArmor" disabled.
>> The bad server has "AppArmor" enabled. !! <-- I'm
>> hoping this could be the cause and I'm going to test tonight without it.
>>
>>
>> Thanks for checking this, I was so lost I actually went ahead and did
>> "kamctl trap" last night too just in case. When I run it, it didn't stop by
>> itself (I stopped it with CTRL+C after some time) but it did generate a
>> file with a lot of output. Can I send you it privately? I'm not sure how to
>> interpret it.
>>
>> Before anything I'm going to test tonight:
>>
>> 1- disabling apparmor
>> 2- unloading nf_conntrack
>>
>> I'll report back with the resutls.
>>
>>
>>
>>
>>
>> On Wed, Sep 14, 2022 at 5:27 AM Daniel-Constantin Mierla <
>> mico...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> be sure you do not hit some limits set by Kamailio (e.g., pike) or the
>>> system/firewall (e.g., selinux, conntrack).
>>>
>>> You should install gdb and run "kamctl trap" when it stops processing
>>> and inspect the written file to see where each kamailio process is in the
>>> execution stack.
>>>
>>> Cheers,
>>> Daniel
>>> On 14.09.22 10:20, Joel Serrano wrote:
>>>
>>> Bumping this! Any comments? Or suggestions on what to check? I'm feeling
>>> it has to be something stupid but I can't see it :(
>>>
>>>
>>> On Sun, Sep 11, 2022 at 12:56 AM Joel Serrano  wrote:
>>>
 Hello,

 I have a cluster of 2 kamailios v5.5 on debian 9 working flawlessly.

 We have added a third node, also on v5.5 but debian 11. Kamailio
 doesn't work correctly for some reason that I'm not seeing.

 The symptoms are:

 1- Kamailio receives INVITEs and starts to process them per routing
 script.
 2- There is a specific place where something happens and the calls are
 dropped (Kamailio is not even replying to the source). Note that the config
 is exactly the same on all 3 servers, only one of the three is having the
 issue.

 I enabled debug logs and I could see:

 [...]
 Sep 10 12:30:48 sbc03 sbc[956340]: exec: {1 102 INVITE
 5111a7c71ec485e443099844491ef...@sip02.example.com} ***
 cfgtrace:dbg_cfg_trace(): request_route=[GET_OUTBOUND_API_DATA]
 c=[/etc/kamailio/sbc/api.cfg] l=61 a=5 n=route
 Sep 10 12:30:48 sbc03 sbc[956340]: exec: {1 102 INVITE
 5111a7c71ec485e443099844491ef...@sip02.example.com} ***
 cfgtrace:dbg_cfg_trace(): request_route=[APPLY_REWRITE_RULE]
 c=[/etc/kamailio/sbc/extras.cfg] l=211 a=26 n=xlog
 Sep 10 12:30:48 sbc03 sbc[956340]: INFO: {1 102 INVITE
 5111a7c71ec485e443099844491ef...@sip02.vozelia.com.pa}