Re: [SR-Users] qm_find_free() Free fragment not found, called from xcap_server, no more pkg

[Cody Herzog]

Excellent.

Thanks again, Henning.

From: Henning Westerholt [mailto:h...@skalatan.de]
Sent: Sunday, September 22, 2019 3:58 AM
To: Kamailio (SER) - Users Mailing List ; Cody 
Herzog 
Subject: Re: [SR-Users] qm_find_free() Free fragment not found, called from 
xcap_server, no more pkg


Hello Cody,

this is only used when memory is freed. The additional code that is executed is 
really small, so there will be no (observable) performance impact.

Cheers,

Henning
Am 21.09.19 um 03:59 schrieb Cody Herzog:
Thanks very much, Henning.

We will experiment with that option.

I wonder if there will be any performance impact with mem_join enabled.

We've still had no luck reproducing the issue, so we will keep trying different 
long-duration load tests to see if we can trigger it.

Thanks again.

From: Henning Westerholt [mailto:h...@skalatan.de]
Sent: Thursday, September 19, 2019 2:22 PM
To: Kamailio (SER) - Users Mailing List 
; Cody Herzog 

Subject: Re: [SR-Users] qm_find_free() Free fragment not found, called from 
xcap_server, no more pkg


Hello Cody,

you only need to enable the mem_join parameter, no compile time switch is 
necessary (anymore).

Cheers,

Henning
Am 19.09.19 um 23:15 schrieb Cody Herzog:
Thanks for the quick and useful response.

Increasing the package memory seems like a simple and safe thing to do. Also, 
we are indeed planning to upgrade soon.

Regarding 'mem_join', based on the documentation, I assume we will have to 
compile with MEM_JOIN_FREE in order for it to work. Is that correct?

We are still hoping to be able to reproduce this issue in our dev environment, 
so that we can prove our changes are helping. Can anyone think of a way we 
might be able to intentionally cause some memory fragmentation? Is there a 
particular type of access pattern, or a series of operations we can repeat many 
times to induce fragmentation?

We've been running heavy SIPp load tests for many days which exercise the 
following:

-Registration
-Subscription to large RLS contact lists
-Subscription to presence of many contacts

So far, we haven't been able to reproduce.

Thanks again.

From: Daniel-Constantin Mierla [mailto:mico...@gmail.com]
Sent: Wednesday, September 18, 2019 11:32 PM
To: Kamailio (SER) - Users Mailing List 
; Cody Herzog 

Subject: Re: [SR-Users] qm_find_free() Free fragment not found, called from 
xcap_server, no more pkg


Hello,

first, I would recommend to upgrade to the latest version in v5.1.x series, the 
5.1.0 was the first there and there were many issues fixed in the 5.1 branch 
that will ensure smoother run in long term.

The issue reported is likely related to fragmentation, try to set global 
parameter:

mem_join=1

As the instance needs to deal with large chunks for xcap documents, I would 
also suggest to increase a bit the pool of pkg memory via -M command line 
parameter -- I see that now is 8MB, make it 12 or 16.

Cheers,
Daniel
On 19.09.19 01:07, Cody Herzog wrote:
Hello.

Recently, for the first time, we experienced an apparent memory issue in our 
production environment. Here's an example of the relevant log messages:

Sep 13 18:55:22 SIPCOMM /usr/local/sbin/kamailio[2302]: ERROR:  
[core/mem/q_malloc.c:286]: qm_find_free(): qm_find_free(0x7fd8bbcfc010, 
134648); Free fragment not found!
Sep 13 18:55:22 SIPCOMM /usr/local/sbin/kamailio[2302]: ERROR:  
[core/mem/q_malloc.c:419]: qm_malloc(): qm_malloc(0x7fd8bbcfc010, 134648) 
called from xcap_server: xcap_server.c: ki_xcaps_put(549), module: xcap_server; 
Free fragment not found!
Sep 13 18:55:22 SIPCOMM /usr/local/sbin/kamailio[2302]: ERROR: xcap_server 
[xcap_server.c:552]: ki_xcaps_put(): no more pkg
Sep 13 18:55:22 SIPCOMM /usr/local/sbin/kamailio[2252]: ERROR: app_perl 
[kamailioxs.xs:487]: XS_Kamailio_log(): 500 Server error

The failed operation was the XCAP server module trying to generate a very large 
RLS contact list for a user. The issue only impacted users with very large 
lists, as though a large contiguous block of memory could not be found, whereas 
other smaller allocations continued to work fine. We believe the requested 
allocation was around 112 KB in size.
The server had been up for 14 days. We were able to work around the issue 
temporarily by just restarting the Kamailio service. It's unusual, because our 
production server is often up for months, and we've never seen this issue 
before. The load on production is increasing slowly due to an increased 
concurrent user count, so that might be related.

Before restarting the service on production, we captured the output of the 
following commands:

kamctl stats shmem
kamcmd mod.stats all shm
kamcmd pkg.stats
kamcmd mod.stats all pkg

Here's the shared mem output:

{
  "jsonrpc":  "2.0",
  "result": [
"shmem:fragments = 27240",
"shmem:free_size = 447203296",
"shmem:max_used_size = 116175576",

Re: [SR-Users] Using static entries with usrloc

[Daniel-Constantin Mierla]

On 20.09.19 16:52, Daniel Tryba wrote:
> On Fri, Sep 20, 2019 at 04:37:20PM +0300, George Diamantopoulos wrote:
>> Am I right in thinking that merely INSERTing respective rows in kamailio's
>> location DB backend will do the trick? Is there anything I need to worry
>> about, like contacts being periodically purged? If yes, how do I prevent
>> that from happening? And will kamailio always query the DB when performing
>> location lookup or do I need to restart upon inserting new records (to
>> purge caches, for example)?
> I did the same (as in clients can choose to setup their endpoints this
> way, but nobody is doing it at this moment).
>
> This depends on what dbmode you have set. db_mode 3 will cache nothing
> and do a DB lookup for all actions. This is also safe for what you are
> wanting to to (in my experience).
>
>> Lastly, I'm not sure how to treat some fields present in usrloc tables like
>> ruid, expires (can I set this to NULL?), callid and cseq (which is
>> irrelevant since there's no REGISTER), flags, cflags and methods.
> ruid: doesn't matter (I set it to username)


Just a note: the ruid should be unique per record, it is fine if you
have one registration per username. It might not have impact if you do
not use gruu extensions of some functions that need the ruid (e.g.,
unregister by ruid, location record attribtues), but it would be safer
to just set it to a unique random value.

Cheers,
Daniel

> expires: I have it set to just before the end of the current unix time,
>  2018-01-01 00:00:00
> callid: a unique random string (substr(md5(time().rand()),0,16))
> cseq: 1
> flags: 0 (?)
> cflag: 128 (?)
> methods: 6879 (?)
>
> But doing it through kamcmd might be the better answer that is already
> given.
>
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- 
https://asipto.com/u/kat


___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] xtrange error but those rules that already work, some bug?

[Daniel-Constantin Mierla]

Hello,

that is not a real error, just a message printed at ERROR level done by
debugger module when cfgtrace is enabled. See:

  -
https://www.kamailio.org/docs/modules/stable/modules/debugger.html#dbg.p.cfgtrace
 

As well as:

  -
https://www.kamailio.org/docs/modules/stable/modules/debugger.html#dbg.p.log_level

and the parameters that follows it.

Cheers,
Daniel

On 23.09.19 15:57, PICCORO McKAY Lenz wrote:
> i have many xtrange errors in the log.. but the code seems as i guess
> completed each rule without error.
>
> In the kamailio i have that error by example:
> |Sep 23 13:38:29 ser-server-02 kamailio[2847]: 4(2855) ERROR: ***
> cfgtrace:request_route=[REGISTRAR]
> c=[/etc/kamailio/kamailio-devel.cfg] l=692 a=25 n=save |
> that line represents a supposed error in the following code, BUT that
> code seem work.. kamailio it’s working perfectly.. so i’m worry about
> it, what it happened?
> |# Handle SIP registrations route[REGISTRAR] { if
> (!is_method("REGISTER")) return; if(isflagset(FLT_NATS)) {
> setbflag(FLB_NATB); #!ifdef WITH_NATSIPPING # uncomment next line to
> do SIP NAT pinging setbflag(FLB_NATSIPPING); #!endif } if
> (!save("location")) sl_reply_error(); #!ifdef WITH_ASTERISK
> route(REGFWD);  #!endif exit; } |
> ||
> why that happened?
>
> Lenz McKAY Gerardo (PICCORO)
> http://qgqlochekone.blogspot.com
>
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- 
https://asipto.com/u/kat

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

[SR-Users] xtrange error but those rules that already work, some bug?

[PICCORO McKAY Lenz]

i have many xtrange errors in the log.. but the code seems as i guess
completed each rule without error.

In the kamailio i have that error by example:

Sep 23 13:38:29 ser-server-02 kamailio[2847]: 4(2855) ERROR: ***
cfgtrace:request_route=[REGISTRAR]
c=[/etc/kamailio/kamailio-devel.cfg] l=692 a=25 n=save

that line represents a supposed error in the following code, BUT that code
seem work.. kamailio it’s working perfectly.. so i’m worry about it, what
it happened?

# Handle SIP registrations
route[REGISTRAR] {
if (!is_method("REGISTER")) return;

if(isflagset(FLT_NATS))
{
setbflag(FLB_NATB);
#!ifdef WITH_NATSIPPING
# uncomment next line to do SIP NAT pinging
setbflag(FLB_NATSIPPING);
#!endif
}
if (!save("location"))
sl_reply_error();
#!ifdef WITH_ASTERISK
route(REGFWD);
 #!endif

exit;
}

why that happened?

Lenz McKAY Gerardo (PICCORO)
http://qgqlochekone.blogspot.com
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Using static entries with usrloc

[George Diamantopoulos]

Thank you all for your input. I'll most likely use the RPC command to add
entries expiring at the end of linux time and see how it goes. Cheers!

On Fri, 20 Sep 2019 at 16:37, George Diamantopoulos 
wrote:

> Hello,
>
> Normally I wouldn't be asking this but I was surprised to find no
> reference to anyone doing this online. I need to have a few AoRs with
> static entries in kami's location table, which will always be reachable at
> their contacts without ever having to register with Kamailio.
>
> Am I right in thinking that merely INSERTing respective rows in kamailio's
> location DB backend will do the trick? Is there anything I need to worry
> about, like contacts being periodically purged? If yes, how do I prevent
> that from happening? And will kamailio always query the DB when performing
> location lookup or do I need to restart upon inserting new records (to
> purge caches, for example)?
>
> Lastly, I'm not sure how to treat some fields present in usrloc tables
> like ruid, expires (can I set this to NULL?), callid and cseq (which is
> irrelevant since there's no REGISTER), flags, cflags and methods.
>
> Thanks,
> George
>
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users