Re: [SR-Users] qm_find_free() Free fragment not found, called from xcap_server, no more pkg

[Cody Herzog]

Thanks very much, Henning.

We will experiment with that option.

I wonder if there will be any performance impact with mem_join enabled.

We've still had no luck reproducing the issue, so we will keep trying different 
long-duration load tests to see if we can trigger it.

Thanks again.

From: Henning Westerholt [mailto:h...@skalatan.de]
Sent: Thursday, September 19, 2019 2:22 PM
To: Kamailio (SER) - Users Mailing List ; Cody 
Herzog 
Subject: Re: [SR-Users] qm_find_free() Free fragment not found, called from 
xcap_server, no more pkg


Hello Cody,

you only need to enable the mem_join parameter, no compile time switch is 
necessary (anymore).

Cheers,

Henning
Am 19.09.19 um 23:15 schrieb Cody Herzog:
Thanks for the quick and useful response.

Increasing the package memory seems like a simple and safe thing to do. Also, 
we are indeed planning to upgrade soon.

Regarding 'mem_join', based on the documentation, I assume we will have to 
compile with MEM_JOIN_FREE in order for it to work. Is that correct?

We are still hoping to be able to reproduce this issue in our dev environment, 
so that we can prove our changes are helping. Can anyone think of a way we 
might be able to intentionally cause some memory fragmentation? Is there a 
particular type of access pattern, or a series of operations we can repeat many 
times to induce fragmentation?

We've been running heavy SIPp load tests for many days which exercise the 
following:

-Registration
-Subscription to large RLS contact lists
-Subscription to presence of many contacts

So far, we haven't been able to reproduce.

Thanks again.

From: Daniel-Constantin Mierla [mailto:mico...@gmail.com]
Sent: Wednesday, September 18, 2019 11:32 PM
To: Kamailio (SER) - Users Mailing List 
; Cody Herzog 

Subject: Re: [SR-Users] qm_find_free() Free fragment not found, called from 
xcap_server, no more pkg


Hello,

first, I would recommend to upgrade to the latest version in v5.1.x series, the 
5.1.0 was the first there and there were many issues fixed in the 5.1 branch 
that will ensure smoother run in long term.

The issue reported is likely related to fragmentation, try to set global 
parameter:

mem_join=1

As the instance needs to deal with large chunks for xcap documents, I would 
also suggest to increase a bit the pool of pkg memory via -M command line 
parameter -- I see that now is 8MB, make it 12 or 16.

Cheers,
Daniel
On 19.09.19 01:07, Cody Herzog wrote:
Hello.

Recently, for the first time, we experienced an apparent memory issue in our 
production environment. Here's an example of the relevant log messages:

Sep 13 18:55:22 SIPCOMM /usr/local/sbin/kamailio[2302]: ERROR:  
[core/mem/q_malloc.c:286]: qm_find_free(): qm_find_free(0x7fd8bbcfc010, 
134648); Free fragment not found!
Sep 13 18:55:22 SIPCOMM /usr/local/sbin/kamailio[2302]: ERROR:  
[core/mem/q_malloc.c:419]: qm_malloc(): qm_malloc(0x7fd8bbcfc010, 134648) 
called from xcap_server: xcap_server.c: ki_xcaps_put(549), module: xcap_server; 
Free fragment not found!
Sep 13 18:55:22 SIPCOMM /usr/local/sbin/kamailio[2302]: ERROR: xcap_server 
[xcap_server.c:552]: ki_xcaps_put(): no more pkg
Sep 13 18:55:22 SIPCOMM /usr/local/sbin/kamailio[2252]: ERROR: app_perl 
[kamailioxs.xs:487]: XS_Kamailio_log(): 500 Server error

The failed operation was the XCAP server module trying to generate a very large 
RLS contact list for a user. The issue only impacted users with very large 
lists, as though a large contiguous block of memory could not be found, whereas 
other smaller allocations continued to work fine. We believe the requested 
allocation was around 112 KB in size.
The server had been up for 14 days. We were able to work around the issue 
temporarily by just restarting the Kamailio service. It's unusual, because our 
production server is often up for months, and we've never seen this issue 
before. The load on production is increasing slowly due to an increased 
concurrent user count, so that might be related.

Before restarting the service on production, we captured the output of the 
following commands:

kamctl stats shmem
kamcmd mod.stats all shm
kamcmd pkg.stats
kamcmd mod.stats all pkg

Here's the shared mem output:

{
  "jsonrpc":  "2.0",
  "result": [
"shmem:fragments = 27240",
"shmem:free_size = 447203296",
"shmem:max_used_size = 116175576",
"shmem:real_used_size = 89667616",
"shmem:total_size = 536870912",
"shmem:used_size = 68824240"
  ],
  "id": 6934
}

Here's the pkg output for the particular PID which was throwing the errors:

{
entry: 34
pid: 2302
rank: 14
used: 2415864
free: 4949688
real_used: 3438920
total_size: 8388608
total_frags: 1951
}

We didn't see anything obvious in the stats output which explains the issue.

We've been trying to reproduce the issue in a dev 

Re: [SR-Users] TLS in-dialog set_forward_no_connect()and upstream TLS LCR gateway

[Anthony Joseph Messina]

Ok, Daniel.  You were correct in pointing out my improper placement of the 
set_forward_no_connect().

I was trying to find a way to insert the command in the most efficient place.  
I have been testing with various nat/no-nat situations and think the following 
will do what it should and avoid the additional calls to "if(is_request())" 
and "if(has_totag())":


route[NATMANAGE] {
#!ifdef WITH_NAT
if(is_request()) {
if(has_totag()) {
if(check_route_param("nat=yes")) {
setbflag(FLB_NATB);
}

##->INSERT HERE: no connect message in a dialog involving NAT traversal
if(isbflagset(FLB_NATB)) {
set_forward_no_connect();
}
}
}
if(!(isflagset(FLT_NATS) || isbflagset(FLB_NATB)))
return;

#   We are sending everything to route(RTPENGINE_MANAGE) separately
#   for more specific handling and bridging, not only based on NAT
#   route(RTPENGINE_MANAGE);

if(is_request()) {
if(!has_totag()) {
if(t_is_branch_route()) {
add_rr_param(";nat=yes");
}
}
}
if(is_reply()) {
if(isbflagset(FLB_NATB)) {
if(is_first_hop()) {
# Skip for GRUU
if(@contact.uri!="" && !is_gruu(@contact.uri))
set_contact_alias();
}
}
}

#   if(isbflagset(FLB_NATB)) {
#   if(is_request()) {
#   if(has_totag()) {
#   set_forward_no_connect();
#   }
#   }
#   }
#!endif
return;
}



On Wednesday, September 18, 2019 1:50:46 AM CDT Daniel-Constantin Mierla 
wrote:
> On 15.09.19 03:34, Anthony Joseph Messina wrote:
> > I'm going to keep testing against the issue I originally reported, and
> > probably wait until after 5.3 is released.  My issue may also be related
> > to a combination of TCPOPS keepalive not keeping the proper connection
> > open for
> > 
> > UAC -> Kamailio/LCR -> TLS Gateway
> > 
> > The connection that's kept open to the TLS Gateway is the original forward
> > of the INVITE
> > 
> > : -> :
> > 
> > The subsequent in-dialog connections (such as BYE from the UAC to the TLS
> > Gateway) don't use the original TLS connection so they are prevented from
> > re- connecting to the TLS Gateway.
> > 
> > Again, I have to do more testing to clear up the root issue on my end.
> > 
> > Also, for a more compact config, would the following achieve the same
> > thing...
> > 
> > 
> > route[NATMANAGE] {
> > #!ifdef WITH_NAT
> > 
> > if(is_request()) {
> > 
> > if(has_totag()) {
> > 
> > if(check_route_param("nat=yes")) {
> > 
> > setbflag(FLB_NATB);
> > 
> > ### Add the command here
> > 
> >  set_forward_no_connect();
> >  
> > }
> > 
> > }
> > 
> > }
> > 
> > ...
> 
> You have to differentiate between calls with one side behind nat and the
> other one on a pubic IP that is like a server/gateway and can accept new
> connections, even for requests within dialog.
> 
> My initial change to the default config file was done in the perspective
> that the respective configuration is routing between local users, where
> is not common for a user device to register, then close the connection,
> because it was using a ephemeral port anyhow.
> 
> Cheers,
> Daniel
> 
> > On Wednesday, September 11, 2019 1:21:34 AM CDT Daniel-Constantin Mierla
> > 
> > wrote:
> >> It no longer looks like an issue related to  set_reply_no_connect() or
> >> set_forward_no_connect() added by the commit you referenced. Those were
> >> added to prevent attempting to connect to devices behind the nat (in
> >> that case the device has to maintain the connection, otherwise nobody
> >> can connect back to it) as well as prevent someone in the wild sending a
> >> request then closing the connection, without waiting for the reply,
> >> which is typically routed back to via, commonly with an ephemeral port.
> >> 
> >> The follow up commit I did it in master recently is no longer setting
> >> the flag for requests within dialog, but I understand you have
> >> connection problems for requests within dialog, am I right?
> >> 
> >> Cheers,
> >> Daniel
> >> 
> >> On 10.09.19 01:08, Anthony Joseph Messina wrote:
> >>> I still ran into some trouble when one side was NAT'd.
> >>> 
> >>> Am I correct in thinking that it would be undesirable to maintain a
> >>> TCP/TLS
> >>> connection to an upstream 

Re: [SR-Users] Using static entries with usrloc

[Daniel Tryba]

On Fri, Sep 20, 2019 at 03:00:34PM +, Henning Westerholt wrote:
> > expires: I have it set to just before the end of the current unix time,
> >   2018-01-01 00:00:00
> 
> Hello,
> 
> you mean probably end of unix time at 03:14:07 UTC on 19 January 2038.

Doh, yes :)


___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Using static entries with usrloc

[Henning Westerholt]

Am 20.09.19 um 16:52 schrieb Daniel Tryba:
On Fri, Sep 20, 2019 at 04:37:20PM +0300, George Diamantopoulos wrote:
>> Am I right in thinking that merely INSERTing respective rows in kamailio's
>> location DB backend will do the trick? Is there anything I need to worry
>> about, like contacts being periodically purged? If yes, how do I prevent
>> that from happening? And will kamailio always query the DB when performing
>> location lookup or do I need to restart upon inserting new records (to
>> purge caches, for example)?
> I did the same (as in clients can choose to setup their endpoints this
> way, but nobody is doing it at this moment).
>
> This depends on what dbmode you have set. db_mode 3 will cache nothing
> and do a DB lookup for all actions. This is also safe for what you are
> wanting to to (in my experience).
>
>> Lastly, I'm not sure how to treat some fields present in usrloc tables like
>> ruid, expires (can I set this to NULL?), callid and cseq (which is
>> irrelevant since there's no REGISTER), flags, cflags and methods.
> ruid: doesn't matter (I set it to username)
> expires: I have it set to just before the end of the current unix time,
>   2018-01-01 00:00:00

Hello,

you mean probably end of unix time at 03:14:07 UTC on 19 January 2038.

Henning

> callid: a unique random string (substr(md5(time().rand()),0,16))
> cseq: 1
> flags: 0 (?)
> cflag: 128 (?)
> methods: 6879 (?)
>
> But doing it through kamcmd might be the better answer that is already
> given.

-- 
Kamailio Merchandising - https://skalatan.de/merchandising/
Kamailio services - https://skalatan.de/services
Henning Westerholt - https://skalatan.de/blog/

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Using static entries with usrloc

[Daniel Tryba]

On Fri, Sep 20, 2019 at 04:37:20PM +0300, George Diamantopoulos wrote:
> Am I right in thinking that merely INSERTing respective rows in kamailio's
> location DB backend will do the trick? Is there anything I need to worry
> about, like contacts being periodically purged? If yes, how do I prevent
> that from happening? And will kamailio always query the DB when performing
> location lookup or do I need to restart upon inserting new records (to
> purge caches, for example)?

I did the same (as in clients can choose to setup their endpoints this
way, but nobody is doing it at this moment).

This depends on what dbmode you have set. db_mode 3 will cache nothing
and do a DB lookup for all actions. This is also safe for what you are
wanting to to (in my experience).

> Lastly, I'm not sure how to treat some fields present in usrloc tables like
> ruid, expires (can I set this to NULL?), callid and cseq (which is
> irrelevant since there's no REGISTER), flags, cflags and methods.

ruid: doesn't matter (I set it to username)
expires: I have it set to just before the end of the current unix time,
 2018-01-01 00:00:00
callid: a unique random string (substr(md5(time().rand()),0,16))
cseq: 1
flags: 0 (?)
cflag: 128 (?)
methods: 6879 (?)

But doing it through kamcmd might be the better answer that is already
given.

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Using static entries with usrloc

[Henning Westerholt]

Hello George,

if you have a static mapping from a AoR to an IP address you could also just 
route this in the cfg or with another database table (e.g. with a sqlops 
query). More or less the whole point of the usrloc/registrar modules is to have 
people register to your server and then maintain this mapping.

About the expiration - the easist would be to just enter a value that is in the 
future. This is e.g. the default value of the location table: 
DEFAULT_LOCATION_EXPIRES "2030-05-28 21:32:15"

Cheers,

Henning

Am 20.09.19 um 15:37 schrieb George Diamantopoulos:
Hello,

Normally I wouldn't be asking this but I was surprised to find no reference to 
anyone doing this online. I need to have a few AoRs with static entries in 
kami's location table, which will always be reachable at their contacts without 
ever having to register with Kamailio.

Am I right in thinking that merely INSERTing respective rows in kamailio's 
location DB backend will do the trick? Is there anything I need to worry about, 
like contacts being periodically purged? If yes, how do I prevent that from 
happening? And will kamailio always query the DB when performing location 
lookup or do I need to restart upon inserting new records (to purge caches, for 
example)?

Lastly, I'm not sure how to treat some fields present in usrloc tables like 
ruid, expires (can I set this to NULL?), callid and cseq (which is irrelevant 
since there's no REGISTER), flags, cflags and methods.

Thanks,
George



___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users


--
Kamailio Merchandising - https://skalatan.de/merchandising/
Kamailio services - https://skalatan.de/services
Henning Westerholt - https://skalatan.de/blog/
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Using static entries with usrloc

[David Villasmil]

Hello George,

You’re better off inserting a permanently, do it like:

kamctl rpc ul.add location usern...@localdomain.com
sip:USERNAME@IP_PERMANENT:5060 0 1 . 0 0 13279 . .

https://kamailio.org/docs/modules/stable/modules/usrloc.html#usrloc.r.add

4.6.  ul.add

Adds a new contact for an user AOR.

Parameters:

   -

   *table name* - table where the contact will be added (Ex: location).
   -

   *AOR* - user AOR in username[@domain] format (domain must be supplied
   only if use_domain option is on).
   -

   *contact* - contact string to be added
   -

   *expires* - expires value of the contact
   -

   *Q* - Q value of the contact
   -

   *path* value with the Path vector (use '0' or '.' if it should not be
   set)
   -

   *flags* - internal USRLOC flags of the contact
   -

   *cflags* - per branch flags of the contact
   -

   *methods* - mask with supported requests of the contact
   -

   *received* (optional) value with the received-from address (source
   address) (use '0' or '.' if it should not be set). Format:
   sip:srcip:srcport;transport=abc
   -

   *socket* (optional) value with the local socket address (use '0' or '.'
   if it should not be set). Format: proto:localip:localport

Note: the position of parameters is relevant, in the case of optional
parameters, use '0' or '.' for parameters that should not be set which are
positioned before any parameter that has to be set.

Hope that helps,

David


On Fri, 20 Sep 2019 at 14:40, George Diamantopoulos 
wrote:

> Hello,
>
> Normally I wouldn't be asking this but I was surprised to find no
> reference to anyone doing this online. I need to have a few AoRs with
> static entries in kami's location table, which will always be reachable at
> their contacts without ever having to register with Kamailio.
>
> Am I right in thinking that merely INSERTing respective rows in kamailio's
> location DB backend will do the trick? Is there anything I need to worry
> about, like contacts being periodically purged? If yes, how do I prevent
> that from happening? And will kamailio always query the DB when performing
> location lookup or do I need to restart upon inserting new records (to
> purge caches, for example)?
>
> Lastly, I'm not sure how to treat some fields present in usrloc tables
> like ruid, expires (can I set this to NULL?), callid and cseq (which is
> irrelevant since there's no REGISTER), flags, cflags and methods.
>
> Thanks,
> George
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

[SR-Users] Using static entries with usrloc

[George Diamantopoulos]

Hello,

Normally I wouldn't be asking this but I was surprised to find no reference
to anyone doing this online. I need to have a few AoRs with static entries
in kami's location table, which will always be reachable at their contacts
without ever having to register with Kamailio.

Am I right in thinking that merely INSERTing respective rows in kamailio's
location DB backend will do the trick? Is there anything I need to worry
about, like contacts being periodically purged? If yes, how do I prevent
that from happening? And will kamailio always query the DB when performing
location lookup or do I need to restart upon inserting new records (to
purge caches, for example)?

Lastly, I'm not sure how to treat some fields present in usrloc tables like
ruid, expires (can I set this to NULL?), callid and cseq (which is
irrelevant since there's no REGISTER), flags, cflags and methods.

Thanks,
George
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users