Re: [SR-Users] DMQ mem leak issues

[Joel Serrano]

Hi Charles,

I'm trying to move _away_ from database for data replication and pass over
to DMQ as much as possible, currently I have db_mode=1 for usrloc so that
is why I asked.

If I understand correctly, on a 2 node cluster, without database for usrloc
and only using DMQ, as long as both nodes don't go down at the same time
you are good... it would be possible to restart the nodes sequentially and
not lose registration info as they would replicate to each other on
startup, right?

Do you see a need for the database (specifically talking about usrloc
replication) other than persistency if *all* nodes in the dmq cluster are
down at the same time?

Thanks,
Joel.

On Tue, Sep 4, 2018 at 11:35 AM, Charles Chance <
charles.cha...@sipcentric.com> wrote:

> Hi Joel,
>
> I have not had much time recently to find the cause of the issue with
> db_mode 1, although I should have some more time over the next couple of
> weeks.
>
> Out of interest, what will you be using the database for?
>
> Cheers,
>
> Charles
>
>
> On Fri, 24 Aug 2018 at 04:00, Joel Serrano  wrote:
>
>> I’m planning to use DMQ + usrloc too! Thanks for sharing the solution!
>>
>> Do you know if It’s fixed also with db_mode=1?
>>
>>
>> On Thu, Aug 23, 2018 at 13:53 Julien Chavanton 
>> wrote:
>>
>>> Hi, I have glad you guys found a solution.
>>>
>>> Thanks for sharing it.
>>>
>>> Regards,
>>> Julien
>>>
>>> On Wed, Aug 22, 2018, 21:43 Rogelio Perez  wrote:
>>>
 Hi Julien,

 Thanks for checking on this.
 I've been working in the background with Charles on this issue and we
 think we've found a solution, although the cause isn't clear to me yet.
 Following Charles advice we changed the usrloc module parameter db_mode
 from 1 (Write-Through) to 2 (Write-Back) and there's been no more memory
 leaks incidents since then.
 I'll report back if we have any further updates.

 Best,
 Rogelio


 On Wed, Aug 22, 2018 at 11:39 AM Julien Chavanton 
 wrote:

> Hi Rogerio, did you have any luck digging this leak further ?
>
> On Wed, Aug 8, 2018 at 3:37 AM Charles Chance <
> charles.cha...@sipcentric.com> wrote:
>
>> Hi Rogelio,
>>
>> I have been running master on a three-node lab (one primary, two
>> secondary) for the past 24 hours or so, maintaining 2000 registrations on
>> the primary, replicating to both secondaries, and memory usage has 
>> remained
>> constant throughout.
>>
>> I will leave it running for another 24 hours to be sure but in the
>> meantime, you mentioned you are loading records from DB - which mode are
>> you using for writing (write-through or write-back)? Do you experience 
>> the
>> same symptoms if you disable the database completely on the secondary 
>> nodes
>> (or just one for testing) and instead, enable sync in dmq_usrloc?
>>
>> Cheers,
>>
>> Charles
>>
>>
>> On 7 August 2018 at 16:42, Julien Chavanton 
>> wrote:
>>
>>> I wonder if this could be introduced by a regression or if you are
>>> facing a specific edge case
>>>
>>> I briefly looked at the commits of DMQ and DMQ_USRLOC
>>> It seems there was significant work done.
>>> I would give a try with 5.0.0 and then we will at least learn that
>>> this is not a recent regression.
>>>
>>> On Mon, Aug 6, 2018 at 1:43 PM, Rogelio Perez 
>>> wrote:
>>>
 Charles, Julien, Daniel,

 The results are pretty much the same, the mem leak is still there
 and we need to restart Kamailio when it reaches certain threshold.
 https://www.dropbox.com/s/enxx6b7t0c8vl49/Selection_539.png?dl=0

 Is there anything else we can try?
 Will a core dump file tell us what's causing it?

 Thanks,
 Rogelio

 On Thu, Aug 2, 2018 at 2:57 PM Rogelio Perez 
 wrote:

> Thanks Charles, it's working now.
> I'm deploying to production and confirming results soon.
>
> Rogelio
>


 --
 
 Rogelio Perez | engineering | telnyx 
 chicago: +1 312 270 8119 | dublin: +353 1 912 6119


 ___
 Kamailio (SER) - Users Mailing List
 sr-users@lists.kamailio.org
 https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users


>>>
>>
>>
>>
>> Sipcentric Ltd. Company registered in England & Wales no. 7365592. 
>> Registered
>> office: Faraday Wharf, Innovation Birmingham Campus, Holt Street,
>> Birmingham Science Park, Birmingham B7 4BB.
>> ___
>> Kamailio (SER) - Users Mailing List
>> sr-users@lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>

Re: [SR-Users] DMQ mem leak issues

[Charles Chance]

Hi Joel,

I have not had much time recently to find the cause of the issue with
db_mode 1, although I should have some more time over the next couple of
weeks.

Out of interest, what will you be using the database for?

Cheers,

Charles


On Fri, 24 Aug 2018 at 04:00, Joel Serrano  wrote:

> I’m planning to use DMQ + usrloc too! Thanks for sharing the solution!
>
> Do you know if It’s fixed also with db_mode=1?
>
>
> On Thu, Aug 23, 2018 at 13:53 Julien Chavanton 
> wrote:
>
>> Hi, I have glad you guys found a solution.
>>
>> Thanks for sharing it.
>>
>> Regards,
>> Julien
>>
>> On Wed, Aug 22, 2018, 21:43 Rogelio Perez  wrote:
>>
>>> Hi Julien,
>>>
>>> Thanks for checking on this.
>>> I've been working in the background with Charles on this issue and we
>>> think we've found a solution, although the cause isn't clear to me yet.
>>> Following Charles advice we changed the usrloc module parameter db_mode
>>> from 1 (Write-Through) to 2 (Write-Back) and there's been no more memory
>>> leaks incidents since then.
>>> I'll report back if we have any further updates.
>>>
>>> Best,
>>> Rogelio
>>>
>>>
>>> On Wed, Aug 22, 2018 at 11:39 AM Julien Chavanton 
>>> wrote:
>>>
 Hi Rogerio, did you have any luck digging this leak further ?

 On Wed, Aug 8, 2018 at 3:37 AM Charles Chance <
 charles.cha...@sipcentric.com> wrote:

> Hi Rogelio,
>
> I have been running master on a three-node lab (one primary, two
> secondary) for the past 24 hours or so, maintaining 2000 registrations on
> the primary, replicating to both secondaries, and memory usage has 
> remained
> constant throughout.
>
> I will leave it running for another 24 hours to be sure but in the
> meantime, you mentioned you are loading records from DB - which mode are
> you using for writing (write-through or write-back)? Do you experience the
> same symptoms if you disable the database completely on the secondary 
> nodes
> (or just one for testing) and instead, enable sync in dmq_usrloc?
>
> Cheers,
>
> Charles
>
>
> On 7 August 2018 at 16:42, Julien Chavanton 
> wrote:
>
>> I wonder if this could be introduced by a regression or if you are
>> facing a specific edge case
>>
>> I briefly looked at the commits of DMQ and DMQ_USRLOC
>> It seems there was significant work done.
>> I would give a try with 5.0.0 and then we will at least learn that
>> this is not a recent regression.
>>
>> On Mon, Aug 6, 2018 at 1:43 PM, Rogelio Perez 
>> wrote:
>>
>>> Charles, Julien, Daniel,
>>>
>>> The results are pretty much the same, the mem leak is still there
>>> and we need to restart Kamailio when it reaches certain threshold.
>>> https://www.dropbox.com/s/enxx6b7t0c8vl49/Selection_539.png?dl=0
>>>
>>> Is there anything else we can try?
>>> Will a core dump file tell us what's causing it?
>>>
>>> Thanks,
>>> Rogelio
>>>
>>> On Thu, Aug 2, 2018 at 2:57 PM Rogelio Perez 
>>> wrote:
>>>
 Thanks Charles, it's working now.
 I'm deploying to production and confirming results soon.

 Rogelio

>>>
>>>
>>> --
>>> 
>>> Rogelio Perez | engineering | telnyx 
>>> chicago: +1 312 270 8119 | dublin: +353 1 912 6119
>>>
>>>
>>> ___
>>> Kamailio (SER) - Users Mailing List
>>> sr-users@lists.kamailio.org
>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>>
>>
>
>
>
> Sipcentric Ltd. Company registered in England & Wales no. 7365592. 
> Registered
> office: Faraday Wharf, Innovation Birmingham Campus, Holt Street,
> Birmingham Science Park, Birmingham B7 4BB.
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>

>>>
>>> --
>>> 
>>> Rogelio Perez | engineering | telnyx 
>>> chicago: +1 312 270 8119 | dublin: +353 1 912 6119
>>>
>>> ___
>> Kamailio (SER) - Users Mailing List
>> sr-users@lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>


-- 
*Charles Chance*
Managing Director

t. 0330 120 1200m. 07932 063 891

-- 
Sipcentric Ltd.
Company registered in England & Wales no. 
7365592. Registered
office: Faraday Wharf, Innovation 
Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
___
Kamailio (SER) - User

Re: [SR-Users] DMQ mem leak issues

[Joel Serrano]

I’m planning to use DMQ + usrloc too! Thanks for sharing the solution!

Do you know if It’s fixed also with db_mode=1?


On Thu, Aug 23, 2018 at 13:53 Julien Chavanton  wrote:

> Hi, I have glad you guys found a solution.
>
> Thanks for sharing it.
>
> Regards,
> Julien
>
> On Wed, Aug 22, 2018, 21:43 Rogelio Perez  wrote:
>
>> Hi Julien,
>>
>> Thanks for checking on this.
>> I've been working in the background with Charles on this issue and we
>> think we've found a solution, although the cause isn't clear to me yet.
>> Following Charles advice we changed the usrloc module parameter db_mode
>> from 1 (Write-Through) to 2 (Write-Back) and there's been no more memory
>> leaks incidents since then.
>> I'll report back if we have any further updates.
>>
>> Best,
>> Rogelio
>>
>>
>> On Wed, Aug 22, 2018 at 11:39 AM Julien Chavanton 
>> wrote:
>>
>>> Hi Rogerio, did you have any luck digging this leak further ?
>>>
>>> On Wed, Aug 8, 2018 at 3:37 AM Charles Chance <
>>> charles.cha...@sipcentric.com> wrote:
>>>
 Hi Rogelio,

 I have been running master on a three-node lab (one primary, two
 secondary) for the past 24 hours or so, maintaining 2000 registrations on
 the primary, replicating to both secondaries, and memory usage has remained
 constant throughout.

 I will leave it running for another 24 hours to be sure but in the
 meantime, you mentioned you are loading records from DB - which mode are
 you using for writing (write-through or write-back)? Do you experience the
 same symptoms if you disable the database completely on the secondary nodes
 (or just one for testing) and instead, enable sync in dmq_usrloc?

 Cheers,

 Charles


 On 7 August 2018 at 16:42, Julien Chavanton 
 wrote:

> I wonder if this could be introduced by a regression or if you are
> facing a specific edge case
>
> I briefly looked at the commits of DMQ and DMQ_USRLOC
> It seems there was significant work done.
> I would give a try with 5.0.0 and then we will at least learn that
> this is not a recent regression.
>
> On Mon, Aug 6, 2018 at 1:43 PM, Rogelio Perez 
> wrote:
>
>> Charles, Julien, Daniel,
>>
>> The results are pretty much the same, the mem leak is still there and
>> we need to restart Kamailio when it reaches certain threshold.
>> https://www.dropbox.com/s/enxx6b7t0c8vl49/Selection_539.png?dl=0
>>
>> Is there anything else we can try?
>> Will a core dump file tell us what's causing it?
>>
>> Thanks,
>> Rogelio
>>
>> On Thu, Aug 2, 2018 at 2:57 PM Rogelio Perez 
>> wrote:
>>
>>> Thanks Charles, it's working now.
>>> I'm deploying to production and confirming results soon.
>>>
>>> Rogelio
>>>
>>
>>
>> --
>> 
>> Rogelio Perez | engineering | telnyx 
>> chicago: +1 312 270 8119 | dublin: +353 1 912 6119
>>
>>
>> ___
>> Kamailio (SER) - Users Mailing List
>> sr-users@lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>>
>



 Sipcentric Ltd. Company registered in England & Wales no. 7365592. 
 Registered
 office: Faraday Wharf, Innovation Birmingham Campus, Holt Street,
 Birmingham Science Park, Birmingham B7 4BB.
 ___
 Kamailio (SER) - Users Mailing List
 sr-users@lists.kamailio.org
 https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

>>>
>>
>> --
>> 
>> Rogelio Perez | engineering | telnyx 
>> chicago: +1 312 270 8119 | dublin: +353 1 912 6119
>>
>> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Julien Chavanton]

Hi, I have glad you guys found a solution.

Thanks for sharing it.

Regards,
Julien

On Wed, Aug 22, 2018, 21:43 Rogelio Perez  wrote:

> Hi Julien,
>
> Thanks for checking on this.
> I've been working in the background with Charles on this issue and we
> think we've found a solution, although the cause isn't clear to me yet.
> Following Charles advice we changed the usrloc module parameter db_mode
> from 1 (Write-Through) to 2 (Write-Back) and there's been no more memory
> leaks incidents since then.
> I'll report back if we have any further updates.
>
> Best,
> Rogelio
>
>
> On Wed, Aug 22, 2018 at 11:39 AM Julien Chavanton 
> wrote:
>
>> Hi Rogerio, did you have any luck digging this leak further ?
>>
>> On Wed, Aug 8, 2018 at 3:37 AM Charles Chance <
>> charles.cha...@sipcentric.com> wrote:
>>
>>> Hi Rogelio,
>>>
>>> I have been running master on a three-node lab (one primary, two
>>> secondary) for the past 24 hours or so, maintaining 2000 registrations on
>>> the primary, replicating to both secondaries, and memory usage has remained
>>> constant throughout.
>>>
>>> I will leave it running for another 24 hours to be sure but in the
>>> meantime, you mentioned you are loading records from DB - which mode are
>>> you using for writing (write-through or write-back)? Do you experience the
>>> same symptoms if you disable the database completely on the secondary nodes
>>> (or just one for testing) and instead, enable sync in dmq_usrloc?
>>>
>>> Cheers,
>>>
>>> Charles
>>>
>>>
>>> On 7 August 2018 at 16:42, Julien Chavanton 
>>> wrote:
>>>
 I wonder if this could be introduced by a regression or if you are
 facing a specific edge case

 I briefly looked at the commits of DMQ and DMQ_USRLOC
 It seems there was significant work done.
 I would give a try with 5.0.0 and then we will at least learn that this
 is not a recent regression.

 On Mon, Aug 6, 2018 at 1:43 PM, Rogelio Perez 
 wrote:

> Charles, Julien, Daniel,
>
> The results are pretty much the same, the mem leak is still there and
> we need to restart Kamailio when it reaches certain threshold.
> https://www.dropbox.com/s/enxx6b7t0c8vl49/Selection_539.png?dl=0
>
> Is there anything else we can try?
> Will a core dump file tell us what's causing it?
>
> Thanks,
> Rogelio
>
> On Thu, Aug 2, 2018 at 2:57 PM Rogelio Perez 
> wrote:
>
>> Thanks Charles, it's working now.
>> I'm deploying to production and confirming results soon.
>>
>> Rogelio
>>
>
>
> --
> 
> Rogelio Perez | engineering | telnyx 
> chicago: +1 312 270 8119 | dublin: +353 1 912 6119
>
>
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
>

>>>
>>>
>>>
>>> Sipcentric Ltd. Company registered in England & Wales no. 7365592. 
>>> Registered
>>> office: Faraday Wharf, Innovation Birmingham Campus, Holt Street,
>>> Birmingham Science Park, Birmingham B7 4BB.
>>> ___
>>> Kamailio (SER) - Users Mailing List
>>> sr-users@lists.kamailio.org
>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>
>
> --
> 
> Rogelio Perez | engineering | telnyx 
> chicago: +1 312 270 8119 | dublin: +353 1 912 6119
>
>
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Rogelio Perez]

Hi Julien,

Thanks for checking on this.
I've been working in the background with Charles on this issue and we think
we've found a solution, although the cause isn't clear to me yet.
Following Charles advice we changed the usrloc module parameter db_mode
from 1 (Write-Through) to 2 (Write-Back) and there's been no more memory
leaks incidents since then.
I'll report back if we have any further updates.

Best,
Rogelio


On Wed, Aug 22, 2018 at 11:39 AM Julien Chavanton 
wrote:

> Hi Rogerio, did you have any luck digging this leak further ?
>
> On Wed, Aug 8, 2018 at 3:37 AM Charles Chance <
> charles.cha...@sipcentric.com> wrote:
>
>> Hi Rogelio,
>>
>> I have been running master on a three-node lab (one primary, two
>> secondary) for the past 24 hours or so, maintaining 2000 registrations on
>> the primary, replicating to both secondaries, and memory usage has remained
>> constant throughout.
>>
>> I will leave it running for another 24 hours to be sure but in the
>> meantime, you mentioned you are loading records from DB - which mode are
>> you using for writing (write-through or write-back)? Do you experience the
>> same symptoms if you disable the database completely on the secondary nodes
>> (or just one for testing) and instead, enable sync in dmq_usrloc?
>>
>> Cheers,
>>
>> Charles
>>
>>
>> On 7 August 2018 at 16:42, Julien Chavanton  wrote:
>>
>>> I wonder if this could be introduced by a regression or if you are
>>> facing a specific edge case
>>>
>>> I briefly looked at the commits of DMQ and DMQ_USRLOC
>>> It seems there was significant work done.
>>> I would give a try with 5.0.0 and then we will at least learn that this
>>> is not a recent regression.
>>>
>>> On Mon, Aug 6, 2018 at 1:43 PM, Rogelio Perez 
>>> wrote:
>>>
 Charles, Julien, Daniel,

 The results are pretty much the same, the mem leak is still there and
 we need to restart Kamailio when it reaches certain threshold.
 https://www.dropbox.com/s/enxx6b7t0c8vl49/Selection_539.png?dl=0

 Is there anything else we can try?
 Will a core dump file tell us what's causing it?

 Thanks,
 Rogelio

 On Thu, Aug 2, 2018 at 2:57 PM Rogelio Perez 
 wrote:

> Thanks Charles, it's working now.
> I'm deploying to production and confirming results soon.
>
> Rogelio
>


 --
 
 Rogelio Perez | engineering | telnyx 
 chicago: +1 312 270 8119 | dublin: +353 1 912 6119


 ___
 Kamailio (SER) - Users Mailing List
 sr-users@lists.kamailio.org
 https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users


>>>
>>
>>
>>
>> Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered
>> office: Faraday Wharf, Innovation Birmingham Campus, Holt Street,
>> Birmingham Science Park, Birmingham B7 4BB.
>> ___
>> Kamailio (SER) - Users Mailing List
>> sr-users@lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>

-- 

Rogelio Perez | engineering | telnyx 
chicago: +1 312 270 8119 | dublin: +353 1 912 6119
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Julien Chavanton]

Hi Rogerio, did you have any luck digging this leak further ?

On Wed, Aug 8, 2018 at 3:37 AM Charles Chance 
wrote:

> Hi Rogelio,
>
> I have been running master on a three-node lab (one primary, two
> secondary) for the past 24 hours or so, maintaining 2000 registrations on
> the primary, replicating to both secondaries, and memory usage has remained
> constant throughout.
>
> I will leave it running for another 24 hours to be sure but in the
> meantime, you mentioned you are loading records from DB - which mode are
> you using for writing (write-through or write-back)? Do you experience the
> same symptoms if you disable the database completely on the secondary nodes
> (or just one for testing) and instead, enable sync in dmq_usrloc?
>
> Cheers,
>
> Charles
>
>
> On 7 August 2018 at 16:42, Julien Chavanton  wrote:
>
>> I wonder if this could be introduced by a regression or if you are facing
>> a specific edge case
>>
>> I briefly looked at the commits of DMQ and DMQ_USRLOC
>> It seems there was significant work done.
>> I would give a try with 5.0.0 and then we will at least learn that this
>> is not a recent regression.
>>
>> On Mon, Aug 6, 2018 at 1:43 PM, Rogelio Perez  wrote:
>>
>>> Charles, Julien, Daniel,
>>>
>>> The results are pretty much the same, the mem leak is still there and we
>>> need to restart Kamailio when it reaches certain threshold.
>>> https://www.dropbox.com/s/enxx6b7t0c8vl49/Selection_539.png?dl=0
>>>
>>> Is there anything else we can try?
>>> Will a core dump file tell us what's causing it?
>>>
>>> Thanks,
>>> Rogelio
>>>
>>> On Thu, Aug 2, 2018 at 2:57 PM Rogelio Perez  wrote:
>>>
 Thanks Charles, it's working now.
 I'm deploying to production and confirming results soon.

 Rogelio

>>>
>>>
>>> --
>>> 
>>> Rogelio Perez | engineering | telnyx 
>>> chicago: +1 312 270 8119 | dublin: +353 1 912 6119
>>>
>>>
>>> ___
>>> Kamailio (SER) - Users Mailing List
>>> sr-users@lists.kamailio.org
>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>>
>>
>
>
>
> Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered
> office: Faraday Wharf, Innovation Birmingham Campus, Holt Street,
> Birmingham Science Park, Birmingham B7 4BB.
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Charles Chance]

Hi Rogelio,

I have been running master on a three-node lab (one primary, two secondary)
for the past 24 hours or so, maintaining 2000 registrations on the primary,
replicating to both secondaries, and memory usage has remained constant
throughout.

I will leave it running for another 24 hours to be sure but in the
meantime, you mentioned you are loading records from DB - which mode are
you using for writing (write-through or write-back)? Do you experience the
same symptoms if you disable the database completely on the secondary nodes
(or just one for testing) and instead, enable sync in dmq_usrloc?

Cheers,

Charles


On 7 August 2018 at 16:42, Julien Chavanton  wrote:

> I wonder if this could be introduced by a regression or if you are facing
> a specific edge case
>
> I briefly looked at the commits of DMQ and DMQ_USRLOC
> It seems there was significant work done.
> I would give a try with 5.0.0 and then we will at least learn that this is
> not a recent regression.
>
> On Mon, Aug 6, 2018 at 1:43 PM, Rogelio Perez  wrote:
>
>> Charles, Julien, Daniel,
>>
>> The results are pretty much the same, the mem leak is still there and we
>> need to restart Kamailio when it reaches certain threshold.
>> https://www.dropbox.com/s/enxx6b7t0c8vl49/Selection_539.png?dl=0
>>
>> Is there anything else we can try?
>> Will a core dump file tell us what's causing it?
>>
>> Thanks,
>> Rogelio
>>
>> On Thu, Aug 2, 2018 at 2:57 PM Rogelio Perez  wrote:
>>
>>> Thanks Charles, it's working now.
>>> I'm deploying to production and confirming results soon.
>>>
>>> Rogelio
>>>
>>
>>
>> --
>> 
>> Rogelio Perez | engineering | telnyx 
>> chicago: +1 312 270 8119 | dublin: +353 1 912 6119
>>
>>
>> ___
>> Kamailio (SER) - Users Mailing List
>> sr-users@lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>>
>

-- 
Sipcentric Ltd.
Company registered in England & Wales no. 
7365592. Registered
office: Faraday Wharf, Innovation 
Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Julien Chavanton]

I wonder if this could be introduced by a regression or if you are facing a
specific edge case

I briefly looked at the commits of DMQ and DMQ_USRLOC
It seems there was significant work done.
I would give a try with 5.0.0 and then we will at least learn that this is
not a recent regression.

On Mon, Aug 6, 2018 at 1:43 PM, Rogelio Perez  wrote:

> Charles, Julien, Daniel,
>
> The results are pretty much the same, the mem leak is still there and we
> need to restart Kamailio when it reaches certain threshold.
> https://www.dropbox.com/s/enxx6b7t0c8vl49/Selection_539.png?dl=0
>
> Is there anything else we can try?
> Will a core dump file tell us what's causing it?
>
> Thanks,
> Rogelio
>
> On Thu, Aug 2, 2018 at 2:57 PM Rogelio Perez  wrote:
>
>> Thanks Charles, it's working now.
>> I'm deploying to production and confirming results soon.
>>
>> Rogelio
>>
>
>
> --
> 
> Rogelio Perez | engineering | telnyx 
> chicago: +1 312 270 8119 | dublin: +353 1 912 6119
>
>
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
>
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Rogelio Perez]

Charles, Julien, Daniel,

The results are pretty much the same, the mem leak is still there and we
need to restart Kamailio when it reaches certain threshold.
https://www.dropbox.com/s/enxx6b7t0c8vl49/Selection_539.png?dl=0

Is there anything else we can try?
Will a core dump file tell us what's causing it?

Thanks,
Rogelio

On Thu, Aug 2, 2018 at 2:57 PM Rogelio Perez  wrote:

> Thanks Charles, it's working now.
> I'm deploying to production and confirming results soon.
>
> Rogelio
>


-- 

Rogelio Perez | engineering | telnyx 
chicago: +1 312 270 8119 | dublin: +353 1 912 6119
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Rogelio Perez]

Thanks Charles, it's working now.
I'm deploying to production and confirming results soon.

Rogelio
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Charles Chance]

Hi Rogelio,

No need to test with method/$rm - the is_method() regression is now fixed.

If you pull the latest master you should be able to test Daniel's patch
again.

Cheers,

Charles


On 2 August 2018 at 13:27, Charles Chance 
wrote:

> Hi Paolo,
>
> How strange...it really shouldn’t (and has never done so for us):
> https://www.kamailio.org/wiki/cookbooks/devel/core#method
>
> Either way, $rm is also fine.
>
> Rogelio - to clarify, my aim here is simply to establish if something
> other than is_method() works in your case.
>
> Best,
>
> Charles
>
>
> On Thu, 2 Aug 2018 at 13:07, Paolo Visintin - evosip.cloud
>  wrote:
>
>> Hi Charles,
>> method == "KDMQ" says "syntax error"
>>
>> we solved using if($rm == "KDMQ")
>>
>> Cheers
>>
>>
>>
>> *Paolo Visintin*
>> *CTO*
>> evosip.cloud
>> [image: Risultati immagini per evosip]
>>
>> 2018-08-02 9:43 GMT+02:00 Charles Chance :
>>
>>> Again out of interest, what happens if you change it to:
>>>
>>> if (method == “KDMQ”) {
>>> ...
>>>
>>> Cheers,
>>>
>>> Charles
>>>
>>>
>>> On Thu, 2 Aug 2018 at 04:45, Rogelio Perez  wrote:
>>>
 Charles, here you go:

 ### Routing Logic 

 # Main SIP request routing logic
 # - processing of any incoming SIP request starts with this route
 # - note: this is the same as route { ... }
 request_route {

   # per request initial checks
   route(REQINIT);

   #!ifdef ENABLE_KDMQ
   # Handle Kamailio DQM messages
   if (is_method("KDMQ")) {
 dmq_handle_message();
   }
   #!endif

>>> --
>>> *Charles Chance*
>>> Managing Director
>>>
>>> t. 0330 120 1200m. 07932 063 891
>>>
>>> Sipcentric Ltd. Company registered in England & Wales no. 7365592. 
>>> Registered
>>> office: Faraday Wharf, Innovation Birmingham Campus, Holt Street,
>>> Birmingham Science Park, Birmingham B7 4BB.
>>>
>>
>>> ___
>>> Kamailio (SER) - Users Mailing List
>>> sr-users@lists.kamailio.org
>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>> ___
>> Kamailio (SER) - Users Mailing List
>> sr-users@lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
> --
> *Charles Chance*
> Managing Director
>
> t. 0330 120 1200m. 07932 063 891
>

-- 
Sipcentric Ltd.
Company registered in England & Wales no. 
7365592. Registered
office: Faraday Wharf, Innovation 
Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Charles Chance]

Hi Paolo,

How strange...it really shouldn’t (and has never done so for us):
https://www.kamailio.org/wiki/cookbooks/devel/core#method

Either way, $rm is also fine.

Rogelio - to clarify, my aim here is simply to establish if something other
than is_method() works in your case.

Best,

Charles


On Thu, 2 Aug 2018 at 13:07, Paolo Visintin - evosip.cloud
 wrote:

> Hi Charles,
> method == "KDMQ" says "syntax error"
>
> we solved using if($rm == "KDMQ")
>
> Cheers
>
>
>
> *Paolo Visintin*
> *CTO*
> evosip.cloud
> [image: Risultati immagini per evosip]
>
> 2018-08-02 9:43 GMT+02:00 Charles Chance :
>
>> Again out of interest, what happens if you change it to:
>>
>> if (method == “KDMQ”) {
>> ...
>>
>> Cheers,
>>
>> Charles
>>
>>
>> On Thu, 2 Aug 2018 at 04:45, Rogelio Perez  wrote:
>>
>>> Charles, here you go:
>>>
>>> ### Routing Logic 
>>>
>>> # Main SIP request routing logic
>>> # - processing of any incoming SIP request starts with this route
>>> # - note: this is the same as route { ... }
>>> request_route {
>>>
>>>   # per request initial checks
>>>   route(REQINIT);
>>>
>>>   #!ifdef ENABLE_KDMQ
>>>   # Handle Kamailio DQM messages
>>>   if (is_method("KDMQ")) {
>>> dmq_handle_message();
>>>   }
>>>   #!endif
>>>
>> --
>> *Charles Chance*
>> Managing Director
>>
>> t. 0330 120 1200m. 07932 063 891
>>
>> Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered
>> office: Faraday Wharf, Innovation Birmingham Campus, Holt Street,
>> Birmingham Science Park, Birmingham B7 4BB.
>>
>
>> ___
>> Kamailio (SER) - Users Mailing List
>> sr-users@lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
-- 
*Charles Chance*
Managing Director

t. 0330 120 1200m. 07932 063 891

-- 
Sipcentric Ltd.
Company registered in England & Wales no. 
7365592. Registered
office: Faraday Wharf, Innovation 
Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Paolo Visintin - evosip.cloud]

Hi Charles,
method == "KDMQ" says "syntax error"

we solved using if($rm == "KDMQ")

Cheers



*Paolo Visintin*
*CTO*
evosip.cloud
[image: Risultati immagini per evosip]

2018-08-02 9:43 GMT+02:00 Charles Chance :

> Again out of interest, what happens if you change it to:
>
> if (method == “KDMQ”) {
> ...
>
> Cheers,
>
> Charles
>
>
> On Thu, 2 Aug 2018 at 04:45, Rogelio Perez  wrote:
>
>> Charles, here you go:
>>
>> ### Routing Logic 
>>
>> # Main SIP request routing logic
>> # - processing of any incoming SIP request starts with this route
>> # - note: this is the same as route { ... }
>> request_route {
>>
>>   # per request initial checks
>>   route(REQINIT);
>>
>>   #!ifdef ENABLE_KDMQ
>>   # Handle Kamailio DQM messages
>>   if (is_method("KDMQ")) {
>> dmq_handle_message();
>>   }
>>   #!endif
>>
> --
> *Charles Chance*
> Managing Director
>
> t. 0330 120 1200m. 07932 063 891
>
> Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered
> office: Faraday Wharf, Innovation Birmingham Campus, Holt Street,
> Birmingham Science Park, Birmingham B7 4BB.
>
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
>
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Charles Chance]

Again out of interest, what happens if you change it to:

if (method == “KDMQ”) {
...

Cheers,

Charles


On Thu, 2 Aug 2018 at 04:45, Rogelio Perez  wrote:

> Charles, here you go:
>
> ### Routing Logic 
>
> # Main SIP request routing logic
> # - processing of any incoming SIP request starts with this route
> # - note: this is the same as route { ... }
> request_route {
>
>   # per request initial checks
>   route(REQINIT);
>
>   #!ifdef ENABLE_KDMQ
>   # Handle Kamailio DQM messages
>   if (is_method("KDMQ")) {
> dmq_handle_message();
>   }
>   #!endif
>
-- 
*Charles Chance*
Managing Director

t. 0330 120 1200m. 07932 063 891

-- 
Sipcentric Ltd.
Company registered in England & Wales no. 
7365592. Registered
office: Faraday Wharf, Innovation 
Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Rogelio Perez]

Charles, here you go:

### Routing Logic 

# Main SIP request routing logic
# - processing of any incoming SIP request starts with this route
# - note: this is the same as route { ... }
request_route {

  # per request initial checks
  route(REQINIT);

  #!ifdef ENABLE_KDMQ
  # Handle Kamailio DQM messages
  if (is_method("KDMQ")) {
dmq_handle_message();
  }
  #!endif
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Charles Chance]

Out of interest, can you show me the beginning of your route block - just
the part where you call dmq_handle_message()?

Cheers,

Charles

On Wed, 1 Aug 2018 at 18:05, Rogelio Perez  wrote:

> Hello,
>
> We had to rollback the changes as dmq_usrloc notifications are not working
> at all with the latest master that includes the DMQ patch.
> On the receiving node we see the following errors:
>
> (394) INFO: 

Re: [SR-Users] DMQ mem leak issues

[Rogelio Perez]

Hello,

We had to rollback the changes as dmq_usrloc notifications are not working
at all with the latest master that includes the DMQ patch.
On the receiving node we see the following errors:

(394) INFO: 

Re: [SR-Users] DMQ mem leak issues

[Rogelio Perez]

Julien,

> Since it seem you are recovering the memory this does not seems like a
real "leak"
I forgot to mention that the recoveries are actual Kamailio manual restarts.

> One hypothesis :
> When you restart a node on the DMQ bus, it can trigger memory usage on
the other nodes since they will start to do a SYNC and send one DMQ message
/ contact
> It could be that one node in the DMQ bus is restarted and not answering
DMQ messages ?
The mem leak periods do not match the moment we restart any of the nodes.

> Few ideas :
> You could search you trace, maybe you will find the DMQ sync requests ...
We verified the traces and we found that at the moment of the mem leak
there was nothing unusual.

> You can also confirm significant increase in active transactions.
Same.

> Verify the state of the bus :
> kamcmd  dmq.list_nodes
The primary node state shows the affected secondary node as inactive.

> Verify the amount of contact on each node (confirm that the cluster is
healthy)
> kamctl stats | grep usrloc | grep contact
I'll run this check the next time we see the mem leak in action.

Daniel's patch is now in production, I'll confirm results soon.

Thanks,
Rogelio
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Julien Chavanton]

Since it seem you are recovering the memory this does not seems like a real
"leak"


One hypothesis :

When you restart a node on the DMQ bus, it can trigger memory usage on the
other nodes since they will start to do a SYNC and send one DMQ message /
contact
It could be that one node in the DMQ bus is restarted and not answering DMQ
messages ?


Few ideas :

You could search you trace, maybe you will find the DMQ sync requests ...

You can also confirm significant increase in active transactions.

Verify the state of the bus :
kamcmd  dmq.list_nodes

Verify the amount of contact on each node (confirm that the cluster is
healthy)
kamctl stats | grep usrloc | grep contact

On Tue, Jul 31, 2018 at 9:21 AM, Rogelio Perez  wrote:

> Thanks Daniel, Charles and Julien.
>
> I confirm we're not getting the error log "running job failed".
> The behavior is always the same, any of the two failover instances would
> run without issues for a day or two and then suddenly start consuming all
> available memory in the span of an hour or less.
> Please check these graphs with some examples for more details:
> https://www.dropbox.com/sh/tu0jxi1vlbq81m8/AABhfz9rDumdCu3l0ROH7Lkla?dl=0
>
> I'll try Daniel's patch and confirm results soon.
>
> Rogelio
>
>
>
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Rogelio Perez]

Thanks Daniel, Charles and Julien.

I confirm we're not getting the error log "running job failed".
The behavior is always the same, any of the two failover instances would
run without issues for a day or two and then suddenly start consuming all
available memory in the span of an hour or less.
Please check these graphs with some examples for more details:
https://www.dropbox.com/sh/tu0jxi1vlbq81m8/AABhfz9rDumdCu3l0ROH7Lkla?dl=0

I'll try Daniel's patch and confirm results soon.

Rogelio
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Julien Chavanton]

Nice finding ! however Rogelio is saying there was no errors in the logs,
looking at the patch he would have seen some.

LM_ERR("running job failed\n");

Hope I am wrong.

Rogerio is this a slow leak of you mean suddenly it is using 1.5G ?

There can be a lot of shm used by sip_msg_shm_clone if Kamailio starts to
re transmit.
2K seems very small.
Normally this kind of high memory usage can take place at startup when DMQ
userloc is getting a full sync from all other nodes.



On Tue, Jul 31, 2018 at 5:59 AM, Charles Chance <
charles.cha...@sipcentric.com> wrote:

> Hi Daniel,
>
> Nice spot! I had tried to reproduce locally, but had not considered the
> possibility that jobs may be failing somewhere in Rogelio’s setup.
>
> Most likely your patch will resolve it but I’m happy to take a look
> further if not.
>
> Cheers,
>
> Charles
>
> On Tue, 31 Jul 2018 at 13:05, Daniel-Constantin Mierla 
> wrote:
>
>> Hello,
>>
>> not using dmq much, but at a quick look in the code, I noticed that there
>> are some cases when the job fields were not released if the processing was
>> not completely done for various reasons.
>>
>> I pushed the commit a1f5fbe2c18246d4afefa44fd8a52612a5182a46, can you
>> try with it and see the results?
>>
>> Maybe Charles Chance can also do a bit of review here, being the one
>> doing most of the work lately for dmq.
>>
>> Cheers,
>> Daniel
>>
>> On Tue, Jul 31, 2018 at 6:58 AM, Rogelio Perez 
>> wrote:
>>
>>> Hello,
>>>
>>> We're running three instances of Kamailo v5.14 as registrars handling
>>> registrations from ~2000 SIP clients, with one instance being primary and
>>> the other two as backups.
>>>
>>> The three of them are using the dmq and dmq_usrloc modules to
>>> synchronize user locations, however after a couple of days of operation the
>>> two failover instances show memory leak behaviors, with mem usage assigned
>>> to the core taking all available resources.
>>>
>>> When this happens we've noticed that:
>>>  - The shared memory used by the function "sip_msg_shm_clone" spikes
>>> (from 1kb to 1.5GB).
>>>   - The shared memory used by the function "dmq:worker.c:job_queue_push"
>>> also increases, but not as much (from 1kb to 1MB)
>>>  - DMQ request are not being answered (with a 200 OK) by the affected
>>> instance during this memory leak, which make us think that DMQ module
>>> becomes unresponsive.
>>>
>>> A few more notes:
>>>  - The failover instances are doing nothing except receiving replicated
>>> contacts.
>>>  - The shared memory grows at the same rate on both instances, but the
>>> critical behavior never happens at the same time.
>>>  - We are allocating 1GB memory on startup to each instance.
>>>  - We store the location DB in a psql DB and we load it at startup.
>>>  - We didn't find any errors in syslog, even at debug level.
>>>
>>> Has anyone experienced a similar issue who can suggest a possible
>>> solution?
>>>
>>> Thanks,
>>> Rogelio Perez
>>> Telnyx
>>>
>>> ___
>>> Kamailio (SER) - Users Mailing List
>>> sr-users@lists.kamailio.org
>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>>
>>
>>
>> --
>> Daniel-Constantin Mierla - http://www.asipto.com
>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>
> --
> *Charles Chance*
> Managing Director
>
> t. 0330 120 1200m. 07932 063 891
>
> Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered
> office: Faraday Wharf, Innovation Birmingham Campus, Holt Street,
> Birmingham Science Park, Birmingham B7 4BB.
>
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
>
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Charles Chance]

Hi Daniel,

Nice spot! I had tried to reproduce locally, but had not considered the
possibility that jobs may be failing somewhere in Rogelio’s setup.

Most likely your patch will resolve it but I’m happy to take a look further
if not.

Cheers,

Charles

On Tue, 31 Jul 2018 at 13:05, Daniel-Constantin Mierla 
wrote:

> Hello,
>
> not using dmq much, but at a quick look in the code, I noticed that there
> are some cases when the job fields were not released if the processing was
> not completely done for various reasons.
>
> I pushed the commit a1f5fbe2c18246d4afefa44fd8a52612a5182a46, can you try
> with it and see the results?
>
> Maybe Charles Chance can also do a bit of review here, being the one doing
> most of the work lately for dmq.
>
> Cheers,
> Daniel
>
> On Tue, Jul 31, 2018 at 6:58 AM, Rogelio Perez  wrote:
>
>> Hello,
>>
>> We're running three instances of Kamailo v5.14 as registrars handling
>> registrations from ~2000 SIP clients, with one instance being primary and
>> the other two as backups.
>>
>> The three of them are using the dmq and dmq_usrloc modules to synchronize
>> user locations, however after a couple of days of operation the two
>> failover instances show memory leak behaviors, with mem usage assigned to
>> the core taking all available resources.
>>
>> When this happens we've noticed that:
>>  - The shared memory used by the function "sip_msg_shm_clone" spikes
>> (from 1kb to 1.5GB).
>>   - The shared memory used by the function "dmq:worker.c:job_queue_push"
>> also increases, but not as much (from 1kb to 1MB)
>>  - DMQ request are not being answered (with a 200 OK) by the affected
>> instance during this memory leak, which make us think that DMQ module
>> becomes unresponsive.
>>
>> A few more notes:
>>  - The failover instances are doing nothing except receiving replicated
>> contacts.
>>  - The shared memory grows at the same rate on both instances, but the
>> critical behavior never happens at the same time.
>>  - We are allocating 1GB memory on startup to each instance.
>>  - We store the location DB in a psql DB and we load it at startup.
>>  - We didn't find any errors in syslog, even at debug level.
>>
>> Has anyone experienced a similar issue who can suggest a possible
>> solution?
>>
>> Thanks,
>> Rogelio Perez
>> Telnyx
>>
>> ___
>> Kamailio (SER) - Users Mailing List
>> sr-users@lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>>
>
>
> --
> Daniel-Constantin Mierla - http://www.asipto.com
> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>
-- 
*Charles Chance*
Managing Director

t. 0330 120 1200m. 07932 063 891

-- 
Sipcentric Ltd.
Company registered in England & Wales no. 
7365592. Registered
office: Faraday Wharf, Innovation 
Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Daniel-Constantin Mierla]

Hello,

not using dmq much, but at a quick look in the code, I noticed that there
are some cases when the job fields were not released if the processing was
not completely done for various reasons.

I pushed the commit a1f5fbe2c18246d4afefa44fd8a52612a5182a46, can you try
with it and see the results?

Maybe Charles Chance can also do a bit of review here, being the one doing
most of the work lately for dmq.

Cheers,
Daniel

On Tue, Jul 31, 2018 at 6:58 AM, Rogelio Perez  wrote:

> Hello,
>
> We're running three instances of Kamailo v5.14 as registrars handling
> registrations from ~2000 SIP clients, with one instance being primary and
> the other two as backups.
>
> The three of them are using the dmq and dmq_usrloc modules to synchronize
> user locations, however after a couple of days of operation the two
> failover instances show memory leak behaviors, with mem usage assigned to
> the core taking all available resources.
>
> When this happens we've noticed that:
>  - The shared memory used by the function "sip_msg_shm_clone" spikes
> (from 1kb to 1.5GB).
>   - The shared memory used by the function "dmq:worker.c:job_queue_push"
> also increases, but not as much (from 1kb to 1MB)
>  - DMQ request are not being answered (with a 200 OK) by the affected
> instance during this memory leak, which make us think that DMQ module
> becomes unresponsive.
>
> A few more notes:
>  - The failover instances are doing nothing except receiving replicated
> contacts.
>  - The shared memory grows at the same rate on both instances, but the
> critical behavior never happens at the same time.
>  - We are allocating 1GB memory on startup to each instance.
>  - We store the location DB in a psql DB and we load it at startup.
>  - We didn't find any errors in syslog, even at debug level.
>
> Has anyone experienced a similar issue who can suggest a possible solution?
>
> Thanks,
> Rogelio Perez
> Telnyx
>
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
>


-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users