Check rtcp mux demux. On Wed, 4 Dec 2019, 20:22 Callum Guy, <callum....@x-on.co.uk> wrote:
> Thanks Johan, I've been through the rtpengine docs and source and can't > see any way to disable the RTCP call stat reports. Having said that I'm no > longer confident that RTPEngine is the problem - I'm losing memory > constantly at a rate of about 130MB/hour at all times, even overnight when > we are not seeing many calls. This seems to rule out the RTP reports or > anything call load related. > > I now wonder if the issue is related to registrations and NAT pings (~3500 > registrations). This is the only constant activity on the system overnight > (other than OpenSIPs clustering?). NAT pings are enabled for all contacts > and many of the handsets are also pinging the server, according to client > config. > > At this time I am not sure where to focus my attention - I've collected > some data from my server (version, relevant module configs, memory dump) in > the hope that someone might be able to spot a problem with my setup - you > can find this at the following link: > > > https://gist.githubusercontent.com/spacetourist/2b880c71eda54bcabf691282d6389209/raw/25c42b90ffa4e592cd761237dffca6f85aabbf4f/gistfile1.txt > > Please let me know if anything else would be useful. All ideas are very > much appreciated, thanks. > > Callum > > > On Tue, 3 Dec 2019 at 17:31, Johan De Clercq <jo...@democon.be> wrote: > >> I think you can. Check the documentation of rtpengine on github. And if >> you can, please use the latest commit. >> >> On Tue, 3 Dec 2019, 18:02 Callum Guy, <callum....@x-on.co.uk> wrote: >> >>> Hi All, >>> >>> I'm working through this memory issue and have some additional data. The >>> server crashed this afternoon due to memory exhaustion on a UDP listener >>> process, as far as I know at this time. Note that the processes all have >>> 4GB assigned so this is a gradual and constant growth issue. >>> >>> At this point I have my suspicions that it could be related to the >>> RTPEngine module - the first memory allocation error messages are all >>> related to what I believe the be RTCP session reports coming in from the >>> RTP engine servers. These came through about every second for an hour >>> before OpenSIPs finally gave up the ghost and restarted. >>> >>> Is this normal behaviour? We have only recently moved to RTPEngine from >>> RTPProxy so I am new to this software. Does anyone know if it is possible >>> to prevent RTPEngine from sending this data to OpenSIPs - its not something >>> that we require and I'd like to check if this is at all related to the >>> memory growth. Any other ideas would also be appreciated! >>> >>> Many thanks, >>> >>> Callum >>> >>> 2019-12-03T11:48:14.653225+00:00 TH-P-SIPREG-1 opensips[2521]: >>> ERROR:core:fm_malloc: not enough free pkg memory (214628248 bytes left, >>> need 536), please increase the "-M" command line parameter! >>> 2019-12-03T11:48:14.653851+00:00 TH-P-SIPREG-1 opensips[2521]: >>> INFO:core:fm_malloc: attempting defragmentation... >>> 2019-12-03T11:48:15.651495+00:00 TH-P-SIPREG-1 opensips[2521]: >>> INFO:core:fm_malloc: unable to alloc a big enough fragment! >>> 2019-12-03T11:48:15.652218+00:00 TH-P-SIPREG-1 opensips[2521]: >>> ERROR:rtpengine:rtpe_function_call: failed to decode bencoded reply from >>> proxy: d7:createdi1575373650e10:created_usi505483e11:last >>> signali1575373670e4:SSRCd10:3831331386d11:average >>> MOSd3:MOSi44e15:round-trip timei6754e6:jitteri0e11:packet >>> lossi0e7:samplesi9ee10:lowest MOSd3:MOSi44e15:round-trip >>> timei6175e6:jitteri0e11:packet lossi0e11:reported ati1575373656ee11:highest >>> MOSd3:MOSi44e15:round-trip timei6175e6:jitteri0e11:packet >>> lossi0e11:reported ati1575373656ee15:MOS >>> progressiond8:intervali3e7:entriesld3:MOSi44e15:round-trip >>> timei6175e6:jitteri0e11:packet lossi0e11:reported >>> ati1575373656eed3:MOSi44e15:round-trip timei6307e6:jitteri0e11:packet >>> lossi0e11:reported ati1575373660eed3:MOSi44e15:round-trip >>> timei6884e6:jitteri0e11:packet lossi0e11:reported >>> ati1575373664eed3:MOSi44e15:round-trip timei7016e6:jitteri0e11:packet >>> lossi0e11:reported ati1575373668eed3:MOSi44e15:round-trip >>> timei6368e6:jitteri0e11:packet lossi0e11:reported >>> ati1575373674eed3:MOSi44e15:round-trip timei6929e6:jitteri0e11:packet >>> lossi0e11:reported ati1575373679eed3:MOSi44e15:round-trip >>> timei6884e6:jitteri0e11:packet lossi0e11:reported >>> ati1575373684eed3:MOSi44e15:round-trip timei7241e6:jitteri0e11:packet >>> lossi0e11:reported ati1575373689eed3:MOSi44e15:round-trip >>> timei6988e6:jitteri0e11:packet lossi0e11:reported >>> ati1575373694eeeee10:1028099660d11:average MOSd3:MOSi43e15:round-trip >>> timei29936e6:jitteri0e11:packet lossi0e7:samplesi7ee10:lowest >>> MOSd3:MOSi43e15:round-trip timei29515e6:jitteri0e11:packet >>> lossi0e11:reported ati1575373661ee11:highest MOSd3:MOSi43e15:round-trip >>> timei29515e6:jitteri0e11:packet lossi0e11:reported ati1575373661ee15:MOS >>> progressiond8:intervali3e7:entriesld3:MOSi43e15:round-trip >>> timei29515e6:jitteri0e11:packet lossi0e11:reported >>> ati1575373661eed3:MOSi43e15:round-trip timei29817e6:jitteri0e11:packet >>> lossi0e11:reported ati1575373666eed3:MOSi43e15:round-trip >>> timei30695e6:jitteri0e11:packet lossi0e11:reported >>> ati1575373671eed3:MOSi43e15:round-trip timei29541e6:jitteri0e11:packet >>> lossi0e11:reported ati1575373676eed3:MOSi43e15:round-trip >>> timei30073e6:jitteri0e11:packet lossi0e11:reported >>> ati1575373681eed3:MOSi43e15:round-trip timei29526e6:jitteri0e11:packet >>> lossi0e11:reported ati1575373686eed3:MOSi43e15:round-trip >>> timei30391e6:jitteri0e11:packet lossi0e11:reported >>> ati1575373691eeeeee4:tagsd10:3044367218d3:tag10:30443672187:createdi1575373650e16:in >>> dialogue >>> with13:N8Nr5mDpcNt1a6:mediasld5:indexi1e4:type5:audio8:protocol7:RTP/AVP7:streamsld10:local >>> porti32494e8:endpointd6:family4:IPv47:address13:51.51.51.514:porti47360ee19:advertised >>> endpointd6:family4:IPv47:address9:10.0.0.154:porti12672ee11:last >>> packeti1575373694e5:flagsl3:RTP6:filled9:confirmed10:kernelizede4:SSRCi1028099660e5:statsd7:packetsi2045e5:bytesi351740e6:errorsi0eeed10:local >>> porti32495e8:endpointd6:family4:IPv47:address13:51.51.51.514:porti47361ee19:advertised >>> endpointd6:family4:IPv47:address9:10.0.0.154:porti12673ee11:last >>> packeti1575373691e5:flagsl4:RTCP6:filled9:confirmed10:kernelized17:no >>> kernel >>> supporte4:SSRCi1028099660e5:statsd7:packetsi8e5:bytesi1240e6:errorsi0eeee5:flagsl11:initialized4:send4:recveeee13:N8Nr5mDpcNt1ad3:tag13:N8Nr5mDpcNt1a7:createdi1575373650e16:in >>> dialogue >>> with10:30443672186:mediasld5:indexi1e4:type5:audio8:protocol7:RTP/AVP7:streamsld10:local >>> porti34226e8:endpointd6:family4:IPv47:address15:192.168.163.2224:porti11484ee19:advertised >>> endpointd6:family4:IPv47:address15:192.168.163.2224:porti11484ee11:last >>> packeti1575373694e5:flagsl3:RTP6:filled9:confirmed10:kernelizede4:SSRCi3831331386e5:statsd7:packetsi2018e5:bytesi347096e6:errorsi0eeed10:local >>> porti34227e8:endpointd6:family4:IPv47:address15:192.168.163.2224:porti11485ee19:advertised >>> endpointd6:family4:IPv47:address15:192.168.163.2224:porti11485ee11:last >>> packeti1575373694e5:flagsl4:RTCP6:filled9:confirmed10:kernelized17:no >>> kernel >>> supporte4:SSRCi3831331386e5:statsd7:packetsi9e5:bytesi1008e6:errorsi0eeee5:flagsl11:initialized4:send4:recv15:ICE >>> controllingeeeee6:totalsd3:RTPd7:packetsi4063e5:bytesi698836e6:errorsi0ee4:RTCPd7:packetsi17e5:bytesi2248e6:errorsi0eee6:result2:oke >>> >>> >>> On Sat, 30 Nov 2019 at 22:51, Callum Guy <callum....@x-on.co.uk> wrote: >>> >>>> Hi Ben, >>>> >>>> Thank you for your reply and insight here, very helpful to know you're >>>> running a drastically different setting for the package memory. >>>> >>>> I presumed if it preallocated that I would have seen some issues during >>>> testing, hence I've ended up with figures that were intended to provide 75% >>>> of the system memory to the application. >>>> >>>> Memory usage had been creeping up all day at the time of writing >>>> however migrations to this platform had been on hold since the initial >>>> capture of memory usage although call traffic would have been relatively >>>> even during daytime hours where the increase continued. On that basis I'm >>>> still concerned that there is an issue with my config causing this growth >>>> however I've now increased available memory and restarted so I should have >>>> ample time to investigate this week, I'll report back any findings for the >>>> community benefit. I will give some serious thought to lowering the package >>>> allocation value once I've got to grips with the situation. >>>> >>>> Usefully this implementation shares a lot of common components to >>>> another variant which acts as a pure proxy and does not deal with >>>> registrations where I'm not seeing this issue so that will narrow down the >>>> search area somewhat. >>>> >>>> Thanks again for your time, >>>> >>>> Callum >>>> >>>> On Sat, 30 Nov 2019, 15:46 Ben Newlin, <ben.new...@genesys.com> wrote: >>>> >>>>> Callum, >>>>> >>>>> >>>>> >>>>> It’s my understanding that OpenSIPS does not release memory back to >>>>> the OS, but it also pre-allocates all memory at startup into its private >>>>> pool and then allocates from that internally. Normally shared memory >>>>> should >>>>> be significantly higher than package memory. For reference, on our system >>>>> we run with “-m 1024 -M 64” and that is sufficient for us to process very >>>>> high traffic volume. We don’t do registration though, so that may affect >>>>> the sizes you need. >>>>> >>>>> >>>>> >>>>> You are setting your package memory size to 4G, so that will allocate >>>>> 4G memory for every package (process) that loads and then 2G for shared >>>>> memory. That will use up all the memory on your machine extremely quickly >>>>> for sure. The statistics you provided seem like the memory increase is >>>>> consistent with higher traffic levels on the second reading. You can see >>>>> in >>>>> your case that all of your “pkmem” processes have an extremely high amount >>>>> of free memory (~3GB!). But that memory is still allocated from the OS, so >>>>> you are instructing OpenSIPS to allocate much more than your system memory >>>>> right at startup. >>>>> >>>>> >>>>> >>>>> Your shared memory also has just under 2GB free, so you have a lot of >>>>> headroom there too. Since OpenSIPS pre-allocates, the amount of memory >>>>> being used by the system overall should be fairly steady; if it is >>>>> continuously increasing that implies a leak somewhere. IIRC there are a >>>>> few >>>>> processes/modules/commands in OpenSIPS or libraries it uses that do >>>>> allocate memory directly from the system and not from OpenSIPS’ pool. You >>>>> may need to investigate some of those to find out where your memory is >>>>> going, or look at other processes/daemons you have running that could be >>>>> using that memory. >>>>> >>>>> >>>>> >>>>> Ben Newlin >>>>> >>>>> >>>>> >>>>> *From: *Users <users-boun...@lists.opensips.org> on behalf of Callum >>>>> Guy <callum....@x-on.co.uk> >>>>> *Reply-To: *OpenSIPS users mailling list <users@lists.opensips.org> >>>>> *Date: *Friday, November 29, 2019 at 10:57 AM >>>>> *To: *OpenSIPS users mailling list <users@lists.opensips.org> >>>>> *Subject: *[OpenSIPS-Users] Memory Leak - runtime flags? >>>>> >>>>> >>>>> >>>>> Hi All, >>>>> >>>>> >>>>> >>>>> I have recently deployed a new registrar and have been seeing a >>>>> gradual increase in the memory footprint - enough that I'm having to >>>>> expand >>>>> the RAM (its virtualised) to ensure it doesn't run out. >>>>> >>>>> >>>>> >>>>> You can see a diff of the statistics collected last night at 11pm and >>>>> today at 3pm here: >>>>> https://gist.github.com/spacetourist/2103503674e134bd598c7f1e3a82674c/revisions >>>>> >>>>> >>>>> >>>>> Processes 5-9 are my UDP SIP receiver threads (autoscaled down from an >>>>> initial footprint of 20 threads). >>>>> >>>>> >>>>> >>>>> Using 3.0.1 on CentOS 7 8GB RAM (soon to be 32GB!). Currently OpenSIPs >>>>> is using all the RAM (minus OS usage) and 2GB of swap. Trying to use >>>>> dialog >>>>> and dr clustering if that is significant. Alos have NAT pings configured >>>>> for all registrations (4000 at time of writing). >>>>> >>>>> >>>>> >>>>> I am using runtime configuration flags of "*-m 2048 -M 4096*" and am >>>>> concerned that these were (way) too high, I think I've misinterpreted >>>>> their >>>>> meaning during initial setup. Is this a ridiculous setting for my >>>>> environment? Is it just as simple as OpenSIPs being greedy with the memory >>>>> such that it doesn't bother to free anything while each process free space >>>>> remaining? Should my -M value * max number of processes fit into my RAM? I >>>>> guess with an 8GB system that would mean dropping this to "-M 256"? >>>>> >>>>> >>>>> >>>>> I've done some research into the issue however I haven't found >>>>> anything else that would be an obvious target so wondered if the community >>>>> might have some ideas of where I can begin investigations. >>>>> >>>>> >>>>> >>>>> Many thanks, >>>>> >>>>> >>>>> >>>>> Callum >>>>> >>>>> >>>>> >>>>> [image: Image removed by sender.] >>>>> >>>>> >>>>> >>>>> *0333 332 0000 | www.x-on.co.uk <http://www.x-on.co.uk> | * *[image: >>>>> Image removed by sender.] <https://www.linkedin.com/company/x-on> [image: >>>>> Image removed by sender.] <https://www.facebook.com/XonTel> [image: Image >>>>> removed by sender.] <https://twitter.com/xonuk>* >>>>> >>>>> X-on is a trading name of Storacall Technology Ltd a limited company >>>>> registered in England and Wales. >>>>> Registered Office : Avaland House, 110 London Road, Apsley, Hemel >>>>> Hempstead, Herts, HP3 9SD. Company Registration No. 2578478. >>>>> The information in this e-mail is confidential and for use by the >>>>> addressee(s) only. If you are not the intended recipient, please notify >>>>> X-on immediately on +44(0)333 332 0000 and delete the >>>>> message from your computer. If you are not a named addressee you must >>>>> not use, disclose, disseminate, distribute, copy, print or reply to this >>>>> email. Views or opinions expressed by an individual >>>>> within this email may not necessarily reflect the views of X-on or its >>>>> associated companies. Although X-on routinely screens for viruses, >>>>> addressees should scan this email and any attachments >>>>> for viruses. X-on makes no representation or warranty as to the >>>>> absence of viruses in this email or any attachments. >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users@lists.opensips.org >>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users >>>>> >>>> >>> >>> *0333 332 0000 | www.x-on.co.uk <http://www.x-on.co.uk> | ** >>> <https://www.linkedin.com/company/x-on> <https://www.facebook.com/XonTel> >>> <https://twitter.com/xonuk> * >>> >>> X-on is a trading name of Storacall Technology Ltd a limited company >>> registered in England and Wales. >>> Registered Office : Avaland House, 110 London Road, Apsley, Hemel >>> Hempstead, Herts, HP3 9SD. Company Registration No. 2578478. >>> The information in this e-mail is confidential and for use by the >>> addressee(s) only. If you are not the intended recipient, please notify >>> X-on immediately on +44(0)333 332 0000 and delete the >>> message from your computer. If you are not a named addressee you must >>> not use, disclose, disseminate, distribute, copy, print or reply to this >>> email. Views or opinions expressed by an individual >>> within this email may not necessarily reflect the views of X-on or its >>> associated companies. Although X-on routinely screens for viruses, >>> addressees should scan this email and any attachments >>> for viruses. X-on makes no representation or warranty as to the absence >>> of viruses in this email or any attachments. >>> >>> _______________________________________________ >>> Users mailing list >>> Users@lists.opensips.org >>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users >>> >> _______________________________________________ >> Users mailing list >> Users@lists.opensips.org >> http://lists.opensips.org/cgi-bin/mailman/listinfo/users >> > > > *0333 332 0000 | www.x-on.co.uk <http://www.x-on.co.uk> | ** > <https://www.linkedin.com/company/x-on> <https://www.facebook.com/XonTel> > <https://twitter.com/xonuk> * > > X-on is a trading name of Storacall Technology Ltd a limited company > registered in England and Wales. > Registered Office : Avaland House, 110 London Road, Apsley, Hemel > Hempstead, Herts, HP3 9SD. Company Registration No. 2578478. > The information in this e-mail is confidential and for use by the > addressee(s) only. If you are not the intended recipient, please notify > X-on immediately on +44(0)333 332 0000 and delete the > message from your computer. If you are not a named addressee you must not > use, disclose, disseminate, distribute, copy, print or reply to this email. > Views > or opinions expressed by an individual > within this email may not necessarily reflect the views of X-on or its > associated companies. Although X-on routinely screens for viruses, > addressees should scan this email and any attachments > for viruses. X-on makes no representation or warranty as to the absence of > viruses in this email or any attachments. > > _______________________________________________ > Users mailing list > Users@lists.opensips.org > http://lists.opensips.org/cgi-bin/mailman/listinfo/users >
_______________________________________________ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users