here is another trace http://pastebin.com/9Ge2NEVQ
I see lot of alloc request but no free. On Mon, Mar 6, 2017 at 6:57 PM, John Nash <john.nash...@gmail.com> wrote: > Ok will try that. Is it possible that wrong usage of drouting may cause > this to happen instead of actual leak?... What are the things private > memory is used for? > > On Mon, Mar 6, 2017 at 6:48 PM, Răzvan Crainea <raz...@opensips.org> > wrote: > >> Hi, John! >> >> From the dump you sent, I don't see any leaks. Perhaps some of those >> fragments increase over time. Can you make a memory dump after the server >> runs some time, like after it gets 100 messages? >> >> Best regards, >> >> Răzvan Crainea >> OpenSIPS Solutionswww.opensips-solutions.com >> >> On 03/06/2017 03:02 PM, John Nash wrote: >> >> Here is the dump >> http://pastebin.com/DTEHF5Vc >> >> On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <raz...@opensips.org> >> wrote: >> >>> None of the "actions" you are talking about have big impact on private >>> memory, but the shared one. Better do the dump and send it over to point >>> out what is "eating" memory. >>> >>> Best regards, >>> >>> Răzvan Crainea >>> OpenSIPS Solutionswww.opensips-solutions.com >>> >>> On 03/06/2017 02:39 PM, John Nash wrote: >>> >>> with every call attempt it decreases. I tried some changes by rejecting >>> invite before drouting call (That means after auth , dispatcher) and found >>> memory is stable but when drouting sends Invite to external gateway and >>> external gateway rejects it. Then this issue happens. >>> >>> Inuse transactions and active dialogs also 0. Somthing wrong happening >>> in handling of failure replies. But apart from use_next_gw and setting >>> some avps for CDR not much going on there. >>> >>> On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <raz...@opensips.org> >>> wrote: >>> >>>> Ok, so it is the first listener for the private IP that leaks. Next, is >>>> the memory stabilizing in time? Or it is continously decreasing? >>>> Yes, that's how you should make the dump. >>>> >>>> Best regards, >>>> >>>> Răzvan Crainea >>>> OpenSIPS Solutionswww.opensips-solutions.com >>>> >>>> On 03/06/2017 10:57 AM, John Nash wrote: >>>> >>>> Dear Razvan, >>>> >>>> Below is the info on my processes >>>> Process:: ID=0 PID=17351 Type=attendant >>>> Process:: ID=1 PID=17352 Type=MI FIFO >>>> Process:: ID=2 PID=17353 Type=MI Datagram >>>> Process:: ID=3 PID=17354 Type=time_keeper >>>> Process:: ID=4 PID=17355 Type=timer >>>> Process:: ID=5 PID=17356 Type=SIP receiver udp:1.1.1.1:9094 >>>> Process:: ID=6 PID=17357 Type=SIP receiver udp:1.1.1.1:5060 >>>> Process:: ID=7 PID=17358 Type=SIP receiver udp:192.168.45.5:5064 >>>> Process:: ID=8 PID=17359 Type=Timer handler >>>> >>>> 1.1.1.1 is public IP (I changed). The decrease in memory I see is for >>>> Process:: ID=7 PID=17358 mainly. My call flow is as following >>>> >>>> - New Invite hits the opensips on 1.1.1.1:9094 >>>> - Apart from message validity checks I query DB to check if its a valid >>>> user (Using local cache also there) >>>> - Create dialog, Topology_hiding functions are called along with some >>>> avp population >>>> - Using dispatcher ds_select_domain Call sent to udp:192.168.45.2:7060 >>>> (using force socket). This 192.168.45.2:7060 is actually freeswitch >>>> - Call again comes back to opensips on udp:192.168.45.5:5064 >>>> - New dialog is created and topology_hiding is called >>>> - Drouting function do_routing is called which tries one gateway and >>>> fails >>>> >>>> >>>> Dump i need to create with memlog=4 memdump=1 right? >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Mon, Mar 6, 2017 at 2:05 PM, Răzvan Crainea <raz...@opensips.org> >>>> wrote: >>>> >>>>> Hi, John! >>>>> >>>>> Transactions are stored in shared memory, not in the private one. So >>>>> the possible leak you are facing its not related to transactions. >>>>> During runtime, OpenSIPS might resize some internal structures, which >>>>> may lead to increase memory usage. However, after a while, these >>>>> allocations should stabilize . >>>>> Can you post the output of the kill -SIGUSR1 on pastebin so we can >>>>> take a look? Also, what type of process is the one you are seeing the leak >>>>> into? You can find out using the 'opensipsctl ps' command. >>>>> >>>>> Best regards, >>>>> >>>>> Răzvan Crainea >>>>> OpenSIPS Solutionswww.opensips-solutions.com >>>>> >>>>> On 03/06/2017 09:55 AM, John Nash wrote: >>>>> >>>>> I am using OpenSIPS (2.1.5 (x86_64/linux)) in production. I observed >>>>> private memory is decreasing constantly for one process mainly and >>>>> ultimately leading to memory errors and crash. >>>>> >>>>> To debug this issue I prepared a test server and compiled opensips as >>>>> per https://www.opensips.org/Documentation/TroubleShooting-OutOfMem >>>>> >>>>> I made only one single call (which was rejected by opensips as it was >>>>> not authorized user) and I saw private free memory decreased. I was hoping >>>>> since transaction is done ideally it should release memory and should show >>>>> me same memory as startup but it did not. I verified this with many call >>>>> attempts and i see free memory is always decreasing slowly. >>>>> >>>>> I used kill -SIGUSR1 <lowest pid> to create memory dump. But i am >>>>> unable to make sense of it. It shows log like ... >>>>> >>>>> r 6 07:29:19 Server3021 opensips[13276]: Memory status (pkg): >>>>> Mar 6 07:29:19 Server3021 opensips[13276]: qm_status (0x7f5b8ebba010): >>>>> Mar 6 07:29:19 Server3021 opensips[13276]: heap size= 4194304 >>>>> Mar 6 07:29:19 Server3021 opensips[13276]: used= 346768, >>>>> used+overhead=848792, free=3345512 >>>>> Mar 6 07:29:19 Server3021 opensips[13276]: max used (+overhead)= >>>>> 931920 >>>>> Mar 6 07:29:19 Server3021 opensips[13276]: dumping all alloc'ed. >>>>> fragments: >>>>> Mar 6 07:29:19 Server3021 opensips[13276]: 0. N >>>>> address=0x7f5b8ebef528 frag=0x7f5b8ebef4f8 size=40 used=1 >>>>> Mar 6 07:29:19 Server3021 opensips[13276]: alloc'd from >>>>> script_cb.c: add_callback(60) >>>>> Mar 6 07:29:19 Server3021 opensips[13276]: start >>>>> check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed >>>>> Mar 6 07:29:19 Server3021 opensips[13276]: 1. N >>>>> address=0x7f5b8ebef5b0 >>>>> >>>>> I pasted only few lines in this mail. What should be my next >>>>> step?...How can i really trace what is wrong in my script or any other >>>>> memory leak? >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Users mailing >>>>> listUsers@lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users >>>>> >>>>> _______________________________________________ Users mailing list >>>>> Users@lists.opensips.org http://lists.opensips.org/cgi- >>>>> bin/mailman/listinfo/users >>>> >>>> _______________________________________________ >>>> Users mailing >>>> listUsers@lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users >>>> >>>> _______________________________________________ Users mailing list >>>> Users@lists.opensips.org http://lists.opensips.org/cgi- >>>> bin/mailman/listinfo/users >>> >>> _______________________________________________ >>> Users mailing >>> listUsers@lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users >>> >>> _______________________________________________ Users mailing list >>> Users@lists.opensips.org http://lists.opensips.org/cgi- >>> bin/mailman/listinfo/users >> >> _______________________________________________ >> Users mailing >> listUsers@lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users >> >> >> _______________________________________________ >> Users mailing list >> Users@lists.opensips.org >> http://lists.opensips.org/cgi-bin/mailman/listinfo/users >> >> >
_______________________________________________ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users