Hi Jan, I see you find the fix for the mem error :)
Regards, Bogdan Jan Rozhon wrote: > Hi Bogdan, > > I am using the latest 1.6.1 version of opensips, so I hope it should not > be the problem. > > My syslog got messy, so I ran the test again, here are the reports: > > first I got about 10 of these errors: > Feb 23 03:18:53 chieftec /usr/sbin/opensips[32454]: > ERROR:tm:new_t: out of mem > Feb 23 03:18:53 chieftec /usr/sbin/opensips[32454]: > ERROR:tm:t_newtran: new_t failed > > Then these errors: > > Feb 23 03:18:53 chieftec /usr/sbin/opensips[32462]: > ERROR:usrloc:shm_str_dup: no shared memory left > Feb 23 03:18:53 chieftec /usr/sbin/opensips[32462]: > ERROR:usrloc:new_ucontact: no more shm memory > Feb 23 03:18:53 chieftec /usr/sbin/opensips[32462]: > ERROR:usrloc:mem_insert_ucontact: failed to create new contact > Feb 23 03:18:53 chieftec /usr/sbin/opensips[32462]: > ERROR:usrloc:insert_ucontact: failed to insert contact > Feb 23 03:18:53 chieftec /usr/sbin/opensips[32462]: > ERROR:registrar:update_contacts: failed to insert contact > > Then these 2 groups of errors occure again and again. > > Somehow one of my SIPp instances managed to quit properly, so I was able > to read trace_screen log - all messages have quite low number of > retransmissions (hundreds), however 2nd REGISTER and 2nd INVITE (both > with authentication) have huge number of retransmissions (hundreds of > thousands) and timeouts (tens of thousands) - so I guess the problem has > something to do with the speed of mysql. > > Am I right? Is there some way to improve the performance? > > Thanks in advance, Jan. > > > Dne 22.2.2010 14:49, Bogdan-Andrei Iancu napsal(a): > >> No other errors before (from same process) ? I mean errors from the DB >> backend itself ? >> >> Regards, >> Bogdan >> >> Jan Rozhon wrote: >> >> >>> Now I found block of these errors - >>> >>> Feb 21 03:30:57 chieftec /usr/sbin/opensips[4795]: >>> ERROR:auth_db:get_ha1: failed to query database >>> Feb 21 03:30:57 chieftec /usr/sbin/opensips[4782]: >>> ERROR:auth_db:get_ha1: failed to query database >>> Feb 21 03:30:57 chieftec /usr/sbin/opensips[4791]: >>> ERROR:auth_db:get_ha1: failed to query database >>> Feb 21 03:30:57 chieftec /usr/sbin/opensips[4792]: >>> ERROR:auth_db:get_ha1: failed to query database >>> Feb 21 03:30:57 chieftec /usr/sbin/opensips[4783]: >>> ERROR:auth_db:get_ha1: failed to query database >>> >>> So there is a problem with database as well. Does it mean, that even if >>> system utilization tools (SAR) shows, that only about 1200 MB is used, >>> opensips has run out of memory? >>> >>> >>> Dne 22.2.2010 10:05, Bogdan-Andrei Iancu napsal(a): >>> >>> >>> >>>> If you received a 500 reply from opensips, you may check opensips logs >>>> for errors - maybe opensips is running out of memory or some other >>>> internal error. >>>> >>>> Regards, >>>> Bogdan >>>> >>>> Jan Rozhon wrote: >>>> >>>> >>>> >>>> >>>>> Hi Bogdan, >>>>> >>>>> I also thought, that the problem may be in SIPp, however most error I >>>>> get in SIPp are 500 Server Error Occured, so It directed me to >>>>> Opensips, moreover I tried to increase the load on a single VM and the >>>>> problems didnt occure until I reached 100% CPU utilization on SIPp VM. >>>>> >>>>> Disabling the authentication is not an option for me, because I need >>>>> it in my thesis, but I will try it to see, if it is not the >>>>> bottleneck. >>>>> >>>>> Thanks >>>>> Jan >>>>> >>>>> 2010/2/22 Bogdan-Andrei Iancu<bog...@voice-system.ro>: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Hi Jan, >>>>>> >>>>>> Jan Rozhon wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Hi Bogdan, >>>>>>> >>>>>>> thank you for your advice, but no help so far. I ran a test again using >>>>>>> a load of 280 calls per second (2800 simultaneous calls) and watched >>>>>>> receive buffer errors using netstat -su, but only 2930 errors occured >>>>>>> out of almost 3 000 000 packets received. So if I understand it well, >>>>>>> the problem is not caused by udp buffer size, because despite the low >>>>>>> number of udp datagrams discarded more than 60 000 calls were not >>>>>>> succesful (aproximately 25% of total calls). >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> In sipp, what are the the cps and max parallel call you are using ? >>>>>> Also, what kind of failures does the sipp report (missing 200ok ? >>>>>> missing BYE? etc). >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Then I checked the database setting - I am using MySQL with "0" >>>>>>> parameter (no database persistency), so I dont know how else I can make >>>>>>> it run faster (I cannot use separate computer for the database). >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> so no DB persistence for usrloc, but you mentioned using auth - is this >>>>>> correct? as auth is doing real time DB queries maybe you should try >>>>>> disabling the auth also to see how this affects the performance. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> As an opensips newbie, I dont have a clue, where the real problem could >>>>>>> be, so any further help would be appreciated. >>>>>>> >>>>>>> Thanks, Jan >>>>>>> >>>>>>> Dne 19.2.2010 14:56, Bogdan-Andrei Iancu napsal(a): >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Hi Jan, >>>>>>>> >>>>>>>> You need to see where the bottleneck is. If it is not the CPU, it must >>>>>>>> be some I/O blocking opensips. For example the DB may be too slow >>>>>>>> blocking opesips processes -> no process to read traffic from >>>>>>>> network -> >>>>>>>> traffic discarded, package lost. >>>>>>>> >>>>>>>> So, you should first look with netstat at the UDP sock, to see if the >>>>>>>> in-buffer is full or not (if full, the kernel will discard packages). >>>>>>>> >>>>>>>> Regarding the timeouts (for calls, for retransmissions) see the TM >>>>>>>> module: >>>>>>>> >>>>>>>> http://www.opensips.org/html/docs/modules/1.6.x/tm.html#id228443 >>>>>>>> >>>>>>>> Regards, >>>>>>>> Bogdan >>>>>>>> >>>>>>>> Jan Rozhon wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> as a part of my diploma thesis I'm trying to do some performance >>>>>>>>> testing >>>>>>>>> of opensips running on low cost machine with 4GB of RAM and dual-core >>>>>>>>> AMD Athlon processor. As a generator of SIP traffic I use SIPp v3.1 >>>>>>>>> running on 4 virtual computers as UAC and two computers as UAS all >>>>>>>>> connected just by gigabit ports of a single switch. My proublem is, >>>>>>>>> that >>>>>>>>> as soon as I reach the load of 200 calls per second (2000 simultaneous >>>>>>>>> calls) many of those calls (aroud 50%) don't finish succesfully. CPU >>>>>>>>> utilization of opensips machine is around 40% and memory around 25%, >>>>>>>>> so >>>>>>>>> the opensips has still a lot of free resources, but it doesn't use >>>>>>>>> them. >>>>>>>>> Could you please advise me, how to change default configuration script >>>>>>>>> and opensips settings to achieve better results? >>>>>>>>> >>>>>>>>> Right now, the only changes I made is : >>>>>>>>> -enabling registrar/proxy authentication; >>>>>>>>> -disabled tcp, since all the traffic is carried by udp; >>>>>>>>> -set shared memory to 2048 MB; >>>>>>>>> -set number of children to 16 >>>>>>>>> -limit debug level to 1 >>>>>>>>> >>>>>>>>> PS. How can I increase the time, after which opensips retransmits SIP >>>>>>>>> messages? >>>>>>>>> >>>>>>>>> Thanks in advance, >>>>>>>>> Jan >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Users mailing list >>>>>>>>> Users@lists.opensips.org >>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>> >>>> >>> _______________________________________________ >>> Users mailing list >>> Users@lists.opensips.org >>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users >>> >>> >>> >>> >> >> > > > _______________________________________________ > Users mailing list > Users@lists.opensips.org > http://lists.opensips.org/cgi-bin/mailman/listinfo/users > > -- Bogdan-Andrei Iancu www.voice-system.ro _______________________________________________ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users