Re: [OpenSIPS-Devel] Memory Lick problem
Hello Liviu , This opensips is on a small carrier, we expected in some weeks reach almost 2000 simultaneos calls, 750 is something that can happen right now, we have all kind of customers like call centers, small companies , residencial customers and other carriers, so we realy want to reach a high number of calls with opensips, but 750 cps is diferent than simultaneos calls (i think), i will check if we are getting some kind of DOS attack i will change the memory parameters as you suggest. Just one question for you, how you interpret the value below as 750 cps ? shmem:used_size = 3401120 shmem:max_used_size = 226816344 2014-03-04 11:16 GMT-05:00 Liviu Chircu : > Hello Mike, > > MySQL data looks like peanuts. We can safely ignore it. The following > stats are quite interesting: > > shmem:used_size = 3401120 > shmem:max_used_size = 226816344 > > From the *tm* module's point of view, this means that your server was > hit, at some point in time, by a traffic of over 750 transactions/sec! If > we add the memory consumption of other modules (dialog, usrloc), and a more > aggressive peak hour, it may be quite normal for memory consumption to > reach 1GB! > > Can you share some details about the nature of your traffic? Do you expect > traffic of over 750 cps regularly? If yes, then: > > * -m 7192 is a good idea. You can probably reduce it a bit, like 4096. > * *-M 512 is too much*. 64MB should be more than enough for an OpenSIPS > listener on *any* scenario. > > If you don't expect such traffic, then it looks like a denial of service > attack, case in which you should look into the *ratelimit* module. > > > Best regards, > > Liviu Chircu > OpenSIPS Developerhttp://www.opensips-solutions.com > > On 03/04/2014 03:43 PM, Mike Tesliuk wrote: > >Hello Liviu, > > The address table have 110 records, and the subscriber have 1738 records, > i have 342 users registered > > and im using the loadbalancer with 10 records > > About recompile yes, i can recompile, i already have done this before , > you said that the logs that i send was ok, nothing like a memory leak, but > i can do, as i told you i change the memory parameteres as you can see below > > /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f > /etc/opensips/opensips.cfg -m 7168 -M 512 > > > im running now since March 1st . > > Right now my fifo get_statistics show me > > shmem:total_size = 7516192768 > shmem:used_size = 3401120 > shmem:real_used_size = 46974408 > shmem:max_used_size = 226816344 > shmem:free_size = 7469218360 > shmem:fragments = 449173 > > pkmem:0-total_size = 536870912 > pkmem:0-used_size = 105424 > pkmem:0-real_used_size = 412856 > pkmem:0-max_used_size = 555464 > pkmem:0-free_size = 536458056 > pkmem:0-fragments = 3 > pkmem:1-total_size = 0 > pkmem:1-used_size = 0 > pkmem:1-real_used_size = 0 > pkmem:1-max_used_size = 0 > pkmem:1-free_size = 536442160 > pkmem:1-fragments = 0 > pkmem:2-total_size = 536870912 > pkmem:2-used_size = 133888 > pkmem:2-real_used_size = 480680 > pkmem:2-max_used_size = 555464 > pkmem:2-free_size = 536390232 > pkmem:2-fragments = 307 > pkmem:3-total_size = 536870912 > pkmem:3-used_size = 122928 > pkmem:3-real_used_size = 441400 > pkmem:3-max_used_size = 555464 > pkmem:3-free_size = 536429512 > pkmem:3-fragments = 109 > pkmem:4-total_size = 536870912 > pkmem:4-used_size = 122928 > pkmem:4-real_used_size = 441496 > pkmem:4-max_used_size = 555464 > pkmem:4-free_size = 536429416 > pkmem:4-fragments = 110 > pkmem:5-total_size = 536870912 > pkmem:5-used_size = 123080 > pkmem:5-real_used_size = 457584 > pkmem:5-max_used_size = 555464 > pkmem:5-free_size = 536413328 > pkmem:5-fragments = 276 > pkmem:6-total_size = 536870912 > pkmem:6-used_size = 123088 > pkmem:6-real_used_size = 460184 > pkmem:6-max_used_size = 555464 > pkmem:6-free_size = 536410728 > pkmem:6-fragments = 303 > pkmem:7-total_size = 536870912 > pkmem:7-used_size = 103360 > pkmem:7-real_used_size = 408680 > pkmem:7-max_used_size = 555464 > pkmem:7-free_size = 536462232 > pkmem:7-fragments = 4 > pkmem:8-total_size = 536870912 > pkmem:8-used_size = 119736 > pkmem:8-real_used_size = 431584 > pkmem:8-max_used_size = 555464 > pkmem:8-free_size = 536439328 > pkmem:8-fragments = 38 > > > But im not on my peak of use now, is holiday in Brazil so we dont have > too much users running > > > ___ > Devel mailing > listDevel@lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/devel > > > > ___ > Devel mailing list > Devel@lists.opensips.org > http://lists.opensips.org/cgi-bin/mailman/listinfo/devel > > ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Re: [OpenSIPS-Devel] Memory Lick problem
Hello Liviu, The address table have 110 records, and the subscriber have 1738 records, i have 342 users registered and im using the loadbalancer with 10 records About recompile yes, i can recompile, i already have done this before , you said that the logs that i send was ok, nothing like a memory leak, but i can do, as i told you i change the memory parameteres as you can see below /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f /etc/opensips/opensips.cfg -m 7168 -M 512 im running now since March 1st . Right now my fifo get_statistics show me shmem:total_size = 7516192768 shmem:used_size = 3401120 shmem:real_used_size = 46974408 shmem:max_used_size = 226816344 shmem:free_size = 7469218360 shmem:fragments = 449173 pkmem:0-total_size = 536870912 pkmem:0-used_size = 105424 pkmem:0-real_used_size = 412856 pkmem:0-max_used_size = 555464 pkmem:0-free_size = 536458056 pkmem:0-fragments = 3 pkmem:1-total_size = 0 pkmem:1-used_size = 0 pkmem:1-real_used_size = 0 pkmem:1-max_used_size = 0 pkmem:1-free_size = 536442160 pkmem:1-fragments = 0 pkmem:2-total_size = 536870912 pkmem:2-used_size = 133888 pkmem:2-real_used_size = 480680 pkmem:2-max_used_size = 555464 pkmem:2-free_size = 536390232 pkmem:2-fragments = 307 pkmem:3-total_size = 536870912 pkmem:3-used_size = 122928 pkmem:3-real_used_size = 441400 pkmem:3-max_used_size = 555464 pkmem:3-free_size = 536429512 pkmem:3-fragments = 109 pkmem:4-total_size = 536870912 pkmem:4-used_size = 122928 pkmem:4-real_used_size = 441496 pkmem:4-max_used_size = 555464 pkmem:4-free_size = 536429416 pkmem:4-fragments = 110 pkmem:5-total_size = 536870912 pkmem:5-used_size = 123080 pkmem:5-real_used_size = 457584 pkmem:5-max_used_size = 555464 pkmem:5-free_size = 536413328 pkmem:5-fragments = 276 pkmem:6-total_size = 536870912 pkmem:6-used_size = 123088 pkmem:6-real_used_size = 460184 pkmem:6-max_used_size = 555464 pkmem:6-free_size = 536410728 pkmem:6-fragments = 303 pkmem:7-total_size = 536870912 pkmem:7-used_size = 103360 pkmem:7-real_used_size = 408680 pkmem:7-max_used_size = 555464 pkmem:7-free_size = 536462232 pkmem:7-fragments = 4 pkmem:8-total_size = 536870912 pkmem:8-used_size = 119736 pkmem:8-real_used_size = 431584 pkmem:8-max_used_size = 555464 pkmem:8-free_size = 536439328 pkmem:8-fragments = 38 But im not on my peak of use now, is holiday in Brazil so we dont have too much users running ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Re: [OpenSIPS-Devel] Memory Lick problem
I put the -m to 7168 and -M to 512 , i dont think this is a good way and as i never see this happen before on a real server (not VM) i think that the kvm can be the problem, i have 400 call with 8MB private memory without problem i cant understand how this server got this problem otherwise the VM thing. i have the system working now from two days, let see what happen 2014-02-28 15:24 GMT-05:00 Mike Tesliuk : > Hello Guys, > > I got the same problem again > > this is how im executing > > /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f > /etc/opensips/opensips.cfg -m 1024 -M 64 > /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f > /etc/opensips/opensips.cfg -m 1024 -M 64 > /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f > /etc/opensips/opensips.cfg -m 1024 -M 64 > /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f > /etc/opensips/opensips.cfg -m 1024 -M 64 > /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f > /etc/opensips/opensips.cfg -m 1024 -M 64 > /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f > /etc/opensips/opensips.cfg -m 1024 -M 64 > /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f > /etc/opensips/opensips.cfg -m 1024 -M 64 > /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f > /etc/opensips/opensips.cfg -m 1024 -M 64 > /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f > /etc/opensips/opensips.cfg -m 1024 -M 64 > > > Just to remember i have 8 GB Ram for this server > > Feb 28 15:35:08 sipproxy1 /usr/local/opensips/sbin/opensips[27539]: Grupo > de gateways: 1 > Feb 28 15:35:08 sipproxy1 /usr/local/opensips/sbin/opensips[27539]: > ERROR:dialog:build_new_dlg: no more shm mem (300) > Feb 28 15:35:08 sipproxy1 /usr/local/opensips/sbin/opensips[27539]: > ERROR:dialog:dlg_create_dialog: failed to create new dialog > Feb 28 15:35:08 sipproxy1 /usr/local/opensips/sbin/opensips[27539]: > ERROR:load_balancer:do_load_balance: failed to create dialog > > > > The amount of calls was like 400 calls when i got the message. > > As i say before, im running this host on a kvm server, this is the first > server that i put in prodcution using virtualization, i have real servers > running without problem , the server run fine from yeasterday to now > > Below the link to the script, i double check the script and dont see > anything that can generate this error > > http://204.16.1.50/opensips.cfg > > I will appreciate your help. > > Thanks > > > > 2014-02-27 12:52 GMT-05:00 Mike Tesliuk : > > Ok, i will try that, sorry my mistake with the subject so e thanks for >> your help, on my "man opensips" i didint see thes -M option (just -m) >> >> 2014-02-27 12:45 GMT-05:00 Liviu Chircu : >> > It doesn't matter how many memory you have. OpenSIPS only uses what it >> is >> > configured to start with. >> > >> > Restart your OpenSIPS daemon using the "-m 512 -M 32" options (512MB >> shared >> > between children, 32MB private for each one). It should be more than >> enough >> > to handle 600 concurrent calls. >> > >> > >> > Best regards, >> > >> > Liviu Chircu >> > OpenSIPS Developer >> > http://www.opensips-solutions.com >> > >> > On 02/27/2014 07:40 PM, Mike Tesliuk wrote: >> >> >> >> Hello Liviu, >> >> >> >> Thanks for you help. >> >> >> >> Im receiving messages of no memmory >> >> >> >> : WARNING:core:fm_malloc: Not enough free memory, will atempt >> >> defragmenation >> >> Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: >> >> ERROR:dialog:dlg_add_leg_info: Failed to resize legs array >> >> Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: >> >> ERROR:dialog:init_leg_info: dlg_add_leg_info failed >> >> Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: >> >> ERROR:dialog:dlg_create_dialog: could not add further info to the >> >> dialog >> >> Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: >> >> ERROR:load_balancer:do_load_balance: failed to create dialog >> >> >> >> >> >> The server is virtual host (kvm) with 8 GB dedicated memory , i got >> >> something like 600 simultaneos calls >> >> >> >> >> >> The memory status error that you ask is on the link below >> >> >> >> >> >> http://204.16.1.50/memorystatus-shm.txt >> >> >> >> >> >> and i get some ERROR messages on the next link >> >> >> >> http://204.16.1.50/process-errors.txt >> >> >> >> ___ >> >> Devel mailing list >> >> Devel@lists.opensips.org >> >> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel >> > >> > >> > >> > ___ >> > Devel mailing list >> > Devel@lists.opensips.org >> > http://lists.opensips.org/cgi-bin/mailman/listinfo/devel >> > > ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Re: [OpenSIPS-Devel] Memory Lick problem
Hello Guys, I got the same problem again this is how im executing /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f /etc/opensips/opensips.cfg -m 1024 -M 64 /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f /etc/opensips/opensips.cfg -m 1024 -M 64 /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f /etc/opensips/opensips.cfg -m 1024 -M 64 /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f /etc/opensips/opensips.cfg -m 1024 -M 64 /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f /etc/opensips/opensips.cfg -m 1024 -M 64 /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f /etc/opensips/opensips.cfg -m 1024 -M 64 /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f /etc/opensips/opensips.cfg -m 1024 -M 64 /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f /etc/opensips/opensips.cfg -m 1024 -M 64 /usr/local/opensips/sbin/opensips -P /var/run/opensips.pid -f /etc/opensips/opensips.cfg -m 1024 -M 64 Just to remember i have 8 GB Ram for this server Feb 28 15:35:08 sipproxy1 /usr/local/opensips/sbin/opensips[27539]: Grupo de gateways: 1 Feb 28 15:35:08 sipproxy1 /usr/local/opensips/sbin/opensips[27539]: ERROR:dialog:build_new_dlg: no more shm mem (300) Feb 28 15:35:08 sipproxy1 /usr/local/opensips/sbin/opensips[27539]: ERROR:dialog:dlg_create_dialog: failed to create new dialog Feb 28 15:35:08 sipproxy1 /usr/local/opensips/sbin/opensips[27539]: ERROR:load_balancer:do_load_balance: failed to create dialog The amount of calls was like 400 calls when i got the message. As i say before, im running this host on a kvm server, this is the first server that i put in prodcution using virtualization, i have real servers running without problem , the server run fine from yeasterday to now Below the link to the script, i double check the script and dont see anything that can generate this error http://204.16.1.50/opensips.cfg I will appreciate your help. Thanks 2014-02-27 12:52 GMT-05:00 Mike Tesliuk : > Ok, i will try that, sorry my mistake with the subject so e thanks for > your help, on my "man opensips" i didint see thes -M option (just -m) > > 2014-02-27 12:45 GMT-05:00 Liviu Chircu : > > It doesn't matter how many memory you have. OpenSIPS only uses what it is > > configured to start with. > > > > Restart your OpenSIPS daemon using the "-m 512 -M 32" options (512MB > shared > > between children, 32MB private for each one). It should be more than > enough > > to handle 600 concurrent calls. > > > > > > Best regards, > > > > Liviu Chircu > > OpenSIPS Developer > > http://www.opensips-solutions.com > > > > On 02/27/2014 07:40 PM, Mike Tesliuk wrote: > >> > >> Hello Liviu, > >> > >> Thanks for you help. > >> > >> Im receiving messages of no memmory > >> > >> : WARNING:core:fm_malloc: Not enough free memory, will atempt > >> defragmenation > >> Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: > >> ERROR:dialog:dlg_add_leg_info: Failed to resize legs array > >> Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: > >> ERROR:dialog:init_leg_info: dlg_add_leg_info failed > >> Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: > >> ERROR:dialog:dlg_create_dialog: could not add further info to the > >> dialog > >> Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: > >> ERROR:load_balancer:do_load_balance: failed to create dialog > >> > >> > >> The server is virtual host (kvm) with 8 GB dedicated memory , i got > >> something like 600 simultaneos calls > >> > >> > >> The memory status error that you ask is on the link below > >> > >> > >> http://204.16.1.50/memorystatus-shm.txt > >> > >> > >> and i get some ERROR messages on the next link > >> > >> http://204.16.1.50/process-errors.txt > >> > >> ___ > >> Devel mailing list > >> Devel@lists.opensips.org > >> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel > > > > > > > > ___ > > Devel mailing list > > Devel@lists.opensips.org > > http://lists.opensips.org/cgi-bin/mailman/listinfo/devel > ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Re: [OpenSIPS-Devel] Memory Lick problem
Ok, i will try that, sorry my mistake with the subject so e thanks for your help, on my "man opensips" i didint see thes -M option (just -m) 2014-02-27 12:45 GMT-05:00 Liviu Chircu : > It doesn't matter how many memory you have. OpenSIPS only uses what it is > configured to start with. > > Restart your OpenSIPS daemon using the "-m 512 -M 32" options (512MB shared > between children, 32MB private for each one). It should be more than enough > to handle 600 concurrent calls. > > > Best regards, > > Liviu Chircu > OpenSIPS Developer > http://www.opensips-solutions.com > > On 02/27/2014 07:40 PM, Mike Tesliuk wrote: >> >> Hello Liviu, >> >> Thanks for you help. >> >> Im receiving messages of no memmory >> >> : WARNING:core:fm_malloc: Not enough free memory, will atempt >> defragmenation >> Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: >> ERROR:dialog:dlg_add_leg_info: Failed to resize legs array >> Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: >> ERROR:dialog:init_leg_info: dlg_add_leg_info failed >> Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: >> ERROR:dialog:dlg_create_dialog: could not add further info to the >> dialog >> Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: >> ERROR:load_balancer:do_load_balance: failed to create dialog >> >> >> The server is virtual host (kvm) with 8 GB dedicated memory , i got >> something like 600 simultaneos calls >> >> >> The memory status error that you ask is on the link below >> >> >> http://204.16.1.50/memorystatus-shm.txt >> >> >> and i get some ERROR messages on the next link >> >> http://204.16.1.50/process-errors.txt >> >> ___ >> Devel mailing list >> Devel@lists.opensips.org >> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel > > > > ___ > Devel mailing list > Devel@lists.opensips.org > http://lists.opensips.org/cgi-bin/mailman/listinfo/devel ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Re: [OpenSIPS-Devel] Memory Lick problem
Hello Liviu, Thanks for you help. Im receiving messages of no memmory : WARNING:core:fm_malloc: Not enough free memory, will atempt defragmenation Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: ERROR:dialog:dlg_add_leg_info: Failed to resize legs array Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: ERROR:dialog:init_leg_info: dlg_add_leg_info failed Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: ERROR:dialog:dlg_create_dialog: could not add further info to the dialog Feb 25 15:13:02 sipproxy1 /usr/local/opensips/sbin/opensips[8125]: ERROR:load_balancer:do_load_balance: failed to create dialog The server is virtual host (kvm) with 8 GB dedicated memory , i got something like 600 simultaneos calls The memory status error that you ask is on the link below http://204.16.1.50/memorystatus-shm.txt and i get some ERROR messages on the next link http://204.16.1.50/process-errors.txt ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
[OpenSIPS-Devel] Memory Lick problem
Hello Guys, Im getting a problem with out of memory on a opensips, this opensips is running on kvm server and i have just two servers on this server, i have enough memory, can you guys try to help me to understand the debug on the link below ? http://204.16.1.50/memoryfail.txt This opensips have simple rules and load_balance I will apreciate any help. Thanks ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
