Re: [Freeswitch-users] Performance bottleneck
Michael Collins wrote: That begs the question… is there a mechanism in sqlite or Linux that allows for the RAM drive to be backed up periodically? That would be a cool feature to get documented for those power users like Ken! ;) Interesting thought: http://tldp.org/HOWTO/LVM-HOWTO/snapshots_backup.html -MC *From:* [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] *On Behalf Of *Ken Rice *Sent:* Tuesday, August 12, 2008 11:07 AM *To:* freeswitch-users@lists.freeswitch.org *Subject:* Re: [Freeswitch-users] Performance bottleneck The Disk IO on sqlite can be quite a bit... One work around for this is to create a ram drive of sufficient size and mount it to /usr/local/freeswitch/db (or whatever your db dir is for freeswitch) this helps out greatly... But anything in the db will not be saved across system reboots unless you do something about that yourself K *From: *Michael Jerris [EMAIL PROTECTED] *Reply-To: *freeswitch-users@lists.freeswitch.org *Date: *Tue, 12 Aug 2008 13:59:13 -0400 *To: *freeswitch-users@lists.freeswitch.org *Subject: *Re: [Freeswitch-users] Performance bottleneck It's going to be the disk io from sqlite. The presense states are all stored in sqlite (or odbc) data source. Mike On Aug 12, 2008, at 1:53 PM, UV wrote: Turning the presence off did the trick, although it would be important (to me, at least) to understand why as it changes the performance significantly. Is the presence mechanism waiting for some response from the network? I’m assuming it’s waiting on something external because I couldn’t find any CPU activity… *From:* [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] *On Behalf Of *Anthony Minessale *Sent:* Wednesday, August 13, 2008 12:55 AM *To:* freeswitch-users@lists.freeswitch.org *Subject:* Re: [Freeswitch-users] Performance bottleneck 9996 is not a good test extension because it does not generate any audio unless it gets some. 9998 that generates a tone or make up an ext that plays a file is a better one. Processing of the sip calls can be delayed by the presence stuff which is very intensive, you can try turning it off and see if you get more calls. Also you should compare it to what happens with the test exten first in the dial plan. On Tue, Aug 12, 2008 at 2:58 AM, UV [EMAIL PROTECTED] wrote: I'm trying to determine the FS resource bottleneck when operating under load (in windows environment), but can't get the FS to load for some unseen reason. FS environment (a weak PC on purpose): CPU 2x Intel Pentium 4 3GHz RAM 2x 512MB DDR II RAM Chipset - Intel E7221 (Copper River) chipset ICH6R + FWH + BCM5721 LAN 1x Broadcom Giga LAN Windows 2003 Server – Service pack 2 FS version 9235 Running Release build on highest priority Load script: A different machine running sipP Running rtp_echo load, 50 cps, limit of 1000 calls, 30sec call duration, extension 9996 (echo test): sipp -rtp_echo -r 50 -l 1000 -d 3 -s 9996 -sf auc.xml -mp 25000 -i 192.168.1.1 http://192.168.1.1 -mi 192.168.1.1 http://192.168.1.1 192.168.1.2 http://192.168.1.2 Results: Test ran for 9.5 hours Total of 48828 calls - all successful No timeouts, retransmissions or unexpected messages. ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
[Freeswitch-users] Performance bottleneck
I’m trying to determine the FS resource bottleneck when operating under load (in windows environment), but can’t get the FS to load for some unseen reason. FS environment (a weak PC on purpose): CPU 2x Intel Pentium 4 3GHz RAM 2x 512MB DDR II RAM Chipset - Intel E7221 (Copper River) chipset ICH6R + FWH + BCM5721 LAN 1x Broadcom Giga LAN Windows 2003 Server – Service pack 2 FS version 9235 Running Release build on highest priority Load script: A different machine running sipP Running rtp_echo load, 50 cps, limit of 1000 calls, 30sec call duration, extension 9996 (echo test): sipp -rtp_echo -r 50 -l 1000 -d 3 -s 9996 -sf auc.xml -mp 25000 -i 192.168.1.1 -mi 192.168.1.1 192.168.1.2 Results: Test ran for 9.5 hours Total of 48828 calls - all successful No timeouts, retransmissions or unexpected messages. Peak was 1003 calls after 4563 seconds (actual 0.2 cps) Total of 1448750 RTP packets Average response time: 11min 21 seconds CPU usage 8% ~ 21%. Average 11%. Memory usage: Started with 26,000KB RAM, 27,660KB VM, 25 threads Peak at 136,000KB RAM,,367,004KB VM, 1024 threads Ended with 88,220KB RAM, 141,684KB VM, 24 threads Disk usage wasn’t monitored. My question is what is slowing the response time so much but keeps the CPU running low? NB Following Patrick Grondin’s post from 17-Jul-08, I intentionally didn’t change the default dialplan as I’m trying to load up the CPU. ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] Performance bottleneck
9996 is not a good test extension because it does not generate any audio unless it gets some. 9998 that generates a tone or make up an ext that plays a file is a better one. Processing of the sip calls can be delayed by the presence stuff which is very intensive, you can try turning it off and see if you get more calls. Also you should compare it to what happens with the test exten first in the dial plan. On Tue, Aug 12, 2008 at 2:58 AM, UV [EMAIL PROTECTED] wrote: I'm trying to determine the FS resource bottleneck when operating under load (in windows environment), but can't get the FS to load for some unseen reason. FS environment (a weak PC on purpose): CPU 2x Intel Pentium 4 3GHz RAM 2x 512MB DDR II RAM Chipset - Intel E7221 (Copper River) chipset ICH6R + FWH + BCM5721 LAN 1x Broadcom Giga LAN Windows 2003 Server – Service pack 2 FS version 9235 Running Release build on highest priority Load script: A different machine running sipP Running rtp_echo load, 50 cps, limit of 1000 calls, 30sec call duration, extension 9996 (echo test): sipp -rtp_echo -r 50 -l 1000 -d 3 -s 9996 -sf auc.xml -mp 25000 -i 192.168.1.1 -mi 192.168.1.1 192.168.1.2 Results: Test ran for 9.5 hours Total of 48828 calls - all successful No timeouts, retransmissions or unexpected messages. Peak was 1003 calls after 4563 seconds (actual 0.2 cps) Total of 1448750 RTP packets Average response time: 11min 21 seconds CPU usage 8% ~ 21%. Average 11%. Memory usage: Started with 26,000KB RAM, 27,660KB VM, 25 threads Peak at 136,000KB RAM,,367,004KB VM, 1024 threads Ended with 88,220KB RAM, 141,684KB VM, 24 threads Disk usage wasn't monitored. My question is what is slowing the response time so much but keeps the CPU running low? NB Following Patrick Grondin's post from 17-Jul-08, I intentionally didn't change the default dialplan as I'm trying to load up the CPU. ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ AIM: anthm MSN:[EMAIL PROTECTED] [EMAIL PROTECTED] GTALK/JABBER/PAYPAL:[EMAIL PROTECTED][EMAIL PROTECTED] IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:[EMAIL PROTECTED] [EMAIL PROTECTED] iax:[EMAIL PROTECTED]/888 googletalk:[EMAIL PROTECTED][EMAIL PROTECTED] pstn:213-799-1400 ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] Performance bottleneck
Plus there are dozens of events fired per call leg to describe the presence changes which does not scale well to 50cps. This is one of the many shortcomings to SIP, the presence stuff is not scalable. On Tue, Aug 12, 2008 at 12:59 PM, Michael Jerris [EMAIL PROTECTED] wrote: It's going to be the disk io from sqlite. The presense states are all stored in sqlite (or odbc) data source. Mike On Aug 12, 2008, at 1:53 PM, UV wrote: Turning the presence off did the trick, although it would be important (to me, at least) to understand why as it changes the performance significantly. Is the presence mechanism waiting for some response from the network? I'm assuming it's waiting on something external because I couldn't find any CPU activity… -- *From:* [EMAIL PROTECTED] [ mailto:[EMAIL PROTECTED][EMAIL PROTECTED] ] *On Behalf Of *Anthony Minessale *Sent:* Wednesday, August 13, 2008 12:55 AM *To:* freeswitch-users@lists.freeswitch.org *Subject:* Re: [Freeswitch-users] Performance bottleneck 9996 is not a good test extension because it does not generate any audio unless it gets some. 9998 that generates a tone or make up an ext that plays a file is a better one. Processing of the sip calls can be delayed by the presence stuff which is very intensive, you can try turning it off and see if you get more calls. Also you should compare it to what happens with the test exten first in the dial plan. On Tue, Aug 12, 2008 at 2:58 AM, UV [EMAIL PROTECTED] wrote: I'm trying to determine the FS resource bottleneck when operating under load (in windows environment), but can't get the FS to load for some unseen reason. FS environment (a weak PC on purpose): CPU 2x Intel Pentium 4 3GHz RAM 2x 512MB DDR II RAM Chipset - Intel E7221 (Copper River) chipset ICH6R + FWH + BCM5721 LAN 1x Broadcom Giga LAN Windows 2003 Server – Service pack 2 FS version 9235 Running Release build on highest priority Load script: A different machine running sipP Running rtp_echo load, 50 cps, limit of 1000 calls, 30sec call duration, extension 9996 (echo test): sipp -rtp_echo -r 50 -l 1000 -d 3 -s 9996 -sf auc.xml -mp 25000 -i 192.168.1.1 -mi 192.168.1.1 192.168.1.2 Results: Test ran for 9.5 hours Total of 48828 calls - all successful No timeouts, retransmissions or unexpected messages. Peak was 1003 calls after 4563 seconds (actual 0.2 cps) Total of 1448750 RTP packets Average response time: 11min 21 seconds CPU usage 8% ~ 21%. Average 11%. Memory usage: Started with 26,000KB RAM, 27,660KB VM, 25 threads Peak at 136,000KB RAM,,367,004KB VM, 1024 threads Ended with 88,220KB RAM, 141,684KB VM, 24 threads Disk usage wasn't monitored. My question is what is slowing the response time so much but keeps the CPU running low? NB Following Patrick Grondin's post from 17-Jul-08, I intentionally didn't change the default dialplan as I'm trying to load up the CPU. ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ AIM: anthm MSN:[EMAIL PROTECTED] [EMAIL PROTECTED] GTALK/JABBER/PAYPAL:[EMAIL PROTECTED][EMAIL PROTECTED] IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:[EMAIL PROTECTED] [EMAIL PROTECTED] iax:[EMAIL PROTECTED]/888 googletalk:[EMAIL PROTECTED][EMAIL PROTECTED] pstn:213-799-1400 No virus found in this incoming message. Checked by AVG - http://www.avg.com Version: 8.0.138 / Virus Database: 270.6.1/1605 - Release Date: 11/08/2008 16:59 ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ AIM: anthm MSN:[EMAIL PROTECTED] [EMAIL PROTECTED] GTALK/JABBER/PAYPAL:[EMAIL PROTECTED][EMAIL PROTECTED] IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:[EMAIL PROTECTED] [EMAIL PROTECTED] iax:[EMAIL PROTECTED]/888 googletalk:[EMAIL PROTECTED][EMAIL PROTECTED] pstn:213-799-1400 ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo
Re: [Freeswitch-users] Performance bottleneck
That begs the question... is there a mechanism in sqlite or Linux that allows for the RAM drive to be backed up periodically? That would be a cool feature to get documented for those power users like Ken! ;) -MC From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ken Rice Sent: Tuesday, August 12, 2008 11:07 AM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] Performance bottleneck The Disk IO on sqlite can be quite a bit... One work around for this is to create a ram drive of sufficient size and mount it to /usr/local/freeswitch/db (or whatever your db dir is for freeswitch) this helps out greatly... But anything in the db will not be saved across system reboots unless you do something about that yourself K From: Michael Jerris [EMAIL PROTECTED] Reply-To: freeswitch-users@lists.freeswitch.org Date: Tue, 12 Aug 2008 13:59:13 -0400 To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] Performance bottleneck It's going to be the disk io from sqlite. The presense states are all stored in sqlite (or odbc) data source. Mike On Aug 12, 2008, at 1:53 PM, UV wrote: Turning the presence off did the trick, although it would be important (to me, at least) to understand why as it changes the performance significantly. Is the presence mechanism waiting for some response from the network? I'm assuming it's waiting on something external because I couldn't find any CPU activity... From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] On Behalf Of Anthony Minessale Sent: Wednesday, August 13, 2008 12:55 AM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] Performance bottleneck 9996 is not a good test extension because it does not generate any audio unless it gets some. 9998 that generates a tone or make up an ext that plays a file is a better one. Processing of the sip calls can be delayed by the presence stuff which is very intensive, you can try turning it off and see if you get more calls. Also you should compare it to what happens with the test exten first in the dial plan. On Tue, Aug 12, 2008 at 2:58 AM, UV [EMAIL PROTECTED] wrote: I'm trying to determine the FS resource bottleneck when operating under load (in windows environment), but can't get the FS to load for some unseen reason. FS environment (a weak PC on purpose): CPU 2x Intel Pentium 4 3GHz RAM 2x 512MB DDR II RAM Chipset - Intel E7221 (Copper River) chipset ICH6R + FWH + BCM5721 LAN 1x Broadcom Giga LAN Windows 2003 Server - Service pack 2 FS version 9235 Running Release build on highest priority Load script: A different machine running sipP Running rtp_echo load, 50 cps, limit of 1000 calls, 30sec call duration, extension 9996 (echo test): sipp -rtp_echo -r 50 -l 1000 -d 3 -s 9996 -sf auc.xml -mp 25000 -i 192.168.1.1 http://192.168.1.1 http://192.168.1.1 -mi 192.168.1.1 http://192.168.1.1 http://192.168.1.1 192.168.1.2 http://192.168.1.2 http://192.168.1.2 Results: Test ran for 9.5 hours Total of 48828 calls - all successful No timeouts, retransmissions or unexpected messages. ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] Performance bottleneck
Actually I don¹t know of any mechanism that will back up the DB... Where sqlite does work well for small to medium installations it only scales to a point... Sqlite does not reuse nodes¹ in the db on an update... It marks them as dead and creates a new entry... While this works ok on smaller tables w/ light to medium updates after a while you have to compress or vacuum the tables... This requires a table level lock with sqlite... FS does have some things built in to handle this, but under load this can cause the switch to appear to hang. Switching over to use something like Postgresql (my prefered db) helps out a good bit here, but keep in mind that in doing so you greatly increase the resources required for the db. Also don¹t forget that pgsql has a similar mechanism on how it handles updates, just don¹t forget to enable auto-vacuuming on pgsql... That is a discussion for a different list tho K From: Brian West [EMAIL PROTECTED] Reply-To: freeswitch-users@lists.freeswitch.org Date: Tue, 12 Aug 2008 13:24:40 -0500 To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] Performance bottleneck Well putting the db in ram does help a bit but it has to keep states of everything going on and do extra work for that... its a heavy task in itself. On Aug 12, 2008, at 1:19 PM, Michael Collins wrote: That begs the question is there a mechanism in sqlite or Linux that allows for the RAM drive to be backed up periodically? That would be a cool feature to get documented for those power users like Ken! ;) -MC Brian West sip:[EMAIL PROTECTED] ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] Performance bottleneck
I dont know if this makes any sense - it's just an idea. If you're willing to take the hit of running MySQL, I know that it's replication features could potentially be used. You can have the primary MySQL server run in ramdisk and get all the performance benefits of doing so while also writing log files to the ram disk in a seperate area. Those logfiles can, using MySQL's built in replication features, be copied over to a backup server and played backup, giving you both a hot spare as well as a disk based backup. This does three things for you: 1) Gives you backup on disk, while preserving performance in RAM 2) Gives you a live backup that you can quickly shunt things over to if for some reason the primary dies 3) Allows you to handle spikes in volume. MySQL by default will just write to the log files and they can be played back later by the (slower) backup server, so a spike in volume of calls should not cause the server to slow down per say. There is a small risk your data will be lost if there is a failure for whatever is not copied over to the (slower) backup server, but that's unlikely to be that huge a lag (better then nothing). As to whether any of this applies (like why the heck you'd install MySQL on a ramdisk to start), I can't say. but it's a thought...Oh, and you need a lot of RAM ;-) _ From: Ken Rice [mailto:[EMAIL PROTECTED] Sent: Tuesday, August 12, 2008 11:44 AM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] Performance bottleneck Actually I don't know of any mechanism that will back up the DB... Where sqlite does work well for small to medium installations it only scales to a point... Sqlite does not reuse 'nodes' in the db on an update... It marks them as dead and creates a new entry... While this works ok on smaller tables w/ light to medium updates after a while you have to compress or vacuum the tables... This requires a table level lock with sqlite... FS does have some things built in to handle this, but under load this can cause the switch to appear to hang. Switching over to use something like Postgresql (my prefered db) helps out a good bit here, but keep in mind that in doing so you greatly increase the resources required for the db. Also don't forget that pgsql has a similar mechanism on how it handles updates, just don't forget to enable auto-vacuuming on pgsql... That is a discussion for a different list tho K _ From: Brian West [EMAIL PROTECTED] Reply-To: freeswitch-users@lists.freeswitch.org Date: Tue, 12 Aug 2008 13:24:40 -0500 To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] Performance bottleneck Well putting the db in ram does help a bit but it has to keep states of everything going on and do extra work for that... its a heavy task in itself. On Aug 12, 2008, at 1:19 PM, Michael Collins wrote: That begs the question. is there a mechanism in sqlite or Linux that allows for the RAM drive to be backed up periodically? That would be a cool feature to get documented for those power users like Ken! ;) -MC Brian West sip:[EMAIL PROTECTED] _ ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
