Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality
Yes it's hard to trust virtualized stuff because you have no idea what they skimp on in terms of realtime access. I won't endorse using FS on a VM as i have not done it very extensively beyond openvz but I can point out a few reasons why it has a fighting chance. FS uses a timer architecture designed to amplify the work of one timer thread into every timer open by FS. This single thread uses the monotonic clock on the system to try and perfrom a 1ms accurate loop. This single loop updates a soft value for current epoch time and microsecond epoch time with the goal of (again) being as close as possible to being accurate to 1 ms. The timer loop also has a global matrix to all of the timing intervals being subscribed to by a timer open by FS. The loop will tick a counter in each unique timing interval (10ms, 20ms, 60ms etc) and fire a conditional broadcast to all of the timers who are blocking for a tick. This is not perfectly accurate but close enough to end up plus or minus 2ms in resulting rtp traces. So as long as the VM will expose the syscall down to the real monotonic clock rather than doing it's own soft timing technique you have a better chance for success. The other issue with VM is with vmware, the bridged networking mode seems to send 2 of every RTP packet to the channel resulting in garbled audio from the obvious timing issue introduced from too many packets. Anyway evaluating FS with a VM is a good way to get acquainted but, with all the money saved choosing FS, many can afford to buy it a nice 8 core box for it to live on and still have money left over to support the project or ClueCon 2009 this august ;) On Fri, Jan 16, 2009 at 2:17 AM, Kristian Kielhofner < kristian.kielhof...@gmail.com> wrote: > Speaking of networking... > > After timing that's the next "achilles heel" of RTP handing with > virtualization. > > Very, very few of these platforms were designed to handle massive > numbers of very small RTP packets. Everything from interrupt handling > on the actual ethernet adapter to getting the data into userspace > within the virtual instance is worrisome: > > http://www.xen.org/files/xensummit_4/NetworkIO_Santos.pdf > http://forum.openvz.org/index.php?t=msg&goto=11619&; > > Interestingly enough the Xen paper makes it out to be really bad yet > the OpenVZ post praises Xen's performance. Without any real testing, > who knows? I just know that scaling 50pps per RTP stream (20ms > packetization) can be difficult enough on native hardware, let alone > [virtualization technology du jour]. > > On Thu, Jan 15, 2009 at 5:02 PM, Remko Kloosterman > wrote: > > Lot's of experience and suggestions here. Thanks. > > > > I believe it should be theoretically possible to have blip-free RTP > > streaming through the appliance. Most windows ethernet drivers allow for > > QoS packet scheduling. If the VMware network bridge driver honors this > > and syncs the buffers at 20ms frames (or whatever frame size applies) > > you should be able to schale up a bit and maintain low jitter. > > > > Anyone knows how the VMware network bridge exactly works? > > > > > > -Oorspronkelijk bericht- > > Van: freeswitch-users-boun...@lists.freeswitch.org > > [mailto:freeswitch-users-boun...@lists.freeswitch.org] Namens Gregory > > Boehnlein > > Verzonden: donderdag 15 januari 2009 21:37 > > Aan: freeswitch-users@lists.freeswitch.org > > Onderwerp: Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality > > > >> To the contrary, we have had quite good results in virtualized > >> environments and you don't really need timing that is that accurate to > > > >> make it work. > > > > If you don't handle RTP, I'm sure it is amazing. However, if you have to > > do voicemail, stream audio from the server or do any kind of actual > > time/latency/jitter sensitive processing, I don't care how much you tune > > your hypervisor, it's never going to scale. > > > >> We work quite well on amazon EC2 for example. There are 2 issues I > >> know about with vmware, 1 is you need to set a setting on the host to > >> extend somewhat sane clocks being available, the second is I have seen > > > >> issues with the bridged network adapter actually doubling up all > >> packets causing very strange issues, I suggest not using bridged > >> networking if you experience this. > > > > I've not seen this behavior on Vmware ESX 3.5u2. Maybe an issue on > > Vmware Server or Workstation? > > > > > > ___ > > Freeswitch-users mailing list > > Freeswitch-users@lists
Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality
Speaking of networking... After timing that's the next "achilles heel" of RTP handing with virtualization. Very, very few of these platforms were designed to handle massive numbers of very small RTP packets. Everything from interrupt handling on the actual ethernet adapter to getting the data into userspace within the virtual instance is worrisome: http://www.xen.org/files/xensummit_4/NetworkIO_Santos.pdf http://forum.openvz.org/index.php?t=msg&goto=11619&; Interestingly enough the Xen paper makes it out to be really bad yet the OpenVZ post praises Xen's performance. Without any real testing, who knows? I just know that scaling 50pps per RTP stream (20ms packetization) can be difficult enough on native hardware, let alone [virtualization technology du jour]. On Thu, Jan 15, 2009 at 5:02 PM, Remko Kloosterman wrote: > Lot's of experience and suggestions here. Thanks. > > I believe it should be theoretically possible to have blip-free RTP > streaming through the appliance. Most windows ethernet drivers allow for > QoS packet scheduling. If the VMware network bridge driver honors this > and syncs the buffers at 20ms frames (or whatever frame size applies) > you should be able to schale up a bit and maintain low jitter. > > Anyone knows how the VMware network bridge exactly works? > > > -Oorspronkelijk bericht- > Van: freeswitch-users-boun...@lists.freeswitch.org > [mailto:freeswitch-users-boun...@lists.freeswitch.org] Namens Gregory > Boehnlein > Verzonden: donderdag 15 januari 2009 21:37 > Aan: freeswitch-users@lists.freeswitch.org > Onderwerp: Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality > >> To the contrary, we have had quite good results in virtualized >> environments and you don't really need timing that is that accurate to > >> make it work. > > If you don't handle RTP, I'm sure it is amazing. However, if you have to > do voicemail, stream audio from the server or do any kind of actual > time/latency/jitter sensitive processing, I don't care how much you tune > your hypervisor, it's never going to scale. > >> We work quite well on amazon EC2 for example. There are 2 issues I >> know about with vmware, 1 is you need to set a setting on the host to >> extend somewhat sane clocks being available, the second is I have seen > >> issues with the bridged network adapter actually doubling up all >> packets causing very strange issues, I suggest not using bridged >> networking if you experience this. > > I've not seen this behavior on Vmware ESX 3.5u2. Maybe an issue on > Vmware Server or Workstation? > > > ___ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > > ___ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > -- Kristian Kielhofner http://blog.krisk.org http://www.submityoursip.com http://www.astlinux.org http://www.star2star.com ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality
Lot's of experience and suggestions here. Thanks. I believe it should be theoretically possible to have blip-free RTP streaming through the appliance. Most windows ethernet drivers allow for QoS packet scheduling. If the VMware network bridge driver honors this and syncs the buffers at 20ms frames (or whatever frame size applies) you should be able to schale up a bit and maintain low jitter. Anyone knows how the VMware network bridge exactly works? -Oorspronkelijk bericht- Van: freeswitch-users-boun...@lists.freeswitch.org [mailto:freeswitch-users-boun...@lists.freeswitch.org] Namens Gregory Boehnlein Verzonden: donderdag 15 januari 2009 21:37 Aan: freeswitch-users@lists.freeswitch.org Onderwerp: Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality > To the contrary, we have had quite good results in virtualized > environments and you don't really need timing that is that accurate to > make it work. If you don't handle RTP, I'm sure it is amazing. However, if you have to do voicemail, stream audio from the server or do any kind of actual time/latency/jitter sensitive processing, I don't care how much you tune your hypervisor, it's never going to scale. > We work quite well on amazon EC2 for example. There are 2 issues I > know about with vmware, 1 is you need to set a setting on the host to > extend somewhat sane clocks being available, the second is I have seen > issues with the bridged network adapter actually doubling up all > packets causing very strange issues, I suggest not using bridged > networking if you experience this. I've not seen this behavior on Vmware ESX 3.5u2. Maybe an issue on Vmware Server or Workstation? ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality
> On that note the OpenVZ instances could live migrate from box to box > without dropping calls and usually had a small acceptable blip in > audio. OpenVZ is not a hypervisor. It essentially runs all of it's applications natively on the CPU. I would expect that it would work under OpenVZ or other container based (chrooted / jailed setups) well. ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality
On Jan 15, 2009, at 1:02 PM, Brian West wrote: > On that note the OpenVZ instances could live migrate from box to box > without dropping calls and usually had a small acceptable blip in > audio. > I'd say a small blip is quite acceptable compared to the alternative! -MC > /b > > On Jan 15, 2009, at 2:59 PM, Gregory Boehnlein wrote: > >>> We have people running FreeSWITCH in vmware and xen with media and >>> considerable load and it doesn't have a problem. We also work very >>> well inside OpenVZ. >> >> I'd be very interested in seeing that, and knowing how it was don > > > ___ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality
On that note the OpenVZ instances could live migrate from box to box without dropping calls and usually had a small acceptable blip in audio. /b On Jan 15, 2009, at 2:59 PM, Gregory Boehnlein wrote: >> We have people running FreeSWITCH in vmware and xen with media and >> considerable load and it doesn't have a problem. We also work very >> well inside OpenVZ. > > I'd be very interested in seeing that, and knowing how it was don ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality
Ok if can summarize a little of the intention of releasing this VMWare image. Its really there so you guys can get it and check it out. I personally don't believe in running such services on a virtual machine (too many nightmare stories from the 'day job' from such things) However, for testing and developing applications that ride on top of FreeSWITCH, this is a quick way to get up and running. Remember Voice application especially where you are interacting with the media streams will be affected by latency and jitter much more readily then store and forward things like IRC, Web, eMail and instant messaging. K On 1/15/09 2:12 PM, "Gregory Boehnlein" wrote: > That won't eliminate the problem. Just reduce the possibility of it > happening. > > Trust me... I've got a large ESX infrastructure, and there is no way that a > software based Voice platform is going to provide skip free audio in a > virtualized environment. ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality
> We have people running FreeSWITCH in vmware and xen with media and > considerable load and it doesn't have a problem. We also work very > well inside OpenVZ. I'd be very interested in seeing that, and knowing how it was done. ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality
We have people running FreeSWITCH in vmware and xen with media and considerable load and it doesn't have a problem. We also work very well inside OpenVZ. /b On Jan 15, 2009, at 2:37 PM, Gregory Boehnlein wrote: > If you don't handle RTP, I'm sure it is amazing. However, if you > have to do > voicemail, stream audio from the server or do any kind of actual > time/latency/jitter sensitive processing, I don't care how much you > tune > your hypervisor, it's never going to scale. ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality
> To the contrary, we have had quite good results in virtualized > environments and you don't really need timing that is that accurate to > make it work. If you don't handle RTP, I'm sure it is amazing. However, if you have to do voicemail, stream audio from the server or do any kind of actual time/latency/jitter sensitive processing, I don't care how much you tune your hypervisor, it's never going to scale. > We work quite well on amazon EC2 for example. There > are 2 issues I know about with vmware, 1 is you need to set a setting > on the host to extend somewhat sane clocks being available, the second > is I have seen issues with the bridged network adapter actually > doubling up all packets causing very strange issues, I suggest not > using bridged networking if you experience this. I've not seen this behavior on Vmware ESX 3.5u2. Maybe an issue on Vmware Server or Workstation? ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality
To the contrary, we have had quite good results in virtualized environments and you don't really need timing that is that accurate to make it work. We work quite well on amazon EC2 for example. There are 2 issues I know about with vmware, 1 is you need to set a setting on the host to extend somewhat sane clocks being available, the second is I have seen issues with the bridged network adapter actually doubling up all packets causing very strange issues, I suggest not using bridged networking if you experience this. Mike On Jan 15, 2009, at 3:12 PM, Gregory Boehnlein wrote: > That won't eliminate the problem. Just reduce the possibility of it > happening. > > Trust me... I've got a large ESX infrastructure, and there is no way > that a > software based Voice platform is going to provide skip free audio in a > virtualized environment. > >> -Original Message- >> From: freeswitch-dev-boun...@lists.freeswitch.org [mailto:freeswitch- >> dev-boun...@lists.freeswitch.org] On Behalf Of Ken Rice >> Sent: Thursday, January 15, 2009 12:15 PM >> To: freeswitch-users@lists.freeswitch.org; Remko Kloosterman; >> freeswitch-...@lists.freeswitch.org >> Subject: Re: [Freeswitch-dev] [Freeswitch-users] VMWare voice quality >> >> On 1/15/09 11:01 AM, "Remko Kloosterman" >> wrote: >> >>> Hello Ken, hello all, >>> >>> I just read about the FreeSWITCH VMware applicance. I'm curious >>> about >>> your experiences with the audio quality on VMWare, so here's a new >>> thread. >>> >>> I've installed freeswitch on VMware Server for Windows. The IVR >>> audio >>> always plays choppy, while the server itself has no performance >> issues. >>> The same poor voice quality also goes for Asterisk or Yate, even >>> on a >>> very fast VMware ESX system. >>> >>> Did you experience the same and/or do you have pointers on how to >>> troubleshoot and fix this? >> >> >> There is a high resolution timer you need to enable on vmware... I'm >> not >> familiar enuff with all the versions of vmware to advise there that >> switch >> is, but they have a couple of articles on it in their knowledge base >> >> ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality
That won't eliminate the problem. Just reduce the possibility of it happening. Trust me... I've got a large ESX infrastructure, and there is no way that a software based Voice platform is going to provide skip free audio in a virtualized environment. > -Original Message- > From: freeswitch-dev-boun...@lists.freeswitch.org [mailto:freeswitch- > dev-boun...@lists.freeswitch.org] On Behalf Of Ken Rice > Sent: Thursday, January 15, 2009 12:15 PM > To: freeswitch-users@lists.freeswitch.org; Remko Kloosterman; > freeswitch-...@lists.freeswitch.org > Subject: Re: [Freeswitch-dev] [Freeswitch-users] VMWare voice quality > > On 1/15/09 11:01 AM, "Remko Kloosterman" wrote: > > > Hello Ken, hello all, > > > > I just read about the FreeSWITCH VMware applicance. I'm curious about > > your experiences with the audio quality on VMWare, so here's a new > > thread. > > > > I've installed freeswitch on VMware Server for Windows. The IVR audio > > always plays choppy, while the server itself has no performance > issues. > > The same poor voice quality also goes for Asterisk or Yate, even on a > > very fast VMware ESX system. > > > > Did you experience the same and/or do you have pointers on how to > > troubleshoot and fix this? > > > There is a high resolution timer you need to enable on vmware... I'm > not > familiar enuff with all the versions of vmware to advise there that > switch > is, but they have a couple of articles on it in their knowledge base > > > > ___ > Freeswitch-dev mailing list > freeswitch-...@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev > http://www.freeswitch.org > > -- > This message has been scanned for viruses and > dangerous content by N2Net Mailshield, and is > believed to be clean. ___ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
