Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Ken Rice kr...@freeswitch.org wrote: I know there has already been some discussion on several fronts of atleast getting the core and several other pieces to where they need to be for stateful failover and I'm not sure if its been mentioned here, but sofia is going to be a bit of work and estimates run in the 100K USD range. Now if we could get a get a couple of corporate sponsors to help here it would be great. I agree. Perhaps some of those corporations which are saving money over proprietary solutions by using FreeSWITCH, and who also want the high-availability functionality, would be ideal sponsors for the work. ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Shouldn't be so hard, right? But there's human nature, even if the corporations get to save, like, 100k worth of licenses and hardware and other costs it is very hard to get them to spend a little back to the community which helped to achive that goal. I think my first, very ambitous goal to make the sofia rewrite a personal long-term hobby is not going to be very realistic, because of my daytime job and my private life. But I will try to persuade my employer to make at least a little contribution, as I really want this in FS, maybe we should * make it a bounty project * try to bring in some big sponsors (who are the biggest companys using FS anyway?) * make some awareness in the voip scene?? seriously what else can be done to get the estimated 100k to become reality? I really want this feature in FS! :-) best, -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 31, 2009, at 7:57 AM, Jason White wrote: Ken Rice kr...@freeswitch.org wrote: I know there has already been some discussion on several fronts of atleast getting the core and several other pieces to where they need to be for stateful failover and I'm not sure if its been mentioned here, but sofia is going to be a bit of work and estimates run in the 100K USD range. Now if we could get a get a couple of corporate sponsors to help here it would be great. I agree. Perhaps some of those corporations which are saving money over proprietary solutions by using FreeSWITCH, and who also want the high-availability functionality, would be ideal sponsors for the work. ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch- users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Actually, getting the 100K is probably the easiest part of the battle. The problem will come in with the strings attached to the 100K. I could persuade my company to give FS devs the money, but there would be conditions, such as:- There would need to be a defined timeline, milestones, and release date- That release cannot be too far in the future (no one will fund an 18 month project any more)- If those dates are missed, penalties will be applied- There will be some of branding/marketing/PR agreement (turning the payment of development into a marketing tool)- There will be all sorts of legal waivers the devs will have to sign (indemnifications, perpetual licenses, etc)From previous OSS experience, the "baggage" that comes with a large payment is usually undesirable to dev groups. Which leave collecting a lot of little payments from many places. Anyone that has worked for some time fund raising for an organization will know how much of a logistical mess that is. Not to mention people that commit to pay, then back out.One good way to get funding for a project like this is to target the government or non-industry related corporate entities. Both cases you target groups that don't compete in your market space (or rely on your competitors), but would benefit from the outcome of the project. An example would be Dell. Dell would not see FS as competition in any way, but if the case was made that adding feature X would increase sales of Dell computers in the telcom space, or that Company X would buy Dells to run FS if this features existed, then Dell could look at investing in the project as a marketing expense.There are down sides with this method. Devs need to walk a fine line with the corporation, ensuring that their product does not conflict with some corporate initiative or large client. Many times you will be partnering with VARS or SIs to present the proposal, as there has to be a project to justify the work. Also, expect a long sales cycle (5+ months). And, some purists would argue that the app is not FOSS once your get some large corp to fund all/part of it. Other are just content with getting the job done. I offer my help in this area if the devs are interested in exploring this style of fund raising. -pete Original Message Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing From: Raimund Sacherer r...@runsolutions.com Date: Mon, August 31, 2009 12:04 am To: freeswitch-users@lists.freeswitch.org Shouldn't be so hard, right? But there's human nature, even if the corporations get to save, like, 100k worth of licenses and hardware and other costs it is very hard to get them to spend a little back to the community which helped to achive that goal. I think my first, very ambitous goal to make the sofia rewrite a personal long-term hobby is not going to be very realistic, because of my daytime job and my private life. But I will try to persuade my employer to make at least a little contribution, as I really want this in FS, maybe we should * make it a bounty project * try to bring in some big sponsors (who are the biggest companys using FS anyway?) * make some awareness in the voip scene?? seriously what else can be done to get the estimated 100k to become reality? I really want this feature in FS! :-) best, -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 31, 2009, at 7:57 AM, Jason White wrote: Ken Rice kr...@freeswitch.org wrote: I know there has already been some discussion on several fronts of atleast getting the core and several other pieces to where they need to be for stateful failover and I'm not sure if its been mentioned here, but sofia is going to be a bit of work and estimates run in the 100K USD range. Now if we could get a get a couple of corporate sponsors to help here it would be great. I agree. Perhaps some of those corporations which are saving money over proprietary solutions by using FreeSWITCH, and who also want the high-availability functionality, would be ideal sponsors for the work. ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch- users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.o
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Hello Pete, wow, i like it when you say getting the money is the easiest part! I think if you are able to help procure the money we should have, as Jim Burke indicated, a rather good brainstorming session behind us. I will create today a Wiki Page where we can define the Project, break up the development parts, generelly layout what has to be done and what will be the best and cleanest way to do it, as well define what and when to failover, etc. If we have a pretty clear Idea on what will be the best way, you can get and fetch the money :-) Seriously, i think we should discuss the possibility with Anthony how, if you/the community can come up with the money, this could be integrated into the development cycle! Let's keep this rolling, best -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 31, 2009, at 12:33 PM, Pete Mueller wrote: Actually, getting the 100K is probably the easiest part of the battle. The problem will come in with the strings attached to the 100K. I could persuade my company to give FS devs the money, but there would be conditions, such as: - There would need to be a defined timeline, milestones, and release date - That release cannot be too far in the future (no one will fund an 18 month project any more) - If those dates are missed, penalties will be applied - There will be some of branding/marketing/PR agreement (turning the payment of development into a marketing tool) - There will be all sorts of legal waivers the devs will have to sign (indemnifications, perpetual licenses, etc) From previous OSS experience, the baggage that comes with a large payment is usually undesirable to dev groups. Which leave collecting a lot of little payments from many places. Anyone that has worked for some time fund raising for an organization will know how much of a logistical mess that is. Not to mention people that commit to pay, then back out. One good way to get funding for a project like this is to target the government or non-industry related corporate entities. Both cases you target groups that don't compete in your market space (or rely on your competitors), but would benefit from the outcome of the project. An example would be Dell. Dell would not see FS as competition in any way, but if the case was made that adding feature X would increase sales of Dell computers in the telcom space, or that Company X would buy Dells to run FS if this features existed, then Dell could look at investing in the project as a marketing expense. There are down sides with this method. Devs need to walk a fine line with the corporation, ensuring that their product does not conflict with some corporate initiative or large client. Many times you will be partnering with VARS or SIs to present the proposal, as there has to be a project to justify the work. Also, expect a long sales cycle (5+ months). And, some purists would argue that the app is not FOSS once your get some large corp to fund all/part of it. Other are just content with getting the job done. I offer my help in this area if the devs are interested in exploring this style of fund raising. -pete Original Message Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing From: Raimund Sacherer r...@runsolutions.com Date: Mon, August 31, 2009 12:04 am To: freeswitch-users@lists.freeswitch.org Shouldn't be so hard, right? But there's human nature, even if the corporations get to save, like, 100k worth of licenses and hardware and other costs it is very hard to get them to spend a little back to the community which helped to achive that goal. I think my first, very ambitous goal to make the sofia rewrite a personal long-term hobby is not going to be very realistic, because of my daytime job and my private life. But I will try to persuade my employer to make at least a little contribution, as I really want this in FS, maybe we should * make it a bounty project * try to bring in some big sponsors (who are the biggest companys using FS anyway?) * make some awareness in the voip scene?? seriously what else can be done to get the estimated 100k to become reality? I really want this feature in FS! :-) best, -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 31, 2009, at 7:57 AM, Jason White wrote: Ken Rice kr...@freeswitch.org wrote: I know there has already been some discussion on several fronts of atleast getting the core and several other pieces to where they need to be for stateful failover and I'm not sure if its been mentioned here, but sofia is going to be a bit of work and estimates run in the 100K USD range. Now if we could get a get a couple of corporate sponsors
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
A random thought. It strikes me that adding support for live call failover to FreeSWITCH might well be the wrong thing to do - it's something which is probably not too difficult to achieve if starting from the ground up, but which is going to be a right PITA to do retrospectively. And, once you've done switching, there's conferencing, IVR stuff... Here's an alternative approach. Have two FS boxes (which I'll call FS1 and FS2 - you can't fault *my* imagination) and a front-end B2BUA/RTP proxy with a bit of a difference. The B2BUA sends incoming stuff to both FS1 and FS2 as two separate call legs, and the RTP proxy forwards inbound RTP to both FS1 and FS2. Assuming that FS1 is the current master, it forwards RTP and call state transitions from FS1 back to the caller. If it detects that FS1 has gone away, then - as FS2 should be in roughly the same state as FS1 - it flips to using FS2 as its source. The internal state of the B2BUA/RTP proxy should be relatively simple, and therefore should be able to be duplicated to a hot standby. I can see a collection of gotchas, such as registration (the B2BUA needs access to the user database), and I haven't even begun to try to think of how to deal with calls *from* the FS boxes. Actually, that's not true: I have, and they clearly need to go through the same sort of mechanism in reverse, which is fine until I start trying to work out how to pair up calls from FS1 and FS2 reliably, and then my head hurts. But I think that this idea has merit. Firstly, it abstracts the failover mechanism away into one place; secondly, it is general - FS1 and FS2 could just as easily be Asterisk1 and Asterisk2. And I'll do it for just $98,650.. --Dave Your forgetting that with RTP theres the duplication of RTP that may be required along with a mechanism that keeps media timers working properly... Where many things don't support RTP timers they sure come in handy on a regular basis especially dealing with several of the other software based platforms out there that just love to have calls end but never want to send a BYE From: Jim Burke j...@evolutiontel.net Reply-To: freeswitch-users@lists.freeswitch.org Date: Mon, 31 Aug 2009 12:54:50 +1000 To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing Conceptually in a pure SIP installation of Freeswitch, hot standby or active standby could be achieved through duplication of messages (INVITE, 200OK and BYE for calls) between 2 boxes and then using VRRP to change the IP address when the box falls over. Going this way allows call state to be as up to date as possible. Obviously this would require logic added to sofia to transfer the RTP port information and UUID information between the FS instances (Easily achieved using additional SIP headers). It would also require that sofia does not forward duplicated messages to gateways or user agents. CDR information would also need to be generated correctly (My experience of Radius with Opensips is that it generates start and stop messages on INVITE/200OK then BYE respectively so this could be an easy solution to the CDR issue). Regards, On Sun, Aug 30, 2009 at 11:19 PM, Raimund Sachererr...@runsolutions.com wrote: So, the first way would be to have a look on the sip stack, which is, in fact, sofia ..., well, sound's like a nice, fun, long-going hobby project to me :-) I will definitly at least look at the sip code to check if it is something i would myself willingly give over to ... But I *really* do want a setup where it does not matter if one specific FS instance is UP or NOT ... -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 29, 2009, at 10:04 PM, Brian West wrote: The sip stack needs to be modified to spin that data up into the state machine so that it can take over calls once the fail over takes place... its not an easy task. /b On Aug 29, 2009, at 2:56 PM, Mitul Limbani wrote: Is itnpossible to have a db cluster know the state of each and every call and then use Heartbeat on this db + fs cluster so that clients see only one ip where as internally all fs boxes refer db for call states, db again is under replication. This in the thioery can be written, but I am sure if we think bit more on this direction the problem seem to be getting addressed. Other guys also chip in their 2 cents, we just need 50 of em to make a full dollar. Thanks Regards, Mitul Limbani, Founder CEO, Enterux Solutions Pvt. Ltd., The Enterprise Linux Company (r), http://www.enterux.com http://www.entVoice.com ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Hi David, On Aug 31, 2009, at 2:22 PM, David Knell wrote: A random thought. It strikes me that adding support for live call failover to FreeSWITCH might well be the wrong thing to do - it's something which is probably not too difficult to achieve if starting from the ground up, but which is going to be a right PITA to do retrospectively. And, once you've done switching, there's conferencing, IVR stuff... Here's an alternative approach. Have two FS boxes (which I'll call FS1 and FS2 - you can't fault *my* imagination) and a front-end B2BUA/RTP proxy with a bit of a difference. The B2BUA sends incoming stuff to both FS1 and FS2 as two separate call legs, and the RTP proxy forwards inbound RTP to both FS1 and FS2. Assuming that FS1 is the current master, it forwards RTP and call state transitions from FS1 back to the caller. If it detects that FS1 has gone away, then - as FS2 should be in roughly the same state as FS1 - it flips to using FS2 as its source. The internal state of the B2BUA/RTP proxy should be relatively simple, and therefore should be able to be duplicated to a hot standby. I can see a collection of gotchas, such as registration (the B2BUA needs access to the user database), and I haven't even begun to try to think of how to deal with calls *from* the FS boxes. Actually, that's not true: I have, and they clearly need to go through the same sort of mechanism in reverse, which is fine until I start trying to work out how to pair up calls from FS1 and FS2 reliably, and then my head hurts. Yeah, sounds like a real challenge But I think that this idea has merit. Firstly, it abstracts the failover mechanism away into one place; secondly, it is general - FS1 and FS2 could just as easily be Asterisk1 and Asterisk2. this, in theory, sound quite interesting ... would be nice to have a proof of concept setup thought. - Sound's not quite THAT nice than a full-built-in solution to FS ... but if it works ... And I'll do it for just $98,650.. Haha, that's really funny :-) --Dave Your forgetting that with RTP theres the duplication of RTP that may be required along with a mechanism that keeps media timers working properly... Where many things don't support RTP timers they sure come in handy on a regular basis especially dealing with several of the other software based platforms out there that just love to have calls end but never want to send a BYE From: Jim Burke j...@evolutiontel.net Reply-To: freeswitch-users@lists.freeswitch.org Date: Mon, 31 Aug 2009 12:54:50 +1000 To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing Conceptually in a pure SIP installation of Freeswitch, hot standby or active standby could be achieved through duplication of messages (INVITE, 200OK and BYE for calls) between 2 boxes and then using VRRP to change the IP address when the box falls over. Going this way allows call state to be as up to date as possible. Obviously this would require logic added to sofia to transfer the RTP port information and UUID information between the FS instances (Easily achieved using additional SIP headers). It would also require that sofia does not forward duplicated messages to gateways or user agents. CDR information would also need to be generated correctly (My experience of Radius with Opensips is that it generates start and stop messages on INVITE/200OK then BYE respectively so this could be an easy solution to the CDR issue). Regards, On Sun, Aug 30, 2009 at 11:19 PM, Raimund Sachererr...@runsolutions.com wrote: So, the first way would be to have a look on the sip stack, which is, in fact, sofia ..., well, sound's like a nice, fun, long-going hobby project to me :-) I will definitly at least look at the sip code to check if it is something i would myself willingly give over to ... But I *really* do want a setup where it does not matter if one specific FS instance is UP or NOT ... -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 29, 2009, at 10:04 PM, Brian West wrote: The sip stack needs to be modified to spin that data up into the state machine so that it can take over calls once the fail over takes place... its not an easy task. /b On Aug 29, 2009, at 2:56 PM, Mitul Limbani wrote: Is itnpossible to have a db cluster know the state of each and every call and then use Heartbeat on this db + fs cluster so that clients see only one ip where as internally all fs boxes refer db for call states, db again is under replication. This in the thioery can be written, but I am sure if we think bit more on this direction the problem seem to be getting addressed. Other guys also chip in their 2 cents, we just need 50 of em to make a full
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
And I'll do it for just $98,650.. Haha, that's really funny :-) Dave is talking New Zealand Dollars ;) On Mon, Aug 31, 2009 at 10:36 PM, Raimund Sachererr...@runsolutions.com wrote: Hi David, On Aug 31, 2009, at 2:22 PM, David Knell wrote: A random thought. It strikes me that adding support for live call failover to FreeSWITCH might well be the wrong thing to do - it's something which is probably not too difficult to achieve if starting from the ground up, but which is going to be a right PITA to do retrospectively. And, once you've done switching, there's conferencing, IVR stuff... Here's an alternative approach. Have two FS boxes (which I'll call FS1 and FS2 - you can't fault *my* imagination) and a front-end B2BUA/RTP proxy with a bit of a difference. The B2BUA sends incoming stuff to both FS1 and FS2 as two separate call legs, and the RTP proxy forwards inbound RTP to both FS1 and FS2. Assuming that FS1 is the current master, it forwards RTP and call state transitions from FS1 back to the caller. If it detects that FS1 has gone away, then - as FS2 should be in roughly the same state as FS1 - it flips to using FS2 as its source. The internal state of the B2BUA/RTP proxy should be relatively simple, and therefore should be able to be duplicated to a hot standby. I can see a collection of gotchas, such as registration (the B2BUA needs access to the user database), and I haven't even begun to try to think of how to deal with calls *from* the FS boxes. Actually, that's not true: I have, and they clearly need to go through the same sort of mechanism in reverse, which is fine until I start trying to work out how to pair up calls from FS1 and FS2 reliably, and then my head hurts. Yeah, sounds like a real challenge But I think that this idea has merit. Firstly, it abstracts the failover mechanism away into one place; secondly, it is general - FS1 and FS2 could just as easily be Asterisk1 and Asterisk2. this, in theory, sound quite interesting ... would be nice to have a proof of concept setup thought. - Sound's not quite THAT nice than a full-built-in solution to FS ... but if it works ... And I'll do it for just $98,650.. Haha, that's really funny :-) --Dave Your forgetting that with RTP theres the duplication of RTP that may be required along with a mechanism that keeps media timers working properly... Where many things don't support RTP timers they sure come in handy on a regular basis especially dealing with several of the other software based platforms out there that just love to have calls end but never want to send a BYE From: Jim Burke j...@evolutiontel.net Reply-To: freeswitch-users@lists.freeswitch.org Date: Mon, 31 Aug 2009 12:54:50 +1000 To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing Conceptually in a pure SIP installation of Freeswitch, hot standby or active standby could be achieved through duplication of messages (INVITE, 200OK and BYE for calls) between 2 boxes and then using VRRP to change the IP address when the box falls over. Going this way allows call state to be as up to date as possible. Obviously this would require logic added to sofia to transfer the RTP port information and UUID information between the FS instances (Easily achieved using additional SIP headers). It would also require that sofia does not forward duplicated messages to gateways or user agents. CDR information would also need to be generated correctly (My experience of Radius with Opensips is that it generates start and stop messages on INVITE/200OK then BYE respectively so this could be an easy solution to the CDR issue). Regards, On Sun, Aug 30, 2009 at 11:19 PM, Raimund Sachererr...@runsolutions.com wrote: So, the first way would be to have a look on the sip stack, which is, in fact, sofia ..., well, sound's like a nice, fun, long-going hobby project to me :-) I will definitly at least look at the sip code to check if it is something i would myself willingly give over to ... But I *really* do want a setup where it does not matter if one specific FS instance is UP or NOT ... -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 29, 2009, at 10:04 PM, Brian West wrote: The sip stack needs to be modified to spin that data up into the state machine so that it can take over calls once the fail over takes place... its not an easy task. /b On Aug 29, 2009, at 2:56 PM, Mitul Limbani wrote: Is itnpossible to have a db cluster know the state of each and every call and then use Heartbeat on this db + fs cluster so that clients see only one ip where as internally all fs boxes refer db for call states, db again is under replication. This in the thioery can be written, but I am sure if we think bit
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Actually Dave, our idea's are very similar. The difference being is that you fork the calls into 2 separate legs at a B2BUA and sending each to an FS box, while I would duplicate messages to an FS box acting in Standby mode using extra SIP headers to let the standby machine know the RTP port number and the UUID. These 2 FS boxes would essentially become the core switching units in Active/Standby configuration and be used to handle switching logic and billing. Thus if FS2 became the Active box, it would have exactly the same call information as FS1 had. Theoretically there would nothing stopping FS1 and FS2 being master and standby for each other at the same time and both handling traffic, although this has the potential to cause over load if something did fail. i.e FS1 receives INVITE, handles call and forwards INVITE to Gateway and then sends a second INVITE (Duplication of Original message, utilising extra SIP headers to pass on required information) to FS2. FS1 fails FS2 takes over and handles call. Then you would look at way's to distribute functionality across other boxes to allow scalability. An example would be to use dedicated FS boxes for TDM cards and Opensips to handle registrations, SIP proxy or SBC controllers. Of course this assumes you are looking to build a softswitch that can handle large call numbers and have the $$$ to do so. Sadly, I have niether the money or the requirement. On Mon, Aug 31, 2009 at 10:22 PM, David Knelld...@3c.co.uk wrote: A random thought. It strikes me that adding support for live call failover to FreeSWITCH might well be the wrong thing to do - it's something which is probably not too difficult to achieve if starting from the ground up, but which is going to be a right PITA to do retrospectively. And, once you've done switching, there's conferencing, IVR stuff... Here's an alternative approach. Have two FS boxes (which I'll call FS1 and FS2 - you can't fault *my* imagination) and a front-end B2BUA/RTP proxy with a bit of a difference. The B2BUA sends incoming stuff to both FS1 and FS2 as two separate call legs, and the RTP proxy forwards inbound RTP to both FS1 and FS2. Assuming that FS1 is the current master, it forwards RTP and call state transitions from FS1 back to the caller. If it detects that FS1 has gone away, then - as FS2 should be in roughly the same state as FS1 - it flips to using FS2 as its source. The internal state of the B2BUA/RTP proxy should be relatively simple, and therefore should be able to be duplicated to a hot standby. I can see a collection of gotchas, such as registration (the B2BUA needs access to the user database), and I haven't even begun to try to think of how to deal with calls *from* the FS boxes. Actually, that's not true: I have, and they clearly need to go through the same sort of mechanism in reverse, which is fine until I start trying to work out how to pair up calls from FS1 and FS2 reliably, and then my head hurts. But I think that this idea has merit. Firstly, it abstracts the failover mechanism away into one place; secondly, it is general - FS1 and FS2 could just as easily be Asterisk1 and Asterisk2. And I'll do it for just $98,650.. --Dave Your forgetting that with RTP theres the duplication of RTP that may be required along with a mechanism that keeps media timers working properly... Where many things don't support RTP timers they sure come in handy on a regular basis especially dealing with several of the other software based platforms out there that just love to have calls end but never want to send a BYE From: Jim Burke j...@evolutiontel.net Reply-To: freeswitch-users@lists.freeswitch.org Date: Mon, 31 Aug 2009 12:54:50 +1000 To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing Conceptually in a pure SIP installation of Freeswitch, hot standby or active standby could be achieved through duplication of messages (INVITE, 200OK and BYE for calls) between 2 boxes and then using VRRP to change the IP address when the box falls over. Going this way allows call state to be as up to date as possible. Obviously this would require logic added to sofia to transfer the RTP port information and UUID information between the FS instances (Easily achieved using additional SIP headers). It would also require that sofia does not forward duplicated messages to gateways or user agents. CDR information would also need to be generated correctly (My experience of Radius with Opensips is that it generates start and stop messages on INVITE/200OK then BYE respectively so this could be an easy solution to the CDR issue). Regards, On Sun, Aug 30, 2009 at 11:19 PM, Raimund Sachererr...@runsolutions.com wrote: So, the first way would be to have a look on the sip stack, which is, in fact, sofia ..., well, sound's like a nice, fun, long-going hobby project to me :-) I
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
So, the first way would be to have a look on the sip stack, which is, in fact, sofia ..., well, sound's like a nice, fun, long-going hobby project to me :-) I will definitly at least look at the sip code to check if it is something i would myself willingly give over to ... But I *really* do want a setup where it does not matter if one specific FS instance is UP or NOT ... -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 29, 2009, at 10:04 PM, Brian West wrote: The sip stack needs to be modified to spin that data up into the state machine so that it can take over calls once the fail over takes place... its not an easy task. /b On Aug 29, 2009, at 2:56 PM, Mitul Limbani wrote: Is itnpossible to have a db cluster know the state of each and every call and then use Heartbeat on this db + fs cluster so that clients see only one ip where as internally all fs boxes refer db for call states, db again is under replication. This in the thioery can be written, but I am sure if we think bit more on this direction the problem seem to be getting addressed. Other guys also chip in their 2 cents, we just need 50 of em to make a full dollar. Thanks Regards, Mitul Limbani, Founder CEO, Enterux Solutions Pvt. Ltd., The Enterprise Linux Company (r), http://www.enterux.com http://www.entVoice.com ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch- users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
High Availability is always an interesting problem. PSTN equipment generally overcomes it through duplication of messaging and hardware. Obviously at an E1 (T1 for the US folks) or Subscribers Line Level this is not possible. This duplication comes at a cost and lowers the calls per minute a softswitch can handle for the amount of hardware purchased. Like everything it is a trade-off between cost and reliability. Per Dave's comment, duplication on a network level can be achieved easily through duplicating network switches and using OSPF, VRRP or network bonding to ensure continuity if a path is compromised. From experience this is exactly what the vendors do and it does work. During failover the calls experience a minut breakup of voice quality but they still hold up ok. Conceptually in a pure SIP installation of Freeswitch, hot standby or active standby could be achieved through duplication of messages (INVITE, 200OK and BYE for calls) between 2 boxes and then using VRRP to change the IP address when the box falls over. Going this way allows call state to be as up to date as possible. Obviously this would require logic added to sofia to transfer the RTP port information and UUID information between the FS instances (Easily achieved using additional SIP headers). It would also require that sofia does not forward duplicated messages to gateways or user agents. CDR information would also need to be generated correctly (My experience of Radius with Opensips is that it generates start and stop messages on INVITE/200OK then BYE respectively so this could be an easy solution to the CDR issue). Regards, On Sun, Aug 30, 2009 at 11:19 PM, Raimund Sachererr...@runsolutions.com wrote: So, the first way would be to have a look on the sip stack, which is, in fact, sofia ..., well, sound's like a nice, fun, long-going hobby project to me :-) I will definitly at least look at the sip code to check if it is something i would myself willingly give over to ... But I *really* do want a setup where it does not matter if one specific FS instance is UP or NOT ... -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 29, 2009, at 10:04 PM, Brian West wrote: The sip stack needs to be modified to spin that data up into the state machine so that it can take over calls once the fail over takes place... its not an easy task. /b On Aug 29, 2009, at 2:56 PM, Mitul Limbani wrote: Is itnpossible to have a db cluster know the state of each and every call and then use Heartbeat on this db + fs cluster so that clients see only one ip where as internally all fs boxes refer db for call states, db again is under replication. This in the thioery can be written, but I am sure if we think bit more on this direction the problem seem to be getting addressed. Other guys also chip in their 2 cents, we just need 50 of em to make a full dollar. Thanks Regards, Mitul Limbani, Founder CEO, Enterux Solutions Pvt. Ltd., The Enterprise Linux Company (r), http://www.enterux.com http://www.entVoice.com ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Jim Burke Director Evolutiontel. http://www.evolutiontel.net ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Your forgetting that with RTP theres the duplication of RTP that may be required along with a mechanism that keeps media timers working properly... Where many things don't support RTP timers they sure come in handy on a regular basis especially dealing with several of the other software based platforms out there that just love to have calls end but never want to send a BYE From: Jim Burke j...@evolutiontel.net Reply-To: freeswitch-users@lists.freeswitch.org Date: Mon, 31 Aug 2009 12:54:50 +1000 To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing Conceptually in a pure SIP installation of Freeswitch, hot standby or active standby could be achieved through duplication of messages (INVITE, 200OK and BYE for calls) between 2 boxes and then using VRRP to change the IP address when the box falls over. Going this way allows call state to be as up to date as possible. Obviously this would require logic added to sofia to transfer the RTP port information and UUID information between the FS instances (Easily achieved using additional SIP headers). It would also require that sofia does not forward duplicated messages to gateways or user agents. CDR information would also need to be generated correctly (My experience of Radius with Opensips is that it generates start and stop messages on INVITE/200OK then BYE respectively so this could be an easy solution to the CDR issue). Regards, On Sun, Aug 30, 2009 at 11:19 PM, Raimund Sachererr...@runsolutions.com wrote: So, the first way would be to have a look on the sip stack, which is, in fact, sofia ..., well, sound's like a nice, fun, long-going hobby project to me :-) I will definitly at least look at the sip code to check if it is something i would myself willingly give over to ... But I *really* do want a setup where it does not matter if one specific FS instance is UP or NOT ... -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 29, 2009, at 10:04 PM, Brian West wrote: The sip stack needs to be modified to spin that data up into the state machine so that it can take over calls once the fail over takes place... its not an easy task. /b On Aug 29, 2009, at 2:56 PM, Mitul Limbani wrote: Is itnpossible to have a db cluster know the state of each and every call and then use Heartbeat on this db + fs cluster so that clients see only one ip where as internally all fs boxes refer db for call states, db again is under replication. This in the thioery can be written, but I am sure if we think bit more on this direction the problem seem to be getting addressed. Other guys also chip in their 2 cents, we just need 50 of em to make a full dollar. Thanks Regards, Mitul Limbani, Founder CEO, Enterux Solutions Pvt. Ltd., The Enterprise Linux Company (r), http://www.enterux.com http://www.entVoice.com ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Jim Burke Director Evolutiontel. http://www.evolutiontel.net ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Hi Ken, I don't disagree, the scenario you mentioned is just the tip of the iceberg when it comes to getting something like this working smoothly. However to try and cover them all in a single post (assuming that I could) would have resulted in something akin to War and Peace and probably bored the pants off of most readers :) For any HA solution to move from a concept into reality, it would require some serious brainstorming to identify all scenarios and how to handle them. Given that many posts in this thread mentioned transfer of call state at time of failure or some form of virtualisation, I wanted to provide another solution that would provide real time call state updates between FS boxes utilising mechanisms that are largely already in-place and thus (hopefully) reducing software development time if someone was willing to cough up the $. Regards, On Mon, Aug 31, 2009 at 1:23 PM, Ken Ricekr...@freeswitch.org wrote: Your forgetting that with RTP theres the duplication of RTP that may be required along with a mechanism that keeps media timers working properly... Where many things don't support RTP timers they sure come in handy on a regular basis especially dealing with several of the other software based platforms out there that just love to have calls end but never want to send a BYE From: Jim Burke j...@evolutiontel.net Reply-To: freeswitch-users@lists.freeswitch.org Date: Mon, 31 Aug 2009 12:54:50 +1000 To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing Conceptually in a pure SIP installation of Freeswitch, hot standby or active standby could be achieved through duplication of messages (INVITE, 200OK and BYE for calls) between 2 boxes and then using VRRP to change the IP address when the box falls over. Going this way allows call state to be as up to date as possible. Obviously this would require logic added to sofia to transfer the RTP port information and UUID information between the FS instances (Easily achieved using additional SIP headers). It would also require that sofia does not forward duplicated messages to gateways or user agents. CDR information would also need to be generated correctly (My experience of Radius with Opensips is that it generates start and stop messages on INVITE/200OK then BYE respectively so this could be an easy solution to the CDR issue). Regards, On Sun, Aug 30, 2009 at 11:19 PM, Raimund Sachererr...@runsolutions.com wrote: So, the first way would be to have a look on the sip stack, which is, in fact, sofia ..., well, sound's like a nice, fun, long-going hobby project to me :-) I will definitly at least look at the sip code to check if it is something i would myself willingly give over to ... But I *really* do want a setup where it does not matter if one specific FS instance is UP or NOT ... -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 29, 2009, at 10:04 PM, Brian West wrote: The sip stack needs to be modified to spin that data up into the state machine so that it can take over calls once the fail over takes place... its not an easy task. /b On Aug 29, 2009, at 2:56 PM, Mitul Limbani wrote: Is itnpossible to have a db cluster know the state of each and every call and then use Heartbeat on this db + fs cluster so that clients see only one ip where as internally all fs boxes refer db for call states, db again is under replication. This in the thioery can be written, but I am sure if we think bit more on this direction the problem seem to be getting addressed. Other guys also chip in their 2 cents, we just need 50 of em to make a full dollar. Thanks Regards, Mitul Limbani, Founder CEO, Enterux Solutions Pvt. Ltd., The Enterprise Linux Company (r), http://www.enterux.com http://www.entVoice.com ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Jim Burke Director Evolutiontel. http://www.evolutiontel.net ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ FreeSWITCH-users mailing list
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
I was thinking more like something the size of war and peace but written with the style of an RFC (and about as interesting and an RFC) I know there has already been some discussion on several fronts of atleast getting the core and several other pieces to where they need to be for stateful failover and I'm not sure if its been mentioned here, but sofia is going to be a bit of work and estimates run in the 100K USD range. Now if we could get a get a couple of corporate sponsors to help here it would be great. K From: Jim Burke j...@evolutiontel.net Reply-To: freeswitch-users@lists.freeswitch.org Date: Mon, 31 Aug 2009 14:32:22 +1000 To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing Hi Ken, I don't disagree, the scenario you mentioned is just the tip of the iceberg when it comes to getting something like this working smoothly. However to try and cover them all in a single post (assuming that I could) would have resulted in something akin to War and Peace and probably bored the pants off of most readers :) For any HA solution to move from a concept into reality, it would require some serious brainstorming to identify all scenarios and how to handle them. Given that many posts in this thread mentioned transfer of call state at time of failure or some form of virtualisation, I wanted to provide another solution that would provide real time call state updates between FS boxes utilising mechanisms that are largely already in-place and thus (hopefully) reducing software development time if someone was willing to cough up the $. Regards, On Mon, Aug 31, 2009 at 1:23 PM, Ken Ricekr...@freeswitch.org wrote: Your forgetting that with RTP theres the duplication of RTP that may be required along with a mechanism that keeps media timers working properly... Where many things don't support RTP timers they sure come in handy on a regular basis especially dealing with several of the other software based platforms out there that just love to have calls end but never want to send a BYE From: Jim Burke j...@evolutiontel.net Reply-To: freeswitch-users@lists.freeswitch.org Date: Mon, 31 Aug 2009 12:54:50 +1000 To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing Conceptually in a pure SIP installation of Freeswitch, hot standby or active standby could be achieved through duplication of messages (INVITE, 200OK and BYE for calls) between 2 boxes and then using VRRP to change the IP address when the box falls over. Going this way allows call state to be as up to date as possible. Obviously this would require logic added to sofia to transfer the RTP port information and UUID information between the FS instances (Easily achieved using additional SIP headers). It would also require that sofia does not forward duplicated messages to gateways or user agents. CDR information would also need to be generated correctly (My experience of Radius with Opensips is that it generates start and stop messages on INVITE/200OK then BYE respectively so this could be an easy solution to the CDR issue). Regards, On Sun, Aug 30, 2009 at 11:19 PM, Raimund Sachererr...@runsolutions.com wrote: So, the first way would be to have a look on the sip stack, which is, in fact, sofia ..., well, sound's like a nice, fun, long-going hobby project to me :-) I will definitly at least look at the sip code to check if it is something i would myself willingly give over to ... But I *really* do want a setup where it does not matter if one specific FS instance is UP or NOT ... -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 29, 2009, at 10:04 PM, Brian West wrote: The sip stack needs to be modified to spin that data up into the state machine so that it can take over calls once the fail over takes place... its not an easy task. /b On Aug 29, 2009, at 2:56 PM, Mitul Limbani wrote: Is itnpossible to have a db cluster know the state of each and every call and then use Heartbeat on this db + fs cluster so that clients see only one ip where as internally all fs boxes refer db for call states, db again is under replication. This in the thioery can be written, but I am sure if we think bit more on this direction the problem seem to be getting addressed. Other guys also chip in their 2 cents, we just need 50 of em to make a full dollar. Thanks Regards, Mitul Limbani, Founder CEO, Enterux Solutions Pvt. Ltd., The Enterprise Linux Company (r), http://www.enterux.com http://www.entVoice.com ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
On Sat, Aug 29, 2009 at 2:34 PM, Diego Viola diego.vi...@gmail.com wrote: Yes, FreeSWITCH is a system that you can trust 100%. I have switched my Asterisk servers to FreeSWITCH and have peace now. If I were you I would get rid of Asterisk and use FreeSWITCH, FS will handle all what you want very well. And I agree with David, fail-over is kinda irrelevant since the FS doesn't crash like Asterisk does. You still have hardware failures and fail-over is also useful for hit-less maintenance on boxes. I'd be interested to know how Brian West was approaching his live migration work. Steve ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Thinking about it, maybe we can create a solution, if some of us work together: My strength are in virtualization, linux, development, databases, integration, etc. What I do not now much about is how SIP (and everything else for that matter in the Voice world) works under the hood, and how it's implemented in FS. I know that the state information for a call has to be stored and retrieved somewhere and somehow, only I do not know that part. What I know is that it hast to be do-able to store all the stream information (ip's, port's, current state's, etc.) in a very fast database (e.g. my idea would be memcached) so another FS could just take this information and take over the call, maybe you loose a second of voice, maybe you loose the recorded call file or a part of it, but that should be it. (SipFoundry has a boxed opensource PBX, which, of course is not flexible like FreeSWITCH or Asterisk, but has Call Live Migration and Call Live Failover integrated!). What I want is for my company to be able to sell a 99.99 uptime PBX (we do mostly call-center related stuff), which can scale well, and can grow with the company without lot's of hassles, my Dream would be: To begin with: One Hardware Node with the essential hardware (digium cards for example). On this node are OpenVZ virtualized containers: [VirtCnt1: FS which only talks to the Hardware and forwards everything] = Could be replaced with hardware media gateway, etc. [VirtCnt2: FS which handles the PBX] \___ Loadbalanced, with odbc or xml, Failover, Livetakeover [VirtCnt3: FS which handles the PBX] / [VirtCnt4: Database for state information] (maybe something as resource-friendly as memcached? ressource heavvy database?) With this we can achieve all this: Problem with VirtCnt2 (e.g. crash, lock, ...) * VirtCnt3 can take over. - You are free without stress to investigate the problem, you can debug and analyze whyle the machine is still running - you can also create a machine-state-dump of the virtual container, dump the container as well, copy the data to your lab and restore the machine up the state which it was running with the problem, so you can liveinvestigate it in the lab (some prerequirements given, but easy doable) - just think about the possibility of better bugreports because someone can take the time to read out all the data with GDB to investigate the proper cause of a machine Lock! You want to upgrade to a new FreeSWITCH version? * Take VirtCnt2 out of the LoadBalancing Scheme, * Stop it, Clone it, * Upgrade FreeSWITCH in the cloned Container * Start the cloned container * if there's something wrong, stop it and restart the original VirtCnt2 - No problem at all, you can Test on the Live Hardware, with part of the Live users (maybe a low-volume queue) to be sure everything works out fine before you activate the full loadbalance Server on it's own can't handle the load * Buy new machine * Setup Hardware Node * Livemigrate VirtCnt3 (no downtime) Now the first Server with the VrtCnt1 and VirtCnt2 as well has to much load * Buy new machine * Setup Hardware Node * Livemigrate VirtCnt2 (no downtime) - Now you have a 3 server solution (1 mediaprox, 2 loadbalanced / failover PBXes) out of the first box you bought, without headaches, because the system was built for it from the beginning! The Database drains to much? * Buy new machine * Setup Hardware Node * Livemigrate database VirtCnt4 (no downtime) You want to upgrade Hardware/Kernel in Hardware node 1? * Livemigrate VirtCnt2 to a hotstandby machine, or to the other PBX machine, upgrade the hardware, Re-Livemigrate the containers. (no downtime) * OR just break the loadbalancing, wait until all current calls are teared down correctly, upgrade machine, reenable the loadbalancer You want an exact copy of the first server for Hardware HA? * Buy new machine * Setup Hardware node * Buy hardware PRI switchover box * Clone VirtCnt1 - VirtCnt4 to the new machine * Make basic failover configuration - the sky's the limit, as the saying goes ... So, I can do all the openvz stuff and the integration with database / memcached / heartbeat / whatever is needed here, someone there to be willing to work with me on this on the FreeSWITCH side? or at least provide me with the necessary information about what's needed / how to talk / what states from FreeSWITCH? I know this seems very ambitious but if this could be made in a rather relativly easy to setup package, with good documentation, it would be a boost for FreeSWITCH, i am sure, because after all this is what everyone is grown accustomed to from good old phone companys and the good old pbx's: carrier grade uptimes ... Thanks for everyone reading up until here, all the best, Ray -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
I was able to do this using OpenVZ, You can get away with it on smaller instances... like if you're doing one instance per company but don't expect live migration to work as well on large instances with thousands of calls up at once. You need a fast network, fast disks and to follow the howto on the wiki. /b On Aug 29, 2009, at 4:58 AM, Steve Kurzeja wrote: You still have hardware failures and fail-over is also useful for hit-less maintenance on boxes. I'd be interested to know how Brian West was approaching his live migration work. Steve ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
We have previously estimated the development of live fail over (after a box dies where live migration is no longer possible) to exceed 100k in development costs. It requires several additions to the sofia sip library, freeswitch and a dependancy on some other code we would have to implement to manage it. It may or may not be worth it to raise that kind of funding just to avoid an occasional disaster. Then there is a matter of securing the time of the developers necessary to carry out the implementation. On Aug 29, 2009 10:19 AM, Brian West br...@freeswitch.org wrote: I was able to do this using OpenVZ, You can get away with it on smaller instances... like if you're doing one instance per company but don't expect live migration to work as well on large instances with thousands of calls up at once. You need a fast network, fast disks and to follow the howto on the wiki. /b On Aug 29, 2009, at 4:58 AM, Steve Kurzeja wrote: You still have hardware failures and fail-over... ___ FreeSWITCH-users mailing list freeswitch-us...@lists... ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
On PRI circuits failover with ha is possible using a red tone appliance, but I m sure it's going to be almost impossible to transfer the running calls during the event of fs box going down. Thanks Regards, Mitul Limbani, Founder CEO, Enterux Solutions Pvt. Ltd., The Enterprise Linux Company (r), http://www.enterux.com http://www.entVoice.com On 29-Aug-2009, at 11:35 PM, David Knell d...@3c.co.uk wrote: Hi Raimund, The difficult bit in all of this is having calls seamlessly transferred from one box to another when the first box dies. There's a lot of state associated with a call, not all of which is easy to replicate across. 99.99% uptime implies an average of no more than 15 seconds downtime during a 40 hour business week (or about one unscheduled reboot every couple of months), which is easily achievable using FreeSWITCH (and, in my experience, Asterisk) on standard hardware. People agonize about their four or five nines too much, in my opinion. Folk are used to their phones crashing, needing rebooting and dropping calls - we've cellphones to thank for that - and a half-decent VoIP solution will knock the spots off your favourite mobile carrier for reliability. Plus there's all the external factors - 99.999% uptime on your PRI means that someone can only drive a digger through the cable once every 273 years, assuming that it takes a day to fix, and no telco's going to give you that in an SLA. And your power can't go off. And so on. Lastly, I'm afraid that virtualization and VoIP don't play well together, at least not if you want to achieve a sensible density. The large number of small packets being moved around the network interfaces - both physical and virtual - will quickly chew up your CPU. Cheers -- Dave Thinking about it, maybe we can create a solution, if some of us work together: My strength are in virtualization, linux, development, databases, integration, etc. What I do not now much about is how SIP (and everything else for that matter in the Voice world) works under the hood, and how it's implemented in FS. I know that the state information for a call has to be stored and retrieved somewhere and somehow, only I do not know that part. What I know is that it hast to be do-able to store all the stream information (ip's, port's, current state's, etc.) in a very fast database (e.g. my idea would be memcached) so another FS could just take this information and take over the call, maybe you loose a second of voice, maybe you loose the recorded call file or a part of it, but that should be it. (SipFoundry has a boxed opensource PBX, which, of course is not flexible like FreeSWITCH or Asterisk, but has Call Live Migration and Call Live Failover integrated!). What I want is for my company to be able to sell a 99.99 uptime PBX (we do mostly call-center related stuff), which can scale well, and can grow with the company without lot's of hassles, my Dream would be: To begin with: One Hardware Node with the essential hardware (digium cards for example). On this node are OpenVZ virtualized containers: [VirtCnt1: FS which only talks to the Hardware and forwards everything] = Could be replaced with hardware media gateway, etc. [VirtCnt2: FS which handles the PBX] \___ Loadbalanced, with odbc or xml, Failover, Livetakeover [VirtCnt3: FS which handles the PBX] / [VirtCnt4: Database for state information] (maybe something as resource-friendly as memcached? ressource heavvy database?) With this we can achieve all this: Problem with VirtCnt2 (e.g. crash, lock, ...) * VirtCnt3 can take over. - You are free without stress to investigate the problem, you can debug and analyze whyle the machine is still running - you can also create a machine-state-dump of the virtual container, dump the container as well, copy the data to your lab and restore the machine up the state which it was running with the problem, so you can liveinvestigate it in the lab (some prerequirements given, but easy doable) - just think about the possibility of better bugreports because someone can take the time to read out all the data with GDB to investigate the proper cause of a machine Lock! You want to upgrade to a new FreeSWITCH version? * Take VirtCnt2 out of the LoadBalancing Scheme, * Stop it, Clone it, * Upgrade FreeSWITCH in the cloned Container * Start the cloned container * if there's something wrong, stop it and restart the original VirtCnt2 - No problem at all, you can Test on the Live Hardware, with part of the Live users (maybe a low-volume queue) to be sure everything works out fine before you activate the full loadbalance Server on it's own can't handle the load * Buy new machine * Setup Hardware Node * Livemigrate VirtCnt3 (no downtime) Now the first Server with the VrtCnt1 and VirtCnt2 as well has to much load * Buy new machine * Setup Hardware Node * Livemigrate
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Hmm, so basically 100 interested companys which each chip in 1000 bucks :-) sounds like lots of manpower, but on the other hand, I do not know the issues regarding the SIP protocoll, but, basically, isn't it *just* to tell another FS box to listen on port x for voicetraffic, forward it to ip on port y? ok, i understand there's a lot going on under the hood, i guess it would mean to setup a call, but take care to not really set up the call, just the internal state ... hmm, could it theoretically be done with the event system? ok, i guess I have to dive further into the internals to fully understand the scope. But a live migration, where the box is available, be possible right now? Would be a step i would like to implement just to be able to do work on a hardware node if necesary without interrupting the service ... -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 29, 2009, at 6:01 PM, Anthony Minessale wrote: We have previously estimated the development of live fail over (after a box dies where live migration is no longer possible) to exceed 100k in development costs. It requires several additions to the sofia sip library, freeswitch and a dependancy on some other code we would have to implement to manage it. It may or may not be worth it to raise that kind of funding just to avoid an occasional disaster. Then there is a matter of securing the time of the developers necessary to carry out the implementation. On Aug 29, 2009 10:19 AM, Brian West br...@freeswitch.org wrote: I was able to do this using OpenVZ, You can get away with it on smaller instances... like if you're doing one instance per company but don't expect live migration to work as well on large instances with thousands of calls up at once. You need a fast network, fast disks and to follow the howto on the wiki. /b On Aug 29, 2009, at 4:58 AM, Steve Kurzeja wrote: You still have hardware failures and fail-over... ___ FreeSWITCH-users mailing list freeswitch-us...@lists... ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch- users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
It is not possible to do a live migration at all right now... There is no way to move a call and have all the states required for that call to magically re-appear on a different instance. This will require a fair bit of work to get there. It is possible to configure fs via some back end DB magic to share configurations ie: n+1 or ³warm standby² style fault tolerance but you are going to loose the calls that are up when the failure occurs. From: Raimund Sacherer r...@runsolutions.com Reply-To: freeswitch-users@lists.freeswitch.org Date: Sat, 29 Aug 2009 20:33:21 +0200 To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing Hmm, so basically 100 interested companys which each chip in 1000 bucks :-) sounds like lots of manpower, but on the other hand, I do not know the issues regarding the SIP protocoll, but, basically, isn't it *just* to tell another FS box to listen on port x for voicetraffic, forward it to ip on port y? ok, i understand there's a lot going on under the hood, i guess it would mean to setup a call, but take care to not really set up the call, just the internal state ... hmm, could it theoretically be done with the event system? ok, i guess I have to dive further into the internals to fully understand the scope. But a live migration, where the box is available, be possible right now? Would be a step i would like to implement just to be able to do work on a hardware node if necesary without interrupting the service ... -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 29, 2009, at 6:01 PM, Anthony Minessale wrote: We have previously estimated the development of live fail over (after a box dies where live migration is no longer possible) to exceed 100k in development costs. It requires several additions to the sofia sip library, freeswitch and a dependancy on some other code we would have to implement to manage it. It may or may not be worth it to raise that kind of funding just to avoid an occasional disaster. Then there is a matter of securing the time of the developers necessary to carry out the implementation. On Aug 29, 2009 10:19 AM, Brian West br...@freeswitch.org wrote: I was able to do this using OpenVZ, You can get away with it on smaller instances... like if you're doing one instance per company but don't expect live migration to work as well on large instances with thousands of calls up at once. You need a fast network, fast disks and to follow the howto on the wiki. /b On Aug 29, 2009, at 4:58 AM, Steve Kurzeja wrote: You still have hardware failures and fail-over... ___ FreeSWITCH-users mailing list freeswitch-us...@lists... ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Is itnpossible to have a db cluster know the state of each and every call and then use Heartbeat on this db + fs cluster so that clients see only one ip where as internally all fs boxes refer db for call states, db again is under replication. This in the thioery can be written, but I am sure if we think bit more on this direction the problem seem to be getting addressed. Other guys also chip in their 2 cents, we just need 50 of em to make a full dollar. Thanks Regards, Mitul Limbani, Founder CEO, Enterux Solutions Pvt. Ltd., The Enterprise Linux Company (r), http://www.enterux.com http://www.entVoice.com On 30-Aug-2009, at 12:14 AM, Ken Rice kr...@freeswitch.org wrote: It is not possible to do a live migration at all right now... There is no way to move a call and have all the states required for that call to magically re-appear on a different instance. This will require a fair bit of work to get there. It is possible to configure fs via some back end DB magic to share configurations ie: n+1 or “warm standby” style fault tolerance but you are going to loose the calls that are up when the failure oc curs. From: Raimund Sacherer r...@runsolutions.com Reply-To: freeswitch-users@lists.freeswitch.org Date: Sat, 29 Aug 2009 20:33:21 +0200 To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing Hmm, so basically 100 interested companys which each chip in 1000 bucks :-) sounds like lots of manpower, but on the other hand, I do not know the issues regarding the SIP protocoll, but, basically, isn't it *just* to tell another FS box to listen on port x for voicetraffic, forward it to ip on port y? ok, i understand there's a lot going on under the hood, i guess it would mean to setup a call, but take care to not really set up the call, just the internal state ... hmm, could it theoretically be done with the event system? ok, i guess I have to dive further into the internals to fully understand the scope. But a live migration, where the box is available, be possible right now? Would be a step i would like to implement just to be able to do work on a hardware node if necesary without interrupting the service ... -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 29, 2009, at 6:01 PM, Anthony Minessale wrote: We have previously estimated the development of live fail over (after a box dies where live migration is no longer possible) to exceed 100k in development costs. It requires several additions to the sofia sip library, freeswitch and a dependancy on some other code we would have to implement to manage it. It may or may not be worth it to raise that kind of funding just to avoid an occasional disaster. Then there is a matter of securing the time of the developers necessary to carry out the implementation. On Aug 29, 2009 10:19 AM, Brian West br...@freeswitch.org wrote: I was able to do this using OpenVZ, You can get away with it on smaller instances... like if you're doing one instance per company but don't expect live migration to work as well on large instances with thousands of calls up at once. You need a fast network, fast disks and to follow the howto on the wiki. /b On Aug 29, 2009, at 4:58 AM, Steve Kurzeja wrote: You still have hardware failures and fail-over... ___ FreeSWITCH-users mailing list freeswitch-us...@lists... ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch- users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch- users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch- users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
The sip stack needs to be modified to spin that data up into the state machine so that it can take over calls once the fail over takes place... its not an easy task. /b On Aug 29, 2009, at 2:56 PM, Mitul Limbani wrote: Is itnpossible to have a db cluster know the state of each and every call and then use Heartbeat on this db + fs cluster so that clients see only one ip where as internally all fs boxes refer db for call states, db again is under replication. This in the thioery can be written, but I am sure if we think bit more on this direction the problem seem to be getting addressed. Other guys also chip in their 2 cents, we just need 50 of em to make a full dollar. Thanks Regards, Mitul Limbani, Founder CEO, Enterux Solutions Pvt. Ltd., The Enterprise Linux Company (r), http://www.enterux.com http://www.entVoice.com ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
I am also in the process of playing around with FS running inside Xen Virtual Machines with "mirrored" VMs on a second failover system. So far, the initial tests are promising. I can see a 2-3 sec "hiccup" in the net traffic during the live migration of the Xen VM. My calls do not drop, but I will stress that I'm only running test cases at this time, I am not using real world traffic.Once I figure it all out, I'll report it here.-pete Original Message ---- Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing From: Michael Collins m...@freeswitch.org Date: Fri, August 28, 2009 12:37 pm To: freeswitch-users@lists.freeswitch.org On Fri, Aug 28, 2009 at 12:11 PM, Giovanni Maruzzelli gmar...@celliax.org wrote: Usually you don't need to worry about stability issues with FS. For scalability, peoples tend to use openser or some other sip loadbalancer in fron of fs, but you probably would not need that. Live migration of calls is not yet possible, tough. Brian West has done some testing with live migrations but I don't know where he left off. Brian, were you using OpenVZ? I forget... In any case, FS allows you to try to do this with the hope that it will actually work in a production environment. As for the other things - yes, FS can work with the TDM card and the queues, etc. If you are in a position to install FS on a sandbox machine for testing then that would be your best bet. I recommend diving in, which is probably what you did when you first started learning Asterisk... Have fun!-MC ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
I guess I should also mention that Xen is a side-project. When considering this issue for an existing production systems, we chose to put as much HA into hardware as we can. We are not concerned with FS crashing, as so far we've never seen that happen (except when our module caused it :) So for each of our systems:- We have dual NIC cards (onboad NIC + PCI card) both bridged together in case one fails- We have redundant power supplies.- We use Mirrored Solid State Disks for local storage (far better MTBF than HDD, a lot faster too)- All but OS and speed-critical data is stored on a NAS device - We have redundant DBs with Memcache infront for speedAt the same time we chose to use COTS hardware (SuperMicro chassis/MoBo) rather than the big-boys like IBM or Dell. This kept the overall cost per machine low. Initially some were concerned that not having a name like IBM on our servers would be concerning to some potential clients. The solution was to pay a company to deisgn and build a custom face plate for the SuperMicro boxes. Which oddly looks more impressive to clients that a rack full of IBM faceplates. It was suprisingly low cost for the faceplates too.For scalability, OpenSIPS was our choice. There's a very nice tutorial on their website on how to configure Load Balancing.-pete Original Message Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing From: "Pete Mueller" p...@privateconnect.com Date: Sat, August 29, 2009 2:25 pm To: freeswitch-users@lists.freeswitch.org I am also in the process of playing around with FS running inside Xen Virtual Machines with "mirrored" VMs on a second failover system. So far, the initial tests are promising. I can see a 2-3 sec "hiccup" in the net traffic during the live migration of the Xen VM. My calls do not drop, but I will stress that I'm only running test cases at this time, I am not using real world traffic.Once I figure it all out, I'll report it here.-pete Original Message ---- Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing From: Michael Collins m...@freeswitch.org Date: Fri, August 28, 2009 12:37 pm To: freeswitch-users@lists.freeswitch.org On Fri, Aug 28, 2009 at 12:11 PM, Giovanni Maruzzelli gmar...@celliax.org wrote: Usually you don't need to worry about stability issues with FS. For scalability, peoples tend to use openser or some other sip loadbalancer in fron of fs, but you probably would not need that. Live migration of calls is not yet possible, tough. Brian West has done some testing with live migrations but I don't know where he left off. Brian, were you using OpenVZ? I forget... In any case, FS allows you to try to do this with the hope that it will actually work in a production environment. As for the other things - yes, FS can work with the TDM card and the queues, etc. If you are in a position to install FS on a sandbox machine for testing then that would be your best bet. I recommend diving in, which is probably what you did when you first started learning Asterisk... Have fun!-MC ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
This sounds like so many redundancy projects that will probably offer nothing in the real world. On 08/30/2009 05:52 AM, Pete Mueller wrote: I guess I should also mention that Xen is a side-project. When considering this issue for an existing production systems, we chose to put as much HA into hardware as we can. We are not concerned with FS crashing, as so far we've never seen that happen (except when our module caused it :) So for each of our systems: - We have dual NIC cards (onboad NIC + PCI card) both bridged together in case one fails NICs hardly ever fail. Its the wiring which is the vulnerable area. How independent can you make the two wiring paths, when they come from the same box? - We have redundant power supplies. Even with a good UPS, power fails more often than a high quality power supply. Just how independent are the two power sources feeding your two power supplies? Do you have two completely independent UPS sets? Do you have spacially diverse wiring from them? - We use Mirrored Solid State Disks for local storage (far better MTBF than HDD, a lot faster too) My experience so far is that SSD reliability is very poor, with entire drives disappearing, rather than just getting the odd bad sector. I guess to balance this, hard disk drive reliability seems to have plummeted in the last year or so, after several good years. - All but OS and speed-critical data is stored on a NAS device NAS == more wiring. More wiring == more vulnerabilities. Are you sure your setup is a win? NAS tends to help keep the data secure, but it isn't good for reliable access to that data. - We have redundant DBs with Memcache infront for speed At the same time we chose to use COTS hardware (SuperMicro chassis/MoBo) rather than the big-boys like IBM or Dell. This kept the overall cost per machine low. Initially some were concerned that not having a name like IBM on our servers would be concerning to some potential clients. The solution was to pay a company to deisgn and build a custom face plate for the SuperMicro boxes. Which oddly looks more impressive to clients that a rack full of IBM faceplates. It was suprisingly low cost for the faceplates too. Some years ago we made an entire custom chassis for off the shelf boards. The quotes for fabricating that in small numbers were all over the place, but we ended with a good quality chassis at low cost. Most off the shelf rack mount enclosures are really pricy, so it isn't that hard to match their price with a custom build. We ended up with a better design (at least for our purposes) that cost us no more. It can really make your stuff stand out. A simple respray of the front panel can achieve a distinctive look at low cost too. :-) For scalability, OpenSIPS was our choice. There's a very nice tutorial on their website on how to configure Load Balancing. Regards, Steve ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
On Sun, 2009-08-30 at 11:17 +0800, Steve Underwood wrote: This sounds like so many redundancy projects that will probably offer nothing in the real world. On 08/30/2009 05:52 AM, Pete Mueller wrote: I guess I should also mention that Xen is a side-project. When considering this issue for an existing production systems, we chose to put as much HA into hardware as we can. We are not concerned with FS crashing, as so far we've never seen that happen (except when our module caused it :) So for each of our systems: - We have dual NIC cards (onboad NIC + PCI card) both bridged together in case one fails NICs hardly ever fail. Its the wiring which is the vulnerable area. How independent can you make the two wiring paths, when they come from the same box? This is one area where you can do quite well. A simple setup: two machines (1, 2), two NICs (A, B) in each, two switches (S1, S2) - wire up 1A - S1 - 2A, 1B - S2 - 2B - run OSPF across the links allows you to unplug any cable or any switch without interrupting communications for more than a second or two if the OSPF timers are suitably set. This generalises nicely - we used to run two machines as web servers, each advertising the same IP address via OSPF to the routers via a setup like the one above. Unplug any one thing, and the whole still worked. Three complete power outages in the data center we were in in 18 months, one of which took out a number of power supplies, neatly illustrated Steve's point: our real-world reliability was determined elsewhere. --Dave -- David Knell, Director, 3C Limited T: +44 20 3298 2000 E: d...@3c.co.uk W: http://www.3c.co.uk ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
There is a need for ensuring that calls do not drop, but we must balance that with the cost of making the system redundant. We took some small, inexpensive measures, to improve our odds, but we could spend a lot more, for basically nothing more than giving some client a warm fuzzy. To expand on what's mentioned below, The biggest cause for downtime that we've experienced is human accidents. We only have our solution in Tier 1 co-location facilities, so power/net dying isn't really an issue. (If the power does go out, 1000s of systems are down, and everyone notices). What we do end up with is IT admins tripping over power cords, pulling the wrong Ethernet cable, blowing a fuse on one side of the rack, etc. So we've doubled-up on all our cables. After that, the next biggest cause has been MoBo/CPU failure due to fan failure. This issue doesn't really have a good solution, and is why we began looking at Xen. This is where SUN systems look attractive, as systems like the E1 can shut down one CPU or board and keep the rest of the system running. But the cost for that solution is high, and I think that's SPARC-only. I'd love to head others take on a solution for this, as Xen is really a lot of overhead for a rare problem. Though it is at least technically interesting for me :) As for storage, this was completely personal experience. Our SSD have had no issues, while SATA drives seem to fail about 1 every month. The NAS storage is connected via dual NICs as well (again for the cabling) and is completely separated from the network, very close to DAS, just using GigE as the "cable". We are always looking for ways to improve, but the newest and greatest from EMC and others just doesn't seem to offer anything significant and cost a LOT more. I like the idea about the complete custom chassis. I hadn't considered that due to my thinking it would be expensive. Sounds like it's worth a look. As we consider creating an appliance offering, this may become more important.-pete Original Message Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing From: Steve Underwood ste...@coppice.org Date: Sat, August 29, 2009 8:17 pm To: freeswitch-users@lists.freeswitch.org This sounds like so many "redundancy" projects that will probably offer nothing in the real world. On 08/30/2009 05:52 AM, Pete Mueller wrote: I guess I should also mention that Xen is a side-project. When considering this issue for an existing production systems, we chose to put as much HA into hardware as we can. We are not concerned with FS crashing, as so far we've never seen that happen (except when our module caused it :) So for each of our systems: - We have dual NIC cards (onboad NIC + PCI card) both bridged together in case one fails NICs hardly ever fail. Its the wiring which is the vulnerable area. How independent can you make the two wiring paths, when they come from the same box? - We have redundant power supplies. Even with a good UPS, power fails more often than a high quality power supply. Just how independent are the two power sources feeding your two power supplies? Do you have two completely independent UPS sets? Do you have spacially diverse wiring from them? - We use Mirrored Solid State Disks for local storage (far better MTBF than HDD, a lot faster too) My experience so far is that SSD reliability is very poor, with entire drives disappearing, rather than just getting the odd bad sector. I guess to balance this, hard disk drive reliability seems to have plummeted in the last year or so, after several good years. - All but OS and speed-critical data is stored on a NAS device NAS == more wiring. More wiring == more vulnerabilities. Are you sure your setup is a win? NAS tends to help keep the data secure, but it isn't good for reliable access to that data. - We have redundant DBs with Memcache infront for speed At the same time we chose to use COTS hardware (SuperMicro chassis/MoBo) rather than the big-boys like IBM or Dell. This kept the overall cost per machine low. Initially some were concerned that not having a name like IBM on our servers would be concerning to some potential clients. The solution was to pay a company to deisgn and build a custom face plate for the SuperMicro boxes. Which oddly looks more impressive to clients that a rack full of IBM faceplates. It was suprisingly low cost for the faceplates too. Some years ago we made an entire custom chassis for off the shelf boards. The quotes for fabricating that in small numbers were all over the place, but we ended with a good quality chassis at low cost. Most off the shelf rack mount enclosures are really pricy, so it isn't that hard to match their price with a custom build. We ended up with a better design (at least for our purposes) that cost us no more. It can really make your stuff stand out. A simple respray of the front panel can achieve a distinctive l
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Pete Mueller p...@privateconnect.com wrote: There is a need for ensuring that calls do not drop, but we must balance that with the cost of making the system redundant. We took some small, inexpensive measures, to improve our odds, but we could spend a lot more, for basically nothing more than giving some client a warm fuzzy. I think this is one area where, as indicated earlier in the thread, a lot of development effort would be needed to obtain that extra degree of reliability. From a broader perspective, the question is whether, over the next decade or two, VoIP can compete with the PSTN in reliability. My (limited) understanding is that PSTN equipment typically achieves 99.9% uptime, and if VoIP systems are going to play in that arena, it would be desirable for free/open-source software to do so. If FreeSWITCH itself is working correctly, all you need is a hardware failure or a kernel panic or a network outage to drop that up-time substantially, not to mention dropping the calls as well, which I've never experienced as a user of the PSTN due to equipment at the telephone exchange. I have, however, experienced some rather low-quality PSTN calls over international lines, which have the added disadvantage of being expensive to use. ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
[Freeswitch-users] FreeSWITCH HA + Loadbalancing
Hello List, I have read the current thread about scalability and I would need some advice about a callcenter setup: First where I come from: I have lot's of problems with an asterisk solution. I have regular crash's and lock-ups, with downgrading and other stuff i got it somewhat stable, but have nevertheless regular hickups. I am desperate and want to get rid of asterisk and I hope that freeSwitch will provide me with a more stable solution. Our Setup (really nothing special): * 1 Asterisk box, New IBM Hardware (3 month old), 2 HE rack server, 3 GIG of RAM, Xircom analog switch connected to mobile stations for 4 different providers, Digium 4port cards TP400something * 8 queues * ~60 agents (which logon, logoff, pause, unpause), not more than 40 concurrently online * ~ 7K - 9K calls (well, CDR entries) a day (not that much for a bpx) * Music on Hold in the call-queues * No special announcement * Transfers between calls in queues and different agents as well as non agents (i mention this because we have transfer related chrashes in asterisk) The current Problems: * Lockups with different causes (ranging from calls not terminated to heavy thread locking through the AMI interface) * Crashes and library aborts (pthread, libc, crashes related to music on hold, app_queue, transfers) We used Asterisk: 1.4.23, 1.4.24, 1.4.26rc3, 1.4.26rc5, 1.4.26 and are now back to 1.4.21.2 (stock debian) as anything beyond that is for whatever reason highly unstable for our szenario. Maybe we should have been segmenting the box into one asterisk dedicted to talking to the hardware, one especial for queue/sip handling, i do not know. (all issues are well documented in issues.asterisk.org, but it seems to be very, very difficult to get to the bottom of them as they exist since 1.4.23 as it seems and are open until know and not fixable since month.) Now, I really would appreciate some success-stories on how you guys managed to get a stable pbx system with freeSWITCH in regard of HA and scalability: * How to segment freeSWITCH? Or is it stable enough to handle all in one for such a szenario as outlined above? * What would be the best strategie for High Availability / Failover? - I read in the WIKI (featurelist) that Livemigration of calls from one box to another should be possible? - I was thinking about using memcached for storing all state information so another freeswitch box can takeover calls from the first box if it dies, is this possible? If so, how? - Is there anotherway to somehow configure freeSWITCH that in the event of a crash i do not loose the current established calls? Basically I just want a stable PBX where I do not have to fear every day it will core-dump or abort or Lock up. Is freeSWITCH mature enough so i can sleep at night for at least 3 month without a crash? Thank you for your Time and help in advance, and I am more than willing to take all the information gathered here and create a wiki page to help other people with the same questions/problems. best Ray -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Usually you don't need to worry about stability issues with FS. For scalability, peoples tend to use openser or some other sip loadbalancer in fron of fs, but you probably would not need that. Live migration of calls is not yet possible, tough. -giovanni On Fri, Aug 28, 2009 at 8:52 PM, Raimund Sachererr...@runsolutions.com wrote: Hello List, I have read the current thread about scalability and I would need some advice about a callcenter setup: First where I come from: I have lot's of problems with an asterisk solution. I have regular crash's and lock-ups, with downgrading and other stuff i got it somewhat stable, but have nevertheless regular hickups. I am desperate and want to get rid of asterisk and I hope that freeSwitch will provide me with a more stable solution. Our Setup (really nothing special): * 1 Asterisk box, New IBM Hardware (3 month old), 2 HE rack server, 3 GIG of RAM, Xircom analog switch connected to mobile stations for 4 different providers, Digium 4port cards TP400something * 8 queues * ~60 agents (which logon, logoff, pause, unpause), not more than 40 concurrently online * ~ 7K - 9K calls (well, CDR entries) a day (not that much for a bpx) * Music on Hold in the call-queues * No special announcement * Transfers between calls in queues and different agents as well as non agents (i mention this because we have transfer related chrashes in asterisk) The current Problems: * Lockups with different causes (ranging from calls not terminated to heavy thread locking through the AMI interface) * Crashes and library aborts (pthread, libc, crashes related to music on hold, app_queue, transfers) We used Asterisk: 1.4.23, 1.4.24, 1.4.26rc3, 1.4.26rc5, 1.4.26 and are now back to 1.4.21.2 (stock debian) as anything beyond that is for whatever reason highly unstable for our szenario. Maybe we should have been segmenting the box into one asterisk dedicted to talking to the hardware, one especial for queue/sip handling, i do not know. (all issues are well documented in issues.asterisk.org, but it seems to be very, very difficult to get to the bottom of them as they exist since 1.4.23 as it seems and are open until know and not fixable since month.) Now, I really would appreciate some success-stories on how you guys managed to get a stable pbx system with freeSWITCH in regard of HA and scalability: * How to segment freeSWITCH? Or is it stable enough to handle all in one for such a szenario as outlined above? * What would be the best strategie for High Availability / Failover? - I read in the WIKI (featurelist) that Livemigration of calls from one box to another should be possible? - I was thinking about using memcached for storing all state information so another freeswitch box can takeover calls from the first box if it dies, is this possible? If so, how? - Is there anotherway to somehow configure freeSWITCH that in the event of a crash i do not loose the current established calls? Basically I just want a stable PBX where I do not have to fear every day it will core-dump or abort or Lock up. Is freeSWITCH mature enough so i can sleep at night for at least 3 month without a crash? Thank you for your Time and help in advance, and I am more than willing to take all the information gathered here and create a wiki page to help other people with the same questions/problems. best Ray -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
On Fri, Aug 28, 2009 at 12:11 PM, Giovanni Maruzzelli gmar...@celliax.orgwrote: Usually you don't need to worry about stability issues with FS. For scalability, peoples tend to use openser or some other sip loadbalancer in fron of fs, but you probably would not need that. Live migration of calls is not yet possible, tough. Brian West has done some testing with live migrations but I don't know where he left off. Brian, were you using OpenVZ? I forget... In any case, FS allows you to try to do this with the hope that it will actually work in a production environment. As for the other things - yes, FS can work with the TDM card and the queues, etc. If you are in a position to install FS on a sandbox machine for testing then that would be your best bet. I recommend diving in, which is probably what you did when you first started learning Asterisk... Have fun! -MC ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Hi Raimund, One FreeSWITCH box will be quite enough to handle the call volumes that you're talking about, and it ought to be much more stable than the Asterisk solution which you've outlined below. It's probably best to forget about live failover without calls dropping - this isn't something that's supported, and there'd be a lot of work to do to develop code to keep two boxed in sync. Once you get used to a stable solution - i.e. something which doesn't crash - then live failover, HA, etc., will seem somewhat irrelevant. I recently did some work on an FS-based call center solution - drop me a note if I could be of any help with yours. Cheers -- Dave Hello List, I have read the current thread about scalability and I would need some advice about a callcenter setup: First where I come from: I have lot's of problems with an asterisk solution. I have regular crash's and lock-ups, with downgrading and other stuff i got it somewhat stable, but have nevertheless regular hickups. I am desperate and want to get rid of asterisk and I hope that freeSwitch will provide me with a more stable solution. Our Setup (really nothing special): * 1 Asterisk box, New IBM Hardware (3 month old), 2 HE rack server, 3 GIG of RAM, Xircom analog switch connected to mobile stations for 4 different providers, Digium 4port cards TP400something * 8 queues * ~60 agents (which logon, logoff, pause, unpause), not more than 40 concurrently online * ~ 7K - 9K calls (well, CDR entries) a day (not that much for a bpx) * Music on Hold in the call-queues * No special announcement * Transfers between calls in queues and different agents as well as non agents (i mention this because we have transfer related chrashes in asterisk) The current Problems: * Lockups with different causes (ranging from calls not terminated to heavy thread locking through the AMI interface) * Crashes and library aborts (pthread, libc, crashes related to music on hold, app_queue, transfers) We used Asterisk: 1.4.23, 1.4.24, 1.4.26rc3, 1.4.26rc5, 1.4.26 and are now back to 1.4.21.2 (stock debian) as anything beyond that is for whatever reason highly unstable for our szenario. Maybe we should have been segmenting the box into one asterisk dedicted to talking to the hardware, one especial for queue/sip handling, i do not know. (all issues are well documented in issues.asterisk.org, but it seems to be very, very difficult to get to the bottom of them as they exist since 1.4.23 as it seems and are open until know and not fixable since month.) Now, I really would appreciate some success-stories on how you guys managed to get a stable pbx system with freeSWITCH in regard of HA and scalability: * How to segment freeSWITCH? Or is it stable enough to handle all in one for such a szenario as outlined above? * What would be the best strategie for High Availability / Failover? - I read in the WIKI (featurelist) that Livemigration of calls from one box to another should be possible? - I was thinking about using memcached for storing all state information so another freeswitch box can takeover calls from the first box if it dies, is this possible? If so, how? - Is there anotherway to somehow configure freeSWITCH that in the event of a crash i do not loose the current established calls? Basically I just want a stable PBX where I do not have to fear every day it will core-dump or abort or Lock up. Is freeSWITCH mature enough so i can sleep at night for at least 3 month without a crash? Thank you for your Time and help in advance, and I am more than willing to take all the information gathered here and create a wiki page to help other people with the same questions/problems. best Ray -- David Knell, Director, 3C Limited T: +44 20 3298 2000 E: d...@3c.co.uk W: http://www.3c.co.uk ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Hi David, What have you used on FS for call center, mod_fifo? Can you describe your experience with that, I'm currently interested in call center + FS scenario. Diego On Fri, Aug 28, 2009 at 9:01 PM, David Knell d...@3c.co.uk wrote: Hi Raimund, One FreeSWITCH box will be quite enough to handle the call volumes that you're talking about, and it ought to be much more stable than the Asterisk solution which you've outlined below. It's probably best to forget about live failover without calls dropping - this isn't something that's supported, and there'd be a lot of work to do to develop code to keep two boxed in sync. Once you get used to a stable solution - i.e. something which doesn't crash - then live failover, HA, etc., will seem somewhat irrelevant. I recently did some work on an FS-based call center solution - drop me a note if I could be of any help with yours. Cheers -- Dave Hello List, I have read the current thread about scalability and I would need some advice about a callcenter setup: First where I come from: I have lot's of problems with an asterisk solution. I have regular crash's and lock-ups, with downgrading and other stuff i got it somewhat stable, but have nevertheless regular hickups. I am desperate and want to get rid of asterisk and I hope that freeSwitch will provide me with a more stable solution. Our Setup (really nothing special): * 1 Asterisk box, New IBM Hardware (3 month old), 2 HE rack server, 3 GIG of RAM, Xircom analog switch connected to mobile stations for 4 different providers, Digium 4port cards TP400something * 8 queues * ~60 agents (which logon, logoff, pause, unpause), not more than 40 concurrently online * ~ 7K - 9K calls (well, CDR entries) a day (not that much for a bpx) * Music on Hold in the call-queues * No special announcement * Transfers between calls in queues and different agents as well as non agents (i mention this because we have transfer related chrashes in asterisk) The current Problems: * Lockups with different causes (ranging from calls not terminated to heavy thread locking through the AMI interface) * Crashes and library aborts (pthread, libc, crashes related to music on hold, app_queue, transfers) We used Asterisk: 1.4.23, 1.4.24, 1.4.26rc3, 1.4.26rc5, 1.4.26 and are now back to 1.4.21.2 (stock debian) as anything beyond that is for whatever reason highly unstable for our szenario. Maybe we should have been segmenting the box into one asterisk dedicted to talking to the hardware, one especial for queue/sip handling, i do not know. (all issues are well documented in issues.asterisk.org, but it seems to be very, very difficult to get to the bottom of them as they exist since 1.4.23 as it seems and are open until know and not fixable since month.) Now, I really would appreciate some success-stories on how you guys managed to get a stable pbx system with freeSWITCH in regard of HA and scalability: * How to segment freeSWITCH? Or is it stable enough to handle all in one for such a szenario as outlined above? * What would be the best strategie for High Availability / Failover? - I read in the WIKI (featurelist) that Livemigration of calls from one box to another should be possible? - I was thinking about using memcached for storing all state information so another freeswitch box can takeover calls from the first box if it dies, is this possible? If so, how? - Is there anotherway to somehow configure freeSWITCH that in the event of a crash i do not loose the current established calls? Basically I just want a stable PBX where I do not have to fear every day it will core-dump or abort or Lock up. Is freeSWITCH mature enough so i can sleep at night for at least 3 month without a crash? Thank you for your Time and help in advance, and I am more than willing to take all the information gathered here and create a wiki page to help other people with the same questions/problems. best Ray -- David Knell, Director, 3C Limited T: +44 20 3298 2000 E: d...@3c.co.uk W: http://www.3c.co.uk ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Yes, FreeSWITCH is a system that you can trust 100%. I have switched my Asterisk servers to FreeSWITCH and have peace now. If I were you I would get rid of Asterisk and use FreeSWITCH, FS will handle all what you want very well. And I agree with David, fail-over is kinda irrelevant since the FS doesn't crash like Asterisk does. Diego On Fri, Aug 28, 2009 at 6:52 PM, Raimund Sacherer r...@runsolutions.comwrote: Hello List, I have read the current thread about scalability and I would need some advice about a callcenter setup: First where I come from: I have lot's of problems with an asterisk solution. I have regular crash's and lock-ups, with downgrading and other stuff i got it somewhat stable, but have nevertheless regular hickups. I am desperate and want to get rid of asterisk and I hope that freeSwitch will provide me with a more stable solution. Our Setup (really nothing special): * 1 Asterisk box, New IBM Hardware (3 month old), 2 HE rack server, 3 GIG of RAM, Xircom analog switch connected to mobile stations for 4 different providers, Digium 4port cards TP400something * 8 queues * ~60 agents (which logon, logoff, pause, unpause), not more than 40 concurrently online * ~ 7K - 9K calls (well, CDR entries) a day (not that much for a bpx) * Music on Hold in the call-queues * No special announcement * Transfers between calls in queues and different agents as well as non agents (i mention this because we have transfer related chrashes in asterisk) The current Problems: * Lockups with different causes (ranging from calls not terminated to heavy thread locking through the AMI interface) * Crashes and library aborts (pthread, libc, crashes related to music on hold, app_queue, transfers) We used Asterisk: 1.4.23, 1.4.24, 1.4.26rc3, 1.4.26rc5, 1.4.26 and are now back to 1.4.21.2 (stock debian) as anything beyond that is for whatever reason highly unstable for our szenario. Maybe we should have been segmenting the box into one asterisk dedicted to talking to the hardware, one especial for queue/sip handling, i do not know. (all issues are well documented in issues.asterisk.org, but it seems to be very, very difficult to get to the bottom of them as they exist since 1.4.23 as it seems and are open until know and not fixable since month.) Now, I really would appreciate some success-stories on how you guys managed to get a stable pbx system with freeSWITCH in regard of HA and scalability: * How to segment freeSWITCH? Or is it stable enough to handle all in one for such a szenario as outlined above? * What would be the best strategie for High Availability / Failover? - I read in the WIKI (featurelist) that Livemigration of calls from one box to another should be possible? - I was thinking about using memcached for storing all state information so another freeswitch box can takeover calls from the first box if it dies, is this possible? If so, how? - Is there anotherway to somehow configure freeSWITCH that in the event of a crash i do not loose the current established calls? Basically I just want a stable PBX where I do not have to fear every day it will core-dump or abort or Lock up. Is freeSWITCH mature enough so i can sleep at night for at least 3 month without a crash? Thank you for your Time and help in advance, and I am more than willing to take all the information gathered here and create a wiki page to help other people with the same questions/problems. best Ray -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing
Hi Diego, We didn't use mod_fifo; we built our own queues using an application hanging off the event socket. This was partly down to typical programmer hubris, and partly to allow us to do things that mod_fifo might not without either (a) trawling through mod_fifo, or (b) pestering Anthony. As a brief outline: - operators each live in their own conference - supervisors can add themselves to these conferences for monitoring or coaching - callers sit in queues for the various skills that the operators have. When an operator is free, look at each queue for the skills that the operator has, and choose the caller who's been waiting longest. I used a one-thread-per-call model, so there's a little bit of thought needed to get the access to shared structures right but, essentially, given the low frequency of access to them and the fact that nothing was time-critical at the sub-second level, I just wrapped all of the accesses in a single global mutex, which was easy and is pretty foolproof. The latter is important, given that it was me writing the stuff ;-) Cheers -- Dave Hi David, What have you used on FS for call center, mod_fifo? Can you describe your experience with that, I'm currently interested in call center + FS scenario. Diego On Fri, Aug 28, 2009 at 9:01 PM, David Knell d...@3c.co.uk wrote: Hi Raimund, One FreeSWITCH box will be quite enough to handle the call volumes that you're talking about, and it ought to be much more stable than the Asterisk solution which you've outlined below. It's probably best to forget about live failover without calls dropping - this isn't something that's supported, and there'd be a lot of work to do to develop code to keep two boxed in sync. Once you get used to a stable solution - i.e. something which doesn't crash - then live failover, HA, etc., will seem somewhat irrelevant. I recently did some work on an FS-based call center solution - drop me a note if I could be of any help with yours. Cheers -- Dave Hello List, I have read the current thread about scalability and I would need some advice about a callcenter setup: First where I come from: I have lot's of problems with an asterisk solution. I have regular crash's and lock-ups, with downgrading and other stuff i got it somewhat stable, but have nevertheless regular hickups. I am desperate and want to get rid of asterisk and I hope that freeSwitch will provide me with a more stable solution. Our Setup (really nothing special): * 1 Asterisk box, New IBM Hardware (3 month old), 2 HE rack server, 3 GIG of RAM, Xircom analog switch connected to mobile stations for 4 different providers, Digium 4port cards TP400something * 8 queues * ~60 agents (which logon, logoff, pause, unpause), not more than 40 concurrently online * ~ 7K - 9K calls (well, CDR entries) a day (not that much for a bpx) * Music on Hold in the call-queues * No special announcement * Transfers between calls in queues and different agents as well as non agents (i mention this because we have transfer related chrashes in asterisk) The current Problems: * Lockups with different causes (ranging from calls not terminated to heavy thread locking through the AMI interface) * Crashes and library aborts (pthread, libc, crashes related to music on hold, app_queue, transfers) We used Asterisk: 1.4.23, 1.4.24, 1.4.26rc3, 1.4.26rc5, 1.4.26 and are now back to 1.4.21.2 (stock debian) as anything beyond that is for whatever reason highly unstable for our szenario. Maybe we should have been segmenting the box into one asterisk dedicted to talking to the hardware, one especial for queue/sip handling, i do not know. (all issues are well documented in issues.asterisk.org, but it seems to be very, very difficult to get to the bottom of them as they exist since 1.4.23 as it seems and are open until know and not fixable since month.) Now, I really would appreciate some success-stories on how you guys managed to get a stable pbx system with freeSWITCH in regard of HA and scalability: * How to segment freeSWITCH? Or is it stable enough to
