Re: Enterprise Multihoming
On Fri, 12 Mar 2004, Stephen Fisher wrote: Most of the multi-homing talk has been about failover capabilities between different providers. What about the effects of multiple providers when neither has actually failed; such as different paths for inbound/outbound traffic. One provider may have better connectivity to x site whereas the other provider has better connectivity to y. (Or is this not as important as it used to be?) Capacity and congestion isnt a (big) issue with bandwidth and circuits being so cheap, most corporates just need to know they can get their email and browse the web and whether it takes 70 or 140ms for data to cross the atlantic providing it pops up on their screen within a few seconds they're happy. So in this way I think the answer to your question is its not important to most multihomers but ymmv.. Steve On Fri, Mar 12, 2004 at 09:15:55AM -0700, John Neiberger wrote: In our case, we already are multihoming and I'm considering moving away from that to a simpler solution. It's been my assertion that we didn't need to multihome in the beginning. The decision was made at a level higher than me. However, now that we have it I'm trying to determine the pros and cons related to moving to a single provider.
Re: Enterprise Multihoming
I think its too easy, thats the problem. For $1000 (excluding bandwidth/ccts) you can buy a box, connect to your two providers, get an ASN and IPs and you're away. Compare to the telephone network, to 'multihome' you need to get licenses, allocations of numbers and codes thats not so easy, get some SS7 kit and do your data builds.. you're talking quite a lot more money and certainly a lot more difficult technically. Perhaps we should make the Internet more difficult :) I dont agree that connecting to two+ upstreams makes you better. In my experience end networks have a couple of orders of magnitude more downtime than a PoP in any reasonably large ISP. Ie the percentage theoretical improvement is small. In addition you seriously increase the complexity of your system, chances are you're using the cheapest kit you could find (or at least cheaper and smaller than what I would use).. its not great at BGP and may fall over when you get a minor DoS attack, you probably generate flaps quite a bit from adhoc changes and if you're announcing a /24 then thats going to get you dampened quickly.. so you actually create a new weakest link. Also most of the corporates I've dealt with take defaults rather than full tables.. so if the provider does have an issue you still forward the traffic, theres no failover of outbound routing. Even if you spend (waste) the money on some decent gear, you're on your own and when a problem occurs the ISPs are going to be less helpful to you (not by choice, I mean they dont have control of your network any more.. there knowledge of whats causing problems is limited to the bit that they provide to you), so chances are your problems may be more serious and take longer to diagnose and fix. IMHO avoid multihoming. You will know when you are big enough and you *need* to do it, if you're not sure or you only want to do it cause you heard everyone else is and its real cool then I suggest you dont. Steve On Thu, 11 Mar 2004, John Neiberger wrote: On another list we've been having multihoming discussions again and I wanted to get some fresh opinions from you. For the past few years it has been fairly common for non-ISPs to multihome to different providers for additional redundancy in case a single provider has problems. I know this is frowned upon now, especially since it helped increase the number of autonomous systems and routing table prefixes beyond what was really necessary. It seems to me that a large number of companies that did this could just have well ordered multiple, geographically separate links to the same provider. What is the prevailing wisdom now? At what point do you feel that it is justified for a non-ISP to multihome to multiple providers? I ask because we have three links: two from Sprint and one from Global Crossing. I'm considering dropping the GC circuit and adding another geographically-diverse connection to Sprint, and then removing BGP from our routers. I see a few upsides to this, but are there any real downsides? Flame on. :-) Thanks, John --
Re: Enterprise Multihoming
At 4:06 PM + 3/12/04, Stephen J. Wilcox wrote: I think its too easy, thats the problem. Hoping that I don't sound too much like Bill Clinton, that depends on what you mean by it. If it is multihoming, with your own ASN, to two providers, your raise some valid points. Is there an intermediate alternative before you go all out? Yes, I think so, assuming your current provider has multiple POPs. Let me examine some of your points if we consider RFC 1998-style multi-POPping (I just invented that highly technical term) using PA address space. For $1000 (excluding bandwidth/ccts) you can buy a box, connect to your two providers, get an ASN and IPs and you're away. Alternatively, another POP link, and preferably another router. If you are more concerned with loop failures than router failures, not a completely unreasonable assumption, you could get away with one router that has multiple interfaces, and spend some of the savings on backup power -- possibly a backup power supply in addition to the UPS, such as a Cisco RPS on their smaller routers. While you'll probably take a performance hit, or if you can reduce to critical traffic on an outage, you might get away with a second smaller router. I dont agree that connecting to two+ upstreams makes you better. In my experience end networks have a couple of orders of magnitude more downtime than a PoP in any reasonably large ISP. Ie the percentage theoretical improvement is small. Like everything else, It Depends. My experience is that access links fail more often than provider routing systems, especially with a clueful provider. Since you can't guarantee that your physical connectivity to two different ISPs doesn't involve a shared risk group in the lines, there are still some things you may not be protected against. One option, depending on the plant in your area, is that if you are considering a second router, consider putting it in a nearby building, reachable by WLAN (if you are minimizing costs), where that building minimally has different ducts to the telco end office, and ideally goes to a different end office. Not always possible, but to be considered. Longer-range wireless (radio or optical) links get more expensive. In addition you seriously increase the complexity of your system, chances are you're using the cheapest kit you could find (or at least cheaper and smaller than what I would use).. its not great at BGP and may fall over when you get a minor DoS attack, you probably generate flaps quite a bit from adhoc changes and if you're announcing a /24 then thats going to get you dampened quickly.. That's a motivation for PA address space, where the provider aggregate is less likely to be small and easily damped. so you actually create a new weakest link. Also most of the corporates I've dealt with take defaults rather than full tables.. so if the provider does have an issue you still forward the traffic, theres no failover of outbound routing. Again looking at intermediate solutions, there are always partial routes such as customer routes of the provier. Even if you spend (waste) the money on some decent gear, you're on your own and when a problem occurs the ISPs are going to be less helpful to you (not by choice, I mean they dont have control of your network any more.. there knowledge of whats causing problems is limited to the bit that they provide to you), so chances are your problems may be more serious and take longer to diagnose and fix. Again, an operational advantage of multiPOPping and working with one carrier, although you aren't going to be protected against insanity of their BGP/ IMHO avoid multihoming. You will know when you are big enough and you *need* to do it, if you're not sure or you only want to do it cause you heard everyone else is and its real cool then I suggest you dont. MHO would be to look at multihoming as a spectrum of solutions rather than a binary choice of single-provider-single-link versus multiple-provider. In given situations, you might also want to look at DSL or cable for diversity, tunneling to an ISP since the broadband provider is unlikely to be willing to speak BGP. Even dialup/ISDN, sometimes for critical workstations, has its place. Shameless plug: I do go through these options in my book, Building Service Provider Networks (Wiley). Even there, though, I only run through the alternatives. You will still have to make your own cost-benefit decisions based on business policy, budget, clue level and cost of alternatives.
Re: Enterprise Multihoming
Shameless plug: I do go through these options in my book, Building Service Provider Networks (Wiley). Even there, though, I only run through the alternatives. You will still have to make your own cost-benefit decisions based on business policy, budget, clue level and cost of alternatives. A copy of which I have sitting here at my desk. Ah, yes. Beginning at p. 344, Multlinking and Multihoming: The Customer Side. I suppose I should re-read that section. :-) Regarding our internal network, I wish I could skip ahead to p. 517, VPNs and Related Services. Unfortunately, the VPN products that are available right now are double the cost of our frame relay network. Oh well, perhaps someday my price will come. Thanks, John --
Re: Enterprise Multihoming
As Marshall noted multi-homing gives you the ability to switch providers easily. This ability also gives you leverage with your network providers since vendor lock-in does not exist. This is a strong business case for multihoming and is one the financial types understand and appreciate. In a prior incarnation I worked for a distributor who had a online ordering system. Our telcom coordinator got a great deal on bundled internet service and telephony from a unnamed vendor. Due to the peering arrangements the carrier had major customers were unable to place orders in a timely fashion. I set up a new AS and set up multihoming with another carrier and made our customers happy again. Subsequently said carrier had an outage which took down our link to them for 7 weeks. Since this was an internal problem at our provider multiple links to this carrier would not have benefited us in the least. A multihoming strategy also allows you to select providers who provide connectivty to your business partners and customers which is another win for obvious reasons. Scott C. McGrath On Thu, 11 Mar 2004, Marshall Eubanks wrote: There is another thing - if you are multi-homed, and want to switch providers, it is pretty seamless and painless - no renumbering, no loss of connection, etc., as you always have a redundant path. On Thursday, March 11, 2004, at 12:34 PM, Pekka Savola wrote: On Thu, 11 Mar 2004, Gregory Taylor wrote: Mutli-homing a non-ISP network or system on multiple carriers is a good way to maintain independent links to the internet by means of different peering, uplinks, over-all routing and reliability. My network on NAIS is currently multi-homed through ATT. I use a single provider as both of my redundant links via 100% Fiber network. Even though this is cheaper for me, all it takes is for ATT to have some major outage and I will be screwed. If I have a backup fiber line from say, Global Crossing, then it doesn't matter if ATT takes a nose dive, I still have my redundancy there. Well, I think this, in many cases, boils down to being able to pick the right provider. I mean, some providers go belly-up from time to time. Others are designed/run better. For a major provider, complete outage of all of its customers is such a big thing they'll want to avoid it always. If it happens, for a brief moment, once in five years (for example), for most companies that's an acceptable level of risk. -- Pekka Savola You each name yourselves king, yet the Netcore Oykingdom bleeds. Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings Regards Marshall Eubanks T.M. Eubanks e-mail : [EMAIL PROTECTED] http://www.telesuite.com
RE: Enterprise Multihoming
Address portability all depends on if you IP blocks are assigned by ARIN/RIPE/APNIC/ISP portable or if you are using the ISP's address space. It has been my experience that multi-homing to diverse ISP's with multiple circuits per ISP (i.e. Primary/Secondary with ISP-A and Primary/Secondary with ISP-B) is the best option if you can afford the cost and your bandwidth requires it. Like it was stated before, if you can afford the possible downtime associated with multi-homing to a single ISP then yes there are definitely cost savings to be had and reduced administrative overhead; but, if you cannot afford the possibility of downtime then separate ISP's is the only way to go. Chris Burton Network Engineer Walt Disney Internet Group: Network Services The information contained in this e-mail message is confidential, intended only for the use of the individual or entity named above. If the reader of this e-mail is not the intended recipient, or the employee or agent responsible to deliver it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please contact Walt Disney Internet Group at 206-664-4000. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Scott McGrath Sent: Friday, March 12, 2004 5:50 PM To: [EMAIL PROTECTED] Subject: Re: Enterprise Multihoming As Marshall noted multi-homing gives you the ability to switch providers easily. This ability also gives you leverage with your network providers since vendor lock-in does not exist. This is a strong business case for multihoming and is one the financial types understand and appreciate. In a prior incarnation I worked for a distributor who had a online ordering system. Our telcom coordinator got a great deal on bundled internet service and telephony from a unnamed vendor. Due to the peering arrangements the carrier had major customers were unable to place orders in a timely fashion. I set up a new AS and set up multihoming with another carrier and made our customers happy again. Subsequently said carrier had an outage which took down our link to them for 7 weeks. Since this was an internal problem at our provider multiple links to this carrier would not have benefited us in the least. A multihoming strategy also allows you to select providers who provide connectivty to your business partners and customers which is another win for obvious reasons. Scott C. McGrath On Thu, 11 Mar 2004, Marshall Eubanks wrote: There is another thing - if you are multi-homed, and want to switch providers, it is pretty seamless and painless - no renumbering, no loss of connection, etc., as you always have a redundant path. On Thursday, March 11, 2004, at 12:34 PM, Pekka Savola wrote: On Thu, 11 Mar 2004, Gregory Taylor wrote: Mutli-homing a non-ISP network or system on multiple carriers is a good way to maintain independent links to the internet by means of different peering, uplinks, over-all routing and reliability. My network on NAIS is currently multi-homed through ATT. I use a single provider as both of my redundant links via 100% Fiber network. Even though this is cheaper for me, all it takes is for ATT to have some major outage and I will be screwed. If I have a backup fiber line from say, Global Crossing, then it doesn't matter if ATT takes a nose dive, I still have my redundancy there. Well, I think this, in many cases, boils down to being able to pick the right provider. I mean, some providers go belly-up from time to time. Others are designed/run better. For a major provider, complete outage of all of its customers is such a big thing they'll want to avoid it always. If it happens, for a brief moment, once in five years (for example), for most companies that's an acceptable level of risk. -- Pekka Savola You each name yourselves king, yet the Netcore Oykingdom bleeds. Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings Regards Marshall Eubanks T.M. Eubanks e-mail : [EMAIL PROTECTED] http://www.telesuite.com
Re: Enterprise Multihoming
Most of the multi-homing talk has been about failover capabilities between different providers. What about the effects of multiple providers when neither has actually failed; such as different paths for inbound/outbound traffic. One provider may have better connectivity to x site whereas the other provider has better connectivity to y. (Or is this not as important as it used to be?) On Fri, Mar 12, 2004 at 09:15:55AM -0700, John Neiberger wrote: In our case, we already are multihoming and I'm considering moving away from that to a simpler solution. It's been my assertion that we didn't need to multihome in the beginning. The decision was made at a level higher than me. However, now that we have it I'm trying to determine the pros and cons related to moving to a single provider.
Enterprise Multihoming
On another list we've been having multihoming discussions again and I wanted to get some fresh opinions from you. For the past few years it has been fairly common for non-ISPs to multihome to different providers for additional redundancy in case a single provider has problems. I know this is frowned upon now, especially since it helped increase the number of autonomous systems and routing table prefixes beyond what was really necessary. It seems to me that a large number of companies that did this could just have well ordered multiple, geographically separate links to the same provider. What is the prevailing wisdom now? At what point do you feel that it is justified for a non-ISP to multihome to multiple providers? I ask because we have three links: two from Sprint and one from Global Crossing. I'm considering dropping the GC circuit and adding another geographically-diverse connection to Sprint, and then removing BGP from our routers. I see a few upsides to this, but are there any real downsides? Flame on. :-) Thanks, John --
Re: Enterprise Multihoming
On Thu, Mar 11, 2004 at 09:04:57AM -0700, John Neiberger wrote: For the past few years it has been fairly common for non-ISPs to multihome to different providers for additional redundancy in case a single provider has problems. I know this is frowned upon now, especially since it helped increase the number of autonomous systems and routing table prefixes beyond what was really necessary. Who defines what is really necessary? What is your understanding of really necessary when it comes to the desire to be commercially and technically independent of your suppliers? It's this discussion again. Regards, Daniel
Re: Enterprise Multihoming
On Thu, 11 Mar 2004, John Neiberger wrote: On another list we've been having multihoming discussions again and I wanted to get some fresh opinions from you. For the past few years it has been fairly common for non-ISPs to multihome to different providers for additional redundancy in case a single provider has problems. I know this is frowned upon now, especially since it helped increase the number of autonomous systems and routing table prefixes beyond what was really necessary. It seems to me that a large number of companies that did this could just have well ordered multiple, geographically separate links to the same provider. What is the prevailing wisdom now? At what point do you feel that it is justified for a non-ISP to multihome to multiple providers? I ask because we have three links: two from Sprint and one from Global Crossing. I'm considering dropping the GC circuit and adding another geographically-diverse connection to Sprint, and then removing BGP from our routers. I see a few upsides to this, but are there any real downsides? Many/most of my external connectivity problems are provider-related rather than circuit-related. Having two circuits to a single provider doesn't help when that provider is broken. I'm not saying that multi-ISP BGP-based multi-homing is risk-free, but I don't see multi-circuit single-provider as a viable alternative. Jay Ford, Network Engineering Group, Information Technology Services University of Iowa, Iowa City, IA 52242 email: [EMAIL PROTECTED], phone: 319-335-, fax: 319-335-2951
Re: Enterprise Multihoming
On 11.03.2004 17:04 John Neiberger wrote: What is the prevailing wisdom now? At what point do you feel that it is justified for a non-ISP to multihome to multiple providers? IMHO you do not need a justification. If you think multiple links to the same provider don't buy you what you need (e.g. if the ISP has severe problems with its internal network multiple links do not buy you anything. Same holds when your ISP goes south which still happens now and then these days) go for real multihoming. Arnold
Re: Enterprise Multihoming
John Neiberger wrote: I see a few upsides to this, but are there any real downsides? Connecting to single AS makes you physically resilient but logically dependent on single entity, be that a provisioning system, routing protocol instance, etc. Depending on your requirements, the option of having somebody redistribute all their BGP routes into ISIS or OSPF might not worth looking forward to. Pete
Re: Enterprise Multihoming
Daniel Roesen [EMAIL PROTECTED] 3/11/04 9:13:04 AM On Thu, Mar 11, 2004 at 09:04:57AM -0700, John Neiberger wrote: For the past few years it has been fairly common for non-ISPs to multihome to different providers for additional redundancy in case a single provider has problems. I know this is frowned upon now, especially since it helped increase the number of autonomous systems and routing table prefixes beyond what was really necessary. Who defines what is really necessary? What is your understanding of really necessary when it comes to the desire to be commercially and technically independent of your suppliers? It's this discussion again. That goes off in entirely the wrong direction but I guess I'll clarify that statement. :-) My point was that most companies could have met their connectivity requirements by simply getting multiple connections to the same provider from the beginning. However, among the less-technical managers it seemed to be popular to demand connectivity to multiple ISPs. It seems that me that this was not really necessary from a technical perspective in many cases, it just made people feel good. I don't really want to focus on that, though; I'm more interested in the situation as it stands today. If a company were going to add brand new Internet connectivity where it didn't exist before, what factors would you use to determine if multiple ISPs should even be considered? Given the stability of the larger ISPs and the general lack of true BGP expertise at many companies, is the potential benefit of multihoming to different ISPs worth the added risk and responsbility that comes with using BGP? Our BGP configuration isn't very difficult to understand but we do have a lack of BGP knowledge in the department and some additional training is in order. However, might it not be better to just simplify our connectivity and remove BGP altogether? Sure, I like BGP as much as the next guy but there's no sense in running it just because we can. :-) Thanks, John --
Re: Enterprise Multihoming
At what point do you feel that it is : justified for a non-ISP to multihome to multiple providers? If the business model allows for the downtime caused by putting all your internet connectivity in one bucket. james
Re: Enterprise Multihoming
Thanks to everyone who has responded so far. I'm glad that I got some opinions here before I proceeded. I also participate in another list that has some fairly experienced people on it. They prevailing opinion there was that multihoming to multiple providers was overrated and largely unnecessary, and they just about had me convinced. My current opinion is that since we can't accept much downtime in the case of a single provider failure, it's probably not wise to put all of our eggs in Sprint's basket even if all circuits are geographically diverse. Thanks again, John --
RE: Enterprise Multihoming
Look at it this way: If Multi-homing to ensure maximum reliabilty was not a good thing: why would XYZ isp do it? Take this example: Remember last year (or year before?) when MCI had the routing issue on the east coast? I had a friend that had 2 T-1's to MCI, he lost all reachability for over 5 hours. I had another friend that had a T-1 from MCI and one from ATT. He stayed up, and so did his ecommerce site. So the end questions is: Do you trust your upstream enough to bank your business, or more importantly your reputation as an IT professional, on the ability of everyone at your ISP to maintain their network and everything that gives you access 99.999% of the time? Jim --Original Message- -From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of -Gregory Taylor -Sent: Thursday, March 11, 2004 11:41 AM -To: John Neiberger; [EMAIL PROTECTED] -Subject: Re: Enterprise Multihoming - - - -Mutli-homing a non-ISP network or system on multiple carriers -is a good -way to maintain independent links to the internet by means of -different -peering, uplinks, over-all routing and reliability. My -network on NAIS -is currently multi-homed through ATT. I use a single -provider as both -of my redundant links via 100% Fiber network. Even though this is -cheaper for me, all it takes is for ATT to have some major -outage and I -will be screwed. If I have a backup fiber line from say, Global -Crossing, then it doesn't matter if ATT takes a nose dive, I -still have -my redundancy there. - -That is why most non-ISPs hold multihoming via different providers as -their #1 choice. - -Greg - -John Neiberger wrote: - -On another list we've been having multihoming discussions again and I -wanted to get some fresh opinions from you. - -For the past few years it has been fairly common for non-ISPs to -multihome to different providers for additional redundancy in case a -single provider has problems. I know this is frowned upon now, -especially since it helped increase the number of autonomous -systems and -routing table prefixes beyond what was really necessary. It -seems to me -that a large number of companies that did this could just have well -ordered multiple, geographically separate links to the same provider. - -What is the prevailing wisdom now? At what point do you feel -that it is -justified for a non-ISP to multihome to multiple providers? I ask -because we have three links: two from Sprint and one from Global -Crossing. I'm considering dropping the GC circuit and adding another -geographically-diverse connection to Sprint, and then -removing BGP from -our routers. - -I see a few upsides to this, but are there any real downsides? - -Flame on. :-) - -Thanks, -John --- - - - - - - -
Re: Enterprise Multihoming
On Thu, 11 Mar 2004, Gregory Taylor wrote: Mutli-homing a non-ISP network or system on multiple carriers is a good way to maintain independent links to the internet by means of different peering, uplinks, over-all routing and reliability. My network on NAIS is currently multi-homed through ATT. I use a single provider as both of my redundant links via 100% Fiber network. Even though this is cheaper for me, all it takes is for ATT to have some major outage and I will be screwed. If I have a backup fiber line from say, Global Crossing, then it doesn't matter if ATT takes a nose dive, I still have my redundancy there. Well, I think this, in many cases, boils down to being able to pick the right provider. I mean, some providers go belly-up from time to time. Others are designed/run better. For a major provider, complete outage of all of its customers is such a big thing they'll want to avoid it always. If it happens, for a brief moment, once in five years (for example), for most companies that's an acceptable level of risk. -- Pekka Savola You each name yourselves king, yet the Netcore Oykingdom bleeds. Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
Re: Enterprise Multihoming
PH Date: Thu, 11 Mar 2004 18:21:03 +0200 PH From: Petri Helenius PH Depending on your requirements, the option of having somebody PH redistribute all their BGP routes into ISIS or OSPF might not PH worth looking forward to. Couldn't quite parse this, but it sounds scary. Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _ DO NOT send mail to the following addresses : [EMAIL PROTECTED] -or- [EMAIL PROTECTED] -or- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked.
Re: Enterprise Multihoming
JN Date: Thu, 11 Mar 2004 10:10:17 -0700 JN From: John Neiberger JN My current opinion is that since we can't accept much JN downtime in the case of a single provider failure, it's JN probably not wise to put all of our eggs in Sprint's basket JN even if all circuits are geographically diverse. Use multiple border routers. Keep your IGP lean and nimble. Think about BGP/IGP synchronization. WAN links can fail, but so can ethernet links and entire routers. Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _ DO NOT send mail to the following addresses : [EMAIL PROTECTED] -or- [EMAIL PROTECTED] -or- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked.
Re: Enterprise Multihoming
JN My current opinion is that since we can't accept much JN downtime in the case of a single provider failure, it's JN probably not wise to put all of our eggs in Sprint's basket JN even if all circuits are geographically diverse. Use multiple border routers. Keep your IGP lean and nimble. Think about BGP/IGP synchronization. WAN links can fail, but so can ethernet links and entire routers. We have multiple border routers and are fairly redundant internally. As it is now, any single piece of equipment could fail (except in one case that I intend to rectify soon) or any two of our three Internet connections could fail and no one would notice much except for perhaps slower connections. I've discovered the wonders of fault-tolerant transceivers and I'll be redesigning a portion of that part of the network around them. Once I'm done, quite literally any single device could fail and no one would notice. John --
Re: Enterprise Multihoming
There is another thing - if you are multi-homed, and want to switch providers, it is pretty seamless and painless - no renumbering, no loss of connection, etc., as you always have a redundant path. On Thursday, March 11, 2004, at 12:34 PM, Pekka Savola wrote: On Thu, 11 Mar 2004, Gregory Taylor wrote: Mutli-homing a non-ISP network or system on multiple carriers is a good way to maintain independent links to the internet by means of different peering, uplinks, over-all routing and reliability. My network on NAIS is currently multi-homed through ATT. I use a single provider as both of my redundant links via 100% Fiber network. Even though this is cheaper for me, all it takes is for ATT to have some major outage and I will be screwed. If I have a backup fiber line from say, Global Crossing, then it doesn't matter if ATT takes a nose dive, I still have my redundancy there. Well, I think this, in many cases, boils down to being able to pick the right provider. I mean, some providers go belly-up from time to time. Others are designed/run better. For a major provider, complete outage of all of its customers is such a big thing they'll want to avoid it always. If it happens, for a brief moment, once in five years (for example), for most companies that's an acceptable level of risk. -- Pekka Savola You each name yourselves king, yet the Netcore Oykingdom bleeds. Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings Regards Marshall Eubanks T.M. Eubanks e-mail : [EMAIL PROTECTED] http://www.telesuite.com
Re: Enterprise Multihoming
On Thu, 11 Mar 2004, Marshall Eubanks wrote: There is another thing - if you are multi-homed, and want to switch providers, it is pretty seamless and painless - no renumbering, no loss of connection, etc., as you always have a redundant path. Sure -- though many ISPs will probably let you keep the address space, even if you switch away completely -- as long as you pay them enough (or the other ISP to route it). Bad practice, but has happened a lot, and probably still does :) FWIW, even if you are multihomed, that does not in and of itself require that you own address space. Public AS number is often enough (and even private will do, but that leads to other kind of mess.) -- Pekka Savola You each name yourselves king, yet the Netcore Oykingdom bleeds. Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
Re: Enterprise Multihoming
John Neiberger wrote: On another list we've been having multihoming discussions again and I wanted to get some fresh opinions from you. Whilst the topic's under discussion may I present myself as a lightning rod :) by asking: (a) Has anyone here used any of the 'basement multi-homing in a box' products such as Checkpoint's ISP Redundancy feature? http://www.checkpoint.com/products/connect/vpn-1_isp_redundancy.html (The 'VPN-1' brand is slightly misleading - it's a generic firewall.) This allows edge networks to multihome between separate ISPs. When it was first mentioned around the office I explained that it couldn't possibly work, and my colleagues explained to me that I was full of it and that the product is on the market and in use. (It has subsequently been lab'd here and seemed to work between our main link (UUnet) and a humble BT DSL line.) As far as I understand it, it's a form of NAT - the device keeps track of which session's packets are going where and spreads traffic around. If one ISP goes down it'll fail over to the other link. (b) I suspect the answer will be a vehement 'no!' -- if so, why? Obviously this won't scale terribly well at the service provider level but for edge networks - what's wrong with it? Obviously this only works for outbound sessions but there are plenty of large enterprises happy to keep the majority of inbound services (web etc) off in a nice secure hosting centre where real netops will use BGP for real multihoming. cheers \a -- Andrew Simmons Penetration Tester | Security Consultant MIS Corporate Defence Solutions, Ltd. Hermitage Court, Hermitage Lane, Maidstone, Kent ME16 9NT Tel: 01622 723432 / Mobile: 07739 834833 (sorry about the disclaimer - there's nothing I can do about it :( ) The information contained in this message or any of its attachments may be privileged and confidential and intended for the exclusive use of the intended recipient. If you are not the intended recipient any disclosure, reproduction, distribution or other dissemination or use of this communications is strictly prohibited. The views expressed in this e-mail are those of the individual and not necessarily of MIS Corporate Defence Solutions Ltd. Any prices quoted are only valid if followed up by a formal written quote. If you have received this transmission in error, please contact our Security Manager on +44 (01622) 723410. This email is intended for the recipient only and contains confidential information, some or all of which may be legally privileged. If you are not the intended recipient, you must not use, save, disclose, distribute, copy, print or rely on this email or any information contained within it. Please notify the sender by return and delete it from your computer. Thank you.
Re: Enterprise Multihoming
E.B. Dreger wrote: PH Date: Thu, 11 Mar 2004 18:21:03 +0200 PH From: Petri Helenius PH Depending on your requirements, the option of having somebody PH redistribute all their BGP routes into ISIS or OSPF might not PH worth looking forward to. Couldn't quite parse this, but it sounds scary. I´m refering to the most popular way of causing an IGP meltdown. Obviously there are other ways, like software defects to make your IGP go mad. But when your upstream´s IGP does that, you want to have provider B to switch over to. It probably has gotten better when the Internet has matured but a few years back when I was more involved in day-to-day operations it was a few times a year when excersizing this option was the best course of action. Pete
Re: Enterprise Multihoming
Whilst the topic's under discussion may I present myself as a lightning rod :) by asking: (a) Has anyone here used any of the 'basement multi-homing in a box' products such as Checkpoint's ISP Redundancy feature? http://www.checkpoint.com/products/connect/vpn-1_isp_redundancy.html (The 'VPN-1' brand is slightly misleading - it's a generic firewall.) This allows edge networks to multihome between separate ISPs. When it was first mentioned around the office I explained that it couldn't possibly work, and my colleagues explained to me that I was full of it and that the product is on the market and in use. (It has subsequently been lab'd here and seemed to work between our main link (UUnet) and a humble BT DSL line.) As far as I understand it, it's a form of NAT - the device keeps track of which session's packets are going where and spreads traffic around. If one ISP goes down it'll fail over to the other link. There are similar boxes from FatPipe and Radware (and others) that promise the same thing. I've done some light research on them and while I can see some positives, I don't prefer them to our current solution. My boss asked me to take a look at them, again, because he's concerned that there's little BGP experience in our department apart from me and he thought that might be one possible solution. It still may be but I don't like the hoops you have to jump through to make these devices work. Then again, I don't have any practical experience with them and I hope someone who has will chime in. John --
Re: Enterprise Multihoming
John Neiberger wrote: Whilst the topic's under discussion may I present myself as a lightning rod :) by asking: (a) Has anyone here used any of the 'basement multi-homing in a box' products such as Checkpoint's ISP Redundancy feature? http://www.checkpoint.com/products/connect/vpn-1_isp_redundancy.html (The 'VPN-1' brand is slightly misleading - it's a generic firewall.) You can do the same thing with your existing cisco: http://www.cisco.com/warp/customer/cc/pd/iosw/ioft/ionetn/tech/emios_wp.htm
Re: Enterprise Multihoming
Jay Ford wrote: [snip] Many/most of my external connectivity problems are provider-related rather than circuit-related. Having two circuits to a single provider doesn't help when that provider is broken. I'm not saying that multi-ISP BGP-based multi-homing is risk-free, but I don't see multi-circuit single-provider as a viable alternative. FWIW, I've had almost the exact opposite experience. Almost all of our connectivity problems have been circuit issues. Two T1s to the same ISP at one site has saved us from a lot of pain. OTOH, we also do have some ISP diversity, though we haven't needed it nearly as much as redundant circuits. YMMV. HAND. -- Crist J. Clark [EMAIL PROTECTED] Globalstar Communications(408) 933-4387
Re: Enterprise Multihoming
John As already stated by lots of folks on the list, this is largely a business decision rather than a technical one. However, there are some more useful thoughts: 1. Is the decision to multi-home consistent with your other redundancy plans? For example, why go through all the trouble of multi-homing and setting up BGP, only for both circuits to be plugged into the same router? ..or, two routers but neither of them on UPS. This is akin to insisting on a Class A bank-grade firewall but not bothering to put a lock on the server room door... 2. Multi-homing is usually considered critical when one is discussing hosting of some kind. Could you be served with multiple servers in geographically separate collocation centers inside one ASN? While many MIS departments like to have direct access to their own servers, this can often be an emotional preference rather than a technical one. Often only the public facing servers need BGP redundancy. The back-ends can be set up to fail-over to separate VPN/IPs in separate ASNs. Having said all that, I prefer physical access to my machines too. So I'm a hypocrite. 3. If you are not doing hosting, a two-ISP NAT solution may make more sense than BGP. In addition to burdening the global routing tables; good BGP management is expensive. It involves either hiring someone with the proper expertise/experience or purchasing that expertise. Relatively speaking, there are not a lot good experienced BGP admins out there. 4. What is the price of downtime, in real dollars? For many business, this really can be estimated. Consider lost time (wages, utilities, etc.) and lost sales. Then compare it to the various options. Just my two cents, John At 10:04 AM 3/11/2004, you wrote: On another list we've been having multihoming discussions again and I wanted to get some fresh opinions from you. For the past few years it has been fairly common for non-ISPs to multihome to different providers for additional redundancy in case a single provider has problems. I know this is frowned upon now, especially since it helped increase the number of autonomous systems and routing table prefixes beyond what was really necessary. It seems to me that a large number of companies that did this could just have well ordered multiple, geographically separate links to the same provider. What is the prevailing wisdom now? At what point do you feel that it is justified for a non-ISP to multihome to multiple providers? I ask because we have three links: two from Sprint and one from Global Crossing. I'm considering dropping the GC circuit and adding another geographically-diverse connection to Sprint, and then removing BGP from our routers. I see a few upsides to this, but are there any real downsides? Flame on. :-) Thanks, John --
Re: Enterprise Multihoming
PH Date: Thu, 11 Mar 2004 20:31:52 +0200 PH From: Petri Helenius PH I´m refering to the most popular way of causing an IGP PH meltdown. Obviously there are other ways, like software PH defects to make your IGP go mad. But when your upstream´s IGP PH does that, you want to have provider B to switch over to. Okay. I was unsure if you were referring to a clueless downstream bloating their IGP, or a clueless transit network redistributing downstream routes. Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _ DO NOT send mail to the following addresses : [EMAIL PROTECTED] -or- [EMAIL PROTECTED] -or- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked.
Re: Enterprise Multihoming
There are similar boxes from FatPipe and Radware (and others) that promise the same thing. I've done some light research on them and while I can see some positives, I don't prefer them to our current solution. Then again, I don't have any practical experience with them and I hope someone who has will chime in. On the fatpipe side, I can chime in. I've worked with their Superstream products. As with all products there are good points, but I have a LOT of bad points for the Superstream. It starts with being based on Caldera openlinux and a required Java interface for all management. I wouldn't use this product again if I could help it. They may have other products that work better, particularly in the case of true multihoming (the superstream is really so a business can pay for two DSL connections and get double the bandwidth) and such. If anyone wants more details, let me know.
