Re: Failover how much complexity will it add?
Randy Bush wrote: It has been routinely observed in nanog presentations that settlement free providers by their nature miss a few prefixes that well connected transit purchasing ISPs carry. just trying to understand what you mean, o no transit-free provider actually has all (covering) prefixes needed to cover the active space, but o one or more reasonably small subsets of the set of transit-free providers do cover the whole active space. If your goal is near-complete coverage, under normal circumstances you need more than one (your subset). Settlement-free provider peering spats are a degenerate condition of the normal state of affairs. The non-settlement-free provider has some subset already. Pointing default into a settlement-free provider, that is deliberately not speaking to another is obviously a recipe to lose data, which speaks to the question of whether one should as for a full table from settlement free upstreams. My somewhat obtuse point was that this isn't some wild west frontier justice sort of affair, but rather, the normal state of affairs. randy
Re: Failover how much complexity will it add?
a...@baklawasecrets.com wrote: Actually thinking about this, I still need to understand the implications of not taking a full routing table to my setup. So what is the likely impact going to be if I take partial instead of full routing table. Would appreciate any feedback on this. My organisation is only looking at using BGP as a means of failover between two separate upstream ISPs. We are not an ISP. I'm up against the same issue. I'm having a hard time understanding what partial routes will accomplish in the following scenario. Let's say we were BGP multi homed and accepting partial tables from two top level ISPs (like Sprint, Cogent, Telia, ATT whatever). Let's say they get into a peering spat with another top level ISP. This results in one of your upstreams not routing packets to one or more ASs. In this situation, as far as I can tell, you'd want a full routing tables from your upstreams. Without a full routing table how would your router know that certain ASs are reachable through one upstream, but not the other? In this day of and age of wild-west, cowboy attitudes between some of the biggest players on the Internet, does protecting against these problems require a routing device that can handle multiple full routing tables? It would seem so... Cheers, Stef
Re: Failover how much complexity will it add?
Stef Walter wrote: In this day of and age of wild-west, cowboy attitudes between some of the biggest players on the Internet, does protecting against these problems require a routing device that can handle multiple full routing tables? It would seem so... It has been routinely observed in nanog presentations that settlement free providers by their nature miss a few prefixes that well connected transit purchasing ISPs carry. If business requirements for reachability necessitate multi-homing then carrying the tables if probably also a requirement. joel Cheers, Stef
Re: Failover how much complexity will it add?
It has been routinely observed in nanog presentations that settlement free providers by their nature miss a few prefixes that well connected transit purchasing ISPs carry. just trying to understand what you mean, o no transit-free provider actually has all (covering) prefixes needed to cover the active space, but o one or more reasonably small subsets of the set of transit-free providers do cover the whole active space. randy
Re: Failover how much complexity will it add?
I would have thought that this lesson would still be fresh in the minds of people, as we just passed 256K routes a little while ago and broke a whole bunch of Catalyst 6500/7600 SUP720-3B's (etc). I guess the pain isn't as memorable as I thought. Not all of us forgot... I remember the day our 7606's filled and started trying to software switch. Was a painful day indeed. At least we knew it was coming.. just couldn't do anything about it due to budgets.
Re: Failover how much complexity will it add?
You will laugh, but the budget at the moment looks like £13k. Impossible? Do only linux and openbsd solutions remain in the mix for this pittance? On Sun 11:47 PM , Dale Rumph da...@ibbs.com wrote: What does your budget look like? A pair of Cisco 7246vxr's with G1's sitting on the edge of the network would be very effective and still allow expansion. Or you could go up to the 7609. However this gear may be slightly overkill. You might be ok with a 3660 enterprise and a ton of ram. I have done single sessions on them but not with the level of HA your looking for. Just my 2c - Original Message - From: a...@baklawasecrets.com To: nanog@nanog.org Sent: Sun Nov 08 18:36:31 2009 Subject: Re: Failover how much complexity will it add? Basically the organisation that I'm working for will not have the skills in house to support a linux or bsd box. They will have trouble with supporting the BGP configuration, however I don't think they will be happy with me if I leave them with a linux box when they don't have linux/unix resource internally. At least with a Cisco or Juniper they are familiar with IOS and it won't be too foreign to them. On Sun 11:30 PM , Renato Frederick wrote: There are any problems with quagga+BSD/Linux that you know or something like that? Or in your scenario a cisco/juniper box is a requirement? I'm asking this because I'm always running BGP with upstreams providers using quagga on BSD and everything is fine until now. -- From: Sent: Sunday, November 08, 2009 8:39 PM To: Subject: Re: Failover how much complexity will it add? So if my requirements are as follows: - BGP router capable of holding full Internet routing table. (whether I go for partial or full, I think I want something with full capability). - Capable of pushing 100meg plus of mixed traffic. What are my options? I want to exclude openbsd, or linux with quagga. Probably looking at Cisco or Juniper products, but interested in any other alternatives people suggest. I realise this is quite a broad question, but hoping this will provide a starting point. Oh and if I have missed any specs I should have included above, please let me know. Thanks Adel
Re: Failover how much complexity will it add?
Looking at two 100Mbit/s BGP connections, so I think I want something that will do more than 100 but nowhere close to a gig. So full routing table capability with throughput of mixed traffic around 200Mbit/s. If that makes sense. Do the 2850s fall into that sort of price point? Adel On Mon 11:13 AM , Joe Abley jab...@hopcount.ca wrote: On 2009-11-09, at 19:53, a...@baklawasecrets.com wrote: You will laugh, but the budget at the moment looks like £13k. Impossible? Do only linux and openbsd solutions remain in the mix for this pittance? I don't see an indication of the traffic you need to push (maybe I deleted a message too enthusiastically) but check the 2800 series from cisco. The 2850 will take full tables and has gigabit interfaces, but don't expect them to do wire speed. Other 2800s suffer from reduced RAM, but perhaps you don't need full tables. Also look at Juniper J-series boxes, and maybe Force10 S-series boxes. There's a healthy market in used cisco gear in most places I have ever visited, if you don't need new. Joe
Re: Failover how much complexity will it add?
Basically the organisation that I'm working for will not have the skills in house to support a linux or bsd box. They will have trouble with supporting the BGP configuration, however I don't think they will be happy with me if I leave them with a linux box when they don't have linux/unix resource internally. At least with a Cisco or Juniper they are familiar with IOS and it won't be too foreign to them. On Sun 11:47 PM , Dale Rumph da...@ibbs.com wrote: What does your budget look like? A pair of Cisco 7246vxr's with G1's sitting on the edge of the network would be very effective and still allow expansion. Or you could go up to the 7609. However this gear may be slightly overkill. You might be ok with a 3660 enterprise and a ton of ram. I have done single sessions on them but not with the level of HA your looking for. Just my 2c You will laugh, but the budget at the moment looks like £13k. Impossible? Do only linux and openbsd solutions remain in the mix for this pittance? No, you have the buy-it-off-eBay solutions as well. Beware the fakes. If they're familiar with IOS, then they can be familiar with Quagga about as easily as they could be familiar with a switch or other network gizmo that had a Ciscoesque CLI but wasn't actually Cisco. You've painted yourself into a corner. I have a word for you: Reconsider. I don't care what you reconsider, but reconsider something. You can reconsider taking BGP with a full table. You can reconsider Quagga. Or you can reconsider your budget. This is the end result of the pick any two problem. Most end user organizations have no need of full routes in BGP. To try to take them dooms TCAM-based equipment at some future point, though if you have a lot of money to throw at it, you can make that point be years in the future. It is essentially planned obsolescence. If you discard the requirement for full routes, you open up a bunch of reasonably-priced possibilities. Finding someone knowledgeable in BSD or Linux isn't that rough. Unlike a Cisco 76xx router, the hardest part of a Quagga-based solution is finding the right mix of hardware and software at the beginning. PC hardware has a lot going for AND against it. There is no reason you can't make a good router out of a PC. If you buy the Cisco software-based routers, you're essentially buying a prepackaged version, except that it'll be specced to avoid any real competition with their low-end TCAM-based offerings. A contemporary PC can easily route gigabits. Vyatta makes what I hear is a fantastic canned solution of some sort, for a reasonable cost, and they will sell just software or software/hardware. If you really can't put it together yourself, there's someone to do it for you. Reconsidering your budget is probably the most painful thing to do, but also opens up the just buy big Cisco option. I think my point here would have to be that what you're looking for would have needed big Cisco... ten years ago. Now, dealing with a few hundred megs of traffic, that's not that big a deal, the thing that's killing you is the BGP table size. Your best option may be to see if you can settle for partial routes plus a default. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: Failover how much complexity will it add?
Thanks, Their offering certainly looks appealing. Will be interested to hear user experiences of the Vyatta BGP router range. Having said that I will still be examining the Cisco offering, just because of the support, larger user community and skills base issue. However if I can't meet the price point using Cisco, obviously other solutions are going to come into the picture. Adel On Mon 11:39 AM , Arnold Nipper arn...@nipper.de wrote: On 09.11.2009 11:53 a...@baklawasecrets.com wrote You will laugh, but the budget at the moment looks like £13k. Impossible? Do only linux and openbsd solutions remain in the mix for this pittance? Do you know Vyatta (http://www.vyatta.com/)? [1] CLI and config is Cisco-ish. Prices e.g. Vyatta Appliance, Vyatta 2502, Enterprise Subscription, Basic Warranty, 1 Year (ships with US Power Cord as standard) (Typically ships in 10-12 business days) Price: $2,997.00 Best regards, Arnold -- Arnold Nipper / nIPper consulting, Sandhausen, Germany email: arn...@nipper.de phone: +49 6224 9259 299 mobile: +49 172 2650958 fax: +49 6224 9259 333 Links: -- [1] http://webmail.123-reg.co.uk/parse.php?redirect=http://www.vyatta.com/%29%3 F
Re: Failover how much complexity will it add?
Thanks, I've taken your advice and decided to reconsider my requirement for a full routing table. I believe I'm being greedy and a partial table will be sufficient. With regards to Linux/BSD, its not the CLI of quagga that will be an issue, rather the sysadmin and lack of supporting infrastructure for Linux boxes within the organisation. So things like package management, syslog servers, monitoring, understanding of security issues etc. I don't want to leave them with a linux/bsd solution that they won't be able to maintain/manage effectively when I am gone. Thanks for your comments. Look forward to hearing which solutions come back into the mix having dropped the full routing table requirement. Regards, Adel On Mon 11:45 AM , Joe Greco jgr...@ns.sol.net wrote: Basically the organisation that I'm working for will not have the skills in house to support a linux or bsd box. They will have trouble with supporting the BGP configuration, however I don't think they will be happy with me if I leave them with a linux box when they don't have linux/unix resource internally. At least with a Cisco or Juniper they are familiar with IOS and it won't be too foreign to them. On Sun 11:47 PM , Dale Rumph wrote: What does your budget look like? A pair of Cisco 7246vxr's with G1's sitting on the edge of the network would be very effective and still allow expansion. Or you could go up to the 7609. However this gear may be slightly overkill. You might be ok with a 3660 enterprise and a ton of ram. I have done single sessions on them but not with the level of HA your looking for. Just my 2c You will laugh, but the budget at the moment looks like £13k. Impossible? Do only linux and openbsd solutions remain in the mix for this pittance? No, you have the buy-it-off-eBay solutions as well. Beware the fakes. If they're familiar with IOS, then they can be familiar with Quagga about as easily as they could be familiar with a switch or other network gizmo that had a Ciscoesque CLI but wasn't actually Cisco. You've painted yourself into a corner. I have a word for you: Reconsider. I don't care what you reconsider, but reconsider something. You can reconsider taking BGP with a full table. You can reconsider Quagga. Or you can reconsider your budget. This is the end result of the pick any two problem. Most end user organizations have no need of full routes in BGP. To try to take them dooms TCAM-based equipment at some future point, though if you have a lot of money to throw at it, you can make that point be years in the future. It is essentially planned obsolescence. If you discard the requirement for full routes, you open up a bunch of reasonably-priced possibilities. Finding someone knowledgeable in BSD or Linux isn't that rough. Unlike a Cisco 76xx router, the hardest part of a Quagga-based solution is finding the right mix of hardware and software at the beginning. PC hardware has a lot going for AND against it. There is no reason you can't make a good router out of a PC. If you buy the Cisco software-based routers, you're essentially buying a prepackaged version, except that it'll be specced to avoid any real competition with their low-end TCAM-based offerings. A contemporary PC can easily route gigabits. Vyatta makes what I hear is a fantastic canned solution of some sort, for a reasonable cost, and they will sell just software or software/hardware. If you really can't put it together yourself, there's someone to do it for you. Reconsidering your budget is probably the most painful thing to do, but also opens up the just buy big Cisco option. I think my point here would have to be that what you're looking for would have needed big Cisco... ten years ago. Now, dealing with a few hundred megs of traffic, that's not that big a deal, the thing that's killing you is the BGP table size. Your best option may be to see if you can settle for partial routes plus a default. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net [1] We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples. Links: -- [1] http://webmail.123-reg.co.uk/parse.php?redirect=http://www.sol.net
Re: Failover how much complexity will it add?
Actually thinking about this, I still need to understand the implications of not taking a full routing table to my setup. So what is the likely impact going to be if I take partial instead of full routing table. Would appreciate any feedback on this. My organisation is only looking at using BGP as a means of failover between two separate upstream ISPs. We are not an ISP. Thanks Adel On Mon 1:32 PM , a...@baklawasecrets.com wrote: Thanks, I've taken your advice and decided to reconsider my requirement for a full routing table. I believe I'm being greedy and a partial table will be sufficient. With regards to Linux/BSD, its not the CLI of quagga that will be an issue, rather the sysadmin and lack of supporting infrastructure for Linux boxes within the organisation. So things like package management, syslog servers, monitoring, understanding of security issues etc. I don't want to leave them with a linux/bsd solution that they won't be able to maintain/manage effectively when I am gone. Thanks for your comments. Look forward to hearing which solutions come back into the mix having dropped the full routing table requirement. Regards, Adel On Mon 11:45 AM , Joe Greco wrote: Basically the organisation that I'm working for will not have the skills in house to support a linux or bsd box. They will have trouble with supporting the BGP configuration, however I don't think they will be happy with me if I leave them with a linux box when they don't have linux/unix resource internally. At least with a Cisco or Juniper they are familiar with IOS and it won't be too foreign to them. On Sun 11:47 PM , Dale Rumph wrote: What does your budget look like? A pair of Cisco 7246vxr's with G1's sitting on the edge of the network would be very effective and still allow expansion. Or you could go up to the 7609. However this gear may be slightly overkill. You might be ok with a 3660 enterprise and a ton of ram. I have done single sessions on them but not with the level of HA your looking for. Just my 2c You will laugh, but the budget at the moment looks like £13k. Impossible? Do only linux and openbsd solutions remain in the mix for this pittance? No, you have the buy-it-off-eBay solutions as well. Beware the fakes. If they're familiar with IOS, then they can be familiar with Quagga about as easily as they could be familiar with a switch or other network gizmo that had a Ciscoesque CLI but wasn't actually Cisco. You've painted yourself into a corner. I have a word for you: Reconsider. I don't care what you reconsider, but reconsider something. You can reconsider taking BGP with a full table. You can reconsider Quagga. Or you can reconsider your budget. This is the end result of the pick any two problem. Most end user organizations have no need of full routes in BGP. To try to take them dooms TCAM-based equipment at some future point, though if you have a lot of money to throw at it, you can make that point be years in the future. It is essentially planned obsolescence. If you discard the requirement for full routes, you open up a bunch of reasonably-priced possibilities. Finding someone knowledgeable in BSD or Linux isn't that rough. Unlike a Cisco 76xx router, the hardest part of a Quagga-based solution is finding the right mix of hardware and software at the beginning. PC hardware has a lot going for AND against it. There is no reason you can't make a good router out of a PC. If you buy the Cisco software-based routers, you're essentially buying a prepackaged version, except that it'll be specced to avoid any real competition with their low-end TCAM-based offerings. A contemporary PC can easily route gigabits. Vyatta makes what I hear is a fantastic canned solution of some sort, for a reasonable cost, and they will sell just software or software/hardware. If you really can't put it together yourself, there's someone to do it for you. Reconsidering your budget is probably the most painful thing to do, but also opens up the just buy big Cisco option. I think my point here would have to be that what you're looking for would have needed big Cisco... ten years ago. Now, dealing with a few hundred megs of traffic, that's not that big a deal, the thing that's killing you is the BGP table size. Your best option may be to see if you can settle for partial routes plus a default. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net [1] [1] We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples. Links: -- [1]
Re: Failover how much complexity will it add?
Thanks, I've taken your advice and decided to reconsider my requirement for a full routing table. I believe I'm being greedy and a partial table will be sufficient. With regards to Linux/BSD, its not the CLI of quagga that will be an issue, rather the sysadmin and lack of supporting infrastructure for Linux boxes within the organisation. So things like package management, You don't need to run Apache on your router. syslog servers, If you didn't have syslog servers for the Cisco, you don't need one for the Quagga. monitoring, If you didn't monitor the Cisco, you don't need to monitor the Quagga. understanding of security issues etc. What security issues? The thing is, people get all tied up over this idea that it is some major ongoing burden to support a Linux based device. I have a shocker for you. The CPE your residential broadband relies on may well run Linux, and you didn't even know it. The wifi router you use may run Linux. There are thousands of embedded uses for Linux. I highly doubt that the average TiVo user has a degree in Linux. Many different things you use in day-to-day life run Linux, BSD, VxWorks, or whatever ... mostly without any need of someone to handhold them on security issues. Of course, security issues do come up. But they do with Cisco as well. A proper Linux router doesn't have ports open, aside from bgp and ssh, and those can be firewalled appropriately. This makes it very difficult to have any meaningful security problems relating to the platform... You can expect the occasional issue. Just like anything else. But trying to compare it to security issues on a general Linux platform is only meaningful if you're trying to argue against the solution. (I'm a BSD guy myself, but I don't see any reason for undue Linux paranoia) I don't want to leave them with a linux/bsd solution that they won't be able to maintain/manage effectively when I am gone. If they're unable to maintain something as straightforward as BSD or Linux when you're gone, this raises alarm bells as to whether or not BGP is really suited for them. BGP is *much* more arcane, relatively speaking. You can go to your local bookstore and pick up a ton of Linux or BSD sysadm books, but you'll be lucky to find a book on BGP. Thanks for your comments. Look forward to hearing which solutions come back into the mix having dropped the full routing table requirement. There's a whole plethora of BGP-capable gear that becomes possible once you make that call. Cisco and Juniper both make good gear. A variety of other mfrs do as well. Something as old as an Ascend GRF 400 (fast ethernet, line speed, 150K routes, ~1998?) is perfectly capable of dealing with the load, though I mention this primarily to make the point that there is a lot of equipment within the last decade that can support this. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: Failover how much complexity will it add?
Hi Joe, I agree with most of what you say below regarding linux sysadmin, BSD etc. I'm quite happy and actually would prefer building a linux solution on our own hardware. However, politically I think this is going to be difficult. I just feel that they will be more comfortable with embedded network boxes as a pose to a linux solution. I guess what I'm saying is this is partially a political thing. Adel On Mon 3:20 PM , Joe Greco jgr...@ns.sol.net wrote: Thanks, I've taken your advice and decided to reconsider my requirement for a full routing table. I believe I'm being greedy and a partial table will be sufficient. With regards to Linux/BSD, its not the CLI of quagga that will be an issue, rather the sysadmin and lack of supporting infrastructure for Linux boxes within the organisation. So things like package management, You don't need to run Apache on your router. syslog servers, If you didn't have syslog servers for the Cisco, you don't need one for the Quagga. monitoring, If you didn't monitor the Cisco, you don't need to monitor the Quagga. understanding of security issues etc. What security issues? The thing is, people get all tied up over this idea that it is some major ongoing burden to support a Linux based device. I have a shocker for you. The CPE your residential broadband relies on may well run Linux, and you didn't even know it. The wifi router you use may run Linux. There are thousands of embedded uses for Linux. I highly doubt that the average TiVo user has a degree in Linux. Many different things you use in day-to-day life run Linux, BSD, VxWorks, or whatever ... mostly without any need of someone to handhold them on security issues. Of course, security issues do come up. But they do with Cisco as well. A proper Linux router doesn't have ports open, aside from bgp and ssh, and those can be firewalled appropriately. This makes it very difficult to have any meaningful security problems relating to the platform... You can expect the occasional issue. Just like anything else. But trying to compare it to security issues on a general Linux platform is only meaningful if you're trying to argue against the solution. (I'm a BSD guy myself, but I don't see any reason for undue Linux paranoia) I don't want to leave them with a linux/bsd solution that they won't be able to maintain/manage effectively when I am gone. If they're unable to maintain something as straightforward as BSD or Linux when you're gone, this raises alarm bells as to whether or not BGP is really suited for them. BGP is *much* more arcane, relatively speaking. You can go to your local bookstore and pick up a ton of Linux or BSD sysadm books, but you'll be lucky to find a book on BGP. Thanks for your comments. Look forward to hearing which solutions come back into the mix having dropped the full routing table requirement. There's a whole plethora of BGP-capable gear that becomes possible once you make that call. Cisco and Juniper both make good gear. A variety of other mfrs do as well. Something as old as an Ascend GRF 400 (fast ethernet, line speed, 150K routes, ~1998?) is perfectly capable of dealing with the load, though I mention this primarily to make the point that there is a lot of equipment within the last decade that can support this. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net [1] We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples. Links: -- [1] http://webmail.123-reg.co.uk/parse.php?redirect=http://www.sol.net
Re: Failover how much complexity will it add?
a...@baklawasecrets.com wrote: Actually thinking about this, I still need to understand the implications of not taking a full routing table to my setup. So what is the likely impact going to be if I take partial instead of full routing table. Would appreciate any feedback on this. My organisation is only looking at using BGP as a means of failover between two separate upstream ISPs. We are not an ISP. Some Cisco L3 switches should support this fine. A 3560 or 3750 can speak BGP and route at line rate as long as your total number of routes will fit in its TCAM space. Ask your upstreams how big a partial feed from them is. desktop routing template: The selected template optimizes the resources in the switch to support this level of features for 8 routed interfaces and 1024 VLANs. number of unicast mac addresses: 3K number of IPv4 IGMP groups + multicast routes:1K number of IPv4 unicast routes:11K number of directly-connected IPv4 hosts:3K number of indirect IPv4 routes: 8K number of IPv4 policy based routing aces: 0.5K number of IPv4/MAC qos aces: 0.5K number of IPv4/MAC security aces: 1K If you ever need IPv6 it gets smaller: number of unicast mac addresses: 2K number of IPv4 IGMP groups + multicast routes:1K number of IPv4 unicast routes:3K number of directly-connected IPv4 hosts:2K number of indirect IPv4 routes: 1K number of IPv6 multicast groups: 1.125k number of directly-connected IPv6 addresses: 2K number of indirect IPv6 unicast routes: 1K number of IPv4 policy based routing aces: 0 number of IPv4/MAC qos aces: 0.5K number of IPv4/MAC security aces: 1K number of IPv6 policy based routing aces: 0 number of IPv6 qos aces: 0.625k number of IPv6 security aces: 0.5K Anything in Cisco land that can hold two full tables in hardware and can do line rate is going to be hideously expensive. ~Seth
Re: Failover how much complexity will it add?
On Mon, 09 Nov 2009 13:39:34 GMT, Adam Armstrong said: Sure, if you want to hand over your entire profit margin to a 3rd party. Do you really want to give away the keys to your business, and rely entirely upon a third party organisation? Better to acquire the skills which are vital to your organisation yourself. Umm.. You did that *anyhow* the instant you paid somebody else to run the cables to your location rather than dig your own ditches. Similarly for electricity and everything else you don't create yourself. NOTE: I am not an employee, or paid affiliate of packet exchange... I have used them for services and am promoting them due to my own good experiences with their services. I used to work for them. Then as now, I honestly can see little purpose in their productset. There's little purpose if you're an ISP that's supposed to be good at BGP yourself. If however, your business is running a /24 worth of webservers that sells your company's product, and Best Practices says you should be multi-homed but the in-house skill set runs more to Apache than BGP, a well-designed BGP appliance can be a ghodsend. (I admit I missed the OP's statement of what business line they were in). pgpNkXzugtzja.pgp Description: PGP signature
Re: Failover how much complexity will it add?
On Nov 8, 2009, at 2:39 PM, a...@baklawasecrets.com wrote: So if my requirements are as follows: - BGP router capable of holding full Internet routing table. (whether I go for partial or full, I think I want something with full capability). - Capable of pushing 100meg plus of mixed traffic. What are my options? I want to exclude openbsd, or linux with quagga. Why is that?
RE: Failover how much complexity will it add?
Most purpose-built routing appliances use ternary content addressable memory (TCAM) in order to accomplish deterministic, hardware-based, longest-prefix lookups in large routing tables, such as a full Internet BGP feed. TCAM is used to replace software-based table lookup algorithms which have been empirically shown to lack scalability as the number of routes in the routing table increases, and as the line rate in/out of the routers increases. Current TCAM hardware-based routing engines scale to 10 Gbps, and beyond. Much research has been done in this area in the last 15 years. I don't think general purpose BSD/Linux systems meet the scalability requirement for deterministic network design. Nothing political about that. -Original Message- From: a...@baklawasecrets.com [mailto:a...@baklawasecrets.com] Sent: Monday, November 09, 2009 8:36 AM To: nanog@nanog.org Subject: Re: Failover how much complexity will it add? Hi Joe, I agree with most of what you say below regarding linux sysadmin, BSD etc. I'm quite happy and actually would prefer building a linux solution on our own hardware. However, politically I think this is going to be difficult. I just feel that they will be more comfortable with embedded network boxes as a pose to a linux solution. I guess what I'm saying is this is partially a political thing. Adel On Mon 3:20 PM , Joe Greco jgr...@ns.sol.net wrote: Thanks, I've taken your advice and decided to reconsider my requirement for a full routing table. I believe I'm being greedy and a partial table will be sufficient. With regards to Linux/BSD, its not the CLI of quagga that will be an issue, rather the sysadmin and lack of supporting infrastructure for Linux boxes within the organisation. So things like package management, You don't need to run Apache on your router. syslog servers, If you didn't have syslog servers for the Cisco, you don't need one for the Quagga. monitoring, If you didn't monitor the Cisco, you don't need to monitor the Quagga. understanding of security issues etc. What security issues? The thing is, people get all tied up over this idea that it is some major ongoing burden to support a Linux based device. I have a shocker for you. The CPE your residential broadband relies on may well run Linux, and you didn't even know it. The wifi router you use may run Linux. There are thousands of embedded uses for Linux. I highly doubt that the average TiVo user has a degree in Linux. Many different things you use in day-to-day life run Linux, BSD, VxWorks, or whatever ... mostly without any need of someone to handhold them on security issues. Of course, security issues do come up. But they do with Cisco as well. A proper Linux router doesn't have ports open, aside from bgp and ssh, and those can be firewalled appropriately. This makes it very difficult to have any meaningful security problems relating to the platform... You can expect the occasional issue. Just like anything else. But trying to compare it to security issues on a general Linux platform is only meaningful if you're trying to argue against the solution. (I'm a BSD guy myself, but I don't see any reason for undue Linux paranoia) I don't want to leave them with a linux/bsd solution that they won't be able to maintain/manage effectively when I am gone. If they're unable to maintain something as straightforward as BSD or Linux when you're gone, this raises alarm bells as to whether or not BGP is really suited for them. BGP is *much* more arcane, relatively speaking. You can go to your local bookstore and pick up a ton of Linux or BSD sysadm books, but you'll be lucky to find a book on BGP. Thanks for your comments. Look forward to hearing which solutions come back into the mix having dropped the full routing table requirement. There's a whole plethora of BGP-capable gear that becomes possible once you make that call. Cisco and Juniper both make good gear. A variety of other mfrs do as well. Something as old as an Ascend GRF 400 (fast ethernet, line speed, 150K routes, ~1998?) is perfectly capable of dealing with the load, though I mention this primarily to make the point that there is a lot of equipment within the last decade that can support this. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net [1] We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples. Links: -- [1] http://webmail.123-reg.co.uk/parse.php?redirect=http://www.sol.net
Re: Failover how much complexity will it add?
Most purpose-built routing appliances use ternary content addressable memory (TCAM) in order to accomplish deterministic, hardware-based, longest-prefix lookups in large routing tables, such as a full Internet BGP feed. TCAM is used to replace software-based table lookup algorithms which have been empirically shown to lack scalability as the number of routes in the routing table increases, and as the line rate in/out of the routers increases. Current TCAM hardware-based routing engines scale to 10 Gbps, and beyond. Much research has been done in this area in the last 15 years. I don't think general purpose BSD/Linux systems meet the scalability requirement for deterministic network design. Nothing political about that. Whoa. How'd you manage to get it completely inverted? It's the TCAM based platforms that are a scaling problem. You have to do a forklift upgrade of them every now and then in order to assure yourself of being able to continue to hold a full table. Put another way: Software-based lookup tables do tend to perform more slowly as the number of routes grows. However, a Linux or BSD router that was operational in 1999 will still be functional today, able to route a full table today. It will suffer a mild degradation in forwarding capabilities as the route table grows, but this is mild. Hardware-based lookup tables have a really bad failure mode: they fill. When they fill, generally speaking, parts of the Internet may vanish. It is great to be able to forward at line speed up to the table capacity of the box, but you can do the same thing on a software-based platform... to get line rate simply means you need to ensure you've got sufficient excess resources. Software-based platforms are finicky at high PPS rates, but these days it'd be kinda hard to come up with a platform that *couldn't* route 1Gbps. We're talking a fraction of that for this guy who has a few 100Mbps links. Now, of course, if he plans to scale that few 100Mbps links up to a few 10Gbps links in the next few years, then there is definitely a question about the suitability of a software-based platform, but it is also the case that any inexpensive TCAM-based platform that might be selected will also need to be upgraded ... at significant expense. I would have thought that this lesson would still be fresh in the minds of people, as we just passed 256K routes a little while ago and broke a whole bunch of Catalyst 6500/7600 SUP720-3B's (etc). I guess the pain isn't as memorable as I thought. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Failover how much complexity will it add?
HI, I was recently brought onto a project where some failover is desired, but I think that the number of connections provisioned is excessive. Also hoping to get some guidance with regards to how well I can get the failover to actually work. So currently 4 X 100Mb/s Internet connections have been provisioned. One is to be used for general Internet, out of the organisation, it also terminates VPNs from remote sites belonging to the organisation and some publicly accessible servers -routed DMZ and translated IPs. Second Internet connection to be used for a separate system which has a site-to-site VPN to a third party support vendor. Internet connections 3 and 4 are currently thought of as providing backups for one and two. Both connections firewalled by a Juniper SSG of some description. Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and reliability issues. So my questions regarding this are: - Should I make sure that the backup Internet connection is from a separate provider? - How can I acheive a failover which doesn't require me to change all the remote VPN endpoints in case of a failover? Its possible to configure failover VPNs on the Junipers, which should take care of this, but how do I take care of the DMZ hosts and external translation? - In fact I think I'm asking what are my options with regard to failover between one Internet connection and the other? I'm hoping to figure out whether adding an extra Internet connection actually gives us that much, in fact whether it justifies the complexity and spend. Many Thanks for your comments. Adel
RE: Failover how much complexity will it add?
-Original Message- From: a...@baklawasecrets.com [mailto:a...@baklawasecrets.com] Sent: Sunday, November 08, 2009 4:52 AM To: nanog@nanog.org Subject: Failover how much complexity will it add? HI, I was recently brought onto a project where some failover is desired, but I think that the number of connections provisioned is excessive. Also hoping to get some guidance with regards to how well I can get the failover to actually work. So currently 4 X 100Mb/s Internet connections have been provisioned. One is to be used for general Internet, out of the organisation, it also terminates VPNs from remote sites belonging to the organisation and some publicly accessible servers -routed DMZ and translated IPs. Second Internet connection to be used for a separate system which has a site-to-site VPN to a third party support vendor. Internet connections 3 and 4 are currently thought of as providing backups for one and two. Both connections firewalled by a Juniper SSG of some description. Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and reliability issues. So my questions regarding this are: - Should I make sure that the backup Internet connection is from a separate provider? Yes yes yes yes a thousand times yes. Depending on the criticality of internet connectivity you should also aim to have your redundant connections coming from a complete separate direction. Example, fiber from Level 3 come from the north in a dedicated conduit and fiber from Verizon coming in a dedicated conduit from the south of the building. Why? Put simply we had construction ignore the painted lines and dig up our conduit a few years back. At that point we have 4 bonded T1's from a single carrier. That was a long couple of days... Carrier diversity is not a bad thing, spend some time shopping an additional provider. Make sure they operate their own network for last mile, and also make sure they don’t piggyback off the same network your main carrier does anywhere locally. Comcast Ethernet, Verizon and Cogent make great secondary connections when you need high availability. You don’t need your secondary to have 99.999% uptime. 97% is usually good enough if it's on a separate network. I wouldn't sway from the big names for your primary connections either. - How can I acheive a failover which doesn't require me to change all the remote VPN endpoints in case of a failover? Its possible to configure failover VPNs on the Junipers, which should take care of this, but how do I take care of the DMZ hosts and external translation? With recent experience with the Juniper SSG VPN functions put nicely they suck. VPN failover is in there, but we had issues with the tunnel staying active for extended periods of time. Also depending on if you do a route based or a policy based VPN, it becomes so much of a headache. We used 2 SSG550 devices as a proof of concept and the one thing which annoyed me to no end was the complete and total crap options within then VPN configuration. When I typically set up a VPN, I use a SonicWall NSA or E-class device (yes I know hiss boo) or an ASA. Saying that the Juniper was lacking was a complete understatement. I personally would completely avoid even attempting VPN failover within a Juniper device. I will say they are rock solid though for generic firewall functionality, just try to keep the config simple or they turn into giant slow dogs. - In fact I think I'm asking what are my options with regard to failover between one Internet connection and the other? Considering you have 4x 100mbit lines, have you looked at BGP? Even if you drop line 2 and its associated backup, you have 2x 100mbit lines. Or even if you have 3 unique carriers with a 100mbit from each of them it makes BGP very appealing. I think this would be an ideal situation for a BGP setup using a couple of small routers. You could probably get away with something as small as a Cisco 3825 for each connection (purely redundancy). If the Cisco name scares you Juniper routers are great as well. Don’t forget Vyatta! If you do BGP, you have 1 VPN to configure, you have 1 tunnel to configure, there is no VPN failover configuration and hopefully you are not pushing more than 1 subnet across the VPN otherwise you end up doing a route based VPN instead of a policy based VPN and you will be significantly happier. That’s a Juniper headache for another day however. I'm hoping to figure out whether
Re: Failover how much complexity will it add?
a...@baklawasecrets.com wrote: HI, Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and reliability issues. So my questions regarding this are: I wouldnt jump to any conclusions that everything will work properly if you are terminating multiple connections directly on the SSG, what with egress likely being different than the ingress, even if you are using the same IP range (BGP) on all the links. You could really be asking for trouble if you are planning on using a different ISP provided IP range on each connection for each purpose. Front it all with routers that can policy route, whether or not you also use BGP. Joe
Re: Failover how much complexity will it add?
a...@baklawasecrets.com wrote: HI, I was recently brought onto a project where some failover is desired, but I think that the number of connections provisioned is excessive. Also hoping to get some guidance with regards to how well I can get the failover to actually work. So currently 4 X 100Mb/s Internet connections have been provisioned. One is to be used for general Internet, out of the organisation, it also terminates VPNs from remote sites belonging to the organisation and some publicly accessible servers -routed DMZ and translated IPs. Second Internet connection to be used for a separate system which has a site-to-site VPN to a third party support vendor. Internet connections 3 and 4 are currently thought of as providing backups for one and two. Both connections firewalled by a Juniper SSG of some description. Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and reliability issues. So my questions regarding this are: - Should I make sure that the backup Internet connection is from a separate provider? - How can I acheive a failover which doesn't require me to change all the remote VPN endpoints in case of a failover? Its possible to configure failover VPNs on the Junipers, which should take care of this, but how do I take care of the DMZ hosts and external translation? - In fact I think I'm asking what are my options with regard to failover between one Internet connection and the other? Forget all of that and just multihome to two separate providers with BGP. Also make sure that of the providers you choose that one is not a customer of the other. Instant, painless redundancy. Having multiple circuits to one provider *will not* back anything up if that provider has an outage as they are %99.999 likely to be part of the same larger circuit and certainly share the same infrastructure at the provider. I'm hoping to figure out whether adding an extra Internet connection actually gives us that much, in fact whether it justifies the complexity and spend. Only if you calculate the cost (money, time, angry customers, etc.) of an outage to be greater than the cost of additional connectivity. Many Thanks for your comments. ~Seth
Re: Failover how much complexity will it add?
On 2009-11-08-10:23:41, Blake Pfankuch bpfank...@cpgreeley.com wrote: Make sure they operate their own network for last mile [...] I wouldn't sway from the big names for your primary connections either. Because ownership of the provider/subsidiary delivering the last mile means one hand is talking to the other, and you're going to get good service and reliability as a result? And big names never have any peering-related spats and always deliver the best possible end-user experience, right? :-) (Some good points further on, though important we don't lead the OP down the wrong path or with a false sense of security there...) -a
Re: Failover how much complexity will it add?
Thanks for all your comments guys. With regards to bgp I did think about placing two bgp routers in front of the ssg's. However my limited understanding makes me think that if I had two bgp connections from different providers I would still have issues. So I guess that if my primary Internet goes down I lose connectivity to all the publicly addressed devices on that connection. Like dmz hosts and so on. I would be interested to hear how this can be avoided if at all or do I have to use the same provider. I should add that we currently have provisioned two ssg in ha mode. Also is terminating bgp on the ssg also an option? I really like the flexibility of route based VPN with addresable tun interfaces. Thanks adel On Sun 3:47 PM , Joe Maimon jmai...@ttec.com sent: adel@ baklawasecrets.com wrote: HI, Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and reliability issues. So my questions regarding this are: I wouldnt jump to any conclusions that everything will work properly if you are terminating multiple connections directly on the SSG, what with egress likely being different than the ingress, even if you are using the same IP range (BGP) on all the links. You could really be asking for trouble if you are planning on using a different ISP provided IP range on each connection for each purpose. Front it all with routers that can policy route, whether or not you also use BGP. Joe
RE: Failover how much complexity will it add?
Seth Mattinen [se...@rollernet.us] said: Forget all of that and just multihome to two separate providers with BGP --Assuming that you're advertising PI space or can work around that appropriately with your providers, I agree, that's the ideal situation. Having multiple circuits to one provider *will not* back anything up if that provider has an outage as they are %99.999 likely to be part of the same larger circuit --True - if you don't specify otherwise when you're ordering, then why would they make the effort? Comments made in some of the other responses in this thread are also valid even with a single service provider - diverse entry points into your facility, diverse upstream circuit routing, and homing to different POPs - which may mean backhauling your secondary circuit away from your local POP and taking a hit for the higher latency on that second link. The moral of this is that whether you're using one provider or more than one, state your diversity requirements clearly up front, and then stay involved and make sure that what's presented to you is _actually_ diverse (oldsflash: even the best intentioned people sometimes make mistakes, especially when there's a handoff to a different last mile provider who may not have been clear on the requirement ). Of course, all of this is potentially wasted effort if the data center you're providing connectivity for does not also maintain the same kind of diversity itself in terms of power, connectivity, architecture, etc. and certainly share the same infrastructure at the provider. --If you enter a single provider's network at diverse points, then that local infrastructure isn't the same at least. But by the same measure, if that provider has a major BGP issue for example, then yeah - they're both screwed... in which case we loop back to the dual provider scenario you mentioned in the first place :) Ultimately choosing the appropriate solution will boil down to the what level of service unavailability one can tolerate in the first place, and put a business value on that impact. From that one can derive technical options, then go cap in hand with a business case to the poor soul paying the bill ;-) j.
Re: Failover how much complexity will it add?
a...@baklawasecrets.com wrote: Thanks for all your comments guys. With regards to bgp I did think about placing two bgp routers in front of the ssg's. However my limited understanding makes me think that if I had two bgp connections from different providers I would still have issues. So I guess that if my primary Internet goes down I lose connectivity to all the publicly addressed devices on that connection. Like dmz hosts and so on. I would be interested to hear how this can be avoided if at all or do I have to use the same provider. No, you will announce the same IP addresses (minimum of a /24 which you can easily obtain from one upstream just by saying I want to multihome if you don't already have a /24) over both. That's the whole point of multihoming. If cost is an issue you can just use one BGP speaking router. If you multihome there is no primary like you're thinking. ~Seth
Re: Failover how much complexity will it add?
On Sun, 08 Nov 2009 08:23:41 MST, Blake Pfankuch said: I wouldn't sway from the big names for your primary connections either. This is, of course, dependent on the OP's location and budget. I know when we were getting our NLR connection set up, there was a fair amount of You want 40G worth of DWDM *where*? involved, and the resulting topology was... complicated. At least at one time, there were places where our provider was running our link across lambdas of a subsidiary of ours, which are going across physical fiber owned by the provider... turtles all the way down. ;) pgp48PlDXrseP.pgp Description: PGP signature
Re: Failover how much complexity will it add?
Thanks Seth and James, Things are getting a lot clearer. The BGP multihoming solution sounds like exactly what I want. I have more questions :-) Now I suppose I would get my allocation from RIPE as I am UK based? Do I also need to apply for an AS number? As the IP block is mine, it is ISP independent. i.e. I can take it with me when I decide to use two completely different ISPs? Is the obtaining of this IP block, what is referred to as PI space? Of course internally I split the /24 up however I want - /28 for untrust range and maybe a routed DMZ block etc.? Assuming I apply for IP block and AS number, whats involved and how long does it take to get these babies? I know the SSG550's have BGP capabilites. As I have two of these in HA mode, does it make sense to do the BGP on these, or should I get dedicated BGP routers? Fixing the internal routing policy so traffic is directed at the active BGP connection. Whats involved here, preferring one BGP link over the other? Thanks again, I obviously need to do some reading of my own, but all the suggestions so far have been very valuable and definitely seem to be pointing in some fruitful directions. Adel On Sun 6:31 PM , James Hess mysi...@gmail.com sent: On Sun, Nov 8, 2009 at 11:34 AM, adel@ baklawasecrets.com wrote:[..] connections from different providers I would still have issues. So I guess that if my primary Internet goes down I lose connectivity to all the publicly addressed devices on that connection. Like dmz hosts and so on. I would be interested to hear how this can be avoided if at all or do I have to use the same provider. You assign multi-homed IP address space to your publicly addressed devices,which are not specific to either ISP. You announce to both ISPs, and you accept some routes from both ISPs. You get multi-homed IPs, either by having an existing ARIN allocation, or getting a /22 from ARIN (special allocation available for multi-homing), or ask for a /24 from ISP A or ISP B for multihoming. If Link A fails, the BGP session eventually times out and dies: ISP A's BGP routers withdraw the routes, the IP addresses are then associated only with provider B. And you design your internal routing policy to direct traffic within your network to the router with an active BGP session. Link A's failure is _not_ a total non-event, but a 3-5 minute partial disruption, while the BGP session times out and updates occur in other people's routers, is minimal compared to a 3 day outage, if serious repairs to upstream fiber are required. -- -J
Re: Failover how much complexity will it add?
Hi Adel There are companies like packet exchange (www.packetexchange.net) (whom i have personally used) who will do all of the legwork for you, such as applying for the ASN, address space, transit agreements, and get the tail connections directly to your building. You just need to pay them and buy the equipment (which they can also provide). Probably easier in the long run. NOTE: I am not an employee, or paid affiliate of packet exchange... I have used them for services and am promoting them due to my own good experiences with their services. Regards, Ken 2009/11/8 a...@baklawasecrets.com: Thanks Seth and James, Things are getting a lot clearer. The BGP multihoming solution sounds like exactly what I want. I have more questions :-) Now I suppose I would get my allocation from RIPE as I am UK based? Do I also need to apply for an AS number? As the IP block is mine, it is ISP independent. i.e. I can take it with me when I decide to use two completely different ISPs? Is the obtaining of this IP block, what is referred to as PI space? Of course internally I split the /24 up however I want - /28 for untrust range and maybe a routed DMZ block etc.? Assuming I apply for IP block and AS number, whats involved and how long does it take to get these babies? I know the SSG550's have BGP capabilites. As I have two of these in HA mode, does it make sense to do the BGP on these, or should I get dedicated BGP routers? Fixing the internal routing policy so traffic is directed at the active BGP connection. Whats involved here, preferring one BGP link over the other? Thanks again, I obviously need to do some reading of my own, but all the suggestions so far have been very valuable and definitely seem to be pointing in some fruitful directions. Adel On Sun 6:31 PM , James Hess mysi...@gmail.com sent: On Sun, Nov 8, 2009 at 11:34 AM, adel@ baklawasecrets.com wrote:[..] connections from different providers I would still have issues. So I guess that if my primary Internet goes down I lose connectivity to all the publicly addressed devices on that connection. Like dmz hosts and so on. I would be interested to hear how this can be avoided if at all or do I have to use the same provider. You assign multi-homed IP address space to your publicly addressed devices,which are not specific to either ISP. You announce to both ISPs, and you accept some routes from both ISPs. You get multi-homed IPs, either by having an existing ARIN allocation, or getting a /22 from ARIN (special allocation available for multi-homing), or ask for a /24 from ISP A or ISP B for multihoming. If Link A fails, the BGP session eventually times out and dies: ISP A's BGP routers withdraw the routes, the IP addresses are then associated only with provider B. And you design your internal routing policy to direct traffic within your network to the router with an active BGP session. Link A's failure is _not_ a total non-event, but a 3-5 minute partial disruption, while the BGP session times out and updates occur in other people's routers, is minimal compared to a 3 day outage, if serious repairs to upstream fiber are required. -- -J
Re: Failover how much complexity will it add?
Hi, Thanks for the info on UKNOF. I've started a thread there with regards to RIPE and obtaining ASN numbers and so on., as this is I guess quite UK specific. Adel On Sun 8:40 PM , Arnold Nipper arn...@nipper.de wrote: Hi Adel, On 08.11.2009 21:24 Ken Gilmour wrote There are companies like packet exchange (www.packetexchange.net [1]) I could also comment on PacketExchange, but I do not. If you get more UK specific now you may perhaps want to post to UKNOF (http://lists.uknof.org.uk/cgi-bin/mailman/listinfo/uknof/) [2] as well. For _independant_ consultancy you may want to have a look at Netsumo (http://www.netsumo.com/) [3] Ask for Andy Davidson. Best regards, Arnold -- Arnold Nipper / nIPper consulting, Sandhausen, Germany email: arn...@nipper.de phone: +49 6224 9259 299 mobile: +49 172 2650958 fax: +49 6224 9259 333 Links: -- [1] http://webmail.123-reg.co.uk/parse.php?redirect=http://www.packetexchange.n et[2] http://webmail.123-reg.co.uk/parse.php?redirect=http://lists.uknof.org.uk/c gi-bin/mailman/listinfo/uknof/%29[3] http://webmail.123-reg.co.uk/parse.php?redirect=http://www.netsumo.com/%29
Re: Failover how much complexity will it add?
Don't think I sent the below to the list, so resending: Thanks Seth and James, Things are getting a lot clearer. The BGP multihoming solution sounds like exactly what I want. I have more questions :-) Now I suppose I would get my allocation from RIPE as I am UK based? Do I also need to apply for an AS number? As the IP block is mine, it is ISP independent. i.e. I can take it with me when I decide to use two completely different ISPs? Is the obtaining of this IP block, what is referred to as PI space? Of course internally I split the /24 up however I want - /28 for untrust range and maybe a routed DMZ block etc.? Assuming I apply for IP block and AS number, whats involved and how long does it take to get these babies? I know the SSG550's have BGP capabilites. As I have two of these in HA mode, does it make sense to do the BGP on these, or should I get dedicated BGP routers? Fixing the internal routing policy so traffic is directed at the active BGP connection. Whats involved here, preferring one BGP link over the other? Thanks again, I obviously need to do some reading of my own, but all the suggestions so far have been very valuable and definitely seem to be pointing in some fruitful directions. Adel On Sun 6:31 PM , James Hess mysi...@gmail.com wrote: On Sun, Nov 8, 2009 at 11:34 AM, wrote: [..] connections from different providers I would still have issues. So I guess that if my primary Internet goes down I lose connectivity to all the publicly addressed devices on that connection. Like dmz hosts and so on. I would be interested to hear how this can be avoided if at all or do I have to use the same provider. You assign multi-homed IP address space to your publicly addressed devices, which are not specific to either ISP. You announce to both ISPs, and you accept some routes from both ISPs. You get multi-homed IPs, either by having an existing ARIN allocation, or getting a /22 from ARIN (special allocation available for multi-homing), or ask for a /24 from ISP A or ISP B for multihoming. If Link A fails, the BGP session eventually times out and dies: ISP A's BGP routers withdraw the routes, the IP addresses are then associated only with provider B. And you design your internal routing policy to direct traffic within your network to the router with an active BGP session. Link A's failure is _not_ a total non-event, but a 3-5 minute partial disruption, while the BGP session times out and updates occur in other people's routers, is minimal compared to a 3 day outage, if serious repairs to upstream fiber are required. -- -J
Re: Failover how much complexity will it add?
Hi, Ok thanks for clearing that up. I'm getting some good feedback on applying for PI and ASN through Ripe LIRs over on the UKNOF so I think I have a handle on this. With regards to BGP and using separate BGP routers. I am announcing my PI space to my upstreams, but I don't need to carry a full Internet routing table, correct? So I can get away with some lightweight BGP routers not being an ISP if that makes sense? Adel On Sun 9:26 PM , Ken Gilmour ken.gilm...@gmail.com wrote: Hey, Yes you apply to RIPE for your allocation. You should ask them for a /20 since it's the same price for that as a /24 if you can justify it (at least with LACNIC where i now get my allocations)... You will also need to apply for an ASN Correct- the block belongs to you and as long as you contact the transit provider from the address listed in WHOIS then you should be able to set up a new agreement easily. Yes the block is PI space (provider independent) It can take up to 1 month to get your assignments. I would recommend getting some different routers for this. I use OpenBSD in some of my locations which is extremely easy to work with. I also have some old NS-208 devices running ScreenOS for internal BGP in one other location. I would not recommend using any router with less than 1GB of RAM for BGP. in HA Mode you can connect the two tails, one to each SSG (if they are in active active mode) and announce it that way (check out anycast), we also do this :). The way BGP works is that both connections are active at the same time, there is no primary and backup, if one goes down you just have one less to receive traffic over and more traffic on the other, but unless you stop announcing from one connection traffic will go over both. Regards, Ken 2009/11/8 : Don't think I sent the below to the list, so resending: Thanks Seth and James, Things are getting a lot clearer. The BGP multihoming solution sounds like exactly what I want. I have more questions :-) Now I suppose I would get my allocation from RIPE as I am UK based? Do I also need to apply for an AS number? As the IP block is mine, it is ISP independent. i.e. I can take it with me when I decide to use two completely different ISPs? Is the obtaining of this IP block, what is referred to as PI space? Of course internally I split the /24 up however I want - /28 for untrust range and maybe a routed DMZ block etc.? Assuming I apply for IP block and AS number, whats involved and how long does it take to get these babies? I know the SSG550's have BGP capabilites. As I have two of these in HA mode, does it make sense to do the BGP on these, or should I get dedicated BGP routers? Fixing the internal routing policy so traffic is directed at the active BGP connection. Whats involved here, preferring one BGP link over the other? Thanks again, I obviously need to do some reading of my own, but all the suggestions so far have been very valuable and definitely seem to be pointing in some fruitful directions. Adel On Sun 6:31 PM , James Hess wrote: On Sun, Nov 8, 2009 at 11:34 AM, wrote: [..] connections from different providers I would still have issues. So I guess that if my primary Internet goes down I lose connectivity to all the publicly addressed devices on that connection. Like dmz hosts and so on. I would be interested to hear how this can be avoided if at all or do I have to use the same provider. You assign multi-homed IP address space to your publicly addressed devices, which are not specific to either ISP. You announce to both ISPs, and you accept some routes from both ISPs. You get multi-homed IPs, either by having an existing ARIN allocation, or getting a /22 from ARIN (special allocation available for multi-homing), or ask for a /24 from ISP A or ISP B for multihoming. If Link A fails, the BGP session eventually times out and dies: ISP A's BGP routers withdraw the routes, the IP addresses are then associated only with provider B. And you design your internal routing policy to direct traffic within your network to the router with an active BGP session. Link A's failure is _not_ a total non-event, but a 3-5 minute partial disruption, while the BGP session times out and updates occur in other people's routers, is minimal compared to a 3 day outage, if serious repairs to upstream fiber are required. -- -J
Re: Failover how much complexity will it add?
a...@baklawasecrets.com wrote: Hi, Thanks for the info on UKNOF. I've started a thread there with regards to RIPE and obtaining ASN numbers and so on., as this is I guess quite UK specific. You will need an AS number regardless of what path you get your addresses from to multihome. In ARIN land the minimum for a multihomed end-site is /22, so if I were to do this here, I would ask one of the upstreams for a /24. I don't know the first thing about RIPE policy. ~Seth
Re: Failover how much complexity will it add?
a...@baklawasecrets.com wrote: Hi, Ok thanks for clearing that up. I'm getting some good feedback on applying for PI and ASN through Ripe LIRs over on the UKNOF so I think I have a handle on this. With regards to BGP and using separate BGP routers. I am announcing my PI space to my upstreams, but I don't need to carry a full Internet routing table, correct? So I can get away with some lightweight BGP routers not being an ISP if that makes sense? Most will give you three choices: full routes, partial routes (internal, their customers) with default, and default only. If you can't swing full routes then I would go for partial routes as it will at least send traffic for each ISP and their customers directly to them rather than randomly over the other link. It all depends on what you're going to use as your BGP speaking platform. ~Seth
Re: Failover how much complexity will it add?
I think partial routes makes perfect sense, makes sense that traffic for customers who are connected to each of my upstreams should go out of the correct BGP link as long as they are up! Now I need to start thinking of BGP router choices, sure I have a plethora of choices :-( On Sun 10:01 PM , Seth Mattinen se...@rollernet.us wrote: a...@baklawasecrets.com wrote: Hi, Ok thanks for clearing that up. I'm getting some good feedback on applying for PI and ASN through Ripe LIRs over on the UKNOF so I think I have a handle on this. With regards to BGP and using separate BGP routers. I am announcing my PI space to my upstreams, but I don't need to carry a full Internet routing table, correct? So I can get away with some lightweight BGP routers not being an ISP if that makes sense? Most will give you three choices: full routes, partial routes (internal, their customers) with default, and default only. If you can't swing full routes then I would go for partial routes as it will at least send traffic for each ISP and their customers directly to them rather than randomly over the other link. It all depends on what you're going to use as your BGP speaking platform. ~Seth
Re: Failover how much complexity will it add?
So if my requirements are as follows: - BGP router capable of holding full Internet routing table. (whether I go for partial or full, I think I want something with full capability). - Capable of pushing 100meg plus of mixed traffic. What are my options? I want to exclude openbsd, or linux with quagga. Probably looking at Cisco or Juniper products, but interested in any other alternatives people suggest. I realise this is quite a broad question, but hoping this will provide a starting point. Oh and if I have missed any specs I should have included above, please let me know. Thanks Adel On Sun 10:18 PM , Seth Mattinen se...@rollernet.us wrote: a...@baklawasecrets.com wrote: I think partial routes makes perfect sense, makes sense that traffic for customers who are connected to each of my upstreams should go out of the correct BGP link as long as they are up! Now I need to start thinking of BGP router choices, sure I have a plethora of choices :-( Personally I'll always go for full routes if the router has enough memory (software based) or TCAM space (hardware based). Cheaper to do on software platforms though. An entry level Cisco 2811 can take full tables from multiple upstreams with 786MB RAM or even 512. It won't push 100 meg of mixed traffic though. ~Seth
Re: Failover how much complexity will it add?
So if my requirements are as follows: - BGP router capable of holding full Internet routing table. (whether I go for partial or full, I think I want something with full capability). - Capable of pushing 100meg plus of mixed traffic. What are my options? I want to exclude openbsd, or linux with quagga. Probably looking at Cisco or Juniper products, but interested in any other alternatives people suggest. I realise this is quite a broad question, but hoping this will provide a starting point. Oh and if I have missed any specs I should have included above, please let me know. Thanks Adel On Sun 10:18 PM , Seth Mattinen se...@rollernet.us wrote: a...@baklawasecrets.com wrote: I think partial routes makes perfect sense, makes sense that traffic for customers who are connected to each of my upstreams should go out of the correct BGP link as long as they are up! Now I need to start thinking of BGP router choices, sure I have a plethora of choices :-( Personally I'll always go for full routes if the router has enough memory (software based) or TCAM space (hardware based). Cheaper to do on software platforms though. An entry level Cisco 2811 can take full tables from multiple upstreams with 786MB RAM or even 512. It won't push 100 meg of mixed traffic though. ~Seth
RE: Failover how much complexity will it add?
From: a...@baklawasecrets.com [a...@baklawasecrets.com] - BGP router capable of holding full Internet routing table. (whether I go for partial or full, I think I want something with full capability). --Capable of holding _2_ full internet routing tables if you are looking for diversity. (just being picky ;-) j.
Re: Failover how much complexity will it add?
There are any problems with quagga+BSD/Linux that you know or something like that? Or in your scenario a cisco/juniper box is a requirement? I'm asking this because I'm always running BGP with upstreams providers using quagga on BSD and everything is fine until now. -- From: a...@baklawasecrets.com Sent: Sunday, November 08, 2009 8:39 PM To: nanog@nanog.org Subject: Re: Failover how much complexity will it add? So if my requirements are as follows: - BGP router capable of holding full Internet routing table. (whether I go for partial or full, I think I want something with full capability). - Capable of pushing 100meg plus of mixed traffic. What are my options? I want to exclude openbsd, or linux with quagga. Probably looking at Cisco or Juniper products, but interested in any other alternatives people suggest. I realise this is quite a broad question, but hoping this will provide a starting point. Oh and if I have missed any specs I should have included above, please let me know. Thanks Adel
Re: Failover how much complexity will it add?
Basically the organisation that I'm working for will not have the skills in house to support a linux or bsd box. They will have trouble with supporting the BGP configuration, however I don't think they will be happy with me if I leave them with a linux box when they don't have linux/unix resource internally. At least with a Cisco or Juniper they are familiar with IOS and it won't be too foreign to them. On Sun 11:30 PM , Renato Frederick freder...@dahype.org wrote: There are any problems with quagga+BSD/Linux that you know or something like that? Or in your scenario a cisco/juniper box is a requirement? I'm asking this because I'm always running BGP with upstreams providers using quagga on BSD and everything is fine until now. -- From: Sent: Sunday, November 08, 2009 8:39 PM To: Subject: Re: Failover how much complexity will it add? So if my requirements are as follows: - BGP router capable of holding full Internet routing table. (whether I go for partial or full, I think I want something with full capability). - Capable of pushing 100meg plus of mixed traffic. What are my options? I want to exclude openbsd, or linux with quagga. Probably looking at Cisco or Juniper products, but interested in any other alternatives people suggest. I realise this is quite a broad question, but hoping this will provide a starting point. Oh and if I have missed any specs I should have included above, please let me know. Thanks Adel
