RE: Route Reflection with Multiple POPs [7:48509]
Hi Guy, As others have pointed out, it is not necessary, nor usually desirable, to run separate IGPs in your AS assuming you use your IGP properly. A properly utilized ISP IGP should carry only link and loopback addresses and thus be reasonably small. Use of hierarchy can also help larger domains scale, though modern IGP implementations on reasonably powered routers should be able to handle rather large single areas. BGP never runs entirely on its own as one always requires some method of resolving BGP Next-Hops to forwarding next hops. Static routes can work here to connect otherwise disconnected IGP domains, or, separate IGP domains can be used for this purpose as well. At 06:19 PM 7/10/2002 -0400, Lupi, Guy wrote: I know that you can run confederations and reflectors, and seperate levels of reflection, which Cisco refers to as nested reflection. Now my question is, how would you set up your bgp peering? Due to financial constraints I would imagine that the best thing to do would be to have one circuit from POP router 1 to core router 1, and another circuit from POP router 2 to core router 2. Assuming that you are only running BGP in the core, and that the clients have to have a session with each reflector, how would you communicate the loopback addresses of all the routers to each other? Are static routes used in this situation? Thanks to all of the people who responded by the way, I appreciate the direction. Ever feel like you know so much, only to read a book and find out that you know so little :)? *-Original Message- *From: Peter van Oene [mailto:[EMAIL PROTECTED]] *Sent: Wednesday, July 10, 2002 3:00 PM *To: [EMAIL PROTECTED] *Subject: Re: Route Reflection with Multiple POPs [7:48509] * * *Couple thoughts. First off, one can confederate and reflect *at the same *time (ie rr clusters inside sub-as's). Also, keep in mind that these *techniques deal with control, not forwarding and thus it is *possible, and *somewhat common, to have a RR hierarchy that does not map *directly to the *physical topology. I tend to see an arbitrary set of core *routers (from *2-20ish) that form the central IBGP mesh with the second level of the *hierarchy being formed by lead major pop routers, or dedicated route *servers (ie one armed routers that simply reflect routes). I *personally *try to keep the topology as flat as possible (ie less than 4 *levels in the *hierarchy) but I don't have any particular technical driver *for that other *than to minimize complexity and aid troubleshooting. * *Of note, this assumes you are reflecting full routes throughout your *backbone. I've seen providers try and reflect partial routes *to areas of *their backbone to allow for smaller routers where RR *topologies can lead to *blackholes. In general, if you are reflecting all routes, you *shouldn't run *into issues here. * *I * * * *At 04:42 PM 7/10/2002 +, Lupi, Guy wrote: *Let me preface this by saying that I am trying to learn more *about large *scale BGP design and operation. This question is on route *reflectors when *you have multiple POPs in seperate IGP domains. If you *currently have one *POP and are going to move to 2 within the same AS, you can *either run full *mesh (doesn't scale), reflectors, or confederations. *Assuming you don't *currently have a central core that the POPs connect back to, *how well does *reflection scale? I was reading Building Service Provider Networks *[Berkowitz], and it states that iBGP doesn't scale well once *you go above *15-20 sessions per router. It also states that most ISPs run *reflectors *instead of confederations, but I believe that statement is *being made under *the assumption that the ISP will have a central core to which *the POPs will *connect. This would indicate to me that assuming you don't *have a central *core, one could only connect 6 or 7 POPs (dual reflectors for *redundancy) *together using reflection before you would have to either *create a central *core to reduce the amount of iBGP sessions, or turn to confederations. *Perhaps the best way to accomplish this would be to establish *a core in *one of the POPs and run reflection from there, which is also *presented as a *solution in the book? Any opinions? I have made an attempt at ASCII *drawing below, to me the central core solution makes more sense. * * First Scenario, no central core * * POP 1 POP 2 * Cluster Cluster * o---o * Full Mesh Between Reflectors * o---o * * Second Scenario, central core establised at one of the POPs * * POP 1 POP 2 * Core Reflectors * Clients o- -o Clients * o- \ / -o *o o * POP 3/ \ POP 4 * Clients o- -o Clients
RE: Route Reflection with Multiple POPs [7:48509]
comments inline, At 01:28 AM 7/11/2002 +, Lupi, Guy wrote: I had a feeling that would happen, I will try to clarify. I was not trying to say that there should be a central core site for the ISP's entire network, but for pieces of it. Lets take a state like New York, within it you have 3 POPs, each is in it's own AS and runs an IGP I'm not sure why you would have 3 pops all in different AS's assuming they are all yours? If you just bought three companies, lets suggest that you'd work toward merging those at your earliest convenience. . In each POP you have 2 core routers and 3 aggregation routers, the 2 core routers have EBGP connections and are reflectors for the 3 aggregation routers. Now you want all 4 POPs to be in the same AS, and they have to have direct connectivity to your 3 New Jersey POPs which should also be in the same AS. In that case, would it make sense to choose a NY POP and a NJ POP, install 2 core or backbone routers that do not participate in any IGP, use those to peer with the POPs in that state and in turn peer with the other state's core or backbone routers? This would significantly reduce the amount of peering that would be required. Hopefully the drawing won't be a disaster. If this was my entire network, I'd likely peer all the core routers one with each other, and reflect from each pop core router downward into the pops (ensuring that cisco's bgp-deterministic-med is on!). Alternatively, you could pick two to four routers which are maximally resilient and mesh them, with hierarchical reflection outward toward the rest of your routers. Again, running isolated IGP's in this network, or most others does more harm then good. As Phillip and Howard have pointed out, keep everything customer oriented in BGP and keep your IGP nice and small. If its small, you can get away with flooding your reachability information unconstrained throughout the network which will allow all your routers to make informed forwarding decisions. This assumes no usage of MPLS style TE however. NY POP 1 NJ POP 1 o o o o o o o o o o NY POP 2 NJ POP 2 o o o---o o o o \ / o o o/ \o o o---o NY POP 3 NJ POP 3 o o o o o o o o o o -Original Message- From: Phillip Heller To: [EMAIL PROTECTED] Sent: 7/10/2002 7:33 PM Subject: Re: Route Reflection with Multiple POPs [7:48509] On Wed, Jul 10, 2002 at 10:27:08PM +, Lupi, Guy wrote: I know that you can run confederations and reflectors, and seperate levels of reflection, which Cisco refers to as nested reflection. Now my question is, how would you set up your bgp peering? Due to financial constraints I would imagine that the best thing to do would be to have one circuit from POP router 1 to core router 1, and another circuit from POP router 2 to core router 2. Assuming that you are only running BGP in the core, and that the clients have to have a session with each reflector, how would you communicate the loopback addresses of all the routers to each other? Are static routes used in this situation? Thanks to all of the people who responded by the way, I appreciate the direction. Ever feel like you know so much, only to read a book and find out that you know so little :)? I hate to say it, but the question is somewhat ambiguous and confusing... First and foremost, what is the physical topology of your backbone? How many devices are you dealing with? These days, most large service provider networks are of a partial mesh (physical) design. There is no central core site. Each pop has several aggregation boxes and several core boxes. The core boxes connect to each other and to other pop's core boxes. Also, the core boxes provide connections for the aggregation boxes. While it's common to see route-reflection from the core to aggregation boxes, it's less common to see route-reflection from one pop to other *multihomed* pops. Generally, customers maintain one bgp session per physical connection to the network; further, the bgp sessions are generally between the directly connected devices. Sometimes customers will want to bond multiple equal bandwidth paths. In this instance, customers generally have less than one bgp session per physical connection (ie: ebgp-multihop with static routes and cef, or Multilink PPP). --phil Message Posted at: http://www.groupstudy.com/form
RE: Route Reflection with Multiple POPs [7:48509]
This is where my thinking went wrong then, Phil also said basically the same thing. It seems that I have a basic misunderstanding of how larger providers carry their routing information. Where would I be able to get information about proper addressing structures and routing protocol implementation in large/service provider networks? Are there any books that go into detail on these particular topics, take you step by step from a customer through to the core? -Original Message- From: Peter van Oene To: Lupi, Guy; [EMAIL PROTECTED] Sent: 7/11/2002 8:18 AM Subject: RE: Route Reflection with Multiple POPs [7:48509] Hi Guy, As others have pointed out, it is not necessary, nor usually desirable, to run separate IGPs in your AS assuming you use your IGP properly. A properly utilized ISP IGP should carry only link and loopback addresses and thus be reasonably small. Use of hierarchy can also help larger domains scale, though modern IGP implementations on reasonably powered routers should be able to handle rather large single areas. BGP never runs entirely on its own as one always requires some method of resolving BGP Next-Hops to forwarding next hops. Static routes can work here to connect otherwise disconnected IGP domains, or, separate IGP domains can be used for this purpose as well. At 06:19 PM 7/10/2002 -0400, Lupi, Guy wrote: I know that you can run confederations and reflectors, and seperate levels of reflection, which Cisco refers to as nested reflection. Now my question is, how would you set up your bgp peering? Due to financial constraints I would imagine that the best thing to do would be to have one circuit from POP router 1 to core router 1, and another circuit from POP router 2 to core router 2. Assuming that you are only running BGP in the core, and that the clients have to have a session with each reflector, how would you communicate the loopback addresses of all the routers to each other? Are static routes used in this situation? Thanks to all of the people who responded by the way, I appreciate the direction. Ever feel like you know so much, only to read a book and find out that you know so little :)? *-Original Message- *From: Peter van Oene [mailto:[EMAIL PROTECTED]] *Sent: Wednesday, July 10, 2002 3:00 PM *To: [EMAIL PROTECTED] *Subject: Re: Route Reflection with Multiple POPs [7:48509] * * *Couple thoughts. First off, one can confederate and reflect *at the same *time (ie rr clusters inside sub-as's). Also, keep in mind that these *techniques deal with control, not forwarding and thus it is *possible, and *somewhat common, to have a RR hierarchy that does not map *directly to the *physical topology. I tend to see an arbitrary set of core *routers (from *2-20ish) that form the central IBGP mesh with the second level of the *hierarchy being formed by lead major pop routers, or dedicated route *servers (ie one armed routers that simply reflect routes). I *personally *try to keep the topology as flat as possible (ie less than 4 *levels in the *hierarchy) but I don't have any particular technical driver *for that other *than to minimize complexity and aid troubleshooting. * *Of note, this assumes you are reflecting full routes throughout your *backbone. I've seen providers try and reflect partial routes *to areas of *their backbone to allow for smaller routers where RR *topologies can lead to *blackholes. In general, if you are reflecting all routes, you *shouldn't run *into issues here. * *I * * * *At 04:42 PM 7/10/2002 +, Lupi, Guy wrote: *Let me preface this by saying that I am trying to learn more *about large *scale BGP design and operation. This question is on route *reflectors when *you have multiple POPs in seperate IGP domains. If you *currently have one *POP and are going to move to 2 within the same AS, you can *either run full *mesh (doesn't scale), reflectors, or confederations. *Assuming you don't *currently have a central core that the POPs connect back to, *how well does *reflection scale? I was reading Building Service Provider Networks *[Berkowitz], and it states that iBGP doesn't scale well once *you go above *15-20 sessions per router. It also states that most ISPs run *reflectors *instead of confederations, but I believe that statement is *being made under *the assumption that the ISP will have a central core to which *the POPs will *connect. This would indicate to me that assuming you don't *have a central *core, one could only connect 6 or 7 POPs (dual reflectors for *redundancy) *together using reflection before you would have to either *create a central *core to reduce the amount of iBGP sessions, or turn to confederations. *Perhaps the best way to accomplish this would be to establish *a core in *one of the POPs and run reflection from there, which is also *presented as a *solution in the book? Any opinions? I have made an attempt at ASCII *drawing below, to me the central core solution makes more sense
RE: Route Reflection with Multiple POPs [7:48509]
Not really :) Howard's is one of the few SP oriented books on the market though admittedly I haven't picked it up yet (very broke :) The ISP resources at NANOG are helpful and contain some valuable links and one can glean some valuable information by looking at some of the historical design oriented presentations that many have given over the years. Cisco also has a document (and recently a book based on the document) from Phillip Smith which deals with ISP design tips (called something like IOS Essentials: Design tips every ISP should know -- something similar to that) Some quick tips would include: IGPs: - install only link/loopback addresses (ie no redistribution into the IGP of static/connected/or otherwise) - tune metrics for optimal SPF(usually to link delay - ie 70ms = 70) - see latest nanog for netflow/lots of math based method which seems cool! :) - pre allocate contiguous IP space to non-backbone areas such that summarization _could_ take place - keep in mind summarization when multiple exits from an area exist can cause sub-optimal routing in normal states and blackholes in some failure states - if you _really_ need to put externals in your IGP (say you have a bunch of dial routers and marketing gave some customers static IP's that are radius assigned (shame on you marketing), use a separate instance of the IGP or use NSSA with no type 7/5 conversion or full external filtering at L1L2 border in ISIS and advertise a BGP aggregate from the related ABR's. - authentication can't hurt - hard code RIDs BGP - peer on lookbacks - hard code RIDs - use recognizable /32 space for loopbacks for easy router identification - use bgp-deterministic-med (to reduce some potential for med oriented oscillation) (lots of ietf chat on this on IDR group as howard points out from time to time) - build resilient RR clusters (ie two RR servers per group of clients) - one cluster ID per RR server (subject to some memory vs potential for bad routing debate) (also cisco default) - use communities and do community based advertisements - wrt to advertising to transit providers/peers, use an explicit accept / default reject style policy for safety (ie when you mess it up, you don't send anything vs messing it up and advertising transit) - authenticate IBGP and possibly EBGP where appropriate - use next-hop-self vs passive interface (helps hook mpls into forwarding plane if you decide to deploy it and keeps IGP smaller) - don't dynamically redistribute anything in bgp - ie only redist statics routes for aggregation - try not to leak too much junk to your transit providers (ie aggregate where you can without too much TE loss) - filter customer prefix advertisements - IRR based would be ideal, but manual is cool too - local pref customer routes over peer routes over transit routes - either use meds or zero them in and out (better to know what is going to happen than learn :) Lots of more stuff out there likely to cover, but thats my 10 minute off the top of my head while my soup cooks :) Likely lots there to debate as well! Building networks generally involves a lot of personal design philosophy which makes for healthy debate. Pete At 09:20 AM 7/11/2002 -0400, you wrote: This is where my thinking went wrong then, Phil also said basically the same thing. It seems that I have a basic misunderstanding of how larger providers carry their routing information. Where would I be able to get information about proper addressing structures and routing protocol implementation in large/service provider networks? Are there any books that go into detail on these particular topics, take you step by step from a customer through to the core? -Original Message- From: Peter van Oene To: Lupi, Guy; [EMAIL PROTECTED] Sent: 7/11/2002 8:18 AM Subject: RE: Route Reflection with Multiple POPs [7:48509] Hi Guy, As others have pointed out, it is not necessary, nor usually desirable, to run separate IGPs in your AS assuming you use your IGP properly. A properly utilized ISP IGP should carry only link and loopback addresses and thus be reasonably small. Use of hierarchy can also help larger domains scale, though modern IGP implementations on reasonably powered routers should be able to handle rather large single areas. BGP never runs entirely on its own as one always requires some method of resolving BGP Next-Hops to forwarding next hops. Static routes can work here to connect otherwise disconnected IGP domains, or, separate IGP domains can be used for this purpose as well. At 06:19 PM 7/10/2002 -0400, Lupi, Guy wrote: I know that you can run confederations and reflectors, and seperate levels of reflection, which Cisco refers to as nested reflection. Now my question is, how would you set up your bgp peering? Due to financial constraints I would imagine that the best thing to do would be to have one circuit from POP router 1 to core router 1, and another circuit from POP router 2 to core
RE: Route Reflection with Multiple POPs [7:48509]
I will definitely look into some of the resources you mentioned. Again, thanks to all who responded. ~-Original Message- ~From: Peter van Oene [mailto:[EMAIL PROTECTED]] ~Sent: Thursday, July 11, 2002 10:16 AM ~To: Lupi, Guy; [EMAIL PROTECTED] ~Subject: RE: Route Reflection with Multiple POPs [7:48509] ~ ~ ~Not really :) Howard's is one of the few SP oriented books on ~the market ~though admittedly I haven't picked it up yet (very broke :) ~ ~The ISP resources at NANOG are helpful and contain some ~valuable links and ~one can glean some valuable information by looking at some of the ~historical design oriented presentations that many have given over the ~years. Cisco also has a document (and recently a book based on the ~document) from Phillip Smith which deals with ISP design tips (called ~something like IOS Essentials: Design tips every ISP should know -- ~something similar to that) Some quick tips would include: ~ ~IGPs: ~- install only link/loopback addresses (ie no redistribution ~into the IGP ~of static/connected/or otherwise) ~- tune metrics for optimal SPF(usually to link delay - ie 70ms ~= 70) - see ~latest nanog for netflow/lots of math based method which seems cool! :) ~- pre allocate contiguous IP space to non-backbone areas such that ~summarization _could_ take place ~- keep in mind summarization when multiple exits from an area ~exist can ~cause sub-optimal routing in normal states and blackholes in ~some failure ~states ~- if you _really_ need to put externals in your IGP (say you ~have a bunch ~of dial routers and marketing gave some customers static IP's that are ~radius assigned (shame on you marketing), use a separate ~instance of the ~IGP or use NSSA with no type 7/5 conversion or full external ~filtering at ~L1L2 border in ISIS and advertise a BGP aggregate from the ~related ABR's. ~- authentication can't hurt ~- hard code RIDs ~ ~ ~BGP ~- peer on lookbacks ~- hard code RIDs ~- use recognizable /32 space for loopbacks for easy router ~identification ~- use bgp-deterministic-med (to reduce some potential for med oriented ~oscillation) (lots of ietf chat on this on IDR group as howard ~points out ~from time to time) ~- build resilient RR clusters (ie two RR servers per group of clients) ~- one cluster ID per RR server (subject to some memory vs ~potential for bad ~routing debate) (also cisco default) ~- use communities and do community based advertisements ~- wrt to advertising to transit providers/peers, use an ~explicit accept / ~default reject style policy for safety (ie when you mess it ~up, you don't ~send anything vs messing it up and advertising transit) ~- authenticate IBGP and possibly EBGP where appropriate ~- use next-hop-self vs passive interface (helps hook mpls into ~forwarding ~plane if you decide to deploy it and keeps IGP smaller) ~- don't dynamically redistribute anything in bgp - ie only ~redist statics ~routes for aggregation ~- try not to leak too much junk to your transit providers (ie ~aggregate ~where you can without too much TE loss) ~- filter customer prefix advertisements - IRR based would be ~ideal, but ~manual is cool too ~- local pref customer routes over peer routes over transit routes ~- either use meds or zero them in and out (better to know what ~is going to ~happen than learn :) ~ ~Lots of more stuff out there likely to cover, but thats my 10 ~minute off ~the top of my head while my soup cooks :) Likely lots there ~to debate as ~well! Building networks generally involves a lot of personal design ~philosophy which makes for healthy debate. ~ ~Pete ~ ~ ~At 09:20 AM 7/11/2002 -0400, you wrote: ~This is where my thinking went wrong then, Phil also said ~basically the same ~thing. It seems that I have a basic misunderstanding of how larger ~providers carry their routing information. Where would I be ~able to get ~information about proper addressing structures and routing protocol ~implementation in large/service provider networks? Are there ~any books that ~go into detail on these particular topics, take you step by ~step from a ~customer through to the core? ~ ~-Original Message- ~From: Peter van Oene ~To: Lupi, Guy; [EMAIL PROTECTED] ~Sent: 7/11/2002 8:18 AM ~Subject: RE: Route Reflection with Multiple POPs [7:48509] ~ ~Hi Guy, ~ ~As others have pointed out, it is not necessary, nor usually ~desirable, ~to ~run separate IGPs in your AS assuming you use your IGP properly. A ~properly utilized ISP IGP should carry only link and loopback ~addresses ~and ~thus be reasonably small. Use of hierarchy can also help ~larger domains ~ ~scale, though modern IGP implementations on reasonably powered routers ~should be able to handle rather large single areas. ~ ~BGP never runs entirely on its own as one always requires ~some method of ~ ~resolving BGP Next-Hops to forwarding next hops. Static ~routes can work ~ ~here to connect otherwise disconnected IGP domains, or, separate
RE: Route Reflection with Multiple POPs [7:48509]
I apologize for taking so long to respond. When I was referring to the amount of peering I meant the iBGP peering, not eBGP peering. When you say a unified IGP, you are making the assumption that all customer routes are carried in BGP, correct? I also wanted to clarify something. How are we defining an ISP? Are we considering UUnet and Sprint to be ISPs, or are they in a different category all together? Would a network of that size also run only a single instance of an IGP, assuming that they only carry internal links and loopbacks within it? ~-Original Message- ~From: Phillip Heller [mailto:[EMAIL PROTECTED]] ~Sent: Wednesday, July 10, 2002 10:25 PM ~To: [EMAIL PROTECTED] ~Subject: Re: Route Reflection with Multiple POPs [7:48509] ~ ~ ~On Thu, Jul 11, 2002 at 01:28:08AM +, Lupi, Guy wrote: ~ I had a feeling that would happen, I will try to clarify. I ~was not trying ~ to say that there should be a central core site for the ISP's entire ~ network, but for pieces of it. Lets take a state like New ~York, within it ~ you have 3 POPs, each is in it's own AS and runs an IGP. In ~each POP you ~ have 2 core routers and 3 aggregation routers, the 2 core ~routers have EBGP ~ connections and are reflectors for the 3 aggregation ~routers. Now you want ~ all 4 POPs to be in the same AS, and they have to have ~direct connectivity ~ to your 3 New Jersey POPs which should also be in the same ~AS. In that ~ case, would it make sense to choose a NY POP and a NJ POP, ~install 2 core ~or ~ backbone routers that do not participate in any IGP, use ~those to peer with ~ the POPs in that state and in turn peer with the other ~state's core or ~ backbone routers? This would significantly reduce the ~amount of peering ~ that would be required. Hopefully the drawing won't be a disaster. ~ ~NY POP 1 NJ POP 1 ~ o o o o ~ o o ~ o o o o ~ ~NY POP 2 NJ POP 2 ~ o o o---o o o ~ o \ / o ~ o o/ \o o ~ o---o ~ ~NY POP 3 NJ POP 3 ~ o o o o ~ o o ~ o o o o ~ ~ ~Are you suggesting seperate per-pop AS's as a result of ~confederations, or otherwise? Confederations will certainly work, but ~may be overkill for a 6 pop regional network. ~ ~I would suggest a single externally visible AS with a unified IGP. IGP ~convergence time is much preferrable to BGP convergence time. ~Also, BGP ~is better suited for political/administrative division, which shouldn't ~be necessary between the two regional networks. ~ ~Also, when you say ...reduce the amount of peering..., are you ~referring to the number of ibgp sessions, external peering ~arrangements, ~or transit connections? ~ ~--phil ~ ~ ~ ~ Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48623t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
Re: Route Reflection with Multiple POPs [7:48509]
Couple thoughts. First off, one can confederate and reflect at the same time (ie rr clusters inside sub-as's). Also, keep in mind that these techniques deal with control, not forwarding and thus it is possible, and somewhat common, to have a RR hierarchy that does not map directly to the physical topology. I tend to see an arbitrary set of core routers (from 2-20ish) that form the central IBGP mesh with the second level of the hierarchy being formed by lead major pop routers, or dedicated route servers (ie one armed routers that simply reflect routes). I personally try to keep the topology as flat as possible (ie less than 4 levels in the hierarchy) but I don't have any particular technical driver for that other than to minimize complexity and aid troubleshooting. Of note, this assumes you are reflecting full routes throughout your backbone. I've seen providers try and reflect partial routes to areas of their backbone to allow for smaller routers where RR topologies can lead to blackholes. In general, if you are reflecting all routes, you shouldn't run into issues here. I At 04:42 PM 7/10/2002 +, Lupi, Guy wrote: Let me preface this by saying that I am trying to learn more about large scale BGP design and operation. This question is on route reflectors when you have multiple POPs in seperate IGP domains. If you currently have one POP and are going to move to 2 within the same AS, you can either run full mesh (doesn't scale), reflectors, or confederations. Assuming you don't currently have a central core that the POPs connect back to, how well does reflection scale? I was reading Building Service Provider Networks [Berkowitz], and it states that iBGP doesn't scale well once you go above 15-20 sessions per router. It also states that most ISPs run reflectors instead of confederations, but I believe that statement is being made under the assumption that the ISP will have a central core to which the POPs will connect. This would indicate to me that assuming you don't have a central core, one could only connect 6 or 7 POPs (dual reflectors for redundancy) together using reflection before you would have to either create a central core to reduce the amount of iBGP sessions, or turn to confederations. Perhaps the best way to accomplish this would be to establish a core in one of the POPs and run reflection from there, which is also presented as a solution in the book? Any opinions? I have made an attempt at ASCII drawing below, to me the central core solution makes more sense. First Scenario, no central core POP 1 POP 2 Cluster Cluster o---o Full Mesh Between Reflectors o---o Second Scenario, central core establised at one of the POPs POP 1 POP 2 Core Reflectors Clients o- -o Clients o- \ / -o o o POP 3/ \ POP 4 Clients o- -o Clients o- -o Guy H. Lupi CCIE No. 9275 Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48520t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
Re: Route Reflection with Multiple POPs [7:48509]
On Wed, Jul 10, 2002 at 04:42:25PM +, Lupi, Guy wrote: Let me preface this by saying that I am trying to learn more about large scale BGP design and operation. This question is on route reflectors when you have multiple POPs in seperate IGP domains. If you currently have one POP and are going to move to 2 within the same AS, you can either run full mesh (doesn't scale), reflectors, or confederations. Assuming you don't currently have a central core that the POPs connect back to, how well does reflection scale? I was reading Building Service Provider Networks [Berkowitz], and it states that iBGP doesn't scale well once you go above 15-20 sessions per router. It also states that most ISPs run reflectors instead of confederations, but I believe that statement is being made under the assumption that the ISP will have a central core to which the POPs will connect. This would indicate to me that assuming you don't have a central core, one could only connect 6 or 7 POPs (dual reflectors for redundancy) together using reflection before you would have to either create a central core to reduce the amount of iBGP sessions, or turn to confederations. Perhaps the best way to accomplish this would be to establish a core in one of the POPs and run reflection from there, which is also presented as a solution in the book? Any opinions? I have made an attempt at ASCII drawing below, to me the central core solution makes more sense. In my experience, a core ibgp mesh will scale to at least 70 sessions per device. I would suggest that route reflection certainly be done between core and aggregation devices per pop. Central reflection may be prone to failure depending on the design of the network. Also, when you mention seperate igp domains, are you referring to areas/levels, or instances? Both OSPF and ISIS scale quite well using area or level hierarchy, which mostly mitigates the necessity for seperate igp instances. Regards, --phil Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48521t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
Re: Route Reflection with Multiple POPs [7:48509]
Let me preface this by saying that I am trying to learn more about large scale BGP design and operation. This question is on route reflectors when you have multiple POPs in seperate IGP domains. If you currently have one POP and are going to move to 2 within the same AS, you can either run full mesh (doesn't scale), reflectors, or confederations. Assuming you don't currently have a central core that the POPs connect back to, how well does reflection scale? I was reading Building Service Provider Networks [Berkowitz], and it states that iBGP doesn't scale well once you go above 15-20 sessions per router. Not an absolute number, but a fairly good one based on experience. There are a lot of variables involved. It also states that most ISPs run reflectors instead of confederations, but I believe that statement is being made under the assumption that the ISP will have a central core to which the POPs will connect. Do remember that you can create hierarchies of reflectors. I recognize that it can be argued that the upper levels of hierarchy, de facto, are a core. This would indicate to me that assuming you don't have a central core, one could only connect 6 or 7 POPs (dual reflectors for redundancy) together using reflection before you would have to either create a central core to reduce the amount of iBGP sessions, or turn to confederations. Perhaps the best way to accomplish this would be to establish a core in one of the POPs and run reflection from there, which is also presented as a solution in the book? As you point out, the core doesn't have to be in the exact geographic center. Not every US carrier has its central point in Kansas! The likelihood is that one of the POPs will be colocated with your technical staff, infrastructure servers, etc., and the core might be no more than some LANs at that site. Any opinions? I have made an attempt at ASCII drawing below, to me the central core solution makes more sense. First Scenario, no central core POP 1 POP 2 Cluster Cluster o---o Full Mesh Between Reflectors o---o Second Scenario, central core establised at one of the POPs POP 1 POP 2 Core Reflectors Clients o- -o Clients o- \ / -o o o POP 3/ \ POP 4 Clients o- -o Clients o- -o Guy H. Lupi CCIE No. 9275 Also, one of the philosophies in evolving core design with MPLS is that the core may essentially be IGP and MPLS only, with the distribution tier routers having the BGP knowledge. In other words, geography forces you to have lots of edge routers that still can distribute some of the policy processing load, while the core routers can be big, strong, and stupid. Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48522t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
Re: Route Reflection with Multiple POPs [7:48509]
At 7:17 PM + 7/10/02, Phillip Heller wrote: On Wed, Jul 10, 2002 at 04:42:25PM +, Lupi, Guy wrote: Let me preface this by saying that I am trying to learn more about large scale BGP design and operation. This question is on route reflectors when you have multiple POPs in seperate IGP domains. If you currently have one POP and are going to move to 2 within the same AS, you can either run full mesh (doesn't scale), reflectors, or confederations. Assuming you don't currently have a central core that the POPs connect back to, how well does reflection scale? I was reading Building Service Provider Networks [Berkowitz], and it states that iBGP doesn't scale well once you go above 15-20 sessions per router. It also states that most ISPs run reflectors instead of confederations, but I believe that statement is being made under the assumption that the ISP will have a central core to which the POPs will connect. This would indicate to me that assuming you don't have a central core, one could only connect 6 or 7 POPs (dual reflectors for redundancy) together using reflection before you would have to either create a central core to reduce the amount of iBGP sessions, or turn to confederations. Perhaps the best way to accomplish this would be to establish a core in one of the POPs and run reflection from there, which is also presented as a solution in the book? Any opinions? I have made an attempt at ASCII drawing below, to me the central core solution makes more sense. In my experience, a core ibgp mesh will scale to at least 70 sessions per device. I would suggest that route reflection certainly be done between core and aggregation devices per pop. With more modern routers, I certainly can see that working, especially depending on how well you dampen flap and generally control the churn rate. One also has to be careful about all the issues evolving in the permanent oscillation drafts and discussion in the IDR WG. 15-20 is an old Cisco recommendation that, like the number of areas per OSPF ABR, certainly can be exceeded when you understand the issues. Central reflection may be prone to failure depending on the design of the network. I certainly didn't mean to imply that there is a single reflector. My book examples typically show dual reflectors in an important cluster. One of the questions, of course, is the extent to which you use (G)MPLS in the core. There are traffic engineering, shared risk group management, and other issues that can make a non-traditional, largely BGP-free core attractive. The BGP intelligence lies in the edge and aggregation routers, which inherently distribute control plane workload. Also, when you mention seperate igp domains, are you referring to areas/levels, or instances? Both OSPF and ISIS scale quite well using area or level hierarchy, which mostly mitigates the necessity for seperate igp instances. Agreed. Indeed, there are many operational carriers that have 1000+ routers in a single area, albeit with some tuning and stable facilities. Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48529t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
RE: Route Reflection with Multiple POPs [7:48509]
When you say that OSPF scales very well in a heirarchy, I realize that there are a lot of factors involved. Let's assume that all routers in each POP with the exception of aggregation and core are in NSSA areas to control LSA's but still be allowed to insert external routes, but obviously the routers in the core would have to maintain all of the information. How large can you scale a topology like this? I am not concerned so much with the number of routers, but with the number of routes. Are we talking about 8000, 16000, 24000, 4 routes? I also realize the types of LSA's play a big part in OSPF, but assuming that your aggregation and core routers were very high end Juniper or Cisco routers, what would be a general number using OSPF? ISIS? *-Original Message- *From: Phillip Heller [mailto:[EMAIL PROTECTED]] *Sent: Wednesday, July 10, 2002 3:17 PM *To: [EMAIL PROTECTED] *Subject: Re: Route Reflection with Multiple POPs [7:48509] * * *On Wed, Jul 10, 2002 at 04:42:25PM +, Lupi, Guy wrote: * Let me preface this by saying that I am trying to learn more *about large * scale BGP design and operation. This question is on route *reflectors when * you have multiple POPs in seperate IGP domains. If you *currently have one * POP and are going to move to 2 within the same AS, you can *either run full * mesh (doesn't scale), reflectors, or confederations. *Assuming you don't * currently have a central core that the POPs connect back to, *how well does * reflection scale? I was reading Building Service Provider Networks * [Berkowitz], and it states that iBGP doesn't scale well once *you go above * 15-20 sessions per router. It also states that most ISPs *run reflectors * instead of confederations, but I believe that statement is *being made under * the assumption that the ISP will have a central core to *which the POPs will * connect. This would indicate to me that assuming you don't *have a central * core, one could only connect 6 or 7 POPs (dual reflectors *for redundancy) * together using reflection before you would have to either *create a central * core to reduce the amount of iBGP sessions, or turn to *confederations. * Perhaps the best way to accomplish this would be to *establish a core in * one of the POPs and run reflection from there, which is also *presented as a * solution in the book? Any opinions? I have made an attempt at ASCII * drawing below, to me the central core solution makes more sense. * *In my experience, a core ibgp mesh will scale to at least 70 sessions *per device. I would suggest that route reflection certainly be done *between core and aggregation devices per pop. * *Central reflection may be prone to failure depending on the design of *the network. * *Also, when you mention seperate igp domains, are you referring to *areas/levels, or instances? Both OSPF and ISIS scale quite well using *area or level hierarchy, which mostly mitigates the necessity for *seperate igp instances. * *Regards, * * --phil * * * * *Report misconduct *and Nondisclosure violations to [EMAIL PROTECTED] * Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48533t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
RE: Route Reflection with Multiple POPs [7:48509]
Please also keep in mind that my generalizations of the solutions in the book may leave something to be desired, and the solutions that I pointed out were certainly not the only ones presented. *-Original Message- *From: Howard C. Berkowitz [mailto:[EMAIL PROTECTED]] *Sent: Wednesday, July 10, 2002 4:10 PM *To: [EMAIL PROTECTED] *Subject: Re: Route Reflection with Multiple POPs [7:48509] * * *At 7:17 PM + 7/10/02, Phillip Heller wrote: *On Wed, Jul 10, 2002 at 04:42:25PM +, Lupi, Guy wrote: * Let me preface this by saying that I am trying to learn *more about large * scale BGP design and operation. This question is on route *reflectors *when * you have multiple POPs in seperate IGP domains. If you *currently have *one * POP and are going to move to 2 within the same AS, you can *either run *full * mesh (doesn't scale), reflectors, or confederations. *Assuming you don't * currently have a central core that the POPs connect back *to, how well *does * reflection scale? I was reading Building Service Provider Networks * [Berkowitz], and it states that iBGP doesn't scale well *once you go above * 15-20 sessions per router. It also states that most ISPs *run reflectors * instead of confederations, but I believe that statement is *being made *under * the assumption that the ISP will have a central core to *which the POPs *will * connect. This would indicate to me that assuming you don't have a *central * core, one could only connect 6 or 7 POPs (dual reflectors *for redundancy) * together using reflection before you would have to either create a *central * core to reduce the amount of iBGP sessions, or turn to *confederations. * Perhaps the best way to accomplish this would be to *establish a core in * one of the POPs and run reflection from there, which is *also presented *as a * solution in the book? Any opinions? I have made an *attempt at ASCII * drawing below, to me the central core solution makes more sense. * *In my experience, a core ibgp mesh will scale to at least 70 sessions *per device. I would suggest that route reflection certainly be done *between core and aggregation devices per pop. * *With more modern routers, I certainly can see that working, *especially depending on how well you dampen flap and generally *control the churn rate. One also has to be careful about all the *issues evolving in the permanent oscillation drafts and discussion in *the IDR WG. * *15-20 is an old Cisco recommendation that, like the number of areas *per OSPF ABR, certainly can be exceeded when you understand the *issues. * * *Central reflection may be prone to failure depending on the design of *the network. * *I certainly didn't mean to imply that there is a single reflector. My *book examples typically show dual reflectors in an important cluster. * *One of the questions, of course, is the extent to which you use *(G)MPLS in the core. There are traffic engineering, shared risk group *management, and other issues that can make a non-traditional, largely *BGP-free core attractive. The BGP intelligence lies in the edge and *aggregation routers, which inherently distribute control plane *workload. * * *Also, when you mention seperate igp domains, are you referring to *areas/levels, or instances? Both OSPF and ISIS scale quite well using *area or level hierarchy, which mostly mitigates the necessity for *seperate igp instances. * *Agreed. Indeed, there are many operational carriers that have 1000+ *routers in a single area, albeit with some tuning and stable *facilities. * * * * Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48535t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
Re: Route Reflection with Multiple POPs [7:48509]
On Wed, Jul 10, 2002 at 04:48:30PM -0400, Lupi, Guy wrote: When you say that OSPF scales very well in a heirarchy, I realize that there are a lot of factors involved. Let's assume that all routers in each POP with the exception of aggregation and core are in NSSA areas to control LSA's but still be allowed to insert external routes, but obviously the routers in the core would have to maintain all of the information. How large can you scale a topology like this? I am not concerned so much with the number of routers, but with the number of routes. Are we talking about 8000, 16000, 24000, 4 routes? I also realize the types of LSA's play a big part in OSPF, but assuming that your aggregation and core routers were very high end Juniper or Cisco routers, what would be a general number using OSPF? ISIS? In a single ospf instance, and without summarization or stubby/nssa areas, I've seen ~ 4000 subnets. With summarization and/or stubby/nssa areas, I've seen as many as 6000. n.b. - these above numbers are rounded and are examples observed on networks that are comprised of very high end routers. The key is to carry only infrastructure networks in your igp. Minimize infrastructure networks in igp by aggregating (ie, redist a static /24 for all you /30 point-to-point customers). Use bgp to carry customer prefixes. Networks with 16000 routes in igp are probably on the border of resource starvation with hardware that is deployed. Networks with more than 16000 routes in igp are probably doing something like redisting bgp into igp, which is always a bad idea. --phil Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48539t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
RE: Route Reflection with Multiple POPs [7:48509]
I know that you can run confederations and reflectors, and seperate levels of reflection, which Cisco refers to as nested reflection. Now my question is, how would you set up your bgp peering? Due to financial constraints I would imagine that the best thing to do would be to have one circuit from POP router 1 to core router 1, and another circuit from POP router 2 to core router 2. Assuming that you are only running BGP in the core, and that the clients have to have a session with each reflector, how would you communicate the loopback addresses of all the routers to each other? Are static routes used in this situation? Thanks to all of the people who responded by the way, I appreciate the direction. Ever feel like you know so much, only to read a book and find out that you know so little :)? *-Original Message- *From: Peter van Oene [mailto:[EMAIL PROTECTED]] *Sent: Wednesday, July 10, 2002 3:00 PM *To: [EMAIL PROTECTED] *Subject: Re: Route Reflection with Multiple POPs [7:48509] * * *Couple thoughts. First off, one can confederate and reflect *at the same *time (ie rr clusters inside sub-as's). Also, keep in mind that these *techniques deal with control, not forwarding and thus it is *possible, and *somewhat common, to have a RR hierarchy that does not map *directly to the *physical topology. I tend to see an arbitrary set of core *routers (from *2-20ish) that form the central IBGP mesh with the second level of the *hierarchy being formed by lead major pop routers, or dedicated route *servers (ie one armed routers that simply reflect routes). I *personally *try to keep the topology as flat as possible (ie less than 4 *levels in the *hierarchy) but I don't have any particular technical driver *for that other *than to minimize complexity and aid troubleshooting. * *Of note, this assumes you are reflecting full routes throughout your *backbone. I've seen providers try and reflect partial routes *to areas of *their backbone to allow for smaller routers where RR *topologies can lead to *blackholes. In general, if you are reflecting all routes, you *shouldn't run *into issues here. * *I * * * *At 04:42 PM 7/10/2002 +, Lupi, Guy wrote: *Let me preface this by saying that I am trying to learn more *about large *scale BGP design and operation. This question is on route *reflectors when *you have multiple POPs in seperate IGP domains. If you *currently have one *POP and are going to move to 2 within the same AS, you can *either run full *mesh (doesn't scale), reflectors, or confederations. *Assuming you don't *currently have a central core that the POPs connect back to, *how well does *reflection scale? I was reading Building Service Provider Networks *[Berkowitz], and it states that iBGP doesn't scale well once *you go above *15-20 sessions per router. It also states that most ISPs run *reflectors *instead of confederations, but I believe that statement is *being made under *the assumption that the ISP will have a central core to which *the POPs will *connect. This would indicate to me that assuming you don't *have a central *core, one could only connect 6 or 7 POPs (dual reflectors for *redundancy) *together using reflection before you would have to either *create a central *core to reduce the amount of iBGP sessions, or turn to confederations. *Perhaps the best way to accomplish this would be to establish *a core in *one of the POPs and run reflection from there, which is also *presented as a *solution in the book? Any opinions? I have made an attempt at ASCII *drawing below, to me the central core solution makes more sense. * * First Scenario, no central core * * POP 1 POP 2 * Cluster Cluster * o---o * Full Mesh Between Reflectors * o---o * * Second Scenario, central core establised at one of the POPs * * POP 1 POP 2 * Core Reflectors * Clients o- -o Clients * o- \ / -o *o o * POP 3/ \ POP 4 * Clients o- -o Clients * o- -o * * *Guy H. Lupi *CCIE No. 9275 * * * * Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48550t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
RE: Route Reflection with Multiple POPs [7:48509]
At 8:56 PM + 7/10/02, Lupi, Guy wrote: When you say that OSPF scales very well in a heirarchy, I realize that there are a lot of factors involved. Let's assume that all routers in each POP with the exception of aggregation and core are in NSSA areas to control LSA's but still be allowed to insert external routes, but obviously the routers in the core would have to maintain all of the information. Not necessarily. Ignoring traffic engineeering for the moment, the major purpose of the IGP in the core is finding other core routers and the ABRs. Fundamentally, it is BGP that contains the large number of routes, and the IGP is just providing next hops for the interconnected iBGP routers (or equivalent confederations). The core is likely to be a simple full mesh, or a relatively small cluster with multiple reflectors. How large can you scale a topology like this? I am not concerned so much with the number of routers, but with the number of routes. Are we talking about 8000, 16000, 24000, 4 routes? I also realize the types of LSA's play a big part in OSPF, but assuming that your aggregation and core routers were very high end Juniper or Cisco routers, what would be a general number using OSPF? ISIS? Again, if you are using hot-potato routing, you are far more concerned with the number of border routers (i.e., borders to each hierarchical area) and the routes between them. *-Original Message- *From: Phillip Heller [mailto:[EMAIL PROTECTED]] *Sent: Wednesday, July 10, 2002 3:17 PM *To: [EMAIL PROTECTED] *Subject: Re: Route Reflection with Multiple POPs [7:48509] * * *On Wed, Jul 10, 2002 at 04:42:25PM +, Lupi, Guy wrote: * Let me preface this by saying that I am trying to learn more *about large * scale BGP design and operation. This question is on route *reflectors when * you have multiple POPs in seperate IGP domains. If you *currently have one * POP and are going to move to 2 within the same AS, you can *either run full * mesh (doesn't scale), reflectors, or confederations. *Assuming you don't * currently have a central core that the POPs connect back to, *how well does * reflection scale? I was reading Building Service Provider Networks * [Berkowitz], and it states that iBGP doesn't scale well once *you go above * 15-20 sessions per router. It also states that most ISPs *run reflectors * instead of confederations, but I believe that statement is *being made under * the assumption that the ISP will have a central core to *which the POPs will * connect. This would indicate to me that assuming you don't *have a central * core, one could only connect 6 or 7 POPs (dual reflectors *for redundancy) * together using reflection before you would have to either *create a central * core to reduce the amount of iBGP sessions, or turn to *confederations. * Perhaps the best way to accomplish this would be to *establish a core in * one of the POPs and run reflection from there, which is also *presented as a * solution in the book? Any opinions? I have made an attempt at ASCII * drawing below, to me the central core solution makes more sense. * *In my experience, a core ibgp mesh will scale to at least 70 sessions *per device. I would suggest that route reflection certainly be done *between core and aggregation devices per pop. * *Central reflection may be prone to failure depending on the design of *the network. * *Also, when you mention seperate igp domains, are you referring to *areas/levels, or instances? Both OSPF and ISIS scale quite well using *area or level hierarchy, which mostly mitigates the necessity for *seperate igp instances. * *Regards, * * --phil * * * * *Report misconduct *and Nondisclosure violations to [EMAIL PROTECTED] * Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48553t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
RE: Route Reflection with Multiple POPs [7:48509]
At 10:27 PM + 7/10/02, Lupi, Guy wrote: I know that you can run confederations and reflectors, and seperate levels of reflection, which Cisco refers to as nested reflection. Now my question is, how would you set up your bgp peering? Due to financial constraints I would imagine that the best thing to do would be to have one circuit from POP router 1 to core router 1, and another circuit from POP router 2 to core router 2. Not sure what you are picturing as POP router. If you mean POP aggregation router, yes. There might be an outer core of reflectors connected to multiple POPs, and then a full iBGP mesh among the reflectors in the inner core. Assuming that you are only running BGP in the core, and that the clients have to have a session with each reflector, how would you communicate the loopback addresses of all the routers to each other? I tend to take the opposite view. In a network not using QoS, the core may or may not need to run BGP at all, or a relatively small number of BGP routes (e.g., the aggregate addresses for POPs). The critical thing to realize is the core is infrastructure, and its job is internal interconnection. Are static routes used in this situation? Could very well be. Remember, the core IGP routers are only interested in infrastructure, and, in a carrier environment, there is no excuse for having a very clean hierarchical addressing structure. Thanks to all of the people who responded by the way, I appreciate the direction. Ever feel like you know so much, only to read a book and find out that you know so little :)? *-Original Message- *From: Peter van Oene [mailto:[EMAIL PROTECTED]] *Sent: Wednesday, July 10, 2002 3:00 PM *To: [EMAIL PROTECTED] *Subject: Re: Route Reflection with Multiple POPs [7:48509] * * *Couple thoughts. First off, one can confederate and reflect *at the same *time (ie rr clusters inside sub-as's). Also, keep in mind that these *techniques deal with control, not forwarding and thus it is *possible, and *somewhat common, to have a RR hierarchy that does not map *directly to the *physical topology. I tend to see an arbitrary set of core *routers (from *2-20ish) that form the central IBGP mesh with the second level of the *hierarchy being formed by lead major pop routers, or dedicated route *servers (ie one armed routers that simply reflect routes). I *personally *try to keep the topology as flat as possible (ie less than 4 *levels in the *hierarchy) but I don't have any particular technical driver *for that other *than to minimize complexity and aid troubleshooting. * *Of note, this assumes you are reflecting full routes throughout your *backbone. I've seen providers try and reflect partial routes *to areas of *their backbone to allow for smaller routers where RR *topologies can lead to *blackholes. In general, if you are reflecting all routes, you *shouldn't run *into issues here. * *I * * * *At 04:42 PM 7/10/2002 +, Lupi, Guy wrote: *Let me preface this by saying that I am trying to learn more *about large *scale BGP design and operation. This question is on route *reflectors when *you have multiple POPs in seperate IGP domains. If you *currently have one *POP and are going to move to 2 within the same AS, you can *either run full *mesh (doesn't scale), reflectors, or confederations. *Assuming you don't *currently have a central core that the POPs connect back to, *how well does *reflection scale? I was reading Building Service Provider Networks *[Berkowitz], and it states that iBGP doesn't scale well once *you go above *15-20 sessions per router. It also states that most ISPs run *reflectors *instead of confederations, but I believe that statement is *being made under *the assumption that the ISP will have a central core to which *the POPs will *connect. This would indicate to me that assuming you don't *have a central *core, one could only connect 6 or 7 POPs (dual reflectors for *redundancy) *together using reflection before you would have to either *create a central *core to reduce the amount of iBGP sessions, or turn to confederations. *Perhaps the best way to accomplish this would be to establish *a core in *one of the POPs and run reflection from there, which is also *presented as a *solution in the book? Any opinions? I have made an attempt at ASCII *drawing below, to me the central core solution makes more sense. * * First Scenario, no central core * * POP 1 POP 2 * Cluster Cluster * o---o * Full Mesh Between Reflectors * o---o * * Second Scenario, central core establised at one of the POPs * * POP 1 POP 2 * Core Reflectors * Clients o- -o Clients * o- \ / -o *o o * POP 3/ \ POP 4 * Clients o
Re: Route Reflection with Multiple POPs [7:48509]
At 9:18 PM + 7/10/02, Phillip Heller wrote: On Wed, Jul 10, 2002 at 04:48:30PM -0400, Lupi, Guy wrote: When you say that OSPF scales very well in a heirarchy, I realize that there are a lot of factors involved. Let's assume that all routers in each POP with the exception of aggregation and core are in NSSA areas to control LSA's but still be allowed to insert external routes, but obviously the routers in the core would have to maintain all of the information. How large can you scale a topology like this? I am not concerned so much with the number of routers, but with the number of routes. Are we talking about 8000, 16000, 24000, 4 routes? I also realize the types of LSA's play a big part in OSPF, but assuming that your aggregation and core routers were very high end Juniper or Cisco routers, what would be a general number using OSPF? ISIS? In a single ospf instance, and without summarization or stubby/nssa areas, I've seen ~ 4000 subnets. With summarization and/or stubby/nssa areas, I've seen as many as 6000. n.b. - these above numbers are rounded and are examples observed on networks that are comprised of very high end routers. The key is to carry only infrastructure networks in your igp. Minimize infrastructure networks in igp by aggregating (ie, redist a static /24 for all you /30 point-to-point customers). Use bgp to carry customer prefixes. Networks with 16000 routes in igp are probably on the border of resource starvation with hardware that is deployed. Networks with more than 16000 routes in igp are probably doing something like redisting bgp into igp, which is always a bad idea. --phil Absolutely excellent guidelines! Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48555t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
Re: Route Reflection with Multiple POPs [7:48509]
On Wed, Jul 10, 2002 at 10:27:08PM +, Lupi, Guy wrote: I know that you can run confederations and reflectors, and seperate levels of reflection, which Cisco refers to as nested reflection. Now my question is, how would you set up your bgp peering? Due to financial constraints I would imagine that the best thing to do would be to have one circuit from POP router 1 to core router 1, and another circuit from POP router 2 to core router 2. Assuming that you are only running BGP in the core, and that the clients have to have a session with each reflector, how would you communicate the loopback addresses of all the routers to each other? Are static routes used in this situation? Thanks to all of the people who responded by the way, I appreciate the direction. Ever feel like you know so much, only to read a book and find out that you know so little :)? I hate to say it, but the question is somewhat ambiguous and confusing... First and foremost, what is the physical topology of your backbone? How many devices are you dealing with? These days, most large service provider networks are of a partial mesh (physical) design. There is no central core site. Each pop has several aggregation boxes and several core boxes. The core boxes connect to each other and to other pop's core boxes. Also, the core boxes provide connections for the aggregation boxes. While it's common to see route-reflection from the core to aggregation boxes, it's less common to see route-reflection from one pop to other *multihomed* pops. Generally, customers maintain one bgp session per physical connection to the network; further, the bgp sessions are generally between the directly connected devices. Sometimes customers will want to bond multiple equal bandwidth paths. In this instance, customers generally have less than one bgp session per physical connection (ie: ebgp-multihop with static routes and cef, or Multilink PPP). --phil Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48556t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
Re: Route Reflection with Multiple POPs [7:48509]
All, Being one of the folks currently reading Howard's most recent book titled - Building Service Provider Networks, I must say that I'm enjoying the various points being addressed by this thread. Also, this does remind me of an article in the 2nd Qtr edition of Packet Magazine titled, ISP Dial Network Design using BGP. http://www.cisco.com/warp/public/784/packet/techtips.html#1 Good article! Howard, I just wanted to be sure that I didn't misunderstand what you said... Could very well be. Remember, the core IGP routers are only interested in infrastructure, and, in a carrier environment, there is no excuse for having a very clean hierarchical addressing structure. Did you mean there is no excuse for [not] having a very clean hierarchical addressing structure or I'm I missing something, as always..:- TIA Nigel - Original Message - From: Howard C. Berkowitz To: Sent: Wednesday, July 10, 2002 7:22 PM Subject: RE: Route Reflection with Multiple POPs [7:48509] At 10:27 PM + 7/10/02, Lupi, Guy wrote: I know that you can run confederations and reflectors, and seperate levels of reflection, which Cisco refers to as nested reflection. Now my question is, how would you set up your bgp peering? Due to financial constraints I would imagine that the best thing to do would be to have one circuit from POP router 1 to core router 1, and another circuit from POP router 2 to core router 2. Not sure what you are picturing as POP router. If you mean POP aggregation router, yes. There might be an outer core of reflectors connected to multiple POPs, and then a full iBGP mesh among the reflectors in the inner core. Assuming that you are only running BGP in the core, and that the clients have to have a session with each reflector, how would you communicate the loopback addresses of all the routers to each other? I tend to take the opposite view. In a network not using QoS, the core may or may not need to run BGP at all, or a relatively small number of BGP routes (e.g., the aggregate addresses for POPs). The critical thing to realize is the core is infrastructure, and its job is internal interconnection. Are static routes used in this situation? Could very well be. Remember, the core IGP routers are only interested in infrastructure, and, in a carrier environment, there is no excuse for having a very clean hierarchical addressing structure. Thanks to all of the people who responded by the way, I appreciate the direction. Ever feel like you know so much, only to read a book and find out that you know so little :)? *-Original Message- *From: Peter van Oene [mailto:[EMAIL PROTECTED]] *Sent: Wednesday, July 10, 2002 3:00 PM *To: [EMAIL PROTECTED] *Subject: Re: Route Reflection with Multiple POPs [7:48509] * * *Couple thoughts. First off, one can confederate and reflect *at the same *time (ie rr clusters inside sub-as's). Also, keep in mind that these *techniques deal with control, not forwarding and thus it is *possible, and *somewhat common, to have a RR hierarchy that does not map *directly to the *physical topology. I tend to see an arbitrary set of core *routers (from *2-20ish) that form the central IBGP mesh with the second level of the *hierarchy being formed by lead major pop routers, or dedicated route *servers (ie one armed routers that simply reflect routes). I *personally *try to keep the topology as flat as possible (ie less than 4 *levels in the *hierarchy) but I don't have any particular technical driver *for that other *than to minimize complexity and aid troubleshooting. * *Of note, this assumes you are reflecting full routes throughout your *backbone. I've seen providers try and reflect partial routes *to areas of *their backbone to allow for smaller routers where RR *topologies can lead to *blackholes. In general, if you are reflecting all routes, you *shouldn't run *into issues here. * *I * * * *At 04:42 PM 7/10/2002 +, Lupi, Guy wrote: *Let me preface this by saying that I am trying to learn more *about large *scale BGP design and operation. This question is on route *reflectors when *you have multiple POPs in seperate IGP domains. If you *currently have one *POP and are going to move to 2 within the same AS, you can *either run full *mesh (doesn't scale), reflectors, or confederations. *Assuming you don't *currently have a central core that the POPs connect back to, *how well does *reflection scale? I was reading Building Service Provider Networks *[Berkowitz], and it states that iBGP doesn't scale well once *you go above *15-20 sessions per router. It also states that most ISPs run *reflectors *instead of confederations, but I believe that statement is *being made under *the assumption that the ISP will have a central core to which *the POPs will *connect. This would indicate to me that assuming you don't *have a central *core, one could only connect 6
Re: Route Reflection with Multiple POPs [7:48509]
All, Being one of the folks currently reading Howard's most recent book titled - Building Service Provider Networks, I must say that I'm enjoying the various points being addressed by this thread. Also, this does remind me of an article in the 2nd Qtr edition of Packet Magazine titled, ISP Dial Network Design using BGP. http://www.cisco.com/warp/public/784/packet/techtips.html#1 Good article! Howard, I just wanted to be sure that I didn't misunderstand what you said... Could very well be. Remember, the core IGP routers are only interested in infrastructure, and, in a carrier environment, there is no excuse for having a very clean hierarchical addressing structure. Did you mean there is no excuse for [not] having a very clean hierarchical addressing structure or I'm I missing something, as always..:- TIA Nigel You've got me. A dirty infrastructure addressing system is a sin beyond redemption. - Original Message - From: Howard C. Berkowitz To: Sent: Wednesday, July 10, 2002 7:22 PM Subject: RE: Route Reflection with Multiple POPs [7:48509] At 10:27 PM + 7/10/02, Lupi, Guy wrote: I know that you can run confederations and reflectors, and seperate levels of reflection, which Cisco refers to as nested reflection. Now my question is, how would you set up your bgp peering? Due to financial constraints I would imagine that the best thing to do would be to have one circuit from POP router 1 to core router 1, and another circuit from POP router 2 to core router 2. Not sure what you are picturing as POP router. If you mean POP aggregation router, yes. There might be an outer core of reflectors connected to multiple POPs, and then a full iBGP mesh among the reflectors in the inner core. Assuming that you are only running BGP in the core, and that the clients have to have a session with each reflector, how would you communicate the loopback addresses of all the routers to each other? I tend to take the opposite view. In a network not using QoS, the core may or may not need to run BGP at all, or a relatively small number of BGP routes (e.g., the aggregate addresses for POPs). The critical thing to realize is the core is infrastructure, and its job is internal interconnection. Are static routes used in this situation? Could very well be. Remember, the core IGP routers are only interested in infrastructure, and, in a carrier environment, there is no excuse for having a very clean hierarchical addressing structure. Thanks to all of the people who responded by the way, I appreciate the direction. Ever feel like you know so much, only to read a book and find out that you know so little :)? *-Original Message- *From: Peter van Oene [mailto:[EMAIL PROTECTED]] *Sent: Wednesday, July 10, 2002 3:00 PM *To: [EMAIL PROTECTED] *Subject: Re: Route Reflection with Multiple POPs [7:48509] * * *Couple thoughts. First off, one can confederate and reflect *at the same *time (ie rr clusters inside sub-as's). Also, keep in mind that these *techniques deal with control, not forwarding and thus it is *possible, and *somewhat common, to have a RR hierarchy that does not map *directly to the *physical topology. I tend to see an arbitrary set of core *routers (from *2-20ish) that form the central IBGP mesh with the second level of the *hierarchy being formed by lead major pop routers, or dedicated route *servers (ie one armed routers that simply reflect routes). I *personally *try to keep the topology as flat as possible (ie less than 4 *levels in the *hierarchy) but I don't have any particular technical driver *for that other *than to minimize complexity and aid troubleshooting. * *Of note, this assumes you are reflecting full routes throughout your *backbone. I've seen providers try and reflect partial routes *to areas of *their backbone to allow for smaller routers where RR *topologies can lead to *blackholes. In general, if you are reflecting all routes, you *shouldn't run *into issues here. * *I * * * *At 04:42 PM 7/10/2002 +, Lupi, Guy wrote: *Let me preface this by saying that I am trying to learn more *about large *scale BGP design and operation. This question is on route *reflectors when *you have multiple POPs in seperate IGP domains. If you *currently have one *POP and are going to move to 2 within the same AS, you can *either run full *mesh (doesn't scale), reflectors, or confederations. *Assuming you don't *currently have a central core that the POPs connect back to, *how well does *reflection scale? I was reading Building Service Provider Networks *[Berkowitz], and it states that iBGP doesn't scale well once *you go above *15-20 sessions per router. It also states that most ISPs run *reflectors *instead of confederations, but I believe that statement is *being made under *the assumption
RE: Route Reflection with Multiple POPs [7:48509]
I had a feeling that would happen, I will try to clarify. I was not trying to say that there should be a central core site for the ISP's entire network, but for pieces of it. Lets take a state like New York, within it you have 3 POPs, each is in it's own AS and runs an IGP. In each POP you have 2 core routers and 3 aggregation routers, the 2 core routers have EBGP connections and are reflectors for the 3 aggregation routers. Now you want all 4 POPs to be in the same AS, and they have to have direct connectivity to your 3 New Jersey POPs which should also be in the same AS. In that case, would it make sense to choose a NY POP and a NJ POP, install 2 core or backbone routers that do not participate in any IGP, use those to peer with the POPs in that state and in turn peer with the other state's core or backbone routers? This would significantly reduce the amount of peering that would be required. Hopefully the drawing won't be a disaster. NY POP 1 NJ POP 1 o o o o o o o o o o NY POP 2 NJ POP 2 o o o---o o o o \ / o o o/ \o o o---o NY POP 3 NJ POP 3 o o o o o o o o o o -Original Message- From: Phillip Heller To: [EMAIL PROTECTED] Sent: 7/10/2002 7:33 PM Subject: Re: Route Reflection with Multiple POPs [7:48509] On Wed, Jul 10, 2002 at 10:27:08PM +, Lupi, Guy wrote: I know that you can run confederations and reflectors, and seperate levels of reflection, which Cisco refers to as nested reflection. Now my question is, how would you set up your bgp peering? Due to financial constraints I would imagine that the best thing to do would be to have one circuit from POP router 1 to core router 1, and another circuit from POP router 2 to core router 2. Assuming that you are only running BGP in the core, and that the clients have to have a session with each reflector, how would you communicate the loopback addresses of all the routers to each other? Are static routes used in this situation? Thanks to all of the people who responded by the way, I appreciate the direction. Ever feel like you know so much, only to read a book and find out that you know so little :)? I hate to say it, but the question is somewhat ambiguous and confusing... First and foremost, what is the physical topology of your backbone? How many devices are you dealing with? These days, most large service provider networks are of a partial mesh (physical) design. There is no central core site. Each pop has several aggregation boxes and several core boxes. The core boxes connect to each other and to other pop's core boxes. Also, the core boxes provide connections for the aggregation boxes. While it's common to see route-reflection from the core to aggregation boxes, it's less common to see route-reflection from one pop to other *multihomed* pops. Generally, customers maintain one bgp session per physical connection to the network; further, the bgp sessions are generally between the directly connected devices. Sometimes customers will want to bond multiple equal bandwidth paths. In this instance, customers generally have less than one bgp session per physical connection (ie: ebgp-multihop with static routes and cef, or Multilink PPP). --phil Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48565t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
Re: Route Reflection with Multiple POPs [7:48509]
On Thu, Jul 11, 2002 at 01:28:08AM +, Lupi, Guy wrote: I had a feeling that would happen, I will try to clarify. I was not trying to say that there should be a central core site for the ISP's entire network, but for pieces of it. Lets take a state like New York, within it you have 3 POPs, each is in it's own AS and runs an IGP. In each POP you have 2 core routers and 3 aggregation routers, the 2 core routers have EBGP connections and are reflectors for the 3 aggregation routers. Now you want all 4 POPs to be in the same AS, and they have to have direct connectivity to your 3 New Jersey POPs which should also be in the same AS. In that case, would it make sense to choose a NY POP and a NJ POP, install 2 core or backbone routers that do not participate in any IGP, use those to peer with the POPs in that state and in turn peer with the other state's core or backbone routers? This would significantly reduce the amount of peering that would be required. Hopefully the drawing won't be a disaster. NY POP 1 NJ POP 1 o o o o o o o o o o NY POP 2 NJ POP 2 o o o---o o o o \ / o o o/ \o o o---o NY POP 3 NJ POP 3 o o o o o o o o o o Are you suggesting seperate per-pop AS's as a result of confederations, or otherwise? Confederations will certainly work, but may be overkill for a 6 pop regional network. I would suggest a single externally visible AS with a unified IGP. IGP convergence time is much preferrable to BGP convergence time. Also, BGP is better suited for political/administrative division, which shouldn't be necessary between the two regional networks. Also, when you say ...reduce the amount of peering..., are you referring to the number of ibgp sessions, external peering arrangements, or transit connections? --phil Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48569t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
Re: Route Reflection with Multiple POPs [7:48509]
On Thu, Jul 11, 2002 at 01:14:07AM +, Howard C. Berkowitz wrote: Did you mean there is no excuse for [not] having a very clean hierarchical addressing structure or I'm I missing something, as always..:- TIA Nigel You've got me. A dirty infrastructure addressing system is a sin beyond redemption. The corollary to this being, large, inherited dirty infrastructure addressing systems can be nearly impossible to clean up. :-) --phil Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48571t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
Re: Route Reflection with Multiple POPs [7:48509]
At 2:29 AM + 7/11/02, Phillip Heller wrote: On Thu, Jul 11, 2002 at 01:14:07AM +, Howard C. Berkowitz wrote: Did you mean there is no excuse for [not] having a very clean hierarchical addressing structure or I'm I missing something, as always..:- TIA Nigel You've got me. A dirty infrastructure addressing system is a sin beyond redemption. The corollary to this being, large, inherited dirty infrastructure addressing systems can be nearly impossible to clean up. :-) --phil Flow control CAN include the use of strong detergent and holy water. Message Posted at: http://www.groupstudy.com/form/read.php?f=7i=48575t=48509 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
