Re: routing between provider edge and CPE routers
In article <[EMAIL PROTECTED]>, Mike Bernico <[EMAIL PROTECTED]> wrote: >> So, by accepting routes from CPE you create a huge security >vulnerability >> for your customers, and other parties. This practice was understood >as a >> very bad network engineering for decades. > >Is there someplace I can find tidbits of information like this? I >haven't been alive decades so I must have missed that memo. Other than >this list I don't know where to find anyone with lots of experience >working for a service provider. You could have thought this up yourself. If you put something in production, /always/ ask yourself: if I was a hacker with bad intentions, how could I abuse this. And actually try to. I hacked my own network and machines a couple of times for fun, you learn a lot from it. Mike. -- Anyone who is capable of getting themselves made President should on no account be allowed to do the job -- Douglas Adams.
Re: routing between provider edge and CPE routers
> You don't say whether you're using Cisco, but recent IOSes have no trouble > with huge configurations. You may have to use 'service compress-config'. > Just stay with some specific items on large configurations though. Don´t for example dream of large access lists or your box will crash and burn. (with IOS, JunOS runs fine with anything I´ve thrown at it) Pete
https man in the middle [was: routing between provider edge and CPE routers]
It's even worse, a fake certificate from a man in the middle causes a trustworthy warning! If a certificate is not co-signed by any of the Browser compiled-in authorities, the Browsers will just ask: "...do you want to trust ". The hacker is completely free to fill in when he creates his own certificate on the server side (using plain openssl). This will be the only popup as the fake certificate will match the faked URL. Did M$ expect people to say "no" to the fake question "Do you want to trust Citibank" when they are in fact trying to connect to the real Citibank site? The default behavior of a browser should be to reject unsigned certificates and not even ask the question. Currently, there is even no warning that was learned from an unsigned certificate. /Martin (disclaimer... does not necessarily reflect the opinion of my employer...) > Even supposedly secure things like SSL-protected websites and SSH logins > are vulnerable due to the simple fact that most people won't think twice > to say "yes" to SSH complaining that it detected a new host key; or notice > that they're really talking to a different website (or that the lock icon > is not showing) - if it looks the same, and its URL is similar-looking > (l->1, O->0, etc; and with newish Unicode URLs the fun is unlimited).
RE: routing between provider edge and CPE routers
A few I've found but not tried out yet: OpenSource: http://www.freeipdb.org/ http://www.brownkid.net/NorthStar/ Windows: http://myips.dzoul.com/main.asp http://www.enterpriseip.net/ I make no promises as to applicability or suitability. www.sourceforge.net www.freshmeat.net These two sites might yield some more hits. -Original Message- From: Vadim Antonov [mailto:[EMAIL PROTECTED]] Sent: January 29, 2003 21:50 To: Mike Bernico Cc: [EMAIL PROTECTED] Subject: RE: routing between provider edge and CPE routers On Wed, 29 Jan 2003, Mike Bernico wrote: > Is there someplace I can find tidbits of information like this? I > haven't been alive decades so I must have missed that memo. Other than > this list I don't know where to find anyone with lots of experience > working for a service provider. Well, this list... in the old archives. The current backbone design issues were pretty much tossed around in 93-94, the "defensive networking" concept included. > I've never heard of software like that. Do you have a recommended > vendor? Is it typically developed in house? There's no sustainable market for those, so they're always home-built... Often it is just a collection of scripts and some RCS to keep configs in. > What can I say, I must work cheap! :) --vadim
RE: routing between provider edge and CPE routers
On Wed, 29 Jan 2003, Mike Bernico wrote: > Is there someplace I can find tidbits of information like this? I > haven't been alive decades so I must have missed that memo. Other than > this list I don't know where to find anyone with lots of experience > working for a service provider. Well, this list... in the old archives. The current backbone design issues were pretty much tossed around in 93-94, the "defensive networking" concept included. > I've never heard of software like that. Do you have a recommended > vendor? Is it typically developed in house? There's no sustainable market for those, so they're always home-built... Often it is just a collection of scripts and some RCS to keep configs in. > What can I say, I must work cheap! :) --vadim
RE: routing between provider edge and CPE routers
> So, by accepting routes from CPE you create a huge security vulnerability > for your customers, and other parties. This practice was understood as a > very bad network engineering for decades. Is there someplace I can find tidbits of information like this? I haven't been alive decades so I must have missed that memo. Other than this list I don't know where to find anyone with lots of experience working for a service provider. > 1) for single-homed sites use static routing, period. Dynamic routing > does not add anything useful in this case (if circuit is down, it's down, > there are no alternative ways to reach the customer's network). I agree, and all the feedback I've gotten should help me convince my peers. > The "convinience" of having to configure only CPE box is no excuse. Invest > some resources in a rather trivial configuration management system, which > keeps track of what network addresses were allocated to which customer, > and produces corresponding bits of router configuration automatically. > Most respectable ISPs did that long time ago. That will also reduce your > tech support costs. I've never heard of software like that. Do you have a recommended vendor? Is it typically developed in house? > PS. They should really require a test in "defensive networking" before >letting anyone to touch provider's routers... What can I say, I must work cheap!
RE: routing between provider edge and CPE routers
Thanks so much for all the feedback. All your input has been extremely helpful. Just to clarify: In our network core all customer routes are summarized and carried in iBGP. That was a recent change of mine. We use EIGRP to carry loopback and next hop information. I'm working on migrating us to IS-IS currently. (Hmm...that last sentence probably just opened up another can of worms...) At the network edge we use heavily filtered EIGRP. I was already leaning towards static routes, based on this groups input I would say that it will be a new priority. Thanks again Mike -Original Message- From: Bruce Robertson [mailto:[EMAIL PROTECTED]] Sent: Wednesday, January 29, 2003 11:09 AM To: Mike Bernico Cc: [EMAIL PROTECTED] Subject: Re: routing between provider edge and CPE routers We switched to BGP just recently, before things got out of hand. I highly recommend that you do so. It really does work better. It's very nice seeing your OSPF config carry essentially just the loopback interfaces. > In particular I'm wondering about the thousands of lines of > configuration used to make static routes work. You don't say whether you're using Cisco, but recent IOSes have no trouble with huge configurations. You may have to use 'service compress-config'. -- Bruce Robertson, President/CEO +1-775-348-7299 Great Basin Internet Services, Inc. fax: +1-775-348-9412 http://www.greatbasin.net
Re: routing between provider edge and CPE routers
On Wed, 29 Jan 2003, Christopher L. Morrow wrote: > On Wed, 29 Jan 2003, Mike Bernico wrote: > > > > We currently use an IGP to route between our distribution routers and > > the CPE routers we manage. > > So, if customers bounce your IGP churns away? And customers have access to > your IGP data (provided they break into the CPE, which is trivial, eh?) Worse yet, any customer which is able to feed routing information to the backbone (be it any IGP or BGP), unless filtered properly, is able to trivially create a man-in-the-middle (or trojan horse) attack on systems protected with plain-text passwords. Simply inject a longer-prefix route to someone else's network, and then examine (or modify) and bounce the source-routed packets to the ultimate destination. (Yes, Virginia, source routing IS evil, and has virtually no legitimate use). Even supposedly secure things like SSL-protected websites and SSH logins are vulnerable due to the simple fact that most people won't think twice to say "yes" to SSH complaining that it detected a new host key; or notice that they're really talking to a different website (or that the lock icon is not showing) - if it looks the same, and its URL is similar-looking (l->1, O->0, etc; and with newish Unicode URLs the fun is unlimited). So, by accepting routes from CPE you create a huge security vulnerability for your customers, and other parties. This practice was understood as a very bad network engineering for decades. The additional problems created by taking routing information from CPE are: increased amounts of route flap (because any bouncy tail circuit or malfunctioning/misconfigured CPE box will cause a flood of routing updates, potentially killing your entire network), and dramatically increased incidence of bogus routes (interfering with connectivity of your other customers, or some third parties). (I've seen even stupider things - people configuring CPE boxes to redistribute routes learned from customer's internal LANs! Any compromised PC, and you're toast). The solution is: 1) for single-homed sites use static routing, period. Dynamic routing does not add anything useful in this case (if circuit is down, it's down, there are no alternative ways to reach the customer's network). The "convinience" of having to configure only CPE box is no excuse. Invest some resources in a rather trivial configuration management system, which keeps track of what network addresses were allocated to which customer, and produces corresponding bits of router configuration automatically. Most respectable ISPs did that long time ago. That will also reduce your tech support costs. 2) for muti-homed sites you have to use routing protocols. Use BGP (_NOT_ IGP!) Implement a strict filtering on all routing updates you get from the customer. Manage these filters like you manage static routes. --vadim PS. They should really require a test in "defensive networking" before letting anyone to touch provider's routers...
Re: routing between provider edge and CPE routers
MB> Date: Wed, 29 Jan 2003 12:51:08 -0600 MB> From: Mike Bernico [ snipped and reformatted throughout ] MB> We currently use an IGP to route between our distribution MB> routers and the CPE routers we manage. I hope I'm misreading. If you're, say, running OSPF between your edge routers and CPE routers... MB> This is causing some problems with stability in that edge MB> IGP. ...I'd imagine so. Routes within one administrative domain that are preferred over BGP routes. Yikes. Roguecasting of GTLDs comes to mind as but one way to do evil deeds. MB> Does any other service provider use an IGP all the way to the MB> customer for non BGP customers or are we the only one? I MB> have a feeling we maybe are. Anything that depends on proper configuration of customer gear is inherently evil and dangerous. Of course, nobody ever creates an ethernet loop, redistributes the wrong prefixes, binds the wrong IP address, or anything like that, right? Hopefully I misread. Sharing your IGP with customers is very, very bad. Dynamic routes also need to be filtered at untrusted boundaries. Eddy -- Brotsman & Dreger, Inc. - EverQuick Internet Division Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 (785) 865-5885 Lawrence and [inter]national Phone: +1 (316) 794-8922 Wichita ~ Date: Mon, 21 May 2001 11:23:58 + (GMT) From: A Trap <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <[EMAIL PROTECTED]>, or you are likely to be blocked.
Re: routing between provider edge and CPE routers
My recommendation would be for you to: o redistribute directly connected interfaces via a strict filter into BGP and use iBGP to carry it around the local AS or o use passive interfaces in IGPs to do the same Avoid having to run a topology computation everytime a T1/56k links drops. I prefer the first option to the second based on experience UUNET / Global Crossing has w/ option #1. - Serge Thus spake Mike Bernico ([EMAIL PROTECTED]): > > > Hi, > > I apologize if this has been asked before. I work for an ISP that > started very small (hundreds of T1 and 56k customers) and has grown very > large in the last few years (thousands of T1 customers, as well as DS3 > customers and OC3 customers). > > We currently use an IGP to route between our distribution routers and > the CPE routers we manage. This has historically worked very well. We > have recently begun running into scalability issues however. We have > some distribution routers that have over 1000 T1 interfaces on them. > This is causing some problems with stability in that edge IGP. Does any > other service provider use an IGP all the way to the customer for non > BGP customers or are we the only one? I have a feeling we maybe are. > > If you do use an IGP, have you had any of the scalability issues we have > had? How did you fix them? > > If you use statics/BGP to CPE routers have you had any issues doing > that? In particular I'm wondering about the thousands of lines of > configuration used to make static routes work. > > > Thanks in advance for your advice. > > Mike Bernico
Re: routing between provider edge and CPE routers
On Wed, 29 Jan 2003, Mike Bernico wrote: > > > Hi, > > I apologize if this has been asked before. I work for an ISP that > started very small (hundreds of T1 and 56k customers) and has grown very > large in the last few years (thousands of T1 customers, as well as DS3 > customers and OC3 customers). > > We currently use an IGP to route between our distribution routers and > the CPE routers we manage. This has historically worked very well. We > have recently begun running into scalability issues however. We have > some distribution routers that have over 1000 T1 interfaces on them. > This is causing some problems with stability in that edge IGP. Does any > other service provider use an IGP all the way to the customer for non > BGP customers or are we the only one? I have a feeling we maybe are. So, if customers bounce your IGP churns away? And customers have access to your IGP data (provided they break into the CPE, which is trivial, eh?)
Re: routing between provider edge and CPE routers
We switched to BGP just recently, before things got out of hand. I highly recommend that you do so. It really does work better. It's very nice seeing your OSPF config carry essentially just the loopback interfaces. > In particular I'm wondering about the thousands of lines of > configuration used to make static routes work. You don't say whether you're using Cisco, but recent IOSes have no trouble with huge configurations. You may have to use 'service compress-config'. -- Bruce Robertson, President/CEO +1-775-348-7299 Great Basin Internet Services, Inc. fax: +1-775-348-9412 http://www.greatbasin.net
routing between provider edge and CPE routers
Hi, I apologize if this has been asked before. I work for an ISP that started very small (hundreds of T1 and 56k customers) and has grown very large in the last few years (thousands of T1 customers, as well as DS3 customers and OC3 customers). We currently use an IGP to route between our distribution routers and the CPE routers we manage. This has historically worked very well. We have recently begun running into scalability issues however. We have some distribution routers that have over 1000 T1 interfaces on them. This is causing some problems with stability in that edge IGP. Does any other service provider use an IGP all the way to the customer for non BGP customers or are we the only one? I have a feeling we maybe are. If you do use an IGP, have you had any of the scalability issues we have had? How did you fix them? If you use statics/BGP to CPE routers have you had any issues doing that? In particular I'm wondering about the thousands of lines of configuration used to make static routes work. Thanks in advance for your advice. Mike Bernico
