Re: Patching for Cisco vulnerability
--On Friday, July 18, 2003 12:29:30 -0600 Irwin Lazar [EMAIL PROTECTED] wrote: Just out of curiosity, are folks just applying the Cisco patch or do you go through some sort of testing/validation process to ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch? I'm trying here to gauge the length of time before this vulnerability is closed out. We had a phone conference with our Cisco people thursday lunch MEST and agreed on a testing scheme, where we would upgrade one of the (redundant) core routers and one access router (our network basically has two kinds of Cisco equipment, 12400 as core and 10700 as CPE) and let them run for an hour -- if no problems by then we'd roll the upgrade through the network, trying not to blackhole our customers. We went from 12.0(23)S1 to 12.0(23)S3, and it was mostly painless. -- Måns NilssonSystems Specialist +46 70 681 7204 KTHNOC MN1334-RIPE We're sysadmins. To us, data is a protocol-overhead. pgp0.pgp Description: PGP signature
Patching for Cisco vulnerability
Just out of curiosity, are folks just applying the Cisco patch or do you go through some sort of testing/validation process to ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch? I'm trying here to gauge the length of time before this vulnerability is closed out. irwin
RE: Patching for Cisco vulnerability
I don't see a lot of choice with this exploit. It's easy and quick. I knocked down a 7206 in less than a minute. Expedite where you can. BobG -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Irwin Lazar Sent: Friday, July 18, 2003 2:30 PM To: [EMAIL PROTECTED] Subject: Patching for Cisco vulnerability Just out of curiosity, are folks just applying the Cisco patch or do you go through some sort of testing/validation process to ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch? I'm trying here to gauge the length of time before this vulnerability is closed out. irwin
Re: Patching for Cisco vulnerability
On Fri, Jul 18, 2003 at 12:29:30PM -0600, Irwin Lazar wrote: Just out of curiosity, are folks just applying the Cisco patch or do you go through some sort of testing/validation process to ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch? I'm trying here to gauge the length of time before this vulnerability is closed out. most providers can easily go from (for example) 12.0(21)S3 to 12.0(21)S7 with less testing than from 12.0(21)S to 12.0(25)S The hurdles are still there to maintain the necessary customer notifications, etc.. but aside from that, I think the press is doing their job (good or bad) in that most customers are aware that there's something bad going on and people are moving to protect the internet infrastructure. - jared -- Jared Mauch | pgp key available via finger from [EMAIL PROTECTED] clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: Patching for Cisco vulnerability
On Fri, Jul 18, 2003 at 03:04:45PM -0400, Jared Mauch wrote: most providers can easily go from (for example) 12.0(21)S3 to 12.0(21)S7 with less testing than from 12.0(21)S to 12.0(25)S 12.0(21)S* (at least S5 and above) have broken SNMP interface counters and Cisco refuses to fix the bug in 12.0(21)S*, so people who don't want to lose money (accounting) are forced to upgrade to 12.0(25)S*. I guess they want to force all conservative ISPs to jump over the 12.0(22)S barrier. Some things won't be forgotten (see also recent discussion about the new non-Cisco-GBIC blocking). Voting with pockets takes place. Regards, Daniel
Re: Patching for Cisco vulnerability
On Fri, 2003-07-18 at 14:29, Irwin Lazar wrote: Just out of curiosity, are folks just applying the Cisco patch or do you go through some sort of testing/validation process to ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch? I think a lot of providers are doing the minimum testing required (upgrade, does it boot? can I ping?) and then rolling it out... I guess it's more of an implicit trust type deal rather than having routers running with an increased load because of ACL's and still having the possibility of a vulnerable router because something was overlooked.. Going forward, if you go the ACL route, every time you add a new interface you need to be sure to either apply the any any acl to it, or add the new ip's to the big block by ip acl ... I'm trying here to gauge the length of time before this vulnerability is closed out. I don't think it will ever truly go away.. there are lots of older routers that won't be able to support the newer code, albeit small routers like the 2500's, but they'll exist.. irwin -- --- Jason H. Frisvold Backbone Engineering Supervisor Penteledata Engineering [EMAIL PROTECTED] RedHat Engineer - RHCE # 807302349405893 Cisco Certified - CCNA # CSCO10151622 MySQL Core Certified - ID# 205982910 --- Imagination is more important than knowledge. Knowledge is limited. Imagination encircles the world. -- Albert Einstein [1879-1955] signature.asc Description: This is a digitally signed message part
Re: Patching for Cisco vulnerability
I don't think it will ever truly go away.. there are lots of older routers that won't be able to support the newer code, albeit small routers like the 2500's, but they'll exist.. Yes I have some old routers (2500s) for which no code exists which is patched and small enough to sit in the flash/memory... acl acl ! Steve
Re: Patching for Cisco vulnerability
On Fri, 2003-07-18 at 15:37, Stephen J. Wilcox wrote: I don't think it will ever truly go away.. there are lots of older routers that won't be able to support the newer code, albeit small routers like the 2500's, but they'll exist.. Yes I have some old routers (2500s) for which no code exists which is patched and small enough to sit in the flash/memory... acl acl ! And given the low processing power of those routers, acl's hurt ... :( Steve -- --- Jason H. Frisvold Backbone Engineering Supervisor Penteledata Engineering [EMAIL PROTECTED] RedHat Engineer - RHCE # 807302349405893 Cisco Certified - CCNA # CSCO10151622 MySQL Core Certified - ID# 205982910 --- Imagination is more important than knowledge. Knowledge is limited. Imagination encircles the world. -- Albert Einstein [1879-1955] signature.asc Description: This is a digitally signed message part
Re: Patching for Cisco vulnerability
On Fri, Jul 18, 2003 at 03:31:25PM -0400, Jared Mauch wrote: 12.0(21)S* (at least S5 and above) have broken SNMP interface counters and Cisco refuses to fix the bug in 12.0(21)S*, so people who don't Do you have a DDTS I can reference? Not handy, but from cisco-nsp Archives I've found CSCea35259 and CSCdy30984, and a reference to CSCea63754 which I can't take a look at in BugToolkit. Symptom: SNMP output octet counter stops counting traffic (except some control plane traffic it seems), with every few days jumping by weird amounts producing such funny things like 150mbps spikes on a FE interface. I've seen a box with a nicely loaded FE (30-70mbps) which took (reproducably) just about 48 hours to have this interface stop counting. If this would have been a customer interface, it would have meant reload router every two nights or lose money. This bug is supposed to be (finally) fixed in 12.0(25)S1. Given that you a) don't want to lose money and b) don't want to do two whole-network upgrades within a short time, going to 12.0(21)S7 to fix the vulnerabilty is no real option, so people are more or less forced to put their networks on bigger risk by going from 12.0(21)S* to (25)S1. Regards, Daniel
Re: Patching for Cisco vulnerability
--On Friday, July 18, 2003 21:57:57 +0200 Daniel Roesen [EMAIL PROTECTED] wrote: On Fri, Jul 18, 2003 at 03:31:25PM -0400, Jared Mauch wrote: 12.0(21)S* (at least S5 and above) have broken SNMP interface counters and Cisco refuses to fix the bug in 12.0(21)S*, so people who don't Do you have a DDTS I can reference? Not handy, but from cisco-nsp Archives I've found CSCea35259 and CSCdy30984, and a reference to CSCea63754 which I can't take a look at in BugToolkit. Symptom: SNMP output octet counter stops counting traffic (except some control plane traffic it seems), with every few days jumping by weird amounts producing such funny things like 150mbps spikes on a FE interface. I've seen a box with a nicely loaded FE (30-70mbps) which took (reproducably) just about 48 hours to have this interface stop counting. If this would have been a customer interface, it would have meant reload router every two nights or lose money. This bug is supposed to be (finally) fixed in 12.0(25)S1. Given that you a) don't want to lose money and b) don't want to do two whole-network upgrades within a short time, going to 12.0(21)S7 to fix the vulnerabilty is no real option, so people are more or less forced to put their networks on bigger risk by going from 12.0(21)S* to (25)S1. I'm running 12.0(25.2)S, and it has the bug REALLY squashed. LER -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED] US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
Re: Infrastructure Filtering (was Re: Patching for Cisco vulnerability)
* [EMAIL PROTECTED] (Christopher L. Morrow) [Sat 19 Jul 2003, 01:03 CEST]: hrm, what nodes don't run 55/53/77/103? What do? Do you have a list? Could we have it? I'm sure you know what devices in your network run Mobile IP or Sun ND (to paraphrase Randy Bush, you can probably count them on the fingers of your nose). Router#conf t Router(config)#ip receive-acl 10 no-idiocy Seriously though... the edge networks (as Jared pointed out) should be able to decide what they want to filter and what they don't... perhaps some large ISP would decide you don't want any traffic from 212/8 or perhaps all porn? Or all religious material? You don't want someone deciding what you do and don't get... unless that someone is you :) That's why I said that transit networks could filter only towards their own infrastructure. yes... inside my network I know what my loopbacks and links are, inside yours?? No idea... or Jared's or Tim Battles or... Luckily it's not your responsibility to protect them (only to intervene when advised they're under attack, which I've heard you're doing a very good job at - but that aside). Regards, -- Niels. -- The time of getting fame for your name on its own is over. Artwork that is only about wanting to be famous will never make you famous. Any fame is a bi-product of making something that means something. You don't go to a restaurant and order a meal because you want to have a shit. -- Banksy
RE: Infrastructure Filtering (was Re: Patching for Cisco vulnerability)
As mentioned before, Receive Path ACL (rACL) is already in 12.0(21)S2 (and forward) for the GSR and 12.0(24)S for the 7500. This is one way of doing infrastructure filtering without packet filtering the data plane (customer traffic). The second phase of Receive Path ACL (rACL) is going everywhere. The marketing name is Control Plane Protocol (CPP) ... but it also takes care of any packet punted to the receive path (i.e. packet with destination address = to the router). It is MQC based (ACL + rate-limiting). Think of it as a TCP wrapper for the receive path - but with the rate-limiting. The rate limiting part is important. It will first show up in 12.2S (and forward) and then cross-port/back-port through customer pressure (talk to your Cisco Account Teams). You'll see it on everything for the small boxes (26XX) to switches (CAT6Ks) to the high end (GSRs). Personally, I see this TCP Wrapper with Rate-Limit around a router as something that is going to be a requirement for all vendors on the Net. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Charles Sprickman Sent: Friday, July 18, 2003 1:21 PM To: [EMAIL PROTECTED] Subject: Infrastructure Filtering (was Re: Patching for Cisco vulnerability) This has me wondering if there are any BCPs that touch on the whole idea of filtering traffic destined to your router, or what the advisory called infrastructure filtering. All in all, it seems like a good idea to block any direct access to router interfaces. But as some have probably found already, it's a big pain in the arse. If I recall correctly, Rob's Secure IOS Template touches on filtering known services (the BGP listener, snmp), but what are people's feelings on maintaining filters on all interfaces *after* loading a fixed IOS? Thanks, Charles -- Charles Sprickman [EMAIL PROTECTED] On Fri, 18 Jul 2003, Irwin Lazar wrote: Just out of curiosity, are folks just applying the Cisco patch or do you go through some sort of testing/validation process to ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch? I'm trying here to gauge the length of time before this vulnerability is closed out. irwin
Re: Infrastructure Filtering (was Re: Patching for Cisco vulnerability)
On Sat, 19 Jul 2003, Niels Bakker wrote: * [EMAIL PROTECTED] (Christopher L. Morrow) [Sat 19 Jul 2003, 01:03 CEST]: hrm, what nodes don't run 55/53/77/103? What do? Do you have a list? Could we have it? I'm sure you know what devices in your network run Mobile IP or Sun ND (to paraphrase Randy Bush, you can probably count them on the fingers of your nose). my nose has many fingers... wait, thats hairs! :) though I do agree... So, I must apologize for reading your message's intent in reverse. Router#conf t Router(config)#ip receive-acl 10 no-idiocy Seriously though... the edge networks (as Jared pointed out) should be able to decide what they want to filter and what they don't... perhaps some large ISP would decide you don't want any traffic from 212/8 or perhaps all porn? Or all religious material? You don't want someone deciding what you do and don't get... unless that someone is you :) That's why I said that transit networks could filter only towards their own infrastructure. Agreed, and it does, to some extent... As should anyone elses, eh? It makes sense that if you have either of the 2 main vendor's products you can accomplish this task easily and at 'no cost' yes... inside my network I know what my loopbacks and links are, inside yours?? No idea... or Jared's or Tim Battles or... Luckily it's not your responsibility to protect them (only to intervene when advised they're under attack, which I've heard you're doing a very good job at - but that aside). We thank you, its a group effort... but as I said above, my apologies, this current event has me a bit punchy :)
Re: Infrastructure Filtering (was Re: Patching for Cisco vulnerability)
On Fri, 18 Jul 2003, Niels Bakker wrote: Personally I'd try to filter packets destined for known router interfaces and let the rest pass through. And of course not run known-buggy software (famous last words...). That would mean Windows (pick your flavor.) :-) Sorry, I couldn't resist. Curtis -- Curtis Maurand mailto:[EMAIL PROTECTED] http://www.maurand.com
