Re: UUNET issues?
On Sun, Nov 05, 2006 at 03:39:57AM +, Chris L. Morrow wrote: On Nov 4, 2006, at 7:54 PM, Herb Leong wrote: Hi, Anyone being impacted by UUNET? I'm fairly sure I'm not the only one who's said this in the last (pick a months long period of time, I'll guess 6): Could you be any less descriptive of the problem you are seeing? Perhaps he should see a dentist? Wisdom Teeth are impacted, people are affected by the effects of events. from Rick Jones sig... --bill
Re: UUNET issues?
On Nov 5, 2006, at 1:51 AM, Randy Bush wrote: Could you be any less descriptive of the problem you are seeing? the internet is broken. anyone know why? Did you ping it? is that what broke it? I'm sure it just needs to be rebooted.
Re: UUNET issues?
Speaking on Deep Background, the Press Secretary whispered: On Nov 5, 2006, at 1:51 AM, Randy Bush wrote: Could you be any less descriptive of the problem you are seeing? the internet is broken. anyone know why? Did you ping it? is that what broke it? I'm sure it just needs to be rebooted. Is this the day we disconnect everything and blow all the dirt out? -- A host is a host from coast to [EMAIL PROTECTED] no one will talk to a host that's close[v].(301) 56-LINUX Unless the host (that isn't close).pob 1433 is busy, hung or dead20915-1433
Re: UUNET issues?
David Lesher wrote: Speaking on Deep Background, the Press Secretary whispered: On Nov 5, 2006, at 1:51 AM, Randy Bush wrote: Could you be any less descriptive of the problem you are seeing? the internet is broken. anyone know why? Did you ping it? is that what broke it? I'm sure it just needs to be rebooted. Is this the day we disconnect everything and blow all the dirt out? You only *think* you are joking. I still remember the Day of the Great Restart when everyone on the ARPAnet had to shut down the IMPs and TIPs, and reload the control software. Why? There were literally thousands of rogue packets flying around the net, eating up bandwidth (and in those days, we are talking 56 kbps links!) and boy were those tubie-thingies plugged up! Shortly after that cusp event, per-packet TTL field was added to the NTP protocol, which is why TCP/IP has the TTL field in the IP packet. The network had added to it a self-cleaning function. Think of it as one long continuous sneeze.
Re: UUNET issues?
On Sun, 05 Nov 2006 07:16:07 -0800, Stephen Satchell [EMAIL PROTECTED] wrote: David Lesher wrote: Speaking on Deep Background, the Press Secretary whispered: On Nov 5, 2006, at 1:51 AM, Randy Bush wrote: Could you be any less descriptive of the problem you are seeing? the internet is broken. anyone know why? Did you ping it? is that what broke it? I'm sure it just needs to be rebooted. Is this the day we disconnect everything and blow all the dirt out? You only *think* you are joking. I still remember the Day of the Great Restart when everyone on the ARPAnet had to shut down the IMPs and TIPs, and reload the control software. Why? There were literally thousands of rogue packets flying around the net, eating up bandwidth (and in those days, we are talking 56 kbps links!) and boy were those tubie-thingies plugged up! Shortly after that cusp event, per-packet TTL field was added to the NTP protocol, which is why TCP/IP has the TTL field in the IP packet. I assume you mean NCP rather than NTP. Anyway, I don't think that would have helped if you're talking about the same incident I'm thinking of. There were application-level retransmissions of (corrupted) packets, complete with building new bad packets from bad data structures, all over the net The problem is documented in RFC 789 It and The Bug Heard 'Round the World are two of my favorite how complex systems fail papers; all system designers should read, memorize, and undertand both. The network had added to it a self-cleaning function. Think of it as one long continuous sneeze. --Steven M. Bellovin, http://www.cs.columbia.edu/~smb
Re: UUNET issues?
Anyway, I don't think that would have helped if you're talking about the same incident I'm thinking of. There were application-level retransmissions of (corrupted) packets, complete with building new bad packets from bad data structures, all over the net The problem is documented in RFC 789 It and The Bug Heard 'Round the World are two of my favorite how complex systems fail papers; all system designers should read, memorize, and undertand both. I actually asked Stephen if he was referring to the LSA corruption problem and he said he was referring to an earlier issue (circa 1972). As for the LSA issue- rebooting would have fixed the problem, assuming it was done by all nodes at the same time. All of the Link State tables would have been rebuilt from scratch by the IMPs and the corrupt announcements would have been gone. As I recall the IMP software was actually patched to ignore the problematic announcements from IMP 51. -Don
Re: UUNET issues?
As for the LSA issue- rebooting would have fixed the problem, assuming it was done by all nodes at the same time. All of the Link State tables would have been rebuilt from scratch by the IMPs and the corrupt announcements would have been gone. Turns out this is actually mentioned on page 14 of RFC 789. As I recall the IMP software was actually patched to ignore the problematic announcements from IMP 51. Not that it matters- but it was IMP 50 not 51. -Don
RE: UUNET issues?
Perhaps he should see a dentist? I just had my impacted Cogent taken care of last week with a Sprint... :) - This mail was scanned by BitDefender For more informations please visit http://www.bitdefender.com -
Re: UUNET issues?
Herb Leong wrote: Hi, Anyone being impacted by UUNET? Nothing unusual here, we are AS4927 connecting to AS701 in Los Angeles. -- Jay Hennigan - CCIE #7880 - Network Administration - [EMAIL PROTECTED] NetLojix Communications, Inc. - http://www.netlojix.com/ WestNet: Connecting you to the planet. 805 884-6323 - WB6RDV
Re: UUNET issues?
Two different transits here as well and nothing out of the norm. -- Elijah Savage | AOL IM:layer3rules Senior Network Engineer | When it has to be switched or routed. http://www.digitalrage.org | The Information Technology News Center - http://www.digitalrage.org/?page_id=46 for pgp public key On Nov 4, 2006, at 7:54 PM, Herb Leong wrote: Hi, Anyone being impacted by UUNET? /herb
Re: UUNET issues?
On Nov 4, 2006, at 7:54 PM, Herb Leong wrote: Hi, Anyone being impacted by UUNET? I'm fairly sure I'm not the only one who's said this in the last (pick a months long period of time, I'll guess 6): Could you be any less descriptive of the problem you are seeing? Also, did you contact customer support? (I'mm guessing yoyu aren't a uunet customer based on email/mx locations only though)
Re: UUNET issues?
Chris L. Morrow wrote: Could you be any less descriptive of the problem you are seeing? the internet is broken. anyone know why?
Re: UUNET issues?
On 11/4/06, Randy Bush [EMAIL PROTECTED] wrote: Chris L. Morrow wrote: Could you be any less descriptive of the problem you are seeing? the internet is broken. anyone know why? Because we didn't deploy IPv6 quickly enough? ;P Matt
Re: UUNET issues?
I'm fairly sure I'm not the only one who's said this in the last (pick a months long period of time, I'll guess 6): Could you be any less descriptive of the problem you are seeing? Ya know ... this whole descriptiveness thing has to be my biggest pet peeve. I have a couple of things going at any given time and I just *love* when I get an email entitled can you fix this? Bad enough when its a customer, worse when its the guys I work with - the best response to these sorts of things coming from an internal address is maybe and don't elaborate :-) They either take the hint and put a proper title on it with a cc: to our project manager, or they fix it themselves.
Re: UUNET issues?
the internet is broken. anyone know why? No.
Re: UUNET issues?
On Nov 4, 2006, at 7:45 PM, Randy Bush wrote: Chris L. Morrow wrote: Could you be any less descriptive of the problem you are seeing? the internet is broken. anyone know why? Did you ping it? Mike
Re: UUNET issues?
Could you be any less descriptive of the problem you are seeing? the internet is broken. anyone know why? Did you ping it? is that what broke it?
Re: UUNET issues?
On Nov 4, 2006, at 10:51 PM, Randy Bush wrote: Could you be any less descriptive of the problem you are seeing? the internet is broken. anyone know why? Did you ping it? is that what broke it? Please. That's how you *know* it's broken.
Re: UUNET issues?
On Sat, 4 Nov 2006 22:55:46 -0800 Michael Smith [EMAIL PROTECTED] wrote: On Nov 4, 2006, at 10:51 PM, Randy Bush wrote: Could you be any less descriptive of the problem you are seeing? the internet is broken. anyone know why? Did you ping it? is that what broke it? Please. That's how you *know* it's broken. With the prevalence of firewalls, I've found broken traceroutes to be a much more reliable indicator of broken Internettedness. -- Sheep are slow and tasty, and therefore must remain constantly alert. - Bruce Schneier, Beyond Fear
Re: UUNET connectivity in Minneapolis, MN
Not sure I understand how on earth something like this happens... power is not that confusing to make sure it does not stop working. Is that so? Have you read the report on the Northeast blackout of 2003? https://reports.energy.gov/ --Michael Dillon
Re: UUNET connectivity in Minneapolis, MN
I certainly understand why utility power goes out and that is the reason why MCI loosing power confuses me. I am pretty sure that someone at MCI also realizes why the blackout happens and how fragile things are. It is irresponsible for a Tier 1 infrastructure provider to not be able to generate their own and have large chunks of their network fail do to the inability to power it. I bet you every SBC CO in the affected area was still pushing power out to customer prems. Unless there is some sort of crazy story related to why a service provider could not keep the lights on, this should have not been an issue with proper operations and engineering. JD On Fri, 12 Aug 2005 [EMAIL PROTECTED] wrote: Not sure I understand how on earth something like this happens... power is not that confusing to make sure it does not stop working. Is that so? Have you read the report on the Northeast blackout of 2003? https://reports.energy.gov/ --Michael Dillon
RE: UUNET connectivity in Minneapolis, MN
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of James D. Butt Unless there is some sort of crazy story related to why a service provider could not keep the lights on, this should have not been an issue with proper operations and engineering. The building where one of our nodes sites got hit with an electrical fire in the basement one day, the fire department shut off all electrical to the whole building including the big diesel generators sitting outside the back of the building so all we had was battery power until that ran out 6 hours later. How do you prepare for that? Geo. George Roettger Netlink Services
RE: UUNET connectivity in Minneapolis, MN
Yes that is an exception... not what happened in this case You can come up with a lot of valid exceptions... There are many reasons why a Tier 1 provider does not stick all its eggs in multi-tenant buildings... smart things can be done with site selection. I am not saying ever customer needs to keep their network like this... but the really bug guys at the core of their network yes. JD On Fri, 12 Aug 2005, Geo. wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of James D. Butt Unless there is some sort of crazy story related to why a service provider could not keep the lights on, this should have not been an issue with proper operations and engineering. The building where one of our nodes sites got hit with an electrical fire in the basement one day, the fire department shut off all electrical to the whole building including the big diesel generators sitting outside the back of the building so all we had was battery power until that ran out 6 hours later. How do you prepare for that? Geo. George Roettger Netlink Services
Re: UUNET connectivity in Minneapolis, MN
Unless there is some sort of crazy story related to why a service provider could not keep the lights on, this should have not been an issue with proper operations and engineering. I'll let others tell you about the rat that caused a short circuit when Stanford attempted to switch to backup power. Or the time that fire crews told staff to evacuate a Wiltel colo near San Jose because of a backhoe that broke a gas pipe. The staff were prevented from starting their backup generators after power to the neighborhood was cut. In my opinion, the only way to solve this problem is to locate colos and PoPs in clusters within a city and deliver resilient DC power to these clusters from a central redundant generator plant. The generator plants, transmission lines and clusters can be engineered for resiliency. And then the highly flammable and dangerous quantities of fuel can be localized in a generator plant where they can be kept a safe distance from residential and office buildings. Unfortunately, to do this sort of thing requires vision which is something that has been lacking in the network operations field of late. --Michael Dillon
RE: UUNET connectivity in Minneapolis, MN
-Original Message- From: [EMAIL PROTECTED] On Behalf Of James D. Butt Unless there is some sort of crazy story related to why a service provider could not keep the lights on, this should have not been an issue with proper operations and engineering. 6 stories from the trenches Once a back hoe decided to punch through a high pressure natural gas main, right outside our offices. The fire department had us shut down ANYTHING that MIGHT make a spark. No nothing was able to run. It did not matter that we had uspes and such, all went dark for hours. During the Northridge earthquake (the one during the world series in sf.ba.ca.us) there was a BUNCH of disruption of the infrastructure, drives were shaken til they crashed, power wend down all over the area, Telco lines got knocked down, underground vaults got flooded, and data centers went off line. When ISDN was king(or ya get a t-1), I worked for an ISP in the bay area that was one of the few to have SOME connectivity when mae-w went down. We had a t-1 that went ânorthâ to another exchange point, and even though that little guy had %50+ packet loss, it kept chugging. We were one of the few ispâs that had ANY net connection, most of the people went in through their local MAE , (that was in the days before connecting to a MAE required that you be connected to several other MAEâs) Once while working for a startup in SF, I pushed for upses and backup power gen sets for our rack of boxes, and I was told that we were in the middle of the finintial district of SF, that bart/the cable cars ran near by, and that a big huge sub station with in rock throwing distance of our building, not to mention a power plant a couple miles away. There was no reason for us to invest in backup gen sets, or hours of ups timeâ¦. I asked what the procedure was if we lost power for an extended period of time, and I was told, âwe go homeâ weâ¦â¦ the power went off to the entire SF region, and I was able to shut down the equipment with out to much trouble, cause my laptop was plugged into a ups (at my desk) and the critical servers were on a ups, as well as the hub I was on. After I verified that we were stil up at our co-lo (via my CDPD modem) I stated the facts to my boss, and told him that I was following his established procedure for extended power loss. I was on my way home. (boss=not happy) A backup generator failed at a co-lo because of algae in the diesel fuel. Another time a valve broke in the buildings HVAC system sending pink gooey water under the door , and into the machine room. There are reasons why a bunch of 9âs piled together, weird stuff does happen. This is nanog, each âold timerâ has a few dozen of these events they can relate. The first 2 ya realy canât prepare for other than for all your stuff to be mirrored âsome place elseâ, the rest are preventable, but they were still rare. ( back to an operational slant) Get a microwave t-2 and shoot it over to some other building, get a freaking cable modem as a backup, or find another way to get your lines out. If having things work is important to you, YOU should make sure it happens! If people are preventing you from doing your job (having servers up and reachable) CYA, and point it out in the post mortem. -charles Curse the dark, or light a match. You decide, it's your dark. Valdis.Kletnieks in NANOG
Re: UUNET connectivity in Minneapolis, MN
On Fri, 12 Aug 2005 06:50:47 CDT, James D. Butt said: Unless there is some sort of crazy story related to why a service provider could not keep the lights on, this should have not been an issue with proper operations and engineering. So a while ago, we're in the middle of some major construction to put in infrastructure for a supercomputer. Meanwhile, as an unrelated project we installed a new diesel backup generator to replace an older generator that was undersized for our current systems, and take several hours of downtime on a Saturday to wire the beast in. The next Friday, some contractors are moving the entrance to our machine room about 30 feet to the right, so you don't walk into the middle of the supercomputer. Worker A starts moving a small red switch unit from its location next to where the door used to be to its new location next to where the door was going to be. Unfortunately, he did it before double-checking with Worker B that the small red switch was disarmed... Ka-blammo, a Halon dump... and of course that's interlocked with the power, so once the Halon stopped hissing, it was *very* quiet in there. Moral: It only takes one guy with a screwdriver. pgp0RjP3GJTEP.pgp Description: PGP signature
Re: UUNET connectivity in Minneapolis, MN
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 So I am standing in a datacenter fiddling with some fiber and listening to an electrician explaining to the datacenter owner how he has just finished auditing all of the backup power systems and that the transfer switch will work this time (unlike the last 3 times). This is making me a little nervous, but I keep quiet (unusual for me)... Electrician starts walking out of the DC, looks at the (glowing) Big Red Button (marked Emergency Power Off) and says Hey, why ya'll running on emergency power? and presses BRB. Lights go dark, disks spin down, Warren takes his business elsewhere! This is the same DC that had large basement mounted generators in a windowless building in NYC. Weeks before the above incident they had tried to test the generator (one of the failed transfer switch incidents), but apparently no one knew that there were manual flues at the top of the exhausts Carbon monoxide, building evacuated... Warren On Aug 12, 2005, at 8:27 AM, [EMAIL PROTECTED] wrote: On Fri, 12 Aug 2005 06:50:47 CDT, James D. Butt said: Unless there is some sort of crazy story related to why a service provider could not keep the lights on, this should have not been an issue with proper operations and engineering. So a while ago, we're in the middle of some major construction to put in infrastructure for a supercomputer. Meanwhile, as an unrelated project we installed a new diesel backup generator to replace an older generator that was undersized for our current systems, and take several hours of downtime on a Saturday to wire the beast in. The next Friday, some contractors are moving the entrance to our machine room about 30 feet to the right, so you don't walk into the middle of the supercomputer. Worker A starts moving a small red switch unit from its location next to where the door used to be to its new location next to where the door was going to be. Unfortunately, he did it before double- checking with Worker B that the small red switch was disarmed... Ka-blammo, a Halon dump... and of course that's interlocked with the power, so once the Halon stopped hissing, it was *very* quiet in there. Moral: It only takes one guy with a screwdriver. -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.4 (Darwin) iD8DBQFC/NVFHSkNr4ucEScRAkc9AKCnwraT9DztjAConsyuBZ7wDs/bJACgyrWR e2zcwlIffPxhTKfFJWm3T3A= =qDyJ -END PGP SIGNATURE-
Re: UUNET connectivity in Minneapolis, MN
[ Charset ISO-8859-1 unsupported, converting... ]
During the Northridge earthquake (the one during the
world series in sf.ba.ca.us) there was a BUNCH of
disruption of the infrastructure, drives were shaken
til they crashed, power wend down all over the area,
Telco lines got knocked down, underground vaults got
flooded, and data centers went off line.
Sorry.. wrong earthquake..
The Loma Prieta quake of 10/17/1989 occured during the opening
game of the World Series, featuring the San Francisco Giants,
and the Oakland Athletics in an all SF Bay area series.
The epicenter was in the Santa Cruz mountains, in the vicinity of
Mt Loma Prieta. Commercial power was lost to much of the bay area.
The Northridge quake occured on 1/17/1994, in southern California.
The epicenter was located in the San Fernando Valley, 20 miles NW of
Los Angeles.
As far as I recall, network disruption was minimal following the
Northridge quake, with a few sites offline {due to a machine room flooding
at UCLA?}
-- Welcome My Son, Welcome To The Machine --
Bob Vaughan | techie @ tantivy.net |
| P.O. Box 19792, Stanford, Ca 94309 |
-- I am Me, I am only Me, And no one else is Me, What could be simpler? --
Re: UUNET connectivity in Minneapolis, MN
Hi Chris, It seems all 800 numbers I have is busy. I heard that there was fire around home depot in Down Grove area, and it did hit the power grid, so UUNET/MCI POP lost the power. UUNET/MCI tech - Fortunately, our Network management center tech has the number for him - said he is waiting for generator coming in, but NO estimated time for recovery. Hyun Christopher L. Morrow wrote: traceroute or ping or end-node ip on your end... or did you call the customer support crew and ask them? --Chris (formerly [EMAIL PROTECTED]) ### ## UUNET Technologies, Inc. ## ## Some Security Engineering Group ## ## (W)703-886-3823 (C)703-338-7319 ## ### On Wed, 10 Aug 2005, Erik Amundson wrote: Anyone else having issues with UUNET connectivity in MSP? We were seeing slowness, now we see no traffic flow at all...we make it one hop, then nothin'. Erik Amundson A+, N+, CCNA, CCNP IT and Network Manager Open Access Technology Int'l, Inc. mailto:[EMAIL PROTECTED] CONFIDENTIAL INFORMATION: This email and any attachment(s) contain confidential and/or proprietary information of Open Access Technology International, Inc. Do not copy or distribute without the prior written consent of OATI. If you are not a named recipient to the message, please notify the sender immediately and do not retain the message in any form, printed or electronic.
Re: UUNET connectivity in Minneapolis, MN
On Thu, 11 Aug 2005 [EMAIL PROTECTED] wrote: we had a loss of comercial power(coned) in the downers grove terminal. terminal is up on generator power now. that seems to map to the internal firedrill as well, anyone else hit by this event?
Re: UUNET connectivity in Minneapolis, MN
Date: Thu, 11 Aug 2005 16:06:05 + (GMT) From: Christopher L. Morrow [EMAIL PROTECTED] Subject: Re: UUNET connectivity in Minneapolis, MN On Thu, 11 Aug 2005 [EMAIL PROTECTED] wrote: we had a loss of comercial power(coned) in the downers grove terminal. terminal is up on generator power now. that seems to map to the internal firedrill as well, anyone else hit by this event? Electric utility had a sub-station burn up. resulting in a medium-sized geographic area without power -- something like 17,000 residences according to news reports (no numbers on 'commercial' custeomrs provided). ATT has a facility in the affected area, and were also without utility power. Rumor mill says that Sprint had a (moderately small) number of T-3 circuits affected, as well.
RE: UUNET connectivity in Minneapolis, MN
info from the local news stations http://www.nbc5.com/news/4836579/detail.html?z=dpdpswid=2265994dppid=65192 http://www.chicagotribune.com/news/local/chi-050811outage,0,6108555.story?co ll=chi-news-hed -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Robert Bonomi Sent: Thursday, August 11, 2005 11:17 AM To: nanog@merit.edu Subject: Re: UUNET connectivity in Minneapolis, MN Date: Thu, 11 Aug 2005 16:06:05 + (GMT) From: Christopher L. Morrow [EMAIL PROTECTED] Subject: Re: UUNET connectivity in Minneapolis, MN On Thu, 11 Aug 2005 [EMAIL PROTECTED] wrote: we had a loss of comercial power(coned) in the downers grove terminal. terminal is up on generator power now. that seems to map to the internal firedrill as well, anyone else hit by this event? Electric utility had a sub-station burn up. resulting in a medium-sized geographic area without power -- something like 17,000 residences according to news reports (no numbers on 'commercial' custeomrs provided). ATT has a facility in the affected area, and were also without utility power. Rumor mill says that Sprint had a (moderately small) number of T-3 circuits affected, as well.
Re: UUNET connectivity in Minneapolis, MN
we had a loss of comercial power(coned) in the downers grove terminal. terminal is up on generator power now. that seems to map to the internal firedrill as well, anyone else hit by this event? Electric utility had a sub-station burn up. resulting in a medium-sized geographic area without power -- something like 17,000 residences according to news reports (no numbers on 'commercial' custeomrs provided). ATT has a facility in the affected area, and were also without utility power. Rumor mill says that Sprint had a (moderately small) number of T-3 circuits affected, as well. ATT must adhere to some diffrent engineering standards; as well devices we monitor there were all fine no blips... but all of the MCI customers we have in IL, MI, WI, MN all had issues... Power went out at 4:30 ish and ckts all dumped about 8:30 pm... Then bounced until 6:30 AM this morning. Not sure I understand how on earth something like this happens... power is not that confusing to make sure it does not stop working. JD
Re: UUNET connectivity in Minneapolis, MN
traceroute or ping or end-node ip on your end... or did you call the customer support crew and ask them? --Chris (formerly [EMAIL PROTECTED]) ### ## UUNET Technologies, Inc. ## ## Some Security Engineering Group ## ## (W)703-886-3823 (C)703-338-7319 ## ### On Wed, 10 Aug 2005, Erik Amundson wrote: Anyone else having issues with UUNET connectivity in MSP? We were seeing slowness, now we see no traffic flow at all...we make it one hop, then nothin'. Erik Amundson A+, N+, CCNA, CCNP IT and Network Manager Open Access Technology Int'l, Inc. mailto:[EMAIL PROTECTED] CONFIDENTIAL INFORMATION: This email and any attachment(s) contain confidential and/or proprietary information of Open Access Technology International, Inc. Do not copy or distribute without the prior written consent of OATI. If you are not a named recipient to the message, please notify the sender immediately and do not retain the message in any form, printed or electronic.
Re: UUNET connectivity in Minneapolis, MN
On Thu, Aug 11, 2005 at 03:42:58AM +, Christopher L. Morrow wrote: traceroute or ping or end-node ip on your end... or did you call the customer support crew and ask them? There was apparently a very serious fire at one or more of the Chicago area hubs MCI manages. They have a ticket #204 from today's date tracking this. I've been seeing reachability issues from the mid/west coast to my sites in NYC and NJ. I also have several Internet T1's down in both MN and Cleveland, OH. -- Mike Sawicki ([EMAIL PROTECTED])
Re: UUNET peering policy
On Mon, Jan 03, 2005 at 07:35:20PM -0500, Tom Vest wrote: Hey, did anyone notice when UU peering policy explicitly incorporated a requirement for number of transit customers served, measured by unique AS? It was between 18 and 28 August 2004. I believe it was on Friday the 27th but my policy archiving was not accurate to 48 hours. RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
Re: UUNet-Teleglobe
On Mon, Mar 22, 2004 at 12:29:39AM -0800, John Obi wrote: Hello, Do you see any issue between UUNet and TeleGlobe in East coast? No. 10 POS6-0.BR3.DCA6.ALTER.NET (152.63.38.117) 82.8 ms 82.6 ms 82.6 ms 11 204.255.174.198 (204.255.174.198) 84.0 ms 84.0 ms 84.0 ms 12 if-2-0.core2.Newark.Teleglobe.net (64.86.83.213) 348 ms (ttl=236!) 348 ms (ttl=236!) 349 ms First off, the actual border between UU and Teleglobe is between hops 10 and 11. Since the IP block for the /30 is owned by UU, Teleglobe probably (either intentionally or unintentionally) did not exchange the PTR info w/UU. You can tell this a couple of ways: a) UU's BR router designation (Border Router) b) The lack of the ix name on the first Teleglobe interface you see c) 204.255.174.197 == POS2-3.BR3.DCA6.ALTER.NET The rest, without knowing anything about Teleglobe's infrastructure in particular, looks like a Cisco MPLS-ism. When decrementing ttl through the lsp is turned on, you see the latency for the end of the tunnel on every reply. Since the rtt is a) consistant for every hop afterwards, and b) ends in a place where that rtt makes sense, it is a dead giveaway: 18 if-8-0-0.bb1.HongKong.Teleglobe.net (64.86.80.129) 352 ms (ttl=240!) 353 ms (ttl=240!) 352 ms (ttl=240!) Happy traceroute engineering. -- Richard A Steenbergen [EMAIL PROTECTED] http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: Source address validation (was Re: UUNet Offer New Protection
From: Paul Vixie [EMAIL PROTECTED] Date: 08 Mar 2004 06:35:16 + [EMAIL PROTECTED] (Ken Diliberto) writes: [...snip...] We're now blocking all SMTP traffic leaving the campus from non-blessed sources (read mail servers). The first day doing this we had comments about less junk mail traffic. We block traffic we consider harmful that shouldn't leave the campus. We're trying to do our part. Any suggestions how we can do better? yes. contact the nanog program committee so you can come to san francisco and tell the rest of us how you did it -- both in the ones and zeros, and in the dollars and cents. Paul: This is MY take and not Ken's... Firewall: block port 25 from all internal hosts except those 'recognized' as mail servers. For a user or department to get a mail server set up and 'recognized', they probably have to go through some sort of qualification and scanning process to ensure that the mail host is configured correctly... Going to San Francisco is still a good idea though. Regards, Gregory Hicks -- Paul Vixie - Gregory Hicks | Principal Systems Engineer Cadence Design Systems | Direct: 408.576.3609 555 River Oaks Pkwy M/S 6B1 | Fax: 408.894.3479 San Jose, CA 95134 | Internet: [EMAIL PROTECTED] I am perfectly capable of learning from my mistakes. I will surely learn a great deal today. A democracy is a sheep and two wolves deciding on what to have for lunch. Freedom is a well armed sheep contesting the results of the decision. - Benjamin Franklin The best we can hope for concerning the people at large is that they be properly armed. --Alexander Hamilton
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Mon, Mar 08, 2004 at 12:40:18AM -0500, Sean Donelan wrote: No. The work you've done is expected of you, as a good Internetwork neighbour. If you're not a good neighbour, next time you need my help, or the help of anyone else I know, please expect the finger. But I keep trying to do good work; and you keep giving me the finger. Why should I keep trying to do good work? Remember it works both ways. No I don't! You're a good Internet Neighbour. If I can expect you to do the right thing, you can expect it of me. And if I don't, you give me the finger instead. But don't give it to everyone, as a side of effect of wanting to just flip me off. -- Avleen Vig Systems Administrator Personal: www.silverwraith.com EFnet:irc.mindspring.com (Earthlink user access only)
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
Here is some insight on this issue What is Unicast Reverse Path Forwarding (uRPF)? Can a default route 0.0.0.0/0 be used to perform a uRPF check? http://www.cisco.com/warp/public/105/44.html#Q18 -Henry
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
Christopher L. Morrow wrote: 2. I've not seen large networks talking about their awful experiences with SAV. it melts routers, good enough for you? Specifically it melts linecards :( my experience is only on Cisco equipment though, so the linecard/ios/rev games must be played. If you upgrade, or initially install, E3 cards a large portion of this care is not necessary though. This is a problem that could be migrated out as new equipment/capabilities hit everyone's networks. I suspect that market pressure will push things in this direction anyway over time. That was exactly what I was doing by saying I will only get service from ISPs that run loose-uRPF in cores. (or all edges, including peering links.) I will not take service from ISP X, who is cheaper than ISP Y, if ISP X cannot assure me that I will not get bogon sourced traffic on my link. What you are saying above is not a technical argument against uRPF (as you grant that there is equipment that will do uRPF at core speeds.) - its a business one. So I am giving you a business incentive to take to your managers. Customers want this service which we cannot deliver w/o upgrades. Customers will not give us money unless we spend this money, and they will go to our competitors who have infrastructure that can do it. If your vendors cannot deliver equipment that meets your requirements to meet your customers' needs, you need to say the same thing to your vendors, and vote with dollars for those that can.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Mon, 8 Mar 2004, Steve Francis wrote: That was exactly what I was doing by saying I will only get service from ISPs that run loose-uRPF in cores. (or all edges, including peering links.) I will not take service from ISP X, who is cheaper than ISP Y, if ISP X cannot assure me that I will not get bogon sourced traffic on my link. Why do you care how a provider does X? Your requirement doesn't seem to be run loose-uRPF in cores, although that may be one way a provider chooses to solve the problem. You requirement is not get bogon sourced traffic on your link. I know its tempting to tell other people how to run their networks. But specifying the solution sometimes cuts out alternative solutions which work just as well or maybe better.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
Sean Donelan wrote: On Mon, 8 Mar 2004, Steve Francis wrote: That was exactly what I was doing by saying I will only get service from ISPs that run loose-uRPF in cores. (or all edges, including peering links.) I will not take service from ISP X, who is cheaper than ISP Y, if ISP X cannot assure me that I will not get bogon sourced traffic on my link. Why do you care how a provider does X? Your requirement doesn't seem to be run loose-uRPF in cores, although that may be one way a provider chooses to solve the problem. You requirement is not get bogon sourced traffic on your link. I know its tempting to tell other people how to run their networks. But specifying the solution sometimes cuts out alternative solutions which work just as well or maybe better. Correct. I was overstating my requirement. What I really want is as you described: I want assurance that any packet I receive on my proposed circuit is NOT sourced from a patently false IP address. (i.e. no packets sourced from reserved IP addresses, RFC 1918 IP addresses; addresses from blocks not yet allocated by routing registries, or addresses from blocks that are not currently being announced via BGP to the Internet.) I would also prefer that such packets be dropped as far as possible from the POP I am connected to, to minimise the chance of such packets overloading the carriers circuits into that POP. I know of no way to do this other than loose-uRPF in the core, or at least loose-uRPF on all edges, including peering connections. Can any of the operators that are arguing against loose-uRPF in the core state if they run loose uRPF on all peering connections, regardless of speed, as well as on all their edges? Or propose another way to achieve the same thing?
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sun, Mar 07, 2004 at 02:13:38AM -0500, Sean Donelan wrote: Try saying that after running a major DDoS target, with HIT ME your forehead. No offense Sean but I'd like you to back your claim up with some impirical data first. Has the number of DDOS attacks increased or decreased in the last few years has uRPF has become more widely deployed? Do you have any evidence the number of attacks are decreasing? Without any data to back this up, I'm estimating based on the attacks I've dealt with. I don't believe the number have gone down at all. If it has, it's done that for someone else, not me, I don't have any evidence. Nor do I *believe* the number of attacks is decreasing. If anything, its staying the same or going up, as more people decide it's fun to take networks offline through the greater and greater number of compromised hosts. If you want to do a little test, try this: In the last 5 years, compromised hosts have become a favourite for launching DDoS attacks from. If the number of compromised hosts with outbound Internet access has gone up, then either the frequency of attacks, or the amplitude of said attacks, or both have gone UP. We all know the number of compromised hosts continues to go up. The rest is simple logic. -- Avleen Vig Systems Administrator
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
just a question why is DDoS the only issue mentioned wrt source address validation? i'm sure there's other reasons to make sure your customers can't send spoofed packets. they might not always be as news-worthy, but i feel it's a provider's duty to do this. it shouldn't be optional (talking specifically about urpf on customer interfaces, loose where needed)
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
fingers wrote: just a question why is DDoS the only issue mentioned wrt source address validation? i'm sure there's other reasons to make sure your customers can't send spoofed packets. they might not always be as news-worthy, but i feel it's a provider's duty to do this. it shouldn't be optional (talking specifically about urpf on customer interfaces, loose where needed) Because _Distributed_ is the hot buzzword of the day. At least one of us thinks clean traffic is a Good Thing all the time. Packets that can't possibley be used for anything ought to be dumped at the earliest possible opportunity as soon as it is apparent (or could be if anybody looked) that they are from addresses that can't be reached or have any other obviously fatal defect.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
SD Date: Sat, 6 Mar 2004 22:04:58 -0500 (EST) SD From: Sean Donelan SD Would you rather ISPs spend money to SD 1. Deploying S-BGP? SD 2. Deploying uRPF? SD 3. Respond to incident reports? Let's look at the big picture instead of a taking a shallow mutex approach. If SAV were universal (ha ha ha!), one could discount spoofed traffic when analyzing flows. But, hey, why bother playing nice and helping other networks, eh? Am I the only one who's had IWFs -- even legitimate entities -- complain about packets from your network that weren't? It certainly would have been nice if $other_networks had used SAV. SAV doesn't take long to implement. Considering the time spent discounting spoofing when responding to incidents, I think there would be a _net_ savings (no pun intended) in time spent responding to incidents. Alas, that requires cooperation and doesn't provide instantaneous gratification. If it doesn't make/save a quick buck, why bother? Detection of sarcasm is left as an exercise to the reader. Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _ DO NOT send mail to the following addresses : [EMAIL PROTECTED] -or- [EMAIL PROTECTED] -or- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
SD Date: Sun, 7 Mar 2004 02:13:38 -0500 (EST) SD From: Sean Donelan SD Has the number of DDOS attacks increased or decreased in the SD last few years has uRPF has become more widely deployed? Number of life guards on duty increases in the summer. So does drowning. Therefore, having life guards on duty is not an effective measure to prevent drowning. Ice cream consumption increases in the summer. So does drowning. Therefore, it is ice cream consumption that causes drowning. (Time for arguments over who has the best and worst analogies!) SD Do you have any evidence the number of attacks are decreasing? Is number of attacks the sole metric? Are all attacks created equal? Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _ DO NOT send mail to the following addresses : [EMAIL PROTECTED] -or- [EMAIL PROTECTED] -or- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sun, 2004-03-07 at 11:08, fingers wrote: just a question why is DDoS the only issue mentioned wrt source address validation? uRPF, strict mode, is how I control 1000+ DSL pvc's from leaking private address space via broken NAT. Also, all other customer facing interfaces run uRPF, strict mode. It is a very powerful tool; null route some trouble causing customer space and traffic destined to this space is dropped via this null route AND traffic sourced from this space is dropped via uRPF, strict check. An AS112 NS also takes care of another facet of this problem. As to the question of DDoS'es and spoofed address space; once we close the hole of allowing DDoS'es to come from untraceable address space I feel we gain something very useful. We now know where the bad stuff is coming from. The solution to DDoS is not a black box that will go to Def Con 1 at the first sign of a port scan. You don't put out a fire with more fuel. Criminal investigation techniques are quite advanced. We cannot start to put them to use if attacks come from addresses that do not point back to the attacker. I am just as jaded as the next person with the present lack of law enforcement support in abuse issues but all of this is a quite new form of crime through a new medium. A push back system would give us the ability to quickly bring DDoS/DoS'es under control and complement a system to track down, gather evidence, and prosecute to persons in control of a DDoS/DoS. Based on my limited experience with all of this it seems the place for uRPF is not at the core (core in the context of the Internet backbone) but at the customer edge, where the problem starts. -- James H. Edwards Routing and Security At the Santa Fe Office: Internet at Cyber Mesa [EMAIL PROTECTED] [EMAIL PROTECTED]
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
actually, it would. universal uRPF would stop some attacks, and it would remove a plan B option for some attack-flowcharts. i would *much* rather play defense without facing this latent weapon available to the offense. I'm agreeing here, okay (yet anoter) example.. smurf attacks. These seem to be non-existent these days so shall we stop disabling 'ip directed-broadcast' on our routers? Steve
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sun, 7 Mar 2004, Avleen Vig wrote: On Sun, Mar 07, 2004 at 02:13:38AM -0500, Sean Donelan wrote: Try saying that after running a major DDoS target, with HIT ME your forehead. No offense Sean but I'd like you to back your claim up with some impirical data first. Has the number of DDOS attacks increased or decreased in the last few years has uRPF has become more widely deployed? Do you have any evidence the number of attacks are decreasing? Without any data to back this up, I'm estimating based on the attacks I've dealt with. I don't believe the number have gone down at all. If it has, it's done that for someone else, not me, Is this attacks on 'known magnets' or 'random stuff'. From what I've seen the frequency of attacks on 'all customers' seems to be slowing SOME. There are the normal nuisance points which attract attacks for whichever reason. So, Avleen, can you seperate the 'known magnets' from 'random stuff' and say which direction the trend is moving? As to the 'strength' of attacks. It seems that bandwidth and pps rates have incresed over time. This COULD BE because you can own up 10,000 xp machines in a heartbeat, or it could be a reflection of bigger/better/faster single hosts being taken over. It's hard to tell from my end of the party :( I don't have any evidence. Nor do I *believe* the number of attacks is decreasing. If anything, its staying the same or going up, as more people decide it's fun to take networks offline through the greater and greater number of compromised hosts. The greater number of compromisable hosts seems to be the constant in this arguement. So, like we've said for several years, until the end station is secured 'better' the consistency and strength of attacks will continue that upward trend.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sun, 7 Mar 2004, fingers wrote: just a question why is DDoS the only issue mentioned wrt source address validation? its easier to discuss than other things... for instance the number of broken vpn/nat systems out there that uRPF will break. Also, the folks with private addressed cores that will start appearing 'broken' when traceroute/unreachables stop working across their networks... i'm sure there's other reasons to make sure your customers can't send spoofed packets. they might not always be as news-worthy, but i feel it's a provider's duty to do this. it shouldn't be optional (talking specifically about urpf on customer interfaces, loose where needed) I'm not sure that anyone would argue that uRPF is bad, the arguement is in it's placement. I do think that part still needs to be worked out, that and making sure that your equipment can handle the task. There are certainly some people hampered by early adoption of some technologies which they can't get out from under in any reasonable fashion. --Chris (formerly [EMAIL PROTECTED]) ### ## UUNET Technologies, Inc. ## ## Manager ## ## Customer Router Security Engineering Team ## ## (W)703-886-3823 (C)703-338-7319 ## ###
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sun, 7 Mar 2004, Laurence F. Sheldon, Jr. wrote: fingers wrote: just a question why is DDoS the only issue mentioned wrt source address validation? i'm sure there's other reasons to make sure your customers can't send spoofed packets. they might not always be as news-worthy, but i feel it's a provider's duty to do this. it shouldn't be optional (talking specifically about urpf on customer interfaces, loose where needed) Because _Distributed_ is the hot buzzword of the day. and people offten seperate 'ddos' from 'dos', even though the end is the same as far as your customer is concerned... it's kinda funny really :) At least one of us thinks clean traffic is a Good Thing all the time. Packets that can't possibley be used for anything ought to be dumped at the earliest possible opportunity as soon as it is apparent (or could be if anybody looked) that they are from addresses that can't be reached or have any other obviously fatal defect. Here is a sticky point... There are reasons to allow 10.x.x.x sources to transit a network. Mostly the reasons come back to 'broken' configurations or 'broken' hardware. The reasons still equate to customer calls and 'broken' networking fromm their perspective. I think the thing you are actually driving at is the 'intent' of the packet, which is quite tough for the router to determine. --Chris (formerly [EMAIL PROTECTED]) ### ## UUNET Technologies, Inc. ## ## Manager ## ## Customer Router Security Engineering Team ## ## (W)703-886-3823 (C)703-338-7319 ## ###
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sun, 7 Mar 2004, Stephen J. Wilcox wrote: actually, it would. universal uRPF would stop some attacks, and it would remove a plan B option for some attack-flowcharts. i would *much* rather play defense without facing this latent weapon available to the offense. I'm agreeing here, okay (yet anoter) example.. smurf attacks. These seem to be non-existent these days so shall we stop disabling 'ip directed-broadcast' on our routers? smurf attacks are far from 'non-existent' today, however they are not as popular as in 1999-2000-2001. In fact netscan.org still shows almost 9k networks that are 'broken'.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sun, 7 Mar 2004, E.B. Dreger wrote: If SAV were universal (ha ha ha!), one could discount spoofed traffic when analyzing flows. But, hey, why bother playing nice and helping other networks, eh? SAV doesn't tell you where the packets came from. At best SAV tells you where the packets didn't come from. Am I the only one who's had IWFs -- even legitimate entities -- complain about packets from your network that weren't? It certainly would have been nice if $other_networks had used SAV. You still need to spend the same amount of time tracing the flows because you can't tell from the packet itself if something went wrong with SAV. Even if everyone said they did SAV (and meant it), things like uRPF rely on a number of things to work correctly. If any of those break or aren't secure, you still can't rely on the source address being accurate. Even if you deployed SAV/uRPF on 100% of your network, you probably wouldn't want to tell people about it due to the idiots with firewalls. SAV doesn't take long to implement. Considering the time spent discounting spoofing when responding to incidents, I think there would be a _net_ savings (no pun intended) in time spent responding to incidents. You would be wrong. There are networks that have deployed SAV/uRPF. They saw no _net_ savings. In the real world, it costs more to deploy and maintain SAV/uRPF. Have you noticed this thread is full of people who don't run large networks saying other people who do run networks should deploy SAV/uRPF. But there hasn't been anyone who does run large networks saying they deployed SAV/uRPF and it saved them money, made their network run better or improved the world?
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sun, Mar 07, 2004 at 08:28:53PM +, Christopher L. Morrow wrote: Without any data to back this up, I'm estimating based on the attacks I've dealt with. I don't believe the number have gone down at all. If it has, it's done that for someone else, not me, Is this attacks on 'known magnets' or 'random stuff'. From what I've seen the frequency of attacks on 'all customers' seems to be slowing SOME. There are the normal nuisance points which attract attacks for whichever reason. So, Avleen, can you seperate the 'known magnets' from 'random stuff' and say which direction the trend is moving? If we class popular websites, servers / networks at major ISPs, IRC servers and the latest popular thing as magnets, and small business sites, personal pages etc as the random stuff, then I don't believe attacks on magnets have gone down at all. On the random stuff I cannot comment, as I've had surprisingly little dealing with that. As to the 'strength' of attacks. It seems that bandwidth and pps rates have incresed over time. This COULD BE because you can own up 10,000 xp machines in a heartbeat, or it could be a reflection of bigger/better/faster single hosts being taken over. It's hard to tell from my end of the party :( I don't think it would be unfair to assume it is both. Again that stands to simple logic. More hosts on the internet = more potential drones. More availible global bandwidth = larger volume output from each drone. I don't have any evidence. Nor do I *believe* the number of attacks is decreasing. If anything, its staying the same or going up, as more people decide it's fun to take networks offline through the greater and greater number of compromised hosts. The greater number of compromisable hosts seems to be the constant in this arguement. So, like we've said for several years, until the end station is secured 'better' the consistency and strength of attacks will continue that upward trend. Indeed. I believe the ISP of the end user is the party responsible here. If the ISP is allowing access through their network, they need to be responsible for the data leaving their networl which originates in their network.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
smurf attacks are far from 'non-existent' today, however they are not as popular as in 1999-2000-2001. thats interesting, i've not seen/heard of one for ages.. (guess u have a wider testing ground :) In fact netscan.org still shows almost 9k networks that are 'broken'. actually i just ran that file thro a quick awk and sort to see to what extent these networks exist.. as you can see almost all only reply two or three times, not like in the old days with 100 replies being commonplace.. 5224 2 1834 3 897 4 334 5 167 6 56 7 19 8 15 9 7 10 11 11 6 12 3 13 6 14 1 15 1 16 4 17 5 18 1 23 1 26 1 28 1 100
Re: Source address validation (was Re: UUNet Offer New Protection
[EMAIL PROTECTED] (Sean Donelan) writes: SAV doesn't tell you where the packets came from. At best SAV tells you where the packets didn't come from. ...which is incredibly more valuable than not knowing anything at all. You would be wrong. There are networks that have deployed SAV/uRPF. They saw no _net_ savings. In the real world, it costs more to deploy and maintain SAV/uRPF. in the therefore-unreal world i live in, the ability to tell a GWF (goober with firewall) that the incident report they sent our noc could not possibly have come from here, is a net cost savings over having to prove it every time. Have you noticed this thread is full of people who don't run large networks saying other people who do run networks should deploy SAV/uRPF. distinguishingly, i do help run a network, and i'm not limiting my accusation (you guys are slackers) to uPRF-free networks of any particular size (big). -- Paul Vixie
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
removed paul from the direct reply since his mailserver doesn't like uunet mail servers :) On Sun, 7 Mar 2004, Stephen J. Wilcox wrote: smurf attacks are far from 'non-existent' today, however they are not as popular as in 1999-2000-2001. thats interesting, i've not seen/heard of one for ages.. (guess u have a wider testing ground :) just last week we had one... they do still happen. In fact netscan.org still shows almost 9k networks that are 'broken'. actually i just ran that file thro a quick awk and sort to see to what extent these networks exist.. as you can see almost all only reply two or three times, not like in the old days with 100 replies being commonplace.. Sure, but a list of 9k networks with this leve of response is still enough to do damage. It's getting better, no doubt about it but it's still a factor. --Chris (formerly [EMAIL PROTECTED]) ### ## UUNET Technologies, Inc. ## ## Manager ## ## Customer Router Security Engineering Team ## ## (W)703-886-3823 (C)703-338-7319 ## ###
Re: Source address validation (was Re: UUNet Offer New Protection
On Sun, 7 Mar 2004, Paul Vixie wrote: in the therefore-unreal world i live in, the ability to tell a GWF (goober with firewall) that the incident report they sent our noc could not possibly have come from here, is a net cost savings over having to prove it every time. Of course, some people claim large networks say that anyway so there is not net cost savings :-) In practice, GWF's do not send reports to noc's about packets which could not have possibly have come from here. They send reports about packets which have our IP addresses, but didn't originate here. The last thing you want to admit is you do SAV because GWF think SAV means every packet with that source address must have originated here. Whether or not we do SAV or everyone else does SAV, it doesn't save any time validating if a packet stream originated here. Did the packet actually originate here, or did SAV fail somewhere and it originated somewhere else? Dear NOC, 192.5.5.241 is attacking me. Prove it isn't. Rinse, Lather, Repeat. Maybe you got hacked in the last 5 seconds, and now you really are attacking them.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
SD Date: Sun, 7 Mar 2004 16:17:50 -0500 (EST) SD From: Sean Donelan SD SAV doesn't tell you where the packets came from. At best SD SAV tells you where the packets didn't come from. If SAV were universal, source addresses could not be spoofed. If source addresses could not be spoofed... SD You would be wrong. There are networks that have deployed SD SAV/uRPF. Some. I said all. SD They saw no _net_ savings. SD SD In the real world, it costs more to deploy and maintain SD SAV/uRPF. The benefit is to other networks. When other networks make your life easier, you benefit. If you want others to help you, help them. SD Have you noticed this thread is full of people who don't run SD large networks saying other people who do run networks should SD deploy SAV/uRPF. 1. SAV is most effective at the edge, which often implies the smaller networks should be doing it 2. I've not seen large networks talking about their awful experiences with SAV. Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _ DO NOT send mail to the following addresses : [EMAIL PROTECTED] -or- [EMAIL PROTECTED] -or- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked.
Re: Source address validation (was Re: UUNet Offer New Protection
SD Date: Sun, 7 Mar 2004 17:47:09 -0500 (EST) SD From: Sean Donelan SD In practice, GWF's ... send reports about packets which have SD our IP addresses, but didn't originate here. The last thing Probably because someone else failed to implement SAV. If $origin_net prevented spoofing your IP space, you'd not have had the problem. If other networks prevented spoofed sources, nobody else could source a packet from your address space. In this case, a packet apparently sourced from you network definitely would have come from your network. Therefore you'd no longer need to check to see if a packet was spoofed. Notice how AS_PATHs and netblock announcements tend to get filter. Why? SD you want to admit is you do SAV because GWF think SAV means SD every packet with that source address must have originated SD here. Uh, no... a spoofed packet from someone else's network means you had no control over it. That's pretty obvious. Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _ DO NOT send mail to the following addresses : [EMAIL PROTECTED] -or- [EMAIL PROTECTED] -or- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Mon, 8 Mar 2004, E.B. Dreger wrote: SD Date: Sun, 7 Mar 2004 16:17:50 -0500 (EST) SD From: Sean Donelan SD SAV doesn't tell you where the packets came from. At best SD SAV tells you where the packets didn't come from. If SAV were universal, source addresses could not be spoofed. If source addresses could not be spoofed... in a perfect world yes, for today we still have LOTS of folks that firewall in one direction only. A great example of this is the great firewall of China :( How, if they are filtering every packet that leaves their country, can I still get attacked from them? :( Until this is a default behaviour and you can't screw it up (ala directed-broadcast) this will be something we all have to deal with. SD Have you noticed this thread is full of people who don't run SD large networks saying other people who do run networks should SD deploy SAV/uRPF. 1. SAV is most effective at the edge, which often implies the smaller networks should be doing it excellent, the original point of the conversation has been satisfied... uRPF for the core is not a good plan, edge networks are a great place for this. Doing this on single homed customers is a great step in the right direction. However, as you say, the best place for this is on the edge of the network. So this implies that each edge LAN router will/should have uRPF or atleast an acl permitting only local LAN traffic to source from it, right? I have a question, I wonder if uRPF works on low end platforms without running CEF? Do all low-end platforms gracefully support CEF along with the other things enterprises typically do on routers? (just a question really...) 2. I've not seen large networks talking about their awful experiences with SAV. it melts routers, good enough for you? Specifically it melts linecards :( my experience is only on Cisco equipment though, so the linecard/ios/rev games must be played. If you upgrade, or initially install, E3 cards a large portion of this care is not necessary though. This is a problem that could be migrated out as new equipment/capabilities hit everyone's networks. I suspect that market pressure will push things in this direction anyway over time.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
CLM Date: Mon, 8 Mar 2004 01:32:51 + (GMT) CLM From: Christopher L. Morrow CLM in a perfect world yes[...] CLM Until this is a default behaviour and you can't screw it up CLM (ala directed-broadcast) this will be something we all have CLM to deal with. Yes. But the only way we'll get there is 1) a flag day or 2) if we gradually work in that direction. CLM it melts routers, good enough for you? Specifically it CLM melts linecards :( :-( CLM This is a problem that could be migrated out as new CLM equipment/capabilities hit everyone's networks. I suspect CLM that market pressure will push things in this direction CLM anyway over time. ...and hopefully will be safe-by-default. Anyone who has multihomed downstreams should be clued enough to disable strict SAV as needed -- similar to, yet the opposite of, manually configuring OSPF to treat interfaces as passive by default. As for low-end routers, uRPF is supported on 26xx. I don't know about a 16xx or 25xx... a scary thought, but chances are such a router would have a very small list of reachable netblocks to check. Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _ DO NOT send mail to the following addresses : [EMAIL PROTECTED] -or- [EMAIL PROTECTED] -or- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Mon, 8 Mar 2004, E.B. Dreger wrote: SD They saw no _net_ savings. SD SD In the real world, it costs more to deploy and maintain SD SAV/uRPF. The benefit is to other networks. When other networks make your life easier, you benefit. This confirms my statement. You save nothing by deploying SAV on your network. There may be some indeterminate benefit at some indeterminate time in the future after everyone else in the world correctly implements SAV. But there is no way to verify if every other network in the world has correctly deployed SAV. Even if everyone deploys SAV/uRPF you never know when someone may misconfigure something, so you still have to keep doing everything you were doing. In the mean time, you get to pay for the extra costs for deploying SAV/uRPF in addition to doing everything you were already doing. http://www.rhyolite.com/anti-spam/you-might-be.html If you want others to help you, help them. I've already done my part. I'm still waiting for others to help me. Should I be expecting a check in the mail?
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
Sean Donelan wrote: On Mon, 8 Mar 2004, E.B. Dreger wrote: SD They saw no _net_ savings. SD SD In the real world, it costs more to deploy and maintain SD SAV/uRPF. The benefit is to other networks. When other networks make your life easier, you benefit. This confirms my statement. How much do you save by putting handrails on your stairways? Restrooms in you lobby? Precipitators on your smoke stacks?
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sun, 7 Mar 2004, Sean Donelan wrote: This confirms my statement. You save nothing by deploying SAV on your network. This isnt the point. The point is, why should others suffer the burden of your clients spewing bogon/spoofed/nonsense garbage at them? The effect is cumulative. If everyone takes this lazy apathetic approach to network administration, it hurts everyone. Its the difference between being a good neighbor and being the fat beerbelly neighbor with dogs barking all night and rusting camaro with no tires up on cinderblocks on his beercan littered lawn. Just because everyone else doesnt maintain a good network doesnt mean you shouldnt. -Dan
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
SD Date: Sun, 7 Mar 2004 21:24:44 -0500 (EST) SD From: Sean Donelan SD This confirms my statement. You save nothing by deploying SD SAV on your network. There may be some indeterminate benefit Unless, of course, the traffic originated from your network and it simplifies your backtrace. Tracing flows isn't difficult, but it's more time consuming than a traceroute. SD at some indeterminate time in the future after everyone else SD in the world correctly implements SAV. But there is no way SD to verify if every other network in the world has correctly SD deployed SAV. Even if everyone deploys SAV/uRPF you never s/SAV/AS_PATH filtering and netblock adverts/ in your above statement. While technically true, it's highly disingenuous. Should providers quit filtering those simply because not everyone does it? It's extra cost with no selfish benefit, right? If you want a network to extend that courtesy to you, extend it to them. If you extend the courtesy to them, demand it in return. SD know when someone may misconfigure something, so you still SD have to keep doing everything you were doing. Perhaps on a lesser scale, though. There's benefit in knowing something did not originate from certain sources. SD In the mean time, you get to pay for the extra costs for SD deploying SAV/uRPF in addition to doing everything you were SD already doing. Just like AS_PATH and netblock announcement filters. Just like flow monitoring. Just like chasing down spammers. Just like dealing with pwned systems. Just like most anything else that wouldn't be necessary in a perfect world. Also note various posters' interest in shifting costs to responsible parties. One can argue what is reasonable, but consequences boost motivation. Perhaps if lack of certain precautions were considered [legally] negligent, failure would be the more expensive option. Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _ DO NOT send mail to the following addresses : [EMAIL PROTECTED] -or- [EMAIL PROTECTED] -or- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sun, Mar 07, 2004 at 09:24:44PM -0500, Sean Donelan wrote: On Mon, 8 Mar 2004, E.B. Dreger wrote: SD They saw no _net_ savings. SD SD In the real world, it costs more to deploy and maintain SD SAV/uRPF. [snip] In the real word, there are different networks with different tools and different gear. In some networks, it is a flip of the switch, you are done, and can move on. The direct benefit to my network is eliminating a category of crap from it. I save having to deal with that category. Yes there is other crap, but reducing the workload... reduces the workload. [snip] has correctly deployed SAV. Even if everyone deploys SAV/uRPF you never know when someone may misconfigure something, so you still have to keep doing everything you were doing. You mean internally to the network? Config management must exist for a huge number of reasons. Drop the right knob in your standards and move on. I don't follow 'having to keep doing everything' when I have one less things to do. In the mean time, you get to pay for the extra costs for deploying SAV/uRPF in addition to doing everything you were already doing. I'm sorry your network has such huge costs for trivial changes that follow simple logic.Actually, I've lost track of how many tiers of soapboxes are involved here, so I'm not sure what level of hypothetical-vs-real this [sub]thread is tackling. I'll encourage my competators to let more crap on their networks. I'll take out the trash at the points where I can. -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sun, Mar 07, 2004 at 09:24:44PM -0500, Sean Donelan wrote: If you want others to help you, help them. I've already done my part. I'm still waiting for others to help me. Should I be expecting a check in the mail? No. The work you've done is expected of you, as a good Internetwork neighbour. If you're not a good neighbour, next time you need my help, or the help of anyone else I know, please expect the finger. Yes, I suppose this does sound somewhat like a cross between an old-school network, and rule by bullying. But we don't have a better way (yet). -- Avleen Vig Systems Administrator Personal: www.silverwraith.com EFnet:irc.mindspring.com (Earthlink user access only)
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
[EMAIL PROTECTED] (vijay gill) writes: Putting rubber to the road eventually, we actually went ahead and packetfiltered rfc1918 space on our edge. I know paul and stephen will be crowing with joy here, as we had several arguments about it in previous lives, ... fwiw, in retrospect you were right at the time, but in my defense it was only because of things neither of us could have known. given only what we actually knew and could prove, you were deadass wrong :-). -- Paul Vixie
Re: Source address validation (was Re: UUNet Offer New Protection
[EMAIL PROTECTED] (Dan Hollis) writes: ... This isnt the point. The point is, why should others suffer the burden of your clients spewing bogon/spoofed/nonsense garbage at them? when i found out that two e-mail based service companies who had been acquired by yahoo had stopped doing verification of e-mail addresses as a result of that acquisition, i asked why. it turns out that yahoo's properties are designed for margin rather than for sustainability, and that the reason we all have to bear the burden of the resulting torrent of acquaintant-spam and unverified swill is simply that yahoo knows that most isp's are unwilling to boycott their e-mail. (though i do it here.) earlier that same week, i complained to amazon about lack of verification and heard this same story except this time it was from their abuse-bot. the only amazing thing about it is how bald-faced both companies are about shifting their costs onto the rest of the community. so i think the non-rhetorical answer to your perhaps-rhetorical question above is simply because they must (or because we can, as you please.) The effect is cumulative. If everyone takes this lazy apathetic approach to network administration, it hurts everyone. when we put these CEO's on a quarterly schedule for answering the question, what have you done for us lately? this result was pretty much preordained. Its the difference between being a good neighbor and being the fat beerbelly neighbor with dogs barking all night and rusting camaro with no tires up on cinderblocks on his beercan littered lawn. just for reference, the short form of the latter is chickenboner (because of the pile of same found under their airstream's kitchen window.) Just because everyone else doesnt maintain a good network doesnt mean you shouldnt. yea, verily. -- Paul Vixie
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sun, 7 Mar 2004, Avleen Vig wrote: No. The work you've done is expected of you, as a good Internetwork neighbour. If you're not a good neighbour, next time you need my help, or the help of anyone else I know, please expect the finger. But I keep trying to do good work; and you keep giving me the finger. Why should I keep trying to do good work? Remember it works both ways.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
Sean Donelan wrote: On Sun, 7 Mar 2004, E.B. Dreger wrote: SAV doesn't take long to implement. Considering the time spent discounting spoofing when responding to incidents, I think there would be a _net_ savings (no pun intended) in time spent responding to incidents. You would be wrong. There are networks that have deployed SAV/uRPF. They saw no _net_ savings. In the real world, it costs more to deploy and maintain SAV/uRPF. Have you noticed this thread is full of people who don't run large networks saying other people who do run networks should deploy SAV/uRPF. But there hasn't been anyone who does run large networks saying they deployed SAV/uRPF and it saved them money, made their network run better or improved the world? Where do you draw the line between large and not large? Does a university with a /16 count as large? We do both SAV and a version of uRPF. It makes our network run better, saves us money (reduces the amount of time we spend on support and makes troubled/distressed/evil/mean/nasty boxes easier to track down) and reduces backbone congestion making the network run better. Another benefit is it improves the world (betcha' were wondering if I'd squeeze all that in). We're now blocking all SMTP traffic leaving the campus from non-blessed sources (read mail servers). The first day doing this we had comments about less junk mail traffic. We block traffic we consider harmful that shouldn't leave the campus. We're trying to do our part. Any suggestions how we can do better? Ken
Re: Source address validation (was Re: UUNet Offer New Protection
[EMAIL PROTECTED] (Sean Donelan) writes: If you're not a good neighbour, next time you need my help, or the help of anyone else I know, please expect the finger. But I keep trying to do good work; and you keep giving me the finger. Why should I keep trying to do good work? Remember it works both ways. i guess i don't understand your comments. you have argued for relaxation of vigilance in the practice of uRPF, on the basis of high cost and assymetric benefit. you have attempted to justify your employer's lack of effort in this area on the basis that the benefit is not only assymetric but also negligible. this sure sounds like a copout. did you actually do something good but you aren't allowed to say so in public? -- Paul Vixie
Re: Source address validation (was Re: UUNet Offer New Protection
[EMAIL PROTECTED] (Ken Diliberto) writes: Where do you draw the line between large and not large? Does a university with a /16 count as large? We do both SAV and a version of uRPF. It makes our network run better, saves us money (reduces the amount of time we spend on support and makes troubled / distressed / evil / mean / nasty boxes easier to track down) and reduces backbone congestion making the network run better. Another benefit is it improves the world (betcha' were wondering if I'd squeeze all that in). We're now blocking all SMTP traffic leaving the campus from non-blessed sources (read mail servers). The first day doing this we had comments about less junk mail traffic. We block traffic we consider harmful that shouldn't leave the campus. We're trying to do our part. Any suggestions how we can do better? yes. contact the nanog program committee so you can come to san francisco and tell the rest of us how you did it -- both in the ones and zeros, and in the dollars and cents. -- Paul Vixie
Re: UUNet Offer New Protection Against DDoS
Christopher L. Morrow wrote: miniscule amounts of traffic in uunet's core is still enough to ddos many a victim into oblivion. anyone who has been ddos'd by uunet customers can appreciate that. miniscule is enough to cause problems in anyone's network the point here was: Core isn't the right place for this I wasn't really trying to argue the 'urpf is good' or 'urpf is bad' arguement, just the placement. Sorry if I made that confusing earlier. So we all agree that in the ideal world, everyone has anti-spoofing ACLs and route map filters and what not on every link into their network. But in the real world...given that you are going to be peering with ISPs (or their upstreams) that do not do uRPF or anything at all on their edges, if you want to drop the patently bogus traffic, or your customers don't want to pay you for delivering it to them over links they don't want congested with it, what do you do? I guess you can say peering links are not core, and that's fine if you run loose-uRPF there, and can be assured that all access to your network has filters on all links.I was thinking of large peering routers as part of the core of an ISP, so loose-uRPF is sufficient on those routers, if edges are protected. But if you are going to run loose-uRPF on your peering routers, why not run it on your core? Is there a technogical reason not to? Cisco OC48 line cards not support it (at least some do.), I'm almost sure Juniper does too. But I don't play in that area. And given that there are ISP's running it in the core; that it will block some malicious traffic; and spoofed traffic may well be used as an attack vector again (sometime people are going to have to catch on and patch machines, or worms will patch them for them, and reduce the botnet farm size. Maybe not this year, but sometime...), I still don't see why you are against it. I accept that filtering on all edges, including peering, is a better place to do it. So do you filter on, say, peering links to other tier 1's? Even so, why not have belt AND suspender, and run it in the core?
Re: UUNet Offer New Protection Against DDoS
[EMAIL PROTECTED] (Steve Francis) writes: ... But in the real world...given that you are going to be peering with ISPs (or their upstreams) that do not do uRPF or anything at all on their edges, ... ok, i'll bite. why do we still do this? see the following from june 2001: http://www.cctec.com/maillists/nanog/historical/0106/msg00681.html (and according to that text, it was a 9-year-old idea at that time.) it's now 2004. how much longer do we want to have this problem? -- Paul Vixie
Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sat, 6 Mar 2004, Paul Vixie wrote: (and according to that text, it was a 9-year-old idea at that time.) it's now 2004. how much longer do we want to have this problem? Source address validation (or Cisco's term uRPF) is perhaps more widely deployed than people realize. Its not 100%, but what's interesting is despite its use, it appears to have had very little impact on DDOS or lots of other bad things. Root and other DNS servers bear the brunt of misconfigured (not necessarily malicious attack) devices. So some people's point of view may be different. But relatively few DDOS attacks use spoofed packets. If more did, they would be easier to deal with. After all these years, perhaps its time to re-examine the assumptions.
Re: UUNet Offer New Protection Against DDoS
--On 06 March 2004 23:02 + Paul Vixie [EMAIL PROTECTED] wrote: ok, i'll bite. why do we still do this? see the following from June 2001: http://www.cctec.com/maillists/nanog/historical/0106/msg00681.html Having had almost exactly that phrase in my peering contracts for $n years, the answer is because if you are A, and peer is B, if ( AB ) your spoofed traffic comes (statistically) from elsewhere so you don't notice. You are dealing with traffic from C, where CA else you've signed their peering agreement, and are 'peering' on their terms instead. Was I going to pull peering with $tier1 from whom the occasional DoS came? Nope. The only way this was ever going to work was if the largest networks cascaded the requirements down to the smallest. And the largest networks were the ones for whom (quite understandably) rpf was most difficult. DoS (read unpaid for, unwanted traffic) is one of the best arguments against settlement-free peering (FX: ducks runs). Alex
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
--On 06 March 2004 18:39 -0500 Sean Donelan [EMAIL PROTECTED] wrote: Source address validation (or Cisco's term uRPF) is perhaps more widely deployed than people realize. Its not 100%, but what's interesting is despite its use, it appears to have had very little impact on DDOS or lots of other bad things. ... But relatively few DDOS attacks use spoofed packets. If more did, they would be easier to deal with. AIUI that's cause effect: the gradual implementation of source-address validation has made attacks dependent on spoofing less attractive to perpetrators. Whereas the available of large pools of zombie machines has made the use of source spoofing unnecessary. Cisco et al have shut one door, but another one (some suggest labeled Microsoft) has opened. Those with long memories might draw parallels with the evolution of phreaking from abuse of the core, which became (reasonably) protected to abuse of unprotected PABXen. As I think I said only a couple of days ago, there is nothing new in the world. Alex
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
After all these years, perhaps its time to re-examine the assumptions. it's always fun and useful to re-example assumptions. for example, anyone who assumes that because the attacks they happen to see, or the attacks they hear about lately, don't use spoofed source addresses -- that spoofing is no longer a problem, needs to re-examine that assumption. for one thing, spoofed sources could be occurring outside local viewing. for another thing, spoofed sources could be plan B when other attacks aren't effective. the last thing is, this is war. information warfare. the enemy knows us better than we know them, and their cost of failure is drastically lower than our cost of failure. don't be lulled into some kind of false sense of security by the fact that YOU are not seeing spoofed packets TODAY. let's close the doors we CAN close, and give attackers fewer options.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sun, 7 Mar 2004, Paul Vixie wrote: don't be lulled into some kind of false sense of security by the fact that YOU are not seeing spoofed packets TODAY. let's close the doors we CAN close, and give attackers fewer options. sadly the prevailing thought seems to be 'we cant block every exploit so we will block none'. this (and others) are used as an excuse to not deploy urpf on edge interfaces facing singlehomed customers. its a fatalistic approach to dealing with network abuse, and its retarded. -Dan
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sun, 7 Mar 2004, Paul Vixie wrote: don't be lulled into some kind of false sense of security by the fact that YOU are not seeing spoofed packets TODAY. let's close the doors we CAN close, and give attackers fewer options. I don't have a false sense of security. We have lots of open doors and windows and even missing walls. Let's close the doors we can close, but buying screen doors for igloos may not be the best use of resources. uRPF doesn't actually prevent any attacks. Would you rather ISPs spend money to 1. Deploying S-BGP? 2. Deploying uRPF? 3. Respond to incident reports?
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
Sean Donelan wrote: Would you rather ISPs spend money to 1. Deploying S-BGP? 2. Deploying uRPF? 3. Respond to incident reports? Why are we limited to that set?
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sat, 6 Mar 2004, Dan Hollis wrote: sadly the prevailing thought seems to be 'we cant block every exploit so we will block none'. this (and others) are used as an excuse to not deploy urpf on edge interfaces facing singlehomed customers. This is one of the few locations SAV/uRPF consistently works. SAV/uRPF is widely (but not 100%) deployed int those location. However I think you are mis-stating the issue. I do not know of anyone that has stated your reason as the reason not to deploy SAV/uRPF on non-routing interfaces. The issue which prompt this thread was deploying uRPF on multi-path backbone interfaces using active routing. How many exploits does uRPF block? Biometric smart cards may do wonders for credit card fraud. Why don't credit card companies replace all existing cards with them? Does uRPF solve more problems than it causes, and saves more than it costs?
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
... buying screen doors for igloos may not be the best use of resources. uRPF doesn't actually prevent any attacks. actually, it would. universal uRPF would stop some attacks, and it would remove a plan B option for some attack-flowcharts. i would *much* rather play defense without facing this latent weapon available to the offense. Would you rather ISPs spend money to 1. Deploying S-BGP? 2. Deploying uRPF? 3. Respond to incident reports? yes. and i can remember being sick and tired of competing (on price, no less) against providers who couldn't/wouldn't do #2 or #3. i'm out of the isp business at the moment, but the race to the bottom mentality is still a pain in my hindquarters, both present and remembered.
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sat, Mar 06, 2004 at 06:39:21PM -0500, Sean Donelan wrote: Source address validation (or Cisco's term uRPF) is perhaps more widely deployed than people realize. Its not 100%, but what's interesting is despite its use, it appears to have had very little impact on DDOS or lots of other bad things. Try saying that after running a major DDoS target, with HIT ME your forehead. No offense Sean but I'd like you to back your claim up with some impirical data first. From experience the majority of TCP based denial of service attacks (which usually seem to be balanced with UDP, but ICMP is not as frequent as it once was), use spoofed sources. -- Avleen Vig Systems Administrator Personal: www.silverwraith.com
Re: Source address validation (was Re: UUNet Offer New Protection Against DDoS)
On Sat, 6 Mar 2004, Avleen Vig wrote: On Sat, Mar 06, 2004 at 06:39:21PM -0500, Sean Donelan wrote: Source address validation (or Cisco's term uRPF) is perhaps more widely deployed than people realize. Its not 100%, but what's interesting is despite its use, it appears to have had very little impact on DDOS or lots of other bad things. Try saying that after running a major DDoS target, with HIT ME your forehead. No offense Sean but I'd like you to back your claim up with some impirical data first. Has the number of DDOS attacks increased or decreased in the last few years has uRPF has become more widely deployed? Do you have any evidence the number of attacks are decreasing?
Re: Source address validation (was Re: UUNet Offer New Protection
[EMAIL PROTECTED] (Sean Donelan) writes: How many exploits does uRPF block? that's hard to measure since we end up not receiving those. but one can assume that spoofed-source attacks aren't tried, either because (1) it's easier to just use a high number of windows-xp drones, or because of (2) uRPF deployment. Does uRPF solve more problems than it causes, and saves more than it costs? until you know what percentage of the attacks you don't see is due to (1) vs (2) above, you can't really pose that question meaningfully. anytime there's a way to protect against a whole class of attack weapons, we have to deploy it. this is war, information warfare. let's deprive the enemy of options until we can force them to meet us on our own chosen terms. -- Paul Vixie
Re: Source address validation (was Re: UUNet Offer New Protection
[EMAIL PROTECTED] (Sean Donelan) writes: Try saying that after running a major DDoS target, with HIT ME your forehead. No offense Sean but I'd like you to back your claim up with some impirical data first. Has the number of DDOS attacks increased or decreased in the last few years has uRPF has become more widely deployed? the number of spoofed-source attacks is down only-slightly. Do you have any evidence the number of attacks are decreasing? the overall number of attacks and their volume seems to be decreasing ever-so-slightly, but the ferocity of the attacks that come through seems to be increasing more-than-slightly. and, when defending against one of these, every valid source address is worth its figurative weight in gold, and constitutes a minor compromise for the attacker, even if the host it helps to identify is disposable, easily replaced, and difficult to repair. [ of course, sean, i could just be making that part up. but since i keep saying it and since i get attacked pretty frequently, i might be telling the truth. it could be worth assuming a little credibility and seeing where that leads you. (but, we digress.) ] -- Paul Vixie
Re: Source address validation (was Re: UUNet Offer New Protection
On 7 Mar 2004, Paul Vixie wrote: [EMAIL PROTECTED] (Sean Donelan) writes: Try saying that after running a major DDoS target, with HIT ME your forehead. No offense Sean but I'd like you to back your claim up with some impirical data first. Has the number of DDOS attacks increased or decreased in the last few years has uRPF has become more widely deployed? the number of spoofed-source attacks is down only-slightly. the % of spoofed and bogon traffic was measured recently at several of the root nameservers. iirc it was suprisingly high. -Dan
Re: UUNet Offer New Protection Against DDoS
Terranson, Alif wrote: As long as we're doing Me Too... Savvis has had prefix:666 for around 18 months as well. Do you know if CW does? Or will that wait until the integration? This thread has caused me to add this as a requirement for a new gigabit ISP circuit I am ordering, as well as uRPF in the core, etc. I've had two ISPs say We don't do this yet, but based on the fact you are making it a requirement, we will role those functions out into our core. Steve Voting with his money for better net-security Alif Terranson OpSec Engineering Manager Operations Security Department Savvis Communications Corporation (314) 628-7602 Voice (314) 208-2306 Pager (618) 558-5854 Cell -Original Message- From: Michael Hallgren [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 3:45 PM To: [EMAIL PROTECTED] Subject: RE: UUNet Offer New Protection Against DDoS Global Crossing has this, already in production. Idem, Teleglobe, mh I was on the phone with Qwest yesterday this was one of this things I asked about. Qwest indicated they are going to deploy this shortly. (i.e., send routes tagged with a community which they will set to null) James Edwards Routing and Security [EMAIL PROTECTED] At the Santa Fe Office: Internet at Cyber Mesa Store hours: 9-6 Monday through Friday 505-988-9200 SIP:1(747)669-1965
Re: UUNet Offer New Protection Against DDoS
On Fri, 5 Mar 2004, Steve Francis wrote: Terranson, Alif wrote: As long as we're doing Me Too... Savvis has had prefix:666 for around 18 months as well. Do you know if CW does? Or will that wait until the integration? This thread has caused me to add this as a requirement for a new gigabit ISP circuit I am ordering, as well as uRPF in the core, etc. uRPF in the core seems like a bad plan, what with diverse routes and such. Loose-mode might help SOME, but really spoofing is such a low priority issue why make it a requirement? Customer triggered blackholing is a nice feature though. --Chris (formerly [EMAIL PROTECTED]) ### ## UUNET Technologies, Inc. ## ## Manager ## ## Customer Router Security Engineering Team ## ## (W)703-886-3823 (C)703-338-7319 ## ###
RE: UUNet Offer New Protection Against DDoS
snip uRPF in the core seems like a bad plan, what with diverse routes and such. Loose-mode might help SOME, but really spoofing is such a low priority issue why make it a requirement? Customer triggered blackholing is a nice feature though. /snip Shared view, mh (Teleglobe, btw) --Chris (formerly [EMAIL PROTECTED]) ### ## UUNET Technologies, Inc. ## ## Manager ## ## Customer Router Security Engineering Team ## ## (W)703-886-3823 (C)703-338-7319 ## ###
Re: UUNet Offer New Protection Against DDoS
Christopher L. Morrow wrote: uRPF in the core seems like a bad plan, what with diverse routes and such. Loose-mode might help SOME, but really spoofing is such a low priority issue why make it a requirement? Customer triggered blackholing is a nice feature though. Obviously loose-mode. Spoofing may not be the current weapon of choice, but why not encourage the best net infrastructure?
RE: UUNet Offer New Protection Against DDoS
Terranson, Alif wrote: As long as we're doing Me Too... Savvis has had prefix:666 for around 18 months as well. Do you know if CW does? Or will that wait until the integration? While I am not 100% certain (and there are plenty of new-Savvis folks here who *do* know for sure ;-), I believe the CW network does support a BH tag. This thread has caused me to add this as a requirement for a new gigabit ISP circuit I am ordering, as well as uRPF in the core, Woah! Never said *anything* about that! No plans for it that I am aware of. No reason I can think of to do this either. etc. I've had two ISPs say We don't do this yet, but based on the fact you are making it a requirement, we will role those functions out into our core. This is really not new, and considering how easy it is to implement, I'm surprised it isn't *much* more widely implemented. Steve Voting with his money for better net-security Go Steve! Go!! Alif Terranson OpSec Engineering Mgr. Operations Security Dept. Savvis Communications Corp. (314) 628-7602 Voice (618) 558-5854 Cell (314) 628-7710 Fax
Re: UUNet Offer New Protection Against DDoS
On Fri, 5 Mar 2004, Steve Francis wrote: Christopher L. Morrow wrote: uRPF in the core seems like a bad plan, what with diverse routes and such. Loose-mode might help SOME, but really spoofing is such a low priority issue why make it a requirement? Customer triggered blackholing is a nice feature though. Obviously loose-mode. Spoofing may not be the current weapon of choice, but why not encourage the best net infrastructure? Loose mode will not save you very much, many larger backbones route lots of 'unused' or 'unallocated' ip space internally for various valid reasons, some even related to security issues for their customers. So, does stopping rfc-1918 (maybe) space help much? not really... atleast not that I can see. Many flooding tools now flood from legittimate space, so the ONLY way to limit this is by filtering as close to the device sourcing the packets as possible. Nebulous filtering and dropping of miniscule amounts of traffic in the core of a large network is just a waste of effort and false panacea. --Chris (formerly [EMAIL PROTECTED]) ### ## UUNET Technologies, Inc. ## ## Manager ## ## Customer Router Security Engineering Team ## ## (W)703-886-3823 (C)703-338-7319 ## ###
Re: UUNet Offer New Protection Against DDoS
On Fri, 5 Mar 2004, Christopher L. Morrow wrote: the packets as possible. Nebulous filtering and dropping of miniscule amounts of traffic in the core of a large network is just a waste of effort and false panacea. uunet does operate lots of dialup RAS though correct? any reason why urpf is not reasonable there? just because its not perfect and doesnt solve every problem doesnt mean its useless. miniscule amounts of traffic in uunet's core is still enough to ddos many a victim into oblivion. anyone who has been ddos'd by uunet customers can appreciate that. -Dan
RE: UUNet Offer New Protection Against DDoS
On Fri, 5 Mar 2004, Christopher L. Morrow wrote: the packets as possible. Nebulous filtering and dropping of miniscule amounts of traffic in the core of a large network is just a waste of effort and false panacea. Agreed. uunet does operate lots of dialup RAS though correct? any reason why urpf is not reasonable there? Nobody I know terminates a dial connection on a *core router* ;-) //Alif Alif Terranson OpSec Engineering Mgr. Operations Security Dept. Savvis Communications Corp. (314) 628-7602 Voice (618) 558-5854 Cell (314) 628-7710 Fax
