different flavours of uRPF [RE: register.com down sev0?]

[Pekka Savola]

On Thu, 26 Oct 2006, Tony Li wrote:
> > It was possible to implement BCP38 before the router vendors 
> > came up with uRPF.
> 
> Further, uRPF is frequently a very inefficient means of implementing BCP
> 38.  Consider that you're going to either compare the source address
> against a table of 200,000 routes or against a handful of prefixes that
> you've statically configured in an ACL.

Isn't that only a problem if you want to run a loose mode uRPF?  
Given that loose mode uRPF isn't very useful in most places where 
you'd like to do ingress filtering, this doesn't seem like a big 
issue..

BTW, I still keep wondering why Cisco hasn't implemented something 
like Juniper's feasible-path strict uRPF.  Works quite well with 
multihomed and asymmetric routing as well -- no need to fiddle with 
communities, BGP weights etc. to ensure symmetry.

-- 
Pekka Savola "You each name yourselves king, yet the
Netcore Oykingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

RE: register.com down sev0?

[Tony Li]

 

> It was possible to implement BCP38 before the router vendors 
> came up with uRPF.


Further, uRPF is frequently a very inefficient means of implementing BCP
38.  Consider that you're going to either compare the source address
against a table of 200,000 routes or against a handful of prefixes that
you've statically configured in an ACL.

Yes, I realize that the latter approach is more of a managerial hassle,
but for those of you who feel that your silicon is running a tad too
warm, you may wish to consider this as a possible performance
improvement technique.  YMMV.

Your former router vendor,
Tony

Re: [Fwd: Re: DNS DDoS [was: register.com down sev0?]]

[Hank Nussbacher]

On Thu, 26 Oct 2006, virendra rode // wrote:


Just curious, any ddos vendors want to share their success stories :-)


If you access Cisco as a customer:

http://www.cisco.com/en/US/customer/products/ps5887/products_case_study0900aecd80120478.shtml

"Rackspace Managed Hosting" - Customer Success Story

-Hank Nussbacher
http://www.interall.co.il

Re: register.com down sev0?

[Gadi Evron]

On Thu, 26 Oct 2006, Chris L. Morrow wrote:
> 
> On Wed, 25 Oct 2006, Randy Bush wrote:
> > > I don't want to detract from the heat of this discussion, as
> > > important as it is, but it (the discussion) illustrates a point
> > > that RIPE has recognized -- and is actively perusing -- yet, ISPs
> > > on this continent seem consistently to ignore: The consistent
> > > implementation of BCP 38.
> >
> > oh?  you have knowledge that this botnet attack used spoofed source
> > addresses?
> 
> what's curious, to me atleat, is that folks equate 'botnet' and 'spoofed
> source attacks' more often than I'd think is reasonable. I've not got
> 'hard numbers' but almost every time the attack is determined to be
> 'botnet' it's not spoofed.
> 
> Odd... (not that I'm against bcp38, I just think the distraction in
> conversation from 'bcp38 is good' to 'we must stop bots' is not helpful)
> 

SAT time.

Almost all spoofed attacks are run by botnets.
Almost all attacks are run by botnets
Almost all spoofed attacked are bigger by a large factor

Almost all botnet attacks are spoofed attacks? Not quite.

That's about it.

Re: Extreme Slowness

[Adam Rothschild]

Elijah,

On 2006-10-26-16:34:18, Elijah Savage <[EMAIL PROTECTED]> wrote:
[HTML mail stripped]
> It seems anything traversing level3 has very high latency along with
> what seems overloaded capacity as if they are running in a degraded
> mode I have connections with Time Warner, AT&T, and MCI [...]

On 2006-10-26-16:48:15, Elijah Savage <[EMAIL PROTECTED]> wrote:
[HTML mail stripped]
> Say like this traceroute. This is from TW to a Broadwing DS3.
> 
> 5  tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13)  153.267 ms   
> 207.125 ms
> tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9)  218.920 ms
> 6  ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206)  36.976 ms  26.923  
> ms  57.770 ms
> 7  ge-11-0.core2.Chicago1.Level3.net (4.68.101.37)  254.145 ms
> ge-11-1.core2.Chicago1.Level3.net (4.68.101.101)  258.522 ms
> ge-11-2.core2.Chicago1.Level3.net (4.68.101.165)  227.223 ms
> 8  broadwing-level3-oc12.Chicago1.Level3.net (209.0.225.10)  231.451 ms
> 9  so-1-1-0.c1.gnwd.broadwing.net (216.140.15.1)  53.269 ms  35.568  
> ms  22.511 ms

Your postings appear to be missing two key pieces of information which
would help with the community diagnosis requested: source and
destination IP addresses.  From the information you did provide, one
can deduce that you're behind a TW/RoadRunner cable modem:

  13.216.78.4.IN-ADDR.ARPA domain name pointer 
tenge-3-2.car1.Cincinnati1.Level3.net
  14.216.78.4.IN-ADDR.ARPA domain name pointer 
ROADRUNNER.car1.Cincinnati1.Level3.net
  9.216.78.4.IN-ADDR.ARPA domain name pointer 
tenge-3-1.car1.Cincinnati1.Level3.net
  10.216.78.4.IN-ADDR.ARPA domain name pointer 
ROADRUNNER.car1.Cincinnati1.Level3.net

Now, the jitter and high latency you're seeing could be a result of
one or more factors, including but not limited to RF/plant issues, TWC
running their transport and/or Level(3) transit hot (which seems to be
a common occurrence these days), ECMP across two circuits of uneven
loading, or your neighbor might be jacking wifi and downloading a
bunch of torrents -- we, the readers, just don't know.

Of note when performing armchair troubleshooting across Level(3)'s
network: the 'ebr's (PTR record of ebr*.{pop}.level3.net == Force10
E1200; Experimental Backbone Router?) tend to drop a lot of diagnostic
traffic (such as, say, 'ping' and 'traceroute') as a part of overly
aggressive control-plane policers.  This loss is, of course, strictly
cosmetic, and has no bearing on end-to-end performance.  Hence, the
old "to it, not through it" rule applies.

smokeping[1] and iperf[2] (to end hosts) are your friends.

As an aside, I've noticed your string of postings today were all
HTML-tagged.  While not expressly forbidden (or even discouraged) by
the current Mailing List AUP, this is generally regarded as bad form;
you might wish to reconfigure your mail client accordingly...

Hope this helps,
-a

[1] 
[2] 

Re: 10,352 active botnets (was Re: register.com down sev0?)

[Sean Donelan]

On Thu, 26 Oct 2006, Gadi Evron wrote:

Jose may be a bit conservative with numbers, but he has good data and
shares it, which is more than I can say for some people.


http://www.asu.edu/security/aware/2005/lippard.htm

Re: ICMP & PathMTU (was: Re: Extreme Slowness)

[Randy Bush]

> 1) What value is ICMP if everybody pretty much considers it's accuracy
>suspect?

because for some uses, narrow precision is not needed.  like is it
pingable?  what is the current path?

my eyes are not highly accurate at measuring distance, color, size,
motion, ...  accurately.  but i'll keep them, thanks.

> 2) How does ICMP's suspect nature affect Path MTU?

pmtu is hosed for other sicker reasons

randy

---

on precision
guy is at mummy exhibit in british museum
asks guard how old mummy is
guard says 2007 years
guy asks how he knows 2007
guard replies that he's been here seven years and mummy was 2000
years old when he got here

ICMP & PathMTU (was: Re: Extreme Slowness)

[Jim Popovitch]

On Thu, 2006-10-26 at 18:01 -0400, Elijah Savage wrote:
> For FYI :) I realize that ICMP is not the best way to test and it is
> not a true indication of slowness or the presence of a problem.

Two questions for everybody...(any and all responses appreciated, even
if the reply mentions botnets or hammers ;-) )

1) What value is ICMP if everybody pretty much considers it's accuracy
suspect?

2) How does ICMP's suspect nature affect Path MTU?



-Jim P.

Re: passports for NANOG-39, Toronto

[Steven M. Bellovin]

On Thu, 26 Oct 2006 10:19:18 -0400, Joe Abley <[EMAIL PROTECTED]>
wrote:

> 
> 
> On 26-Oct-2006, at 09:26, [EMAIL PROTECTED] wrote:
> 
> > You could do the same fly-drive via Detroit but there is
> > a lot more driving.
> 
> Indeed. Rough estimates, excluding time taken to cross the border and  
> assuming good weather:
> 
>BUF to Toronto: 2 hours
>DTW to Toronto: 5 hours
>CLE to Toronto: 6 hours
>LGA to Toronto: 9 hours
>BOS to Toronto: 9 hours
>ORD to Toronto: 10 hours
>IAD to Toronto: 10 hours
> 
Don't neglect the border crossing delay.  Driving home from Montreal after
the IETF, we had to wait close to two hours because of congestion at U.S.
Immigration.  (Of course, that was the way home -- folks going into Canada
had virtually no wait, as best we could see...)


--Steven M. Bellovin, http://www.cs.columbia.edu/~smb

Re: Extreme Slowness

[Jeremy Chadwick]

On Thu, Oct 26, 2006 at 06:01:43PM -0400, Elijah Savage wrote:
> For FYI :) I realize that ICMP is not the best way to test and it is  
> not a true indication of slowness or the presence of a problem.

Which begs the same question I've asked in the recent past: then
what *is* a good diagnostic tool?  If ICMP "is not the best way to
test", then what is?  What other globally-implemented layer 3 or
below protocols do we have available for troubleshooting?

Sure, UDP-based traceroute still relies on ICMP TTL exceeded
responses to work.  I've no idea what TCP traceroute relies on,
as I haven't looked at it.

-- 
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networkinghttp://www.parodius.com/ |
| UNIX Systems Administrator   Mountain View, CA, USA |
| Making life hard for others since 1977.   PGP: 4BD6C0CB |

Re: Extreme Slowness

[Elijah Savage]

Yes sir I did. This is now resolved. But thank you for noticing.  On Oct 26, 2006, at 7:11 PM, Aaron Glenn wrote:On 10/26/06, Elijah Savage <[EMAIL PROTECTED]> wrote: Say like this traceroute. This is from TW to a Broadwing DS3.5  tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13)153.267 ms  207.125 ms    tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9)218.920 ms 6  ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206)  36.976 ms  26.923 ms57.770 ms 7  ge-11-0.core2.Chicago1.Level3.net (4.68.101.37)254.145 ms    ge-11-1.core2.Chicago1.Level3.net (4.68.101.101)258.522 ms    ge-11-2.core2.Chicago1.Level3.net (4.68.101.165)227.223 ms 8  broadwing-level3-oc12.Chicago1.Level3.net(209.0.225.10)  231.451 ms 9  so-1-1-0.c1.gnwd.broadwing.net (216.140.15.1)  53.269 ms  35.568 ms22.511 ms10  216.140.14.17 (216.140.14.17)  34.751 ms  39.008 ms  46.644 ms11  p5-0-0.e0.cncn.broadwing.net (216.140.15.78)  32.065 ms  60.797 ms54.766 ms12  67.98.17.122 (67.98.17.122)  44.772 ms  27.631 ms  30.655 ms13  * * * Uhh, you do realize the end to end latency there (to hop 12, at least)is ~30ms...not the 250ms+ you see on intermediate hops, right? 

Re: Extreme Slowness

[Aaron Glenn]

On 10/26/06, Elijah Savage <[EMAIL PROTECTED]> wrote:

Say like this traceroute. This is from TW to a Broadwing DS3.

5  tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13)
153.267 ms  207.125 ms
tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9)
218.920 ms
 6  ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206)  36.976 ms  26.923 ms
57.770 ms
 7  ge-11-0.core2.Chicago1.Level3.net (4.68.101.37)
254.145 ms
ge-11-1.core2.Chicago1.Level3.net (4.68.101.101)
258.522 ms
ge-11-2.core2.Chicago1.Level3.net (4.68.101.165)
227.223 ms
 8  broadwing-level3-oc12.Chicago1.Level3.net
(209.0.225.10)  231.451 ms
 9  so-1-1-0.c1.gnwd.broadwing.net (216.140.15.1)  53.269 ms  35.568 ms
22.511 ms
10  216.140.14.17 (216.140.14.17)  34.751 ms  39.008 ms  46.644 ms
11  p5-0-0.e0.cncn.broadwing.net (216.140.15.78)  32.065 ms  60.797 ms
54.766 ms
12  67.98.17.122 (67.98.17.122)  44.772 ms  27.631 ms  30.655 ms
13  * * *


Uhh, you do realize the end to end latency there (to hop 12, at least)
is ~30ms...not the 250ms+ you see on intermediate hops, right?

Re: register.com down sev0? - More information

[Charles Gucker]

5. AT&T (at least when I've dealt with them in their datacenters) does not
support BGP community strings for null routing (or any strings for that
matter :) Think about that for a second. To stop an attack Register.com
would need to call AT&T and request a filter/null route. Since AT&T
operations is based in Singapore (again this was last time I dealt with
them) I'm sure getting those filters/routes in probably doesn't happen
nearly fast enough. I have heard that AT&T is currently in the process of
setting up communities- maybe someone who knows more could comment.


Well, this is not exactly true.AT&T does support BGP communities,
although their communities aren't all that powerful, IMO.   To my
knowledge, you are correct when you say that they do not support any
null-routing capabilities.   I would love to find out the procedure
and string required to request/implement null routing via a community.

For those who would like to see AT&T's official guide, it can be found at:
http://www.onesc.net/communities/as7018

charles

Re: register.com down sev0?

[Chris L. Morrow]

On Thu, 26 Oct 2006, Fergie wrote:

> Chris,
>
> W.R.T. #2 below:
>
> Be for real: No one ever suggested that backbone service
> providers attempt to ingress filter traffic -- this is an
> edge function.

ah, cause I thought 'everyone should do bcp38' mean 'everyone'... I agree
that it's a great thing, I think 'everyone' should do it, I even thing we
should where possible. I think LOTS of this would go away if people
filtered their lan segments... tey have the horsey's there to do it
without the compromises that must be taken at the 'core' (or 'more central
portions of 'the net')

And I was sorta yanking your chain some :)

>
> Cheers,
>
> - ferg
>
> -- "Chris L. Morrow" <[EMAIL PROTECTED]> wrote:
>
> On Thu, 26 Oct 2006, Fergie wrote:
> > and co-authored -- and likewise, cannot figure out for life of
> > me, why there is such push-back from the Ops community on doing
> > The Right Thing.
>
> you could google answers from other folks but in shor:
> 1) it doesn't always work as advertised
> 2) people don't always tell you the routes the hold
> 3) equipment vendors don't alway splan properly for 'features'
>
> Not everyone is as smart as you (both) and can manage that problem as they
> scale...
>
>
> --
> "Fergie", a.k.a. Paul Ferguson
>  Engineering Architecture for the Internet
>  fergdawg(at)netzero.net
>  ferg's tech blog: http://fergdawg.blogspot.com/
>

Re: register.com down sev0?

[Daniel Senie]

At 05:26 PM 10/26/2006, Fergie wrote:


Chris,

W.R.T. #2 below:

Be for real: No one ever suggested that backbone service
providers attempt to ingress filter traffic -- this is an
edge function.


I guess I'd add some clarification, though it should be obvious without.

Backbone service providers who also sell edge circuits (e.g. 
dedicated T-1's to non-multihomed customers) ARE providing the edge 
function. A provider who claims "we're a backbone, so we should do no 
ingress filtering at all" is being disingenuous, at least for many of 
the largest networks today. I'm not accusing anyone of actually 
making such statements at all. I agree with Paul that this is an edge 
function, but that "edge" is a part of nearly every provider at some 
point in their businesses.





-- "Chris L. Morrow" <[EMAIL PROTECTED]> wrote:

On Thu, 26 Oct 2006, Fergie wrote:
> and co-authored -- and likewise, cannot figure out for life of
> me, why there is such push-back from the Ops community on doing
> The Right Thing.

you could google answers from other folks but in shor:
1) it doesn't always work as advertised
2) people don't always tell you the routes the hold
3) equipment vendors don't alway splan properly for 'features'

Not everyone is as smart as you (both) and can manage that problem as they
scale...


--
"Fergie", a.k.a. Paul Ferguson
 Engineering Architecture for the Internet
 fergdawg(at)netzero.net
 ferg's tech blog: http://fergdawg.blogspot.com/

Re: Extreme Slowness

[Elijah Savage]

Seems to be all cleared up now. I had a couple of my customers even try to pull up their home site and could not get to it. For FYI :) I realize that ICMP is not the best way to test and it is not a true indication of slowness or the presence of a problem. On Oct 26, 2006, at 5:14 PM, Elijah Savage wrote:Here is one from that browdwing ds3 to MCI well Verizon now. 5  tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9)  157.795 ms  179.050 ms    tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13)  205.087 ms 6  * * ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206)  50.134 ms 7  * ae-1-100.ebr1.Chicago1.Level3.net (4.69.132.41)  45.873 ms * 8  ae-2.ebr2.NewYork1.Level3.net (4.69.132.66)  66.346 ms  72.509 ms * -- Elijah Savage               |  AOL IM:layer3rules Senior Network Engineer     |  When it has to be switched or routed. http://www.digitalrage.org  |  The Information Technology News Center- http://www.digitalrage.org/?page_id=46 for pgp public key On Oct 26, 2006, at 4:30 PM, Brandon Galbraith wrote:Can you be more specific?-brandonOn 10/26/06, Elijah Savage <[EMAIL PROTECTED]> wrote: Looks like level3 is having issues. Anyone know what is going on?  -- Elijah Savage               |  AOL IM:layer3rules  Senior Network Engineer     |  When it has to be switched or routed.  http://www.digitalrage.org  |  The Information Technology News Center-  http://www.digitalrage.org/?page_id=46 for pgp public key  -- Brandon GalbraithEmail: [EMAIL PROTECTED]AIM: brandong00Voice: 630.400.6992"A true pirate starts drinking before the sun hits the yard-arm. Ya. --thelost"

[Fwd: Re: DNS DDoS [was: register.com down sev0?]]

[virendra rode //]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

We ran into similar attacks (couple days back) coming from non-spoofed
address range (being initiated from valid prefixes).

In working (w/ a co-worker of mine) on a network attack situation (trace
process) for a 30,000 user location (serving 60 other school districts)
running BCP38 & rate-limit which got ddos'd w/ about 8mpps.
It appears that these attacks were coming from the inside which not only
saturated devices along its way but also got amplified into several
other networks also causing significant flaps to its peered connection
(OC-xx).
Besides being distracted with this incredible among of traffic flow our
goal number one goal was to prevent this bleeding, thanks to the
distributed monitoring sensors (maybe we got lucky) we were able to
identify and sink-hole (null route) certain blocks (vlans) while we
worked with the network/desktop team to isolate the infected machines.
This was certainly a hair-pulling experience.

The point that I'm trying to make here is, you can have data coming from
a herd of comprised hosts (bots, self-propagating worms,
spam-relays,fake http get request, backdoors, etc) that can attack
against a well-protected system(s) so any kind of defense mechanism
can/will get defeated.

Then again, it doesn't mean one wouldn't want to follow well practiced
prevention methods.

Just curious, any ddos vendors want to share their success stories :-)



regards,
/virendra


-  Original Message 
Subject: Re: DNS DDoS [was: register.com down sev0?]
Date: Thu, 26 Oct 2006 17:32:56 +
From: [EMAIL PROTECTED]
Reply-To: [EMAIL PROTECTED]
To: Robert Boyle <[EMAIL PROTECTED]>, [EMAIL PROTECTED],Patrick
W. Gilmore <[EMAIL PROTECTED]>, Nanog 
References:
<[EMAIL PROTECTED]><[EMAIL PROTECTED]>
<[EMAIL PROTECTED]>

The network hardware vendors do need to include the feature to support
BCP-38.  It'll help us out on a number of fronts especially with some of
the recent cyber attacks.

We're in process of reaching out to many of the companies and many
providers to encourage the implementation of BCP-38.  We've gotten a lot
of great feedback from many of you and its greatly appreciated.  You
know who you are :)
Especially some of the feedback related to the hardware OS issues.

- -Jerry
[EMAIL PROTECTED] or [EMAIL PROTECTED]

Sent via BlackBerry from Cingular Wireless

- -Original Message-
From: Robert Boyle <[EMAIL PROTECTED]>
Date: Thu, 26 Oct 2006 12:04:03
To:"Patrick W. Gilmore" <[EMAIL PROTECTED]>, nanog@merit.edu
Subject: Re: DNS DDoS [was: register.com down sev0?]


At 11:21 AM 10/26/2006, you wrote:
Unfortunately, as Jared has pointed out, the equipment vendors have
>to help the operators support this.  So let's all call your favorite
>router vendor and ask them when they will have the "ip bcp38" config
>option. :)

Even better would be the option: "no ip bcp38"

Make it so a conscious action is needed to disable it, but PLEASE put
that in the release notes so when the config doesn't "change" we know
that something really did change... :)

R



Tellurian Networks - Global Hosting Solutions Since 1995
http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin


-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFQS8zpbZvCIJx1bcRAn93AKCSF2JcGTbB/bX/NcxxWdOwBXRDagCbBkY4
OBRqFdIvWojOwTK+K6Mlp2U=
=LumS
-END PGP SIGNATURE-

Re: 10,352 active botnets (was Re: register.com down sev0?

[Jack Bates]

Matthew Crocker wrote:



Maybe the new slogan needs to be "Save the Internet! Train the chimps!"


Shouldnt  'ip verify unicast source reachable-by rx' be a default 
setting on all interfaces?  Only to be removed by trained chimps?




Only if you wish to break existing configurations during IOS upgrades. I could 
see ip verify unicast source reachable-by any (less breakage), but rx will kill 
all types of good asymmetric routing. The largest breakage I have seen caused by 
rx is the link IP breakage caused by the router responding out multiple 
interfaces. It's also a problem when customers are straddling the fence, 
purposefully using asymmetric routing.


It would be nicer to have router support where a packet is acceptable if it's 
network is acceptable in the BGP (or IGP) policy/filter (ie, network may not be 
there, but it is allowed) as well as the link addresses associated with the BGP 
(or IGP) peer.


-Jack

Re: register.com down sev0?

[Fergie]

We all have our opinions, Randy.

Hammers and nails being what they are...

- ferg

-- Randy Bush <[EMAIL PROTECTED]> wrote:

> what's curious, to me atleat, is that folks equate 'botnet' and 'spoofed
> source attacks' more often than I'd think is reasonable. I've not got
> 'hard numbers' but almost every time the attack is determined to be
> 'botnet' it's not spoofed.
> 
> Odd... (not that I'm against bcp38, I just think the distraction in
> conversation from 'bcp38 is good' to 'we must stop bots' is not helpful)

bingo!

when you have religion about a hammer, everything looks like a
nail.

randy



--
"Fergie", a.k.a. Paul Ferguson
 Engineering Architecture for the Internet
 fergdawg(at)netzero.net
 ferg's tech blog: http://fergdawg.blogspot.com/

Re: register.com down sev0?

[Fergie]

Chris,

W.R.T. #2 below:

Be for real: No one ever suggested that backbone service
providers attempt to ingress filter traffic -- this is an
edge function.

Cheers,

- ferg

-- "Chris L. Morrow" <[EMAIL PROTECTED]> wrote:

On Thu, 26 Oct 2006, Fergie wrote:
> and co-authored -- and likewise, cannot figure out for life of
> me, why there is such push-back from the Ops community on doing
> The Right Thing.

you could google answers from other folks but in shor:
1) it doesn't always work as advertised
2) people don't always tell you the routes the hold
3) equipment vendors don't alway splan properly for 'features'

Not everyone is as smart as you (both) and can manage that problem as they
scale...


--
"Fergie", a.k.a. Paul Ferguson
 Engineering Architecture for the Internet
 fergdawg(at)netzero.net
 ferg's tech blog: http://fergdawg.blogspot.com/

Re: register.com down sev0?

[Randy Bush]

> what's curious, to me atleat, is that folks equate 'botnet' and 'spoofed
> source attacks' more often than I'd think is reasonable. I've not got
> 'hard numbers' but almost every time the attack is determined to be
> 'botnet' it's not spoofed.
> 
> Odd... (not that I'm against bcp38, I just think the distraction in
> conversation from 'bcp38 is good' to 'we must stop bots' is not helpful)

bingo!

when you have religion about a hammer, everything looks like a
nail.

randy

Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)

[Chris L. Morrow]

On Thu, 26 Oct 2006, Mikael Abrahamsson wrote:

>
> On Thu, 26 Oct 2006, Fergie wrote:
>
> > The point I'm trying to make is that if the community thinks it is
> > valuable, then the path is clear.
>
> I of course realise that it's best if user cannot spoof at all, but it
> might be easier for ISPs to filter based on their PA blocks than to (in

do your customers:
1) not bring their own ip space?
2) always advertise to you their ip space?

Re: register.com down sev0?

[Chris L. Morrow]

On Thu, 26 Oct 2006, Fergie wrote:
> and co-authored -- and likewise, cannot figure out for life of
> me, why there is such push-back from the Ops community on doing
> The Right Thing.

you could google answers from other folks but in shor:
1) it doesn't always work as advertised
2) people don't always tell you the routes the hold
3) equipment vendors don't alway splan properly for 'features'

Not everyone is as smart as you (both) and can manage that problem as they
scale...

Re: Extreme Slowness

[Elijah Savage]

Here is one from that browdwing ds3 to MCI well Verizon now. 5  tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9)  157.795 ms  179.050 ms    tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13)  205.087 ms 6  * * ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206)  50.134 ms 7  * ae-1-100.ebr1.Chicago1.Level3.net (4.69.132.41)  45.873 ms * 8  ae-2.ebr2.NewYork1.Level3.net (4.69.132.66)  66.346 ms  72.509 ms * -- Elijah Savage               |  AOL IM:layer3rules Senior Network Engineer     |  When it has to be switched or routed. http://www.digitalrage.org  |  The Information Technology News Center- http://www.digitalrage.org/?page_id=46 for pgp public key On Oct 26, 2006, at 4:30 PM, Brandon Galbraith wrote:Can you be more specific?-brandonOn 10/26/06, Elijah Savage <[EMAIL PROTECTED]> wrote: Looks like level3 is having issues. Anyone know what is going on?  -- Elijah Savage               |  AOL IM:layer3rules  Senior Network Engineer     |  When it has to be switched or routed.  http://www.digitalrage.org  |  The Information Technology News Center-  http://www.digitalrage.org/?page_id=46 for pgp public key  -- Brandon GalbraithEmail: [EMAIL PROTECTED]AIM: brandong00Voice: 630.400.6992"A true pirate starts drinking before the sun hits the yard-arm. Ya. --thelost"

Re: register.com down sev0?

[Chris L. Morrow]

On Wed, 25 Oct 2006, Randy Bush wrote:
> > I don't want to detract from the heat of this discussion, as
> > important as it is, but it (the discussion) illustrates a point
> > that RIPE has recognized -- and is actively perusing -- yet, ISPs
> > on this continent seem consistently to ignore: The consistent
> > implementation of BCP 38.
>
> oh?  you have knowledge that this botnet attack used spoofed source
> addresses?

what's curious, to me atleat, is that folks equate 'botnet' and 'spoofed
source attacks' more often than I'd think is reasonable. I've not got
'hard numbers' but almost every time the attack is determined to be
'botnet' it's not spoofed.

Odd... (not that I'm against bcp38, I just think the distraction in
conversation from 'bcp38 is good' to 'we must stop bots' is not helpful)

Re: 10,352 active botnets (was Re: register.com down sev0?

[Matthew Crocker]

Maybe the new slogan needs to be "Save the Internet! Train the  
chimps!"


Shouldnt  'ip verify unicast source reachable-by rx' be a default  
setting on all interfaces?  Only to be removed by trained chimps?


-Matt

--
Matthew S. Crocker
Vice President
Crocker Communications, Inc.
Internet Division
PO BOX 710
Greenfield, MA 01302-0710
http://www.crocker.com

Re: Extreme Slowness

[Elijah Savage]

Say like this traceroute. This is from TW to a Broadwing DS3.5  tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13)  153.267 ms  207.125 ms    tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9)  218.920 ms 6  ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206)  36.976 ms  26.923 ms  57.770 ms 7  ge-11-0.core2.Chicago1.Level3.net (4.68.101.37)  254.145 ms    ge-11-1.core2.Chicago1.Level3.net (4.68.101.101)  258.522 ms    ge-11-2.core2.Chicago1.Level3.net (4.68.101.165)  227.223 ms 8  broadwing-level3-oc12.Chicago1.Level3.net (209.0.225.10)  231.451 ms   9  so-1-1-0.c1.gnwd.broadwing.net (216.140.15.1)  53.269 ms  35.568 ms  22.511 ms10  216.140.14.17 (216.140.14.17)  34.751 ms  39.008 ms  46.644 ms11  p5-0-0.e0.cncn.broadwing.net (216.140.15.78)  32.065 ms  60.797 ms  54.766 ms12  67.98.17.122 (67.98.17.122)  44.772 ms  27.631 ms  30.655 ms13  * * * -- Elijah Savage               |  AOL IM:layer3rules Senior Network Engineer     |  When it has to be switched or routed. http://www.digitalrage.org  |  The Information Technology News Center- http://www.digitalrage.org/?page_id=46 for pgp public key On Oct 26, 2006, at 4:30 PM, Brandon Galbraith wrote:Can you be more specific?-brandonOn 10/26/06, Elijah Savage <[EMAIL PROTECTED]> wrote: Looks like level3 is having issues. Anyone know what is going on?  -- Elijah Savage               |  AOL IM:layer3rules  Senior Network Engineer     |  When it has to be switched or routed.  http://www.digitalrage.org  |  The Information Technology News Center-  http://www.digitalrage.org/?page_id=46 for pgp public key  -- Brandon GalbraithEmail: [EMAIL PROTECTED]AIM: brandong00Voice: 630.400.6992"A true pirate starts drinking before the sun hits the yard-arm. Ya. --thelost"

Re: BCP38 thread 93,871,738,435 + SPF

[Douglas Otis]

On Thu, 2006-10-26 at 13:03 -0400, Steven M. Bellovin wrote:
> On Thu, 26 Oct 2006 17:07:32 +0200, Florian Weimer <[EMAIL PROTECTED]>
> wrote:
> 
> > * Steven M. Bellovin:
> > 
> > > As you note, the 20-25% figure (of addresses) has been pretty
> > > constant for quite a while.  Assuming that subverted machines are
> > > uniformly distributed (a big assumption)
> > 
> > I doubt this assumption about distribution is valid.  At least over
> > here, consumer-grade ISPs (think DSL with dynamic IP addresses) apply
> > ingress filters, while real ISPs don't.  If you're lucky, you get
> > egress filters at some border routers, but it's not standard at all.
> > Customer-facing interfaces are generally unfiltered.
>
> Those are good points.  It would be interesting to look at the raw AS
> data and see what classes of organizations were represented.
> Unfortunately, that data is not publicly available, according to the FAQ
> for that project. 

Microsoft no longer demands licensing for Sender-ID.  Executing SPF
scripts may become increasingly popular for validating SMTP clients
against MAILFROM, PRA, or perhaps DKIM domains.  SPF scripts can be far
more malicious than even spoofed source reflective DNS attacks.  Dozens
of TXT or hundreds of A record transactions might be invoked by the
recipients of each message.

In addition to MTAs processing SPF script, popular applications like
SpamAssassin executes a different script after an elapsed timeout of 5
seconds.  Such short timeouts eliminate congestion avoidance.  SPF is
analogous to a complex lock where there is no obvious place for a key.
One must trust a stranger to operate their own unique DNS based
mechanism.

The macro capability of this script also allows any element of an
email-address to randomize DNS queries against any unseen victim domain.
SMTP logs may not indicate who initiated an attack, nor indicate when an
attack was in progress.  BCP38 filtering or ACLs on recursive DNS will
not offer protection from this exploit, which can achieve gains much
higher than spoofed source reflective DNS attacks.

Spam being sent through Bot farms has already set the stage for
untraceable DNS attacks based upon SPF.  In addition to taking out major
interconnects, these attacks can:

 a) inundate authoritative DNS;

 b) requests A records from anywhere;

 c) probe IP address, port, and the transaction IDs of resolvers;

While not as bad as eavesdropping, it still places the network and the
integrity of DNS at risk.  All of this while the spam is still being
delivered.  What a productivity tool!

How is this attack detected?

How is this attack avoided?

-Doug
 


  

Re: Extreme Slowness

[Elijah Savage]

It seems anything traversing level3 has very high latency along with what seems overloaded capacity as if they are running in a degraded mode I have connections with Time Warner, AT&T, and MCI. Though I know it is not concrete it seems as if something is going on according to this http://www.internetpulse.net/ -- Elijah Savage               |  AOL IM:layer3rules Senior Network Engineer     |  When it has to be switched or routed. http://www.digitalrage.org  |  The Information Technology News Center- http://www.digitalrage.org/?page_id=46 for pgp public key On Oct 26, 2006, at 4:30 PM, Brandon Galbraith wrote:Can you be more specific?-brandonOn 10/26/06, Elijah Savage <[EMAIL PROTECTED]> wrote: Looks like level3 is having issues. Anyone know what is going on?  -- Elijah Savage               |  AOL IM:layer3rules  Senior Network Engineer     |  When it has to be switched or routed.  http://www.digitalrage.org  |  The Information Technology News Center-  http://www.digitalrage.org/?page_id=46 for pgp public key  -- Brandon GalbraithEmail: [EMAIL PROTECTED]AIM: brandong00Voice: 630.400.6992"A true pirate starts drinking before the sun hits the yard-arm. Ya. --thelost"

Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)

[Michael Painter]

- Original Message - 
From: "william(at)elan.net" <[EMAIL PROTECTED]>

To: "Don" <[EMAIL PROTECTED]>
Cc: 
Sent: Thursday, October 26, 2006 8:17 AM
Subject: Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)





On Thu, 26 Oct 2006, Don wrote:

Has anyone put together a centralized system where you can send in a list of 
attacking bots, let it automatically sort by allocation, and then let it 
notify the appropriate admin with a list of [potentially] compromised hosts?


mynetwatchman [1] comes to mind and so does dshield [2]

[1] http://www.mynetwatchman.com
[2] http://www.dshield.org

--
William Leibzon
Elan Networks
[EMAIL PROTECTED]



Anyone familiar with these folks?
http://www.simplicita.com/Simplicita_Research_Data_Partner_Program.html

--Michael

Re: Extreme Slowness

[Brandon Galbraith]

Can you be more specific?-brandonOn 10/26/06, Elijah Savage <[EMAIL PROTECTED]> wrote:
Looks like level3 is having issues. Anyone know what is going on?
 
-- Elijah Savage               |  AOL IM:layer3rules
 Senior Network Engineer     |  When it has to be switched or routed. 
http://www.digitalrage.org  |  The Information Technology News Center- 
http://www.digitalrage.org/?page_id=46 for pgp public key 
-- Brandon GalbraithEmail: [EMAIL PROTECTED]AIM: brandong00Voice: 630.400.6992"A true pirate starts drinking before the sun hits the yard-arm. Ya. --thelost"

Extreme Slowness

[Elijah Savage]

Looks like level3 is having issues. Anyone know what is going on? -- Elijah Savage               |  AOL IM:layer3rules Senior Network Engineer     |  When it has to be switched or routed. http://www.digitalrage.org  |  The Information Technology News Center- http://www.digitalrage.org/?page_id=46 for pgp public key 

Re: register.com down sev0?

[Daniel Senie]

At 07:25 AM 10/26/2006, Jared Mauch wrote:


On Thu, Oct 26, 2006 at 06:03:54AM +, Fergie wrote:
>
> Randy,
>
> I don't think I implied anything of the sort.
>
> I did, however, pipe up when a BCP is mentioned that I endorse,
> and co-authored -- and likewise, cannot figure out for life of
> me, why there is such push-back from the Ops community on doing
> The Right Thing.

The challenge is that the router vendors still haven't
done "The Right Thing".

I have one device that

1) halves its forwarding table space by enabling u-rpf
2) can only do either strict or loose mode rpf *GLOBALLY* so I can
   not strict rpf-check a static customer AND loose rpf someone
   larger for unrouted space.


It was possible to implement BCP38 before the router vendors came up with uRPF.



because of the above (#1 isn't that bad, but #2 is)
I can't enable u-rpf on the device as a policy.  Changing one
interface from loose -> strict silently changes all other u-rpf
interfaces and then customers gripe about dropped packets.

obviously moving these checks closer to the edge
is ideal, such as always doing rpf on the ethernet lan
interface for your customer CPE.


Yes, it is. And does not require uRPF.

I know you're looking to do the right thing. It's important though 
that this not be put entirely on the router vendors. How many 
"managed T1" services out there have routers controlled by the ISP 
providing them? How many of those routers are configured with a 
single line ACL that would implement BCP38 sufficiently?


How many aggregation routers for incoming T1s are not configured with 
a single line ACL per T-1 to ensure the packets coming in are from 
assigned, not-multihomed space?


If scripts are being used to auto-configure routers to ship out to 
T-1 customers, then appropriate ACLs should be written by such 
scripts at the same time. Scripts that configure aggregation switches 
should similarly be reviewed for ACL inclusion.


It's certainly helpful to have implementations such as uRPF to help 
make it easier to deploy BCP38, but deployment of BCP38 is not 
dependent on the existence of uRPF.




> Having said that, botnets don't need to spoof addresses -- the
> sheer dispersion of geographic and AS infection base renders the
> whole point of spoofing almost moot.

yup, it's an evolving threat, even if some solution to the
botnet problem is discovered, it will take years to fix.  Think of
the smurf amplifiers that are still out there[1].


Dan
(the other co-author of the BCP in question) 

Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)

[william(at)elan.net]

On Thu, 26 Oct 2006, Don wrote:

Has anyone put together a centralized system where you can send in a list of 
attacking bots, let it automatically sort by allocation, and then let it 
notify the appropriate admin with a list of [potentially] compromised hosts?


mynetwatchman [1] comes to mind and so does dshield [2]

[1] http://www.mynetwatchman.com
[2] http://www.dshield.org

--
William Leibzon
Elan Networks
[EMAIL PROTECTED]

re: passports for NANOG-39, Toronto

[chuck goolsbee]

At 2:26 PM +0100 10/26/06, [EMAIL PROTECTED] wrote:

 http://travel.state.gov/travel/tips/regional/regional_1170.html



 December 31, 2007 - Passport required for all land border crossings, as
 well as air and sea travel.


FWIW I live near the WA/BC (US/CDN) border and cross it often (at 
least twice a month) for both work and social activities, and have 
been using an expired US passport for the past two years with no 
issues. The Canadians never even ask for it. The US border folks 
occasionally hassle me a tiny bit, but never about the expired 
passport (go figure). The ONLY time the expired passport was an issue 
was on a flight, with my entire family from Seattle to Denver(!) 
where the TSA boarding pass & ID checker in airport security nearly 
didn't let me through. Again, go figure.


Yes, I need to renew, but as my world-travelling days for work are 
behind me I haven't been motivated to do so.


--chuck

Re: DNS DDoS [was: register.com down sev0?]

[jerry]

The network hardware vendors do need to include the feature to support BCP-38.  
It'll help us out on a number of fronts especially with some of the recent 
cyber attacks.  

We're in process of reaching out to many of the companies and many providers to 
encourage the implementation of BCP-38.  We've gotten a lot of great feedback 
from many of you and its greatly appreciated.  You know who you are :)
Especially some of the feedback related to the hardware OS issues.

-Jerry
[EMAIL PROTECTED] or [EMAIL PROTECTED]

Sent via BlackBerry from Cingular Wireless  

-Original Message-
From: Robert Boyle <[EMAIL PROTECTED]>
Date: Thu, 26 Oct 2006 12:04:03 
To:"Patrick W. Gilmore" <[EMAIL PROTECTED]>, nanog@merit.edu
Subject: Re: DNS DDoS [was: register.com down sev0?]


At 11:21 AM 10/26/2006, you wrote:
Unfortunately, as Jared has pointed out, the equipment vendors have
>to help the operators support this.  So let's all call your favorite
>router vendor and ask them when they will have the "ip bcp38" config
>option. :)

Even better would be the option: "no ip bcp38"

Make it so a conscious action is needed to disable it, but PLEASE put 
that in the release notes so when the config doesn't "change" we know 
that something really did change... :)

R



Tellurian Networks - Global Hosting Solutions Since 1995
http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin

Re: BCP38 thread 93,871,738,435

[Steven M. Bellovin]

On Thu, 26 Oct 2006 17:07:32 +0200, Florian Weimer <[EMAIL PROTECTED]>
wrote:

> * Steven M. Bellovin:
> 
> > As you note, the 20-25% figure (of addresses) has been pretty constant
> > for quite a while.  Assuming that subverted machines are uniformly
> > distributed (a big assumption)
> 
> I doubt this assumption about distribution is valid.  At least over
> here, consumer-grade ISPs (think DSL with dynamic IP addresses) apply
> ingress filters, while real ISPs don't.  If you're lucky, you get
> egress filters at some border routers, but it's not standard at all.
> Customer-facing interfaces are generally unfiltered.
> 
Those are good points.  It would be interesting to look at the raw AS
data and see what classes of organizations were represented.
Unfortunately, that data is not publicly available, according to the FAQ
for that project. 


--Steven M. Bellovin, http://www.cs.columbia.edu/~smb

Re: register.com down sev0?

[Randy Bush]

>> the case for which we know bcp 38 is useful, is the dns reflector
>> attack.  so far, botnets seem to have no need to spoof, they just
>> overwhelm you with zombies from real space.
> 
> Incorrect.
> 
> While that is one mode of attack from a botnet, it is not the only  
> mode.  And there are reasons for even botnets to spoof source  
> addresses.  And reasons that the attack-ee would prefer they did not.
> 
> Randy, are you REALLY arguing -against- BCP38?  Or just yanking  
> Fergie's chain 'cause it wouldn't have helped in this particular  
> instance?

i merely said that using this particular attack to launch yet
another bcp38 religious dos against the nanog list was bogus.  have
we learned one new thing from the last day's oratory?

personally, i long ago implemented spoofing blocking in all places
i have been able to do so.  but i am not foolish enough to believe
that religious ranting on mailing lists is gonna change anyone from
doing what makes business sense for their network.  and, as spoofed
attacks other than the dns reflector seem to have been rare, that
perceived interest in anti-spoofing blocks is low when compared to
other priorities in these hard times.  i think we have converted
those who were convertable and the rest watch the religious
zealotry and scratch their heads.

randy

Re: register.com down sev0? - More information

[Donald Stahl]

5. AT&T (at least when I've dealt with them in their datacenters) does not 
support BGP community strings for null routing (or any strings for that 
matter :)
Lest anyone take me too seriously on that last point- AT&T hosting does 
have community strings for certain features- unfortunately not for null 
routing.


-Don

(My apologies for the earlier lack of a full email name)

Re: DNS DDoS [was: register.com down sev0?]

[Robert Boyle]

At 11:21 AM 10/26/2006, you wrote:
Unfortunately, as Jared has pointed out, the equipment vendors have

to help the operators support this.  So let's all call your favorite
router vendor and ask them when they will have the "ip bcp38" config
option. :)


Even better would be the option: "no ip bcp38"

Make it so a conscious action is needed to disable it, but PLEASE put 
that in the release notes so when the config doesn't "change" we know 
that something really did change... :)


R



Tellurian Networks - Global Hosting Solutions Since 1995
http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin

Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)

[Don]

Put another way, anti-spoofing does three things: it makes reflector
attacks harder, it makes it easier to use ACLs to block sources, and it
helps people track down the bot and notify the admin. Are people actually
successfully doing either of the latter two?
I think it's a time constraint- looking up, sorting and notifying admins 
about 10,000 attack sources isn't practical. I'd love to do it- but I 
don't have time. That said- if someone notifies me of a compromised host I 
immediately investigate- and I suspect so would everyone else on this 
list.


Has anyone put together a centralized system where you can send in 
a list of attacking bots, let it automatically sort by allocation, and 
then let it notify the appropriate admin with a list of [potentially] 
compromised hosts?


Then again: Considering how many admins don't care, how many end users 
don't care/know, and how quickly many of thee systems would get 
re-infected maybe it's all a bit pointless.


I'd be surprised if there were much of either.  That leaves reflector 
attacks.  Are those that large a portion of the attacks people are 
seeing?
Everything I have seen of late has been legitimate traffic originating 
from across the globe. With tens of thousands of compromised hosts that's 
all it takes.


-Don

Re: register.com down sev0?

[Patrick W. Gilmore]

On Oct 26, 2006, at 11:24 AM, Randy Bush wrote:


the case for which we know bcp 38 is useful, is the dns reflector
attack.  so far, botnets seem to have no need to spoof, they just
overwhelm you with zombies from real space.


Incorrect.

While that is one mode of attack from a botnet, it is not the only  
mode.  And there are reasons for even botnets to spoof source  
addresses.  And reasons that the attack-ee would prefer they did not.


Randy, are you REALLY arguing -against- BCP38?  Or just yanking  
Fergie's chain 'cause it wouldn't have helped in this particular  
instance?


--
TTFN,
patrick

Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)

[Patrick W. Gilmore]

On Oct 26, 2006, at 9:33 AM, Steven M. Bellovin wrote:


Put another way, anti-spoofing does three things: it makes reflector
attacks harder, it makes it easier to use ACLs to block sources,  
and it
helps people track down the bot and notify the admin. Are people  
actually
successfully doing either of the latter two?  I'd be surprised if  
there
were much of either.  That leaves reflector attacks.  Are those  
that large

a portion of the attacks people are seeing?


I disagree.  As someone who has been attacked by spoof-source  
packets, and not-spoof-source packed, I can say, from personal  
experience, that the former is much, much easier to mitigate.


And, as I posted before, even if all universal adoption of BCP38  
means is that DDoS attacks move to botnets with 100% real source IP  
addresses, that would still be a Very Good Thing, IMHO.


But perhaps others feel differently.  Or perhaps they just haven't  
been attacked enough. :)


--
TTFN,
patrick

Re: register.com down sev0?

[Gadi Evron]

On Thu, 26 Oct 2006, Randy Bush wrote:
> 
> > I don't think I implied anything of the sort.
> 
> ahhh, but you did.
> 
> >>> I don't want to detract from the heat of this discussion, as
> >>> important as it is, but it (the discussion) illustrates a point
> >>> that RIPE has recognized -- and is actively perusing -- yet, ISPs
> >>> on this continent seem consistently to ignore: The consistent
> >>> implementation of BCP 38.
> > 
> > oh?  you have knowledge that this botnet attack used spoofed source
> > addresses?
> 
> if the register.com botnet attack was not from spoofed addresses,
> then bcp 38 would not have helped.
> 
> the case for which we know bcp 38 is useful, is the dns reflector
> attack.  so far, botnets seem to have no need to spoof, they just
> overwhelm you with zombies from real space.

And yet they do anyway.

Before the "reflector attacks" run at the beginning of this year, you
stated you do not see the need to deal with spoofing, as it is not
something being exploited.

It is being exploited, let's deal with it.

Gadi.

> 
> randy

10,352 active botnets (was Re: register.com down sev0?

[Valdis . Kletnieks]

On Thu, 26 Oct 2006 05:11:14 -, Fergie said:
> I don't want to detract from the heat of this discussion, as
> important as it is, but it (the discussion) illustrates a point
> that RIPE has recognized -- and is actively perusing -- yet, ISPs
> on this continent seem consistently to ignore: The consistent
> implementation of BCP 38.
>
> It is nothing less than irresponsible, IMO...
>
> Why _is_ that?

The same people I mentioned the other day as not having enough clue to
do DNS correctly don't have enough clue to do BCP38 correctly either.
As one person mentioned, if stuff still requires pioneer-level skillsets
to use, the pioneers have more work to do.  The problem is that the
following wave seems to be made up mostly of chimpanzees, and nobody's
figured out how to make routers and network services that can be run
by chimps...

Maybe the new slogan needs to be "Save the Internet! Train the chimps!"


pgpFsZMkxDfPo.pgp
Description: PGP signature

Re: register.com down sev0?

[Randy Bush]

> I don't think I implied anything of the sort.

ahhh, but you did.

>>> I don't want to detract from the heat of this discussion, as
>>> important as it is, but it (the discussion) illustrates a point
>>> that RIPE has recognized -- and is actively perusing -- yet, ISPs
>>> on this continent seem consistently to ignore: The consistent
>>> implementation of BCP 38.
> 
> oh?  you have knowledge that this botnet attack used spoofed source
> addresses?

if the register.com botnet attack was not from spoofed addresses,
then bcp 38 would not have helped.

the case for which we know bcp 38 is useful, is the dns reflector
attack.  so far, botnets seem to have no need to spoof, they just
overwhelm you with zombies from real space.

randy

DNS DDoS [was: register.com down sev0?]

[Patrick W. Gilmore]

On Oct 26, 2006, at 1:31 AM, [EMAIL PROTECTED] wrote:


It is essentially impossible to distinguish end-user requests from
(im)properly created DoS packets (especially until BCP38 is widely
adopted - i.e. probably never).  Since there is no single place -  
no 13
places - which can withstand a well crafted DoS, you are  
guaranteed that

some users will not be able to reach any of your listed authorities.

Yeah - I know it hard-to-impossible to do that, and it is a tug-of-war
between worm writers (to generate queries indistinguishable from real
client-resolver-generated queries) and trying-to-detect-malformed- 
queries
(such as duplicated qid, or from IP space that shouldn't be hitting  
this

specific node). You probably dealt with more ddos than rest of us
combined, so I bow to your superior knowledge.


First, thanx for the nod, but there are some here who have dealt with  
more than I have.  But I think I've seen enough to know something  
about it.


You can try things like "filter IP addresses which should not be  
going to node X", but what happens if the DDoS changes the network  
topology enough that you can't be certain users are going where you  
did not?  If the DDoS is large, this is pretty much guaranteed.


Worse, suppose the topology changes for reasons unrelated to a DDoS.   
You could end up DoS'ing end users without an attack!  (You could  
theoretically only put the filters in place when an attack is  
happening, but that has other problems - which may or may not be worse.)


Filtering on things like duplicated query IDs is not possible on  
router hardware doing 10s of Gbps or millions of PPS.  And doing it  
on the server is not useful if there are more bits / pps than the  
router can process.  Remember, servers can't answer packets that are  
dropped before they get to the servers.


Etc., etc., etc.


Overall, we are losing the war.  What good providers, like the roots,  
Ultra, etc., do is to minimize the effect of any attack.  If a  
"miscreant" fires the "DDoS of biblical proportions" and only 5% of  
users are affected, I consider that a success.  Unfortunately, those  
5% don't think so, but one can only do what one can do.  Besides, if  
it truly is an attack of biblical proportion, those 5% are probably  
having much larger problems than name resolution.



Couple other comments:

From all indications I've seen (and most are not authoritative, but  
it's all the info I have), this was not a DDoS of "biblical  
proportions".  There were no whole networks to go offline, there were  
no massive swaths of address space flapping, there were no entire  
peering points being congested, etc.  A few Gbps does not count as  
"biblical" any more.


Whether this attack used spoof-source or not, BCP38 is _VITAL_, IMHO,  
to helping curb these things.  It guarantees, at the very least, that  
you know where the attack is sourced.  Filtering become much easier.   
Reaching the right operators to help with the problem becomes orders  
of magnitude easier.  And if the miscreants just start using BotNets  
with real IP address, GOOD.  It's not the End All Be All answer, but  
it is a _huge_ step in the right direction.


Unfortunately, as Jared has pointed out, the equipment vendors have  
to help the operators support this.  So let's all call your favorite  
router vendor and ask them when they will have the "ip bcp38" config  
option. :)


--
TTFN,
patrick

Re: BCP38 thread 93,871,738,435

[Florian Weimer]

* Steven M. Bellovin:

> As you note, the 20-25% figure (of addresses) has been pretty constant
> for quite a while.  Assuming that subverted machines are uniformly
> distributed (a big assumption)

I doubt this assumption about distribution is valid.  At least over
here, consumer-grade ISPs (think DSL with dynamic IP addresses) apply
ingress filters, while real ISPs don't.  If you're lucky, you get
egress filters at some border routers, but it's not standard at all.
Customer-facing interfaces are generally unfiltered.

(But I have to admit that we recently ran into filters at an
upstream's upstream, so there's at least some BCP 38 adoption.)

Re: passports for NANOG-39, Toronto

[Joe Abley]

On 26-Oct-2006, at 09:26, [EMAIL PROTECTED] wrote:


You could do the same fly-drive via Detroit but there is
a lot more driving.


Indeed. Rough estimates, excluding time taken to cross the border and  
assuming good weather:


  BUF to Toronto: 2 hours
  DTW to Toronto: 5 hours
  CLE to Toronto: 6 hours
  LGA to Toronto: 9 hours
  BOS to Toronto: 9 hours
  ORD to Toronto: 10 hours
  IAD to Toronto: 10 hours


Joe

Re: 10,352 active botnets (was Re: register.com down sev0?)

[Simon Waters]

On Thursday 26 Oct 2006 13:45, you wrote:
> 
> Is there a similar statistic available for Mac OS X ?

Now now.

> > "Of the 4 million computers cleaned by the company's MSRT
> > (malicious software removal tool), about 50 percent (2 million)
> > contained at least one backdoor Trojan. While this is a high
> > percentage, Microsoft notes that this is a decrease from the
> > second half of 2005. During that period, the MSRT data showed
> > that 68 percent of machines cleaned by the tool contained a
> > backdoor Trojan."

A lot depends on the definition.

I've removed some malware trying to exploit an old Microsoft JRE bug. This 
stuff gets everywhere (well anywhere IE goes).

These get downloaded to some cached program folder for Java, and because the 
exploit hasn't worked for years, sit there till some antivirus software comes 
along and removes them, doing nowt but consuming disk space.

If you are the Microsoft malicious software removal tool marketing department, 
that is a trojan removed. To the average person on the street, it is another 
bit of meaningless fluff their PC will lose when they reinstall.

So yes, Microsoft is big enough to have bits who have a vested interest in 
making the other bits look bad (if only incidentally). Thus is the way of big 
companies.

Re: register.com down sev0? - More information

[Chris Adams]

Once upon a time, Don <[EMAIL PROTECTED]> said:
> Some facts:
> 3. The attack was large enough to affect many other customers in the same 
> data center- one with a lot of bandwidth off AT&T's backbone.

Is this what got Red Hat over the last couple of days as well?  I think
they have a lot of their stuff on AT&T's network.

-- 
Chris Adams <[EMAIL PROTECTED]>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)

[Steven M. Bellovin]

On Thu, 26 Oct 2006 02:20:48 -0400 (EDT), Sean Donelan <[EMAIL PROTECTED]>
wrote:

> 
> The only data I have is from the MIT anti-spoofing test project which
> has been pretty consistent for a long time.  About 75%-80% of the nets, 
> addressses, ASNs tests couldn't spoof, and about 20%-25% could.
> 
> The geo-location maps don't show much difference between parts of
> the world.  RIPE countries don't seem to be better or worse than ARIN
> countries or APNIC countries or so on.  ISPs on every continent seem
> to be about the same.
> 
> http://spoofer.csail.mit.edu/summary.php
> 
> If someone finds the silver bullet that will change the remaining 25% or
> so of networks, I think ISPs on every continent would be interested.

That would be nice -- but I wonder how much operational impact that would
have.

As you note, the 20-25% figure (of addresses) has been pretty constant
for quite a while.  Assuming that subverted machines are uniformly
distributed (a big assumption) and assuming that their methodology is
valid (another big assumption), that means we've already knocked out the
75-80% of the sources of spoofed IP address attacks.  Has anyone seen a
commensurate reduction in DDoS attacks?  I sure haven't heard of that.
Are people saying that the problem would be several times worse if
anti-spoofing weren't in place?  As best I can tell, the limiting factor
on attack rates isn't the lack of sources but the lack of a profit motive
for launching the attacks.

Put another way, anti-spoofing does three things: it makes reflector
attacks harder, it makes it easier to use ACLs to block sources, and it
helps people track down the bot and notify the admin. Are people actually
successfully doing either of the latter two?  I'd be surprised if there
were much of either.  That leaves reflector attacks.  Are those that large
a portion of the attacks people are seeing?

I agree that anti-spoofing is a good idea, and I've said so for a long
time.  I was one of the people who insisted that AT&T do it, way back
when.  But I'm not convinced it's a major factor here.


--Steven M. Bellovin, http://www.cs.columbia.edu/~smb

Re: register.com down sev0? - More information

[Don]

As pointed out by Rob Seastrom in private email, RFC2182 addresses things
of biblical proportions - such as dispersion of nameservers geographically
and topologically. Having 3 secondaries, only one of them on separate /24,
and none of them on topologically different network does not qualify.
Register.com offered several models for DNS service including distributed 
anycast based services. Considering what I've heard about the scale of 
the attack I'm glad they chose not host their own domain name on the 
anycast networks- it simply would have taken more people down.


Some facts:
1. I've spoken with some AT&T engineers about what was going on. According 
to them this was (as mentioned earlier) a multi gigabit attack that came 
in through every peer on the AT&T network. Anycasting would not have fixed 
this problem- the attack was too large and too diverse. (I guess if they 
had 10 gige pipes and pops all over the planet- maybe. But that's not 
exactly a valid business model.)


2. These were not spoofed source addresses. This looks like a rather large 
botnet sending real traffic.


3. The attack was large enough to affect many other customers in the same 
data center- one with a lot of bandwidth off AT&T's backbone.


4. DNS is a tiny protocol. It's possible to send a LOT of small, but 
perfectly valid, DNS packets. The fact that the attack was multi gigabit 
per second is bad enough. Couple that with the packets all being really 
tiny and you have a recipe for routing disaster.


5. AT&T (at least when I've dealt with them in their datacenters) does not 
support BGP community strings for null routing (or any strings for that 
matter :) Think about that for a second. To stop an attack Register.com 
would need to call AT&T and request a filter/null route. Since AT&T 
operations is based in Singapore (again this was last time I dealt with 
them) I'm sure getting those filters/routes in probably doesn't happen 
nearly fast enough. I have heard that AT&T is currently in the process of 
setting up communities- maybe someone who knows more could comment.


The truth is that none of us has all the facts about what happened.


Given that register.com is/was public (I think?) - I wonder what are their
sarbox auditors saying about it now ;)
Register.com is not public (If I recall correctly they were bought out a 
couple of years ago by a private firm). Furthermore if they were public I 
would think their stockholders might have something to say about spending 
large sums of money to prevent a DDoS which probably would not work 
anyway.


-Don

re: passports for NANOG-39, Toronto

[Michael . Dillon]

> http://travel.state.gov/travel/tips/regional/regional_1170.html

> December 31, 2007 - Passport required for all land border crossings, as
> well as air and sea travel. 

If someone wants to go but does not have a passport for
whatever reason, i.e. last minute travel plans, then it
is possible to fly to Buffalo NY and make a land crossing
from there, i.e. bus or rental car. If you do want to take
a rental car across the border, you have to notify your
rental company so they can issue a non-resident insurance
card for you. As long as you have a US driver's licence this
is fairly routine. Cross the bridge to Canada and take 
the QEW all the way to Toronto.

http://www.buffaloairport.com/

You could do the same fly-drive via Detroit but there is
a lot more driving.

--Michael Dillon

P.S. Now that you have your shiny new passports, don't 
just stop at Canada. There's a whole world out there.

Re: passports for NANOG-39, Toronto

[Marshall Eubanks]

In DC, at least, you can get an appointment (no Congressional  
pressure required), go to
an office in the AM and pick it up the same day. I have done this  
several times;

it always amazes me how many people are in line who are
leaving the country the same day, but I wouldn't push it that far.  
Here are the offices :


http://travel.state.gov/passport/about/agencies/agencies_913.html

You will spend a good fraction of the day doing this (the appointment  
is really an

appointment to sit in a waiting room, and you have to do it twice).

Regards
Marshall

On Oct 26, 2006, at 1:07 AM, Howard C. Berkowitz wrote:



particularly if you are in the DC area, call your congressman's  
district
(usually) office and ask them to send the Passport Office a  
"congressional
courtesy" request. In practice, this means that you don't stand in  
line, but
go upstairs to the diplomatic processing area, and, with proper  
documents

and photos, you'll probably have the passport in under an hour.
I believe there is also a priority program for cities that have  
Passport

Office branches. Just one of the perks of incumbents.


-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On  
Behalf Of

Robert E. Seastrom
Sent: Wednesday, October 25, 2006 10:26 PM
To: [EMAIL PROTECTED]
Subject: passports for NANOG-39, Toronto



You may have heard that the US and Canada are going to start  
requiring

passports for air travel between them beginning "soon".  That date is
currently set as 8 Jan 2007, which is before February NANOG.  MERIT
has noted this on the web site, but a cursory check of my list
archives didn't turn up mention of it (sorry if I overlooked it; the
last couple of weeks have been hectic), so I figured I'd include the
pointer:

   http://www.nanog.org/mtg-0702/passport.html

as well as a link to the State Department:

   http://travel.state.gov/passport/passport_1738.html

Normal passport processing is "within six weeks", but that probably
doesn't take the holiday season into account.  If you don't have a
passport already and plan to travel from the US to NANOG 39 in
Toronto, getting on that project sometime in the next month or so
would allow plenty of spare time.  No reason to pay expedite fees if
you don't have to.

---Rob

Re: register.com down sev0?

[Rich Kulawiec]

On Thu, Oct 26, 2006 at 12:14:43AM -0400, [EMAIL PROTECTED] wrote:
> 
> On 26 Oct 2006, Paul Vixie wrote:
> > i wonder if that's due to the spam they've been sending out?
> Paul, this isn't nanae. Let's not sling accusations like that wildly. 

There's nothing "wild" about it -- Paul is one of the most sober,
reasoned observers of the spam problem, and if he told me that
my servers were sending spam, then I'd darn well go investigate.

Right now.

Besides -- it's not like this isn't common knowledge in the anti-spam
world.  I'm sure I'm not the only one who's had unsatisfying correspondence
with register.com wherein they refuse to lift a finger to stop the abuse
from/facilitated by their operation.

---Rsk

Re: 10,352 active botnets (was Re: register.com down sev0?)

[Marshall Eubanks]

Dear Fergie;

Is there a similar statistic available for Mac OS X ?

Regards
Marshall

On Oct 26, 2006, at 5:43 AM, Fergie wrote:



Jose's numbers are conservative.

Given some mathematical acrobatics, I'd suggest examining some
of the (shocking) number sin Microsoft's Security Intelligence
Report (Google it) -- these are reflective:

"Of the 4 million computers cleaned by the company's MSRT
(malicious software removal tool), about 50 percent (2 million)
contained at least one backdoor Trojan. While this is a high
percentage, Microsoft notes that this is a decrease from the
second half of 2005. During that period, the MSRT data showed
that 68 percent of machines cleaned by the tool contained a
backdoor Trojan."

Ref: http://www.eweek.com/article2/0,1759,2036439,00.asp

If you're wondering why DDoS attacks are so effective, look
no further than your backyard.

- ferg


-- Sean Donelan <[EMAIL PROTECTED]> wrote:

On Thu, 26 Oct 2006, [EMAIL PROTECTED] wrote:
Well, let's talk about "worst-case ddos". Let's say, 50mpps (I  
have not
heard of ddos larger that that number). Let's say, you can sink/ 
filter
100kpps on each box (not unreasonable on higher-end box with nsd).  
That

means, you should be able to filter this attack with ~500 servers,
appropriately place. Say, because you don't know where the attack  
will
come in, you need 4 times more the estimated number of servers,  
that's
2000 servers. That's not entirely unreasonable number for a large  
enough

company.


Botnets were the topic at today's Info Security conference in New York
City.     Coincidences?  Or just
as random as your iPod shuffle?

Jose Nazario estimated that there were 10,352 botnets active on the
Internet earlier this year. You will probably always be outnumbered on
the public Internet.


--
"Fergie", a.k.a. Paul Ferguson
 Engineering Architecture for the Internet
 fergdawg(at)netzero.net
 ferg's tech blog: http://fergdawg.blogspot.com/

Re: register.com down sev0?

[Per Heldal]

On Thu, 2006-10-26 at 06:03 +, Fergie wrote:
> Having said that, botnets don't need to spoof addresses -- the
> sheer dispersion of geographic and AS infection base renders the
> whole point of spoofing almost moot.

A lot of new possibilities arise if spoofing can be eliminated with near
100% certainty. Some examples:

Automated filtering.

Automated notification to providers. "Cut off host X or..."

Expose compromised systems and hold their owners financially responsible
for damages. Severe punishment of large number of users may cause
outrage, basis for regress, class-action lawsuits, and maybe finally
turn the attention to the real source of the problem; software vendors
whose products are of such a dismal quality that they'd be banned
worldwide from just about any market other than that for computer
software. 


-- 


Per Heldal - http://heldal.eml.cc/

Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)

[Per Heldal]

On Thu, 2006-10-26 at 02:20 -0400, Sean Donelan wrote:
> http://spoofer.csail.mit.edu/summary.php
> 
> If someone finds the silver bullet that will change the remaining 25% or
> so of networks, I think ISPs on every continent would be interested.
> 

Financial incentive is the key. If there is none, those with the most to
gain (backbone operators) also have to power to create such incentive.
It wouldn't be fundamentally different from the basic network policing
that happened on the academic networks which formed the Internet
backbone in the 80s and early 90s.

Keywords:
=

Work with OS and CPE vendors to include probes with equipment/software.

Create lists of badly behaved prefixes.

Drop offending prefixes from the DFZ.

--

Result: BPC compliance or go the "scenic route" (bust). Problem
solved .. move on.



Problems:
=

Politics. Ill-informed politicians can come up with the most incredible
excuses to protect offenders.

Decide who define the criteria used to identify "offending networks",
and administer the filtering recommendation.

Has to tolerate some "collateral damage".

Widespread misconception of an untouchable "public internet". Such a
thing doesn't exist. The net still consist of interconnected privately
owned networks within which the owner/operator is free to implement and
enforce whatever policies they want. Some countries may require that
customers/users are informed about the existence and consequences of
such restrictions, but that shouldn't be much of a problem either. I'd
be more than happy to tell anyone who object to BCP38 to look elsewhere
for network connectivity.



-- 


Per Heldal - http://heldal.eml.cc/

Re: register.com down sev0?

[Jared Mauch]

On Thu, Oct 26, 2006 at 06:03:54AM +, Fergie wrote:
> 
> Randy,
> 
> I don't think I implied anything of the sort.
> 
> I did, however, pipe up when a BCP is mentioned that I endorse,
> and co-authored -- and likewise, cannot figure out for life of
> me, why there is such push-back from the Ops community on doing
> The Right Thing.

The challenge is that the router vendors still haven't
done "The Right Thing".

I have one device that

1) halves its forwarding table space by enabling u-rpf
2) can only do either strict or loose mode rpf *GLOBALLY* so I can
   not strict rpf-check a static customer AND loose rpf someone
   larger for unrouted space.

because of the above (#1 isn't that bad, but #2 is)
I can't enable u-rpf on the device as a policy.  Changing one
interface from loose -> strict silently changes all other u-rpf
interfaces and then customers gripe about dropped packets.

obviously moving these checks closer to the edge
is ideal, such as always doing rpf on the ethernet lan
interface for your customer CPE.

> Having said that, botnets don't need to spoof addresses -- the
> sheer dispersion of geographic and AS infection base renders the
> whole point of spoofing almost moot.

yup, it's an evolving threat, even if some solution to the
botnet problem is discovered, it will take years to fix.  Think of
the smurf amplifiers that are still out there[1].

- jared

1 - http://www.powertech.no/smurf/

-- 
Jared Mauch  | pgp key available via finger from [EMAIL PROTECTED]
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

re: passports for NANOG-39, Toronto

[Alex Rubenstein]

> You may have heard that the US and Canada are going to start requiring
> passports for air travel between them beginning "soon".  That date is
> currently set as 8 Jan 2007, which is before February NANOG.  MERIT
> has noted this on the web site, but a cursory check of my list
> archives didn't turn up mention of it (sorry if I overlooked it; the
> last couple of weeks have been hectic), so I figured I'd include the
> pointer:

FYI, this date only applies to air or sea (which I imagine is the bulk
of people going). However, for land crossings:

http://travel.state.gov/travel/tips/regional/regional_1170.html

"The Intelligence Reform and Terrorism Prevention Act of 2004 requires
that, by January 1, 2008, travelers to and from the Caribbean, Bermuda,
Panama, Mexico and Canada have a passport or other secure, accepted
document to enter or re-enter the United States."

[...]

The travel initiative requirements will be rolled out in phases.   The
proposed implementation timeline is as follows:

December 31, 2006 - Passport required for all air and sea travel to or
from Canada, Mexico, Central and South America, the Caribbean, and
Bermuda. 

December 31, 2007 - Passport required for all land border crossings, as
well as air and sea travel. 

Re: 10,352 active botnets (was Re: register.com down sev0?)

[Gadi Evron]

On Thu, 26 Oct 2006, Fergie wrote:
> 
> Jose's numbers are conservative.
> 
> Given some mathematical acrobatics, I'd suggest examining some
> of the (shocking) number sin Microsoft's Security Intelligence
> Report (Google it) -- these are reflective: 
> 
> "Of the 4 million computers cleaned by the company's MSRT
> (malicious software removal tool), about 50 percent (2 million)
> contained at least one backdoor Trojan. While this is a high
> percentage, Microsoft notes that this is a decrease from the
> second half of 2005. During that period, the MSRT data showed
> that 68 percent of machines cleaned by the tool contained a
> backdoor Trojan."
> 
> Ref: http://www.eweek.com/article2/0,1759,2036439,00.asp
> 
> If you're wondering why DDoS attacks are so effective, look
> no further than your backyard.
> 
> - ferg

Jose may be a bit conservative with numbers, but he has good data and
shares it, which is more than I can say for some people.

Jose is definitely someone who knows what he is talking about when it
comes to botnets.

These numbers are not really relevant in my opinion, but they help get the
message across.

Gadi.

Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)

[Mikael Abrahamsson]

On Thu, 26 Oct 2006, Fergie wrote:

The point I'm trying to make is that if the community thinks it is 
valuable, then the path is clear.


What is the biggest problem to solve? Would it be enough for ISPs to make 
sure that they will not send out packets which didn't belong within their 
PA blocks, or is it that one user shouldn't be able to spoof at all (even 
IPs adjacant to their own)? Would the global problem go away if global 
spoofing stopped working?


I of course realise that it's best if user cannot spoof at all, but it 
might be easier for ISPs to filter based on their PA blocks than to (in 
some cases) purchase new equipment to replace their current equipment that 
cannot do IP spoof filtering.


--
Mikael Abrahamssonemail: [EMAIL PROTECTED]