Re: BCP38 making it work, solving problems
On 20 Oct 2004, at 21:49, Jon Lewis wrote: On Wed, 20 Oct 2004, Patrick W Gilmore wrote: Have you actually done the work to see how many packets it takes to shut down a session with and without MD5 enabled? (The question is rhetorical, since your post shows that you have not.) Just a bit more sauce for the goose...enabling MD5 on BGP peers under certain latest in their train IOS versions will immediately crash IOS. Guess how I know that? In a similar vein, upgrading from certain flavours of 12.0 to certain flavours of 12.2 seems to cause password hashes to become disfunctional, leaving BGP sessions down after the first reboot until the passwords are re-entered in plain text from the command line. (Guess how I know that, too). This is not a statement in support of not using MD5 auth passwords. Just a reminder to include MD5 passwords in your lab testing before deployment, if you use them. Joe
Re: BCP38 making it work, solving problems
On Wed, 20 Oct 2004, Patrick W Gilmore wrote: > Have you actually done the work to see how many packets it takes to > shut down a session with and without MD5 enabled? (The question is > rhetorical, since your post shows that you have not.) Just a bit more sauce for the goose...enabling MD5 on BGP peers under certain latest in their train IOS versions will immediately crash IOS. Guess how I know that? -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: BCP38 making it work, solving problems
> Date: Wed, 20 Oct 2004 03:33:05 GMT > From: "Fergie (Paul Ferguson)" <[EMAIL PROTECTED]> > To: [EMAIL PROTECTED] > Cc: [EMAIL PROTECTED] > Subject: Re: BCP38 making it work, solving problems > Hmm, so let me figure this one out... Are you arguing against > BCP38 implementation/filtering, or what? *pro* but by showing that without *valid* arguments for not implementing you are plain irresponsible by not implementing... (more or less) > - ferg Regards, JP Velders
Re: BCP38 making it work, solving problems
On Oct 19, 2004, at 1:14 PM, JP Velders wrote: jacking the connection is in a completely different class as someone bombarding you with a bunch of forged BGP packets to close down a session. Without that MD5 checksum you are quite vulnerable to that. I haven't seen a vendor come up with a solution to that, because the problem is on a much more vendor-neutral level... We haven't talked about this in a few months, so what the hell Have you actually done the work to see how many packets it takes to shut down a session with and without MD5 enabled? (The question is rhetorical, since your post shows that you have not.) Back on topic, the MD5 debate is not an exact apples-to-apples comparison of BCP38. Stopping people from shutting down your BGP sessions is not the same as letting your customer hurt me while claiming to be a third party. Put another way, MD5 on BGP sessions is a personal choice per network. I made my decision. You are welcome and encouraged to make your own. Neither will effect the other, except where our two networks meet. (And then I am positive we can come to some mutual understanding.) BCP38 is not a personal decision. Not implementing it hurts the whole Internet, not just your little corner. -- TTFN, patrick
Re: BCP38 making it work, solving problems
> Hmm, so let me figure this one out... Are you arguing against > BCP38 implementation/filtering, or what? no one is arguing against it. they're just trying to tell the religious zealots (who seem not to run large networks) why it is not deploying as rapidly as they might like. randy
Re: BCP38 making it work, solving problems
Hmm, so let me figure this one out... Are you arguing against BCP38 implementation/filtering, or what? - ferg -- JP Velders <[EMAIL PROTECTED]> wrote: In most cases it will go like that, the minority of when it doesn't go like that, you start filtering / whatever, just like we do now. Regards, JP Velders -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet [EMAIL PROTECTED] or [EMAIL PROTECTED]
Re: BCP38 making it work, solving problems
>dropped over it's 25 day uptime: > > RPF Failures: Packets: 34889152, Bytes: 12838806927 > RPF Failures: Packets: 4200, Bytes: 449923 > RPF Failures: Packets: 3066337401, Bytes: 122772518288 > RPF Failures: Packets: 30954487, Bytes: 3272647457 > RPF Failures: Packets: 4707582841, Bytes: 227001949225 > RPF Failures: Packets: 11291931, Bytes: 643099278 > RPF Failures: Packets: 291592413, Bytes: 20642951232 > RPF Failures: Packets: 380355, Bytes: 22616137 > RPF Failures: Packets: 607543, Bytes: 31687907 > RPF Failures: Packets: 0, Bytes: 0 > RPF Failures: Packets: 91, Bytes: 6978 > RPF Failures: Packets: 0, Bytes: 0 > RPF Failures: Packets: 0, Bytes: 0 > RPF Failures: Packets: 2, Bytes: 80 > RPF Failures: Packets: 13904, Bytes: 1093686 > > this means the junk isn't reaching root servers, peers, or >our customers. mitigating the need to carry this traffic when it >is of (virtually) no use. > And those you do see it indicates a misconfigured / compromised system. A compromised system that is sending spoofed traffic can also launch attacks using regular traffic. Think of this as a early warning system. The same with those ISP's that block outbound port 25. Think of it as a early warning system. The customer is misconfigured or compromised. You need to find out which. [This is not to say that I agree with the practice of blocking port 25] Apply the same logic to anything else you filter outbound.
Re: BCP38 making it work, solving problems
On Tue, Oct 19, 2004 at 01:36:18PM +, Paul Vixie wrote: > > > ... the first thing is, some nets who want the internet to work this > > > way have to implement BCP38 in their own corner of the internet. > > > then they have to start de-peering with nets who don't do it, and > > > offer a better rate to customers who do it than to those who don't. > > > then they have to de-peer with anyone who doesn't require their > > > peers and customers to do it. then they have to refuse as customers > > > anyone who won't do it. ... > > > > As it was "in the old days": first clean up your own act and then > > start pointing at others that they're doing "it" wrong. > > > > It's a mentality I see very rarely amongst the larger ISP's who've > > been part of that 'I' in their name since the very early beginning. > > i'm pretty depressed at the lack of self regulation among the techiefolk. > responses to BCP38 of the form "but my customer has a good reason to spoof" > or "but my equipment can't do wire speed SAV" or "but BCP38 will not solve > all DDoS problems single-handedly" are small minded, provincial, and wrong. > ... it's just "but if man was meant to fly he'd have wings" all over again. I've been a strong advocate of getting uRPF enabled on our customer links. So much so, that we've found interesting limitations of some of the routers out there. While, I'd like to enable SAV on all our links, it's just not possible. Some major routing vendors have not re-released their time-to-market hardware with versions that can still do full port density and line-rate, while others have re-spun their hardware in order to support new features at the same densities/costs. These are just a few of the challenges that providers face.. The more I see these days though is non-spoofed attacks, that are semi-sophisticated in nature. *BUT* this doesn't mean that you people that aren't u-rpfing your non-multihomed T1/DSL customers should just ignore the need for u-rpf. It will save you a lot of grief in your networks operationally. No more tracking spoofed DoS attacks from your customers. No forwarding packets with these bogus sources across your expensive links. This will only help you and your customers operate a "clean" network. My employers network isn't perfect, nor do I suspect many peoples are, but SAV filtering/u-rpf is important enough that everyone should be doing it. Just picking on one router, I see many billions of packets dropped over it's 25 day uptime: RPF Failures: Packets: 34889152, Bytes: 12838806927 RPF Failures: Packets: 4200, Bytes: 449923 RPF Failures: Packets: 3066337401, Bytes: 122772518288 RPF Failures: Packets: 30954487, Bytes: 3272647457 RPF Failures: Packets: 4707582841, Bytes: 227001949225 RPF Failures: Packets: 11291931, Bytes: 643099278 RPF Failures: Packets: 291592413, Bytes: 20642951232 RPF Failures: Packets: 380355, Bytes: 22616137 RPF Failures: Packets: 607543, Bytes: 31687907 RPF Failures: Packets: 0, Bytes: 0 RPF Failures: Packets: 91, Bytes: 6978 RPF Failures: Packets: 0, Bytes: 0 RPF Failures: Packets: 0, Bytes: 0 RPF Failures: Packets: 2, Bytes: 80 RPF Failures: Packets: 13904, Bytes: 1093686 this means the junk isn't reaching root servers, peers, or our customers. mitigating the need to carry this traffic when it is of (virtually) no use. - Jared (working to make the packets on my corner of the network a little less messy) -- Jared Mauch | pgp key available via finger from [EMAIL PROTECTED] clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: BCP38 making it work, solving problems
> Date: Tue, 19 Oct 2004 13:36:18 + > From: Paul Vixie <[EMAIL PROTECTED]> > To: [EMAIL PROTECTED] > Subject: Re: BCP38 making it work, solving problems > [ ... ] > > As it was "in the old days": first clean up your own act and then > > start pointing at others that they're doing "it" wrong. > > It's a mentality I see very rarely amongst the larger ISP's who've > > been part of that 'I' in their name since the very early beginning. > i found out from verisign that big companies call something > "innovation" if it hasn't been generally done before and if it > immediately increases revenue. > failing the "immediately increases revenue" part, no big company > will do anything that hasn't been generally done before. And continuing on that road would lead to innovation, how ? > [ ... ] > i'm pretty depressed at the lack of self regulation among the techiefolk. Techiefolk are 'commodity' nowadays. The ones that do care are vastly outnumbered. (which doesn't mean they should stop caring!) > responses to BCP38 of the form "but my customer has a good reason to spoof" > or "but my equipment can't do wire speed SAV" or "but BCP38 will not solve > all DDoS problems single-handedly" are small minded, provincial, and wrong. > ... it's just "but if man was meant to fly he'd have wings" all over again. Thinking outside of the box or about "The Greater Good" is something of a quaint idea nowadays :( Think I'll go dust off my wings... ;D Regards, JP Velders
Re: BCP38 making it work, solving problems
> Date: Tue, 19 Oct 2004 13:20:08 -0400 > From: David G. Andersen <[EMAIL PROTECTED]> > Subject: Re: BCP38 making it work, solving problems > [ ... ] > Unless you're worried about an adversary who taps into your > fiber, how is MD5 checksums any better than anti spoofing filters > that protect your BGP peering sessions? The only benefit I see is > that you can actually verify that your peer is using md5 checksums, > instead of having to take them on faith that they won't permit > someone to spoof their router's address. How much control do 'they' have over the ways 'someone' can spoof ? With large providers who don't see any harm in allowing possibly spoofed traffic through, you cannot exclude the possibility that an ISP connected to an IX "leaks" those spoofed packets onto the IX. (or leaks RFC1918 space... I know of a few examples / mails ;D) In the current world - where you cannot exclude either one - you're much better off 'safe' then 'sorry'... Implementing BCP38 (to come back on-topic) is just plain good neighbourhood policy. I don't go building 2.5 meter tall fences around my house because I don't want my neighbour's plants in my garden. No, we come to an understanding that whenever his plants get out of control in my garden I can cut them back, but that he will also trim them more often. In most cases it will go like that, the minority of when it doesn't go like that, you start filtering / whatever, just like we do now. Regards, JP Velders
Re: BCP38 making it work, solving problems
On Tue, Oct 19, 2004 at 07:14:32PM +0200, JP Velders scribed: > > > Date: Tue, 19 Oct 2004 09:21:46 -0700 > > From: Randy Bush <[EMAIL PROTECTED]> > > Subject: Re: BCP38 making it work, solving problems > > > > For example, how many ISPs use TCP MD5 to limit the possibility of a > > > BGP/TCP connection getting hijacked or disrupted by a ddos attack? > > > i hope none use it for the latter, as it will not help. more and > > more use it for the former. why? becuase they perceived the need > > to solve an immediate problem, a weakness in a vendor's code. > > Uhm, you might need to run that by me again... > > Hijacking the connection is in a completely different class as someone > bombarding you with a bunch of forged BGP packets to close down a > session. Without that MD5 checksum you are quite vulnerable to that. I > haven't seen a vendor come up with a solution to that, because the > problem is on a much more vendor-neutral level... Unless you're worried about an adversary who taps into your fiber, how is MD5 checksums any better than anti spoofing filters that protect your BGP peering sessions? The only benefit I see is that you can actually verify that your peer is using md5 checksums, instead of having to take them on faith that they won't permit someone to spoof their router's address. -Dave -- work: [EMAIL PROTECTED] me: [EMAIL PROTECTED] MIT Laboratory for Computer Science http://www.angio.net/
Re: BCP38 making it work, solving problems
> Date: Tue, 19 Oct 2004 09:21:46 -0700 > From: Randy Bush <[EMAIL PROTECTED]> > Subject: Re: BCP38 making it work, solving problems > > For example, how many ISPs use TCP MD5 to limit the possibility of a > > BGP/TCP connection getting hijacked or disrupted by a ddos attack? > i hope none use it for the latter, as it will not help. more and > more use it for the former. why? becuase they perceived the need > to solve an immediate problem, a weakness in a vendor's code. Uhm, you might need to run that by me again... Hijacking the connection is in a completely different class as someone bombarding you with a bunch of forged BGP packets to close down a session. Without that MD5 checksum you are quite vulnerable to that. I haven't seen a vendor come up with a solution to that, because the problem is on a much more vendor-neutral level... Regards, JP Velders PS: ofcourse that MD5 option also causes problems for peerings to come back "up" again if you have to reboot/reload *without* properly closing them... :( Hey, pro's and con's are part of the job ;)
Re: BCP38 making it work, solving problems
> For example, how many ISPs use TCP MD5 to limit the possibility of a > BGP/TCP connection getting hijacked or disrupted by a ddos attack? i hope none use it for the latter, as it will not help. more and more use it for the former. why? becuase they perceived the need to solve an immediate problem, a weakness in a vendor's code. > Where ingress filters don't help, of course, is when the attacks come from > an apparently-legitimate address. many folk see that this is the vast majority of the cases. hence, one reason for the lack of adoption of rfc 2827 randy
Re: BCP38 making it work, solving problems
> > ... the first thing is, some nets who want the internet to work this > > way have to implement BCP38 in their own corner of the internet. > > then they have to start de-peering with nets who don't do it, and > > offer a better rate to customers who do it than to those who don't. > > then they have to de-peer with anyone who doesn't require their > > peers and customers to do it. then they have to refuse as customers > > anyone who won't do it. ... > > As it was "in the old days": first clean up your own act and then > start pointing at others that they're doing "it" wrong. > > It's a mentality I see very rarely amongst the larger ISP's who've > been part of that 'I' in their name since the very early beginning. i found out from verisign that big companies call something "innovation" if it hasn't been generally done before and if it immediately increases revenue. failing the "immediately increases revenue" part, no big company will do anything that hasn't been generally done before. unless their insurance companies or their government insists, that is. the same cfo's who are choking the life out of your backbone (equipment, staff, etc) are perfectly happy to pay a little more to get UL-listed refrigerators for the company's break rooms. i guess the grownups really are in charge? i'm pretty depressed at the lack of self regulation among the techiefolk. responses to BCP38 of the form "but my customer has a good reason to spoof" or "but my equipment can't do wire speed SAV" or "but BCP38 will not solve all DDoS problems single-handedly" are small minded, provincial, and wrong. ... it's just "but if man was meant to fly he'd have wings" all over again.
Re: BCP38 making it work, solving problems
At 01:11 PM 10/19/04 +0200, JP Velders wrote: As it was "in the old days": first clean up your own act and then start pointing at others that they're doing "it" wrong. hear hear... But Paul knows and in fact does that. He is pointing out the difficulty of getting people to do basic things that are for their own benefit. For example, how many ISPs use TCP MD5 to limit the possibility of a BGP/TCP connection getting hijacked or disrupted by a ddos attack? But this has been in the code since ~1990, and was put there because of a fairly serious and specific attack that was made on Internet routing, and benefits primarily the ISP that enables the procedure in that it knows that its routes are coming to it from systems it has chosen to trust. Ingress filters help the ISP that installs them, in that a certain class of attacks are prevented among customers of the ISP. Would it be better if all ISPs and all edge networks put appropriate filters in place? Absolutely. But even if they do not, the ISP saves itself that much trouble. Where ingress filters don't help, of course, is when the attacks come from an apparently-legitimate address. Then we are off to other tools.
Re: BCP38 making it work, solving problems
> Date: 12 Oct 2004 16:22:17 + > From: Paul Vixie <[EMAIL PROTECTED]> > To: [EMAIL PROTECTED] > Subject: Re: BCP38 making it work, solving problems > [ ... ] > how are we going to get there? the first thing is, some nets who want the > internet to work this way have to implement BCP38 in their own corner of the > internet. then they have to start de-peering with nets who don't do it, and > offer a better rate to customers who do it than to those who don't. then > they have to de-peer with anyone who doesn't require their peers and customers > to do it. then they have to refuse as customers anyone who won't do it. > [ ... ] As it was "in the old days": first clean up your own act and then start pointing at others that they're doing "it" wrong. It's a mentality I see very rarely amongst the larger ISP's who've been part of that 'I' in their name since the very early beginning. Regards, JP Velders
Re: BCP38 making it work, solving problems
On 14-okt-04, at 0:17, Fred Baker wrote: Trusting the source when it says that its packets aren't evil might be sub-optimal. Evaluation of evilness is best left up to the receiver. Likely true. Next question is whether the receiver can really determine that in real time. For some things, yes, but for many things it is not as obvious to me. It would be a very good start not having to receive that which you can identify as something you don't want.
Re: BCP38 making it work, solving problems
On Thu, Oct 14, 2004 at 11:48:24AM +0100, [EMAIL PROTECTED] wrote: > > > At 12:01 PM 10/13/04 +0200, Iljitsch van Beijnum wrote: > > >Trusting the source when it says that its packets aren't evil might be > > >sub-optimal. Evaluation of evilness is best left up to the receiver. > > > > Likely true. Next question is whether the receiver can really determine > > that in real time. For some things, yes, but for many things it is not > as > > obvious to me. > > Correct me if I'm wrong here, but my interpretation of this > suggestion was not that we should trust the source to mark > packets but that we should trust our peers to mark packets. ... > > This doesn't mean that the non-evil bit is the only way, > but the idea of network operators marking traffic in some > way to indicate their level of confidence in its normality > seems to be worth pursuing. It seems to be the natural > progression of projects like the selection found at > cymru.com. > > --Michael Dillon ah ... so you have no problems with me marking your packets anyway I choose, right? i suspect that a single tagging scheme will be too prone to abuse and that it will be important to have/allow the source to indicate its preferences. i am reminded of one ISP announcing 128.0.0.0/3 some time back based on the presumption that it could deliver any packet to the correct destination in that range. ... :) --bill
Re: BCP38 making it work, solving problems
> At 12:01 PM 10/13/04 +0200, Iljitsch van Beijnum wrote: > >Trusting the source when it says that its packets aren't evil might be > >sub-optimal. Evaluation of evilness is best left up to the receiver. > > Likely true. Next question is whether the receiver can really determine > that in real time. For some things, yes, but for many things it is not as > obvious to me. Correct me if I'm wrong here, but my interpretation of this suggestion was not that we should trust the source to mark packets but that we should trust our peers to mark packets. This seems to be something that is workable since most people have a manageable number of peers. Presumably each peer could mark the traffic based on what they know about their customer's network. If a customer follows all best practices, they mark it with the non-evil bit, otherwise not. If truly evil traffic is coming in from a peer, then one could apply mitigating actions only to traffic that is not marked non-evil, either blackholing it all or diverting it to a router that will perform complex filtering or heavily rate limiting it. It seems to me that really addressing DDOS, botnets, etc., requires network operators to agree on some sort of common coordinated action and using a network protocol to communicate about this coordinated action would be very useful. This doesn't mean that the non-evil bit is the only way, but the idea of network operators marking traffic in some way to indicate their level of confidence in its normality seems to be worth pursuing. It seems to be the natural progression of projects like the selection found at cymru.com. --Michael Dillon
Re: BCP38 making it work, solving problems
At 12:01 PM 10/13/04 +0200, Iljitsch van Beijnum wrote: Trusting the source when it says that its packets aren't evil might be sub-optimal. Evaluation of evilness is best left up to the receiver. Likely true. Next question is whether the receiver can really determine that in real time. For some things, yes, but for many things it is not as obvious to me.
Re: BCP38 making it work, solving problems
On 13 Oct 2004, Paul Vixie wrote: > > >How many people have seen "forged" spoofed IP addresses being used for DOS > > >attacks lately? > > syn-flood protection, and random TCP ISS, are now common enough that > spoofed-source isn't effective for TCP flows. if you want to bring down > somebody's web server then blackhats really do have to use real addresses. of course the docs were written a couple years ago, and things have changed a lot in that time. the proliference of and ease of establishing bot networks is such that their controllers dont care if you track them and shut them down as they are easily replaced Steve
Re: BCP38 making it work, solving problems
> >How many people have seen "forged" spoofed IP addresses being used > >for DOS attacks lately? syn-flood protection, and random TCP ISS, are now common enough that spoofed-source isn't effective for TCP flows. if you want to bring down somebody's web server then blackhats really do have to use real addresses. however, if you just want to make their web server unreachable, then you can either overload their DNS infrastructure or just congest their upstreams, and you don't need to use real addresses for that. i've never seen a dns attack that didn't have 50% or more packets coming from spoofed sources, though due to loose-mode uRPF, most spoofed sources in the last year or so have been from addresses for which a route exists. -- Paul Vixie
Re: BCP38 making it work, solving problems
> Some medium-big (and up) operators implement 'RFC-1918 source' filters on > their gateways to the 'external internet', ... maybe so. but i want to renew an old complaint, with updated numbers: # ipfw show ... 00400 576234043927757977 deny ip from 10.0.0.0/8 to any in 00500 13930 11937839356 deny ip from 172.16.0.0/12 to any in 00600 358603542414549719 deny ip from 192.168.0.0/16 to any in 00700 6041138223 399700558975 pipe 1 udp from any to any dst-port 53 in 00800 297572981264366050 pipe 2 tcp from any to any dst-port 53 in ... those are rule#, packets, octets, and rule. on this f-root server which is one of about 50 although its routing placement causes it to hear more requests, usually, than most of the others, we see that since last reboot: 233M packets (18G octets) came from an RFC1918 source 6G packets (401G octets) came from non-RFC1918 sources i think it's fair to say that filtering RFC1918 source addresses on egress toward "the core" is not all that common. although i'm happy to report that both AOL and SBC now do it, and the above numbers havn't gotten as much worse as i expected they would in the time since my last complaint. # tcpdump -n -s 0 -c 10 udp port 53 and \ src net \( 10.0.0.0/8 or 172.16.0.0/12 or 192.168.0.0/16 \) ... 16:45:06.164258 192.168.0.1.1030 > 192.5.5.241.53: \ 12540 A? x1.w00pie.nl. (30) 16:45:06.181960 172.16.160.2.53 > 192.5.5.241.53: \ 7929 A? NPIFF12B7.k12.nj.us. (37) 16:45:06.182641 10.8.1.16.1051 > 192.5.5.241.53: \ 15860 [1au] PTR? 98.239.71.156.in-addr.arpa. (55) 16:45:06.320755 192.168.230.161.1041 > 192.5.5.241.53: \ 3026 A? rsthost3.ods.org.cncc.com. (43) 16:45:06.339133 172.20.0.13.53 > 192.5.5.241.53: \ 27425 A? infopak.gov.pk. (32) 16:45:06.378595 10.1.100.51.53 > 192.5.5.241.53: 3217 A? PC0737. (28) 16:45:06.394286 10.8.1.11.53 > 192.5.5.241.53: 8457 A? www.office.ac. (31) 16:45:06.394933 10.8.1.11.53 > 192.5.5.241.53: 6422 A? www.office.ac. (31) 16:45:06.395310 10.8.1.11.53 > 192.5.5.241.53: 288 A? www.office.ac. (31) 16:45:06.395803 10.8.1.11.53 > 192.5.5.241.53: 246 A? www.office.ac. (31) ... i've been wondering if it would be possible to track down these sources by looking at the query-names. i've also been wondering if ISC should have a peering agreement requiring peers to implement BCP38. -- Paul Vixie
Re: BCP38 making it work, solving problems
on Wed, Oct 13, 2004 at 07:09:10AM +0530, Suresh Ramasubramanian wrote: > > [EMAIL PROTECTED] [12/10/04 13:16 -0400]: > > > > > If I, and my little 7-man company, can afford to have me solve the > > > problem on our end, why the heck can't you do the same? > > > > You can do it because you are a 7-man company. So can I. However, companies > > the size of Sprint cannot do it. > > > > Most filtering that I've seen (email, router, whatever) that just works great > for a 7 man company will not work when you serve several million users, > that's a fact. A 7 man Web app development company with ~100 or so hosted POP mail accounts, FYI. Not that it matters. If I can write rules that block spam with forged headers, so can any damn body else. And I'm tired of getting the bounces from those who feel it's not possible. Some examples of headers from mail that other ISPs have felt compelled by their size to accept and then bounce back to me: Received: from dslam129-213-59-62.adsl.zonnet.nl (62.59.213.129) by 0 with SMTP; 27 Aug 2004 21:20:16 - Received: from thewordfactory.com (mail.thewordfactory.com [216.27.21.211]) by dslam129-213-59-62.adsl.zonnet.nl (Postfix) with ESMTP id 0453B70AB1 for <[EMAIL PROTECTED]>; Fri, 27 Aug 2004 15:29:58 -0500 The second Received header is forged. The first is from a dynamic DSL line. The message was accepted by mail.philonline.com ([203.177.71.7]) and bounced back to me in a message that didn't even have a Message-ID header, letting me know they are so dumb they accept forged mail from dynamic IPs and only then analyze it to see if the user exists or if the forged sender should be notified. Received: from ip84-RND_DIGIT[2-3]-RND_DIGIT[2-3]-RND_DIGIT[2-3].4tvMsWC8okzw.customer.aviamail.zzn.com (50-RND_DIGIT[2-3]-RND_DIGIT[2-3]-RND_DIGIT[2-3].customer.aviamail.zzn.com [24.135.29.42]) by ezEG4GA1.aviamail.zzn.com (RND_DIGIT[1-3].RND_DIGIT[1-3].RND_DIGIT[1-3]/RND_DIGIT[1-3].RND_DIGIT[1-3].RND_DIGIT[1-3]) with SMTP id B3tc6UKcaTVY This was in a bounce from mail.cygentech.com (geoanalysis36.cust.viawest.net [216.150.205.36]). We've been seeing these headers for more than a year now, and blocking them almost as long. But these idiots can't see that backscattering them at me is rather stupid. Just a few of the others who've done this (accepted mail with completely bogus RNDizer headers from broken spamware): plesk4.merkury.com (216-86-139-44.tns.net [216.86.139.44] (may be forged)) smtp03.adnc.com (smtp03.adnc.com [206.251.248.23]) cobble.phpwebhosting.com (cobble.phpwebhosting.com [64.65.61.211]) date.marketorder.com ([64.65.150.229]) (by way of postini) exchgkom.TRI.NET (mail.tecolote.com [65.170.33.20]) DNS2.TCBINC.NET (DNS2.TCBINC.NET [64.124.116.30]) mail-in-01.netsonic.net (mail-in-01.netsonic.net [66.180.172.253]) ms1.tcnoc.com (ms1.tcnoc.com [63.150.10.30]) exchange3.corp.bcs-tx.com (ip204.bcs-tx.com [216.136.93.204] (may be forged)) [...etc...] My full list of backscatter hosts is ~17K entries long. And those are just the ones who've hit my servers. Never mind the charter/rr/att/hotmail/etc. hosts that are also listed - I'm not just talking about random Exchange boxes here - I'm talking about every major ISP with which I am familiar. Want to know if you're responsible for backscatter abuse? Just ask. As you know, Suresh, Outblaze does the same thing. Listed here as hosts we no longer accept null sender mail from: mta1-1.us4.outblaze.com BOUNCER spf0.us4.outblaze.com BOUNCER spf10.us4.outblaze.com BOUNCER spf1.us4.outblaze.com BOUNCER spf2.us4.outblaze.com BOUNCER spf4-1.us4.outblaze.com BOUNCER spf5-1.us4.outblaze.com BOUNCER spf7-17.us4.outblaze.comBOUNCER At least you've said you're moving towards fixing it. Thanks. > One false positive report per week from 7 users. How many per week - or per > day - when you have 40 million users, is a question that gets answered real > fast. I don't even want to go into the backscatter messages that show that: - the sender forged the IP/domain of the recipient into HELO/EHLO - the recipient accepts mail for unknown accounts - the relaying host forwarded Webmail from Nigeria without proper Received headers added for blocking purposes - you name the obvious spamsign Come on, there is so much obvious crud that should be checked that isn't being checked - we could reduce backscatter by a third or more simply by refusing during SMTP handshake messages from hosts that forge the receipient IP/hostname/domain in HELO, or to users that don't exist, or if virus filters were clueful enough to drop, rather than emit DSNs, for known virus-originated messages that always forge the sender. > A lot of the bad filtering (or lack of filtering, for that matter) decisions > I've seen at large network providers and ISPs is generally where they are > also unresponsive to their users and to the internet community that reports > stuff to them (quite a few places I could name where most role acco
Re: BCP38 making it work, solving problems
>> any idea why someone(s) is ddosing dark space? seems a bit silly. > No one is DDOSing dark space. The dark space telescope picks up the > "richochets" caused by DDOS. sorry. i guess i should give up getting more sleep and go make coffee. randy
Re: BCP38 making it work, solving problems
Randy Bush wrote: For the week starting Sept 12, our dark space telescope "saw" 1675 spoofed DDOS attacks. any idea why someone(s) is ddosing dark space? seems a bit silly. Something like this I rather fancy ... http://lists.planet-lab.org/pipermail/announce/2004-April/12.html [Planetlab-announce] network telescope Larry Peterson llp at CS.Princeton.EDU Thu Apr 15 16:31:24 EDT 2004 * Previous message: [Planetlab-announce] Cambridge meeting * Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] I have a request to make of PlanetLab sites... There is an effort underway to collect information about worms, ddos backscatter, and other suspect activity on the Internet. Our ability to do this sort of analysis would be greatly enhanced by having access to IP address blocks spread across the Internet, for example, at as many PlanetLab sites as possible. My specific request is for sites to contribute blocks of, say, 16 otherwise unused addresses to PlanetLab. I have attached a note from Vern Paxson outlining the idea, a so called "network telescope". Please let me know if your site would be willing to contribute (assign) some number of addresses to PlanetLab for this purpose. Larry -- A basic challenge for analyzing Internet-scale malicious phenomena such as worms and automated scanning is acquiring sufficiently broad visibility into their workings. Monitoring at a single location may for example miss the early stages of a worm's spread or, more generally, lack the diverse perspectives necessary for capturing large-scale behavior. A powerful tool for acquiring such broader visibility is a "network telescope". Network telescopes monitor traffic sent to communication dead-ends such as unallocated portions of the IP address space or ports on endhosts for which no server is listening. Since there is no legitimate reason for a host to send packets to those destinations, such traffic provides strong evidence of malicious activity - including DDoS backscatter, port scanning, and probe activity from worms. Our goal is to build a large-scale telescope with significantly more sampling breadth and diversity than current telescopes. This telescope will be structured as two layers. Its front-end sensors will be spread across a large number of address blocks and monitoring points to achieve sampling diversity. We'll use both unallocated address blocks (which attackers can learn about fairly easily) but also unused subblocks within allocated blocks. This latter "dark address space" is much more difficult for an attacker to learn about and also enables highly diverse distribution of the sensors. It's this latter that we're hoping can be done in conjunction with PL nodes. In particular, the way we picture it working is that the PL nodes will have multiple addresses assigned to them. A monitor running on the host then tunnels traffic it receives on the extra addresses over to the analysis point. It could also tunnel traffic sent to its "normal" address but for which there's no listener. One crucial issue with building a large telescope is *filtering*. For a very large telescope, the volume of data collected can be enormous. However, for many forms of analysis we can often filter out a great deal of the traffic. For example, for worm detection we can drop traffic seen by the sensor rather than forwarding it if the traffic does not correspond to worm activity of possible interest (e.g., it's instead DoS backscatter, or activity from known worms). Because PL nodes can do computation before they forward traffic over the tunnel (unlike, for example, telescope sensors based on using routers), they make ideal platforms for developing such filtering.
Re: BCP38 making it work, solving problems
At 04:59 AM 13-10-04 -0700, Randy Bush wrote: > For the week starting Sept 12, our dark space telescope "saw" > 1675 spoofed DDOS attacks. any idea why someone(s) is ddosing dark space? seems a bit silly. No one is DDOSing dark space. The dark space telescope picks up the "richochets" caused by DDOS. CAIDA created a nice 90 second video clip explaining this at: http://www.caida.org/outreach/resources/animations/passive_monitoring/backscatter.mpg -Hank randy
Re: BCP38 making it work, solving problems
> For the week starting Sept 12, our dark space telescope "saw" > 1675 spoofed DDOS attacks. any idea why someone(s) is ddosing dark space? seems a bit silly. randy
Re: BCP38 making it work, solving problems
On 12-okt-04, at 7:30, Fred Baker wrote: From an ISP perspective, I would think that it would be of value to offer *not* ingress filtering (whether by ACL or by uRPF) as a service that a customer pays for. So what is our collective position on ISPs filtering their peers? Both the position that this should be done as there are too many clueless peers and the position that it shouldn't as it breaks too much legitimate stuff (especially possible future stuff such as the multiaddress multihoming for IPv6) are reasonable. We need to agree on one or the other, though: half the net doing one and the other half doing the other won't make anyone happy. Steve Bellovin wrote an April Fool's note suggesting an "Evil Bit" (ftp://ftp.rfc-editor.org/in-notes/rfc3514.txt); I actually think that's not such a dumb idea if implemented as a "Not Evil" flag, using a DSCP or extending the RFC 3168 codes to include such, as Steve Crocker has been suggesting. Basically, a customer gets ingress filtered (by whatever means) and certain DSCP settings are treated as "someone not proven to have their act together". Should a ddos happen, such traffic is dumped first. But if the customer pays extra, their traffic is marked "not evil", protected by the above, and ingress filtering may be on or off according to the relevant agreement. I would much rather see a solution where ISPs rate limit their customers except for flows for which the customer can present a token that shows the recipient actually wants to receive the traffic, or the recipient gets to send a message to shut up the flow. This should solve the (D)DoS thing very nicely, although it does require both ends to cooperate and it requires customer facing stuff to look fairly deep into packets. Address spoofing is just one part of the ddos problem; to nail ddos, we also need to police a variety of application patterns. One reason I like the above is that it gives us a handle on what traffic might possibly be "not evil" - someone has done something that demonstrates that it is from a better managed source. Trusting the source when it says that its packets aren't evil might be sub-optimal. Evaluation of evilness is best left up to the receiver.
Re: BCP38 making it work, solving problems
At 09:50 AM 12-10-04 -0700, Bora Akyol wrote: How many people have seen "forged" spoofed IP addresses being used for DOS attacks lately? For the week starting Sept 12, our dark space telescope "saw" 1675 spoofed DDOS attacks. No idea how many non-spoofed attacks took place during the same time period. -Hank Bora
Re: BCP38 making it work, solving problems
> From [EMAIL PROTECTED] Tue Oct 12 20:41:45 2004 > Date: Wed, 13 Oct 2004 07:09:10 +0530 > From: Suresh Ramasubramanian <[EMAIL PROTECTED]> > To: [EMAIL PROTECTED] > Cc: Steven Champeon <[EMAIL PROTECTED]>, [EMAIL PROTECTED] > Subject: Re: BCP38 making it work, solving problems > > > [EMAIL PROTECTED] [12/10/04 13:16 -0400]: > > > > > If I, and my little 7-man company, can afford to have me solve the > > > problem on our end, why the heck can't you do the same? > > > > You can do it because you are a 7-man company. So can I. However, companies > > the size of Sprint cannot do it. > > > > Most filtering that I've seen (email, router, whatever) that just works great > for a 7 man company will not work when you serve several million users, > that's a fact. Certain _basics_ *are* applicable, regardless of scale. e.g. perimeter filtering of inbound packets w/ RFC-1918 a _source_ address, except for specific ICMP status/response messages. e.g. perimeter filtering of inbound packets with a _source_ address that is in *your* assigned address-space. Some medium-big (and up) operators implement 'RFC-1918 source' filters on their gateways to the 'external internet', but *not* on their customer interfaces. Which means that one of their customers can be attacked via such means, by *another* of their customers. And, after the fact, they can't even tell =which= of their customers done the deed. Similarly, one customer can 'spoof' another customer of that same provider. > One false positive report per week from 7 users. How many per week - or per > day - when you have 40 million users, is a question that gets answered real > fast. > > A lot of the bad filtering (or lack of filtering, for that matter) decisions > I've seen at large network providers and ISPs is generally where they are > also unresponsive to their users and to the internet community that reports > stuff to them (quite a few places I could name where most role accounts seem > to funnel straight to /dev/null) > > srs >
Re: BCP38 making it work, solving problems
[EMAIL PROTECTED] [12/10/04 13:16 -0400]: > > > If I, and my little 7-man company, can afford to have me solve the > > problem on our end, why the heck can't you do the same? > > You can do it because you are a 7-man company. So can I. However, companies > the size of Sprint cannot do it. > Most filtering that I've seen (email, router, whatever) that just works great for a 7 man company will not work when you serve several million users, that's a fact. One false positive report per week from 7 users. How many per week - or per day - when you have 40 million users, is a question that gets answered real fast. A lot of the bad filtering (or lack of filtering, for that matter) decisions I've seen at large network providers and ISPs is generally where they are also unresponsive to their users and to the internet community that reports stuff to them (quite a few places I could name where most role accounts seem to funnel straight to /dev/null) srs
Re: BCP38 making it work, solving problems
> If I, and my little 7-man company, can afford to have me solve the > problem on our end, why the heck can't you do the same? You can do it because you are a 7-man company. So can I. However, companies the size of Sprint cannot do it. Alex
Re: BCP38 making it work, solving problems
On Tue, 12 Oct 2004, Bora Akyol wrote: > Excerpt from the text quoted above: > >2.3. For a DDoS attack to succeed more than once, the launch points must >remain anonymous. Therefore, forged IP source addresses are used. From >the victim's point of view, a DDoS attack seems to come from everywhere >at once, even from many IP addresses that are unallocated or otherwise >invalid. > > How many people have seen "forged" spoofed IP addresses being used > for DOS attacks lately? it does still happen... I've not run the numbers for our reactions to say '50% spoofed/50% non-spoofed' but it certainly seems like 'more' are non-spoofed lately. This could be a simple swing of the pendulum, or other 'better' things like more people egress filtering. -Chris
Re: BCP38 making it work, solving problems
On Oct 12, 2004, at 12:50 PM, Bora Akyol wrote: 2.3. For a DDoS attack to succeed more than once, the launch points must remain anonymous. Therefore, forged IP source addresses are used. From the victim's point of view, a DDoS attack seems to come from everywhere at once, even from many IP addresses that are unallocated or otherwise invalid. How many people have seen "forged" spoofed IP addresses being used for DOS attacks lately? Not saying that I have not see non-forged DoS attacks too, or even which is more common, just saying they exist, are happening today, and cause non-trivial problems for some providers. From my _personal_ experience (not my company, not a scientific sampling), it appears non-spoofed sources are a bigger problem. But ignoring spoofed sources would be a mistake, IMHO. -- TTFN, patrick
Re: BCP38 making it work, solving problems
on Tue, Oct 12, 2004 at 12:49:54PM -0400, [EMAIL PROTECTED] wrote: > This is just the cold blooded economic reality. The same reality which > dictates that only smaller companies can enfore strict anti-spam policies, > and prevent their customers from behaving badly. You want cold-blooded economic reality? I'm nominally a consultant, programmer, writer, and sysadmin - in that order. Over the past two years, I've spent over half of my time dealing with spam that I would never see if the rest of you had your acts together. You're costing me money and detracting from my ability to earn my keep. And yet, over those past two years, I've cut my spam load inbound by 98-99% with very few FPs (on the order of one report per week, which is far less time than I spend proactively blocking unwanted mail from badly policed networks). I taught myself the dark art of writing sendmail rulesets, built a database of ~7K rDNS naming conventions, ~18K poorly configured mail sources that inundate my servers with bounces from mail I didn't send, ~1.7K legitimate mail hosts that have relayed spam, 419 advance fee fraud scams, phishing frauds, or otherwise so utterly failed to police your own network egress points that we've blocked mail from you altogether. Where are the rest of you and why aren't you doing this? If I, and my little 7-man company, can afford to have me solve the problem on our end, why the heck can't you do the same? If I can do it, as the lone admin in a 7-man company, I should think that anyone on this list could do it. But I don't see it. All I see is arguing over whether or not to implement this or that BCP, or this or that antispam measure, or this or that anti-backscatter measure, or this or that antivirus measure. Get on with it. Email is dying, if it isn't already dead. The killer app isn't the Web, it's email. If email dies, who will want Internet services? You'll all be out pounding pavement and wishing you'd had the guts to do what you know is right and necessary. It's not possible for anyone here to say they didn't see it coming. The Cassandras have been screaming loudly, and often clearly, for years. Profoundly disappointed in the utter moral bankruptcy and cowardice of the NANOG constituency, Steve -- join us! http://hesketh.com/about/careers/web_designer.html join us! hesketh.com/inc. v: +1(919)834-2552 f: +1(919)834-2554 w: http://hesketh.com join us! http://hesketh.com/about/careers/account_manager.htmljoin us!
Re: BCP38 making it work, solving problems
> uPRF is only one of several ways to implement BCP38. you could do it with > contracts and reverse-SLA's and thus no technology (on your side) at all: > demand that a customer pay 10X his bill, or $1.00 per packet, whichever is > lower, if they emit packets with source addresses no explicitly named in > the contract. why pay for expensive upgrades on your end of the link, when > all you really care about is that BCP38's rules are observed? No reasonably sized provider is going to do that. There is too much competition, most of which is based on price. Until the companies creating the price pressure die (as in die completely, not re-emerge under a new, slightly different name), there is going to be no financial insentive for anyone to spend money improving their network. Let me underline, I am not talking about smaller ISPs, smaller networks or smaller service providers. > that is of course good news. but it demonstrates a pitfall in CFO-think, > which is the belief that participating in assymetric cost:benefit efforts > (where uunet bears the cost of an upgrade in order for all the non-uunet > parts of the internet to get the benefit of less spoofed traffic, and the > abuse incident costs don't drop nearly enough to pay for the upgrades) is > essentially a selfless act. Rubbish. CFO speak is what keeps the companies alive. Engineer-speak typically lands the company in chapter 11. Companies in Chapter 11 have too many operational decisions dictated by the courts, and those that think CFO speak would greally hate to hear courts on the topic. > we all want cleaner ddos flows. when we get ddos'd, we want to be able to > look at the source addresses, look 'em up in whois, and call the launch-isp, > and get things stopped. we want to be able to turn on flow shaping and > know that an attacker can't cause us to use an arbitrarily large number of > buckets. we *all* want these things. even the bad guys, who are often the > victim of ddos attacks by other bad guys, want these things. It is possible that _nanog_ subscribers want this. I am not quite clear how one can make that generalization about those behind kor.net, those in .ru, .ua etc. Finally, > how are we going to get there? the first thing is, some nets who want the > internet to work this way have to implement BCP38 in their own corner of the > internet. then they have to start de-peering with nets who don't do it, and > offer a better rate to customers who do it than to those who don't. then > they have to de-peer with anyone who doesn't require their peers and customers > to do it. then they have to refuse as customers anyone who won't do it. Last time I checked it was 2004, not 1998. The companies are financed by revenues that they generate, not IPOs or VCs based on a promise of enormous payoff sometime down the road. Cash is the king. > it's all very simple, and it's inevitable. you and your CFO's have a couple > of choices to make. first thing is, do you want the insurance companies, > government regulatory agencies, and ISO9000 people to be making these rules > or do you want to make them at the technical and business level? Yes, I do. This will level playing field and hopefully force a few of the big networks out of business completely, decreasing price pressure on this service. A drop in the price pressure will create an opportunity for those companies to spend the money (should they want to or be forced to) to be better internet citizens. This is just the cold blooded economic reality. The same reality which dictates that only smaller companies can enfore strict anti-spam policies, and prevent their customers from behaving badly. Alex
Re: BCP38 making it work, solving problems
On 10/12/04 9:06 AM, "Paul Vixie" <[EMAIL PROTECTED]> wrote: > >> There is, of course, the issue of multihomed networks, or networks that >> have satellite connectivity etc emitting spoofed source packets. > > y'know, SAC004 (http://www.icann.org/committees/security/sac004.txt) is > only four pages long, and one of those is references. you should read it > before you call multihomed networks an "issue" wrt BCP38 deployment. in > fact, you should read it, and BCP38, and BCP84, before participating in > this discussion at all, either here, or at the bar-bofs next week. Excerpt from the text quoted above: 2.3. For a DDoS attack to succeed more than once, the launch points must remain anonymous. Therefore, forged IP source addresses are used. From the victim's point of view, a DDoS attack seems to come from everywhere at once, even from many IP addresses that are unallocated or otherwise invalid. How many people have seen "forged" spoofed IP addresses being used for DOS attacks lately? Bora
Re: BCP38 making it work, solving problems
> ... > Example: > > the 'chris.net' network is a customer of MCI, his customer "bakker.net". > 'bakker.net' decides 'chris.net' has priced transit cheaply this > year/month/day and choses not to accept traffic from 'chris.net' but send > all outbound traffic through 'chris.net'. 'chris.net' never seens routes > for the sources sending this traffic, yet passes it along to the upstream, > which also has no routes for 'bakker.net' via 'chris.net'. uPRF is only one of several ways to implement BCP38. you could do it with contracts and reverse-SLA's and thus no technology (on your side) at all: demand that a customer pay 10X his bill, or $1.00 per packet, whichever is lower, if they emit packets with source addresses no explicitly named in the contract. why pay for expensive upgrades on your end of the link, when all you really care about is that BCP38's rules are observed? > Regardless, the point here is: "Things seem like they may be getting > better, as 'security' requirements are now firmly being included into new > equipment purchases." that is of course good news. but it demonstrates a pitfall in CFO-think, which is the belief that participating in assymetric cost:benefit efforts (where uunet bears the cost of an upgrade in order for all the non-uunet parts of the internet to get the benefit of less spoofed traffic, and the abuse incident costs don't drop nearly enough to pay for the upgrades) is essentially a selfless act. we all want cleaner ddos flows. when we get ddos'd, we want to be able to look at the source addresses, look 'em up in whois, and call the launch-isp, and get things stopped. we want to be able to turn on flow shaping and know that an attacker can't cause us to use an arbitrarily large number of buckets. we *all* want these things. even the bad guys, who are often the victim of ddos attacks by other bad guys, want these things. how are we going to get there? the first thing is, some nets who want the internet to work this way have to implement BCP38 in their own corner of the internet. then they have to start de-peering with nets who don't do it, and offer a better rate to customers who do it than to those who don't. then they have to de-peer with anyone who doesn't require their peers and customers to do it. then they have to refuse as customers anyone who won't do it. it's all very simple, and it's inevitable. you and your CFO's have a couple of choices to make. first thing is, do you want the insurance companies, government regulatory agencies, and ISO9000 people to be making these rules or do you want to make them at the technical and business level? (this is called "industry self regulation" and it's an either-or proposition.) second choice to make is, do you want to do this early enough that you can fold it into your normal purchase/retire cycle, or do you want it rammed down your throat later in an irritating, short-timeframe, expensive, embarrassing way? fergie complained about the lack of chutzpah among a community who used to be distinguishable from other technical communities by that very chutzpah. i agreed but pointed out that the same engineers are here, but with vastly fewer of their buddies to help out, and vastly more supervision from the CFO than used to be the case. HOWEVER, there is still an opportunity to show some leadership, and GET THIS DONE without waiting for fireman's fund and a bunch of ISO9000 wonks to have to recognize it in a corporate risk profile. -- Paul Vixie
Re: BCP38 making it work, solving problems
Paul Vixie wrote: There is, of course, the issue of multihomed networks, or networks that have satellite connectivity etc emitting spoofed source packets. y'know, SAC004 (http://www.icann.org/committees/security/sac004.txt) is only four pages long, and one of those is references. you should read it before you call multihomed networks an "issue" wrt BCP38 deployment. in fact, you should read it, and BCP38, and BCP84, before participating in this discussion at all, either here, or at the bar-bofs next week. Multihomed networks are not an issue. Stupid implementations of these? Yes sure. srs
Re: BCP38 making it work, solving problems
> There is, of course, the issue of multihomed networks, or networks that > have satellite connectivity etc emitting spoofed source packets. y'know, SAC004 (http://www.icann.org/committees/security/sac004.txt) is only four pages long, and one of those is references. you should read it before you call multihomed networks an "issue" wrt BCP38 deployment. in fact, you should read it, and BCP38, and BCP84, before participating in this discussion at all, either here, or at the bar-bofs next week. -- Paul Vixie
Re: BCP38 making it work, solving problems
On Tue, 12 Oct 2004, Niels Bakker wrote: > > * [EMAIL PROTECTED] (Christopher L. Morrow) [Tue 12 Oct 2004, 05:18 CEST]: > > a common occurance we've seen is a customer of a customer NOT > > announcing , nor planning on announcing, their routes to their > > upstream#1 which they use ONLY for outbound traffic (cheap transit for > > instance, and perhaps only for some portions of their total sources) > > though they announce to upstreams#2-N the proper sources to gather the > > return traffic. These things make uRPF 'difficult'. > > You could use uRPF-loose there, or the customer could do: > > ! > route-map outbound-only permit 10 > match prefix-list myprefixes > set community no-export > ! this does not address the problem, the customer's customer isn't announcing routes for this traffic so there is nothing to no-export :( Example: the 'chris.net' network is a customer of MCI, his customer "bakker.net". 'bakker.net' decides 'chris.net' has priced transit cheaply this year/month/day and choses not to accept traffic from 'chris.net' but send all outbound traffic through 'chris.net'. 'chris.net' never seens routes for the sources sending this traffic, yet passes it along to the upstream, which also has no routes for 'bakker.net' via 'chris.net'. Regardless, the point here is: "Things seem like they may be getting better, as 'security' requirements are now firmly being included into new equipment purchases."
Re: BCP38 making it work, solving problems
* [EMAIL PROTECTED] (Christopher L. Morrow) [Tue 12 Oct 2004, 05:18 CEST]: > a common occurance we've seen is a customer of a customer NOT > announcing , nor planning on announcing, their routes to their > upstream#1 which they use ONLY for outbound traffic (cheap transit for > instance, and perhaps only for some portions of their total sources) > though they announce to upstreams#2-N the proper sources to gather the > return traffic. These things make uRPF 'difficult'. You could use uRPF-loose there, or the customer could do: ! route-map outbound-only permit 10 match prefix-list myprefixes set community no-export ! And bash the people who, in this age, don't have "neighbor x.y.z.a send-community" on all their BGP sessions. -- Niels (who recently had a CCIE claim that he was "not aware of a single ISP accepting communities from its peers" - well, my experience begs to differ, with his employer a rare and lonely exception to the rule)
Re: BCP38 making it work, solving problems
At 08:39 AM 10/12/04 +0530, Suresh Ramasubramanian wrote: Yes I know that multihoming customers must make sure packets going out to the internet over a link match the route advertised out that link .. but stupid multihoming implementations do tend to ensure that lots of people will yell loudly, and yell loudly enough for several tickets to be escalated well beyond tier 1 NOC support desks, for ISPs to kind of think twice before they put uRPF filters in .. You might want to take a glance at RFC 3704, which looks at a number of the issues that have been raised in this thread, including the routing of traffic to appropriate enterprise egress points. In my heart of hearts, I would like enterprises to (as a default) match layer 2 and layer 3 addresses on the originating LAN, and quarantine-as-busted any machine that sends an address other than assigned on an interface. It seems that the few cases where a device legitimately sends multiple addresses are exception cases that can be handled separately. Handling it that close to the source solves the problem for everyone. Practically, that is difficult. If you think getting all of the service providers (who wind up having to fix ddos attacks, and pay for bandwidth and services related to ddos attacks) to manage networks well is difficult, consider the prospect of getting all the edge networks to do so... As simple solution is, as someone suggested, pose an idiot tax and bill the customers for doing stupid things. Egress traffic filtering in the enterprise is relatively simple for the average enterprise - it has at most a few prefixes and can write a simple ACL on its upstream router. It can use the ACL either to discard offending packets or to route them to the right egress. It is also relatively simple for the average enterprises' ISP: it knows what prefix(es) it agreed to accept traffic from and can write an ACL. It gets a little dicier when the customer is a lower tier ISP. In that case, there are potentially many prefixes, and they change more frequently. That is the argument for something like uRPF. No, it is not a "sure fix", but it handles that case more readily, both in the sense of being a fast lookup and in the sense of maintaining the table. The problem is, of course, in the asymmetry of routing - it has to be used with the brain engaged. From an ISP perspective, I would think that it would be of value to offer *not* ingress filtering (whether by ACL or by uRPF) as a service that a customer pays for. Steve Bellovin wrote an April Fool's note suggesting an "Evil Bit" (ftp://ftp.rfc-editor.org/in-notes/rfc3514.txt); I actually think that's not such a dumb idea if implemented as a "Not Evil" flag, using a DSCP or extending the RFC 3168 codes to include such, as Steve Crocker has been suggesting. Basically, a customer gets ingress filtered (by whatever means) and certain DSCP settings are treated as "someone not proven to have their act together". Should a ddos happen, such traffic is dumped first. But if the customer pays extra, their traffic is marked "not evil", protected by the above, and ingress filtering may be on or off according to the relevant agreement. The agreement would need to include a provision to the effect that once a ddos is traced in part to the customer, their traffic is marked as "evil" for a period of time afterwards. What the customer is paying for, if you will, is the ability to do their thing during a ddos in a remote part of the network, such as delivering a service to a remote peer. Address spoofing is just one part of the ddos problem; to nail ddos, we also need to police a variety of application patterns. One reason I like the above is that it gives us a handle on what traffic might possibly be "not evil" - someone has done something that demonstrates that it is from a better managed source.
Re: BCP38 making it work, solving problems
On Tue, 12 Oct 2004, Suresh Ramasubramanian wrote: > Daniel Senie wrote: > > One of your arguments presented was that corporate customers weren't > > asking for unicast RPF, and I responded that corporate customers are not > > in need of automated mechanisms to implement BCP38, since in most cases > > their networks are EDGE networks, and it's quite simple to filter your > > egress points to ensure you don't send out any spoofed packets. > > There is, of course, the issue of multihomed networks, or networks that > have satellite connectivity etc emitting spoofed source packets. a common occurance we've seen is a customer of a customer NOT announcing , nor planning on announcing, their routes to their upstream#1 which they use ONLY for outbound traffic (cheap transit for instance, and perhaps only for some portions of their total sources) though they announce to upstreams#2-N the proper sources to gather the return traffic. These things make uRPF 'difficult'. As to the entire conversation I think more folks will implement BCP38, or parts atleast, as they replace gear which is not capable of dealing with parts of the implementation of BCP38. Hopefully new gear is being tested/certified/bought that can do wirespeed filtering on all interfaces. -Chris
Re: BCP38 making it work, solving problems
Daniel Senie wrote: One of your arguments presented was that corporate customers weren't asking for unicast RPF, and I responded that corporate customers are not in need of automated mechanisms to implement BCP38, since in most cases their networks are EDGE networks, and it's quite simple to filter your egress points to ensure you don't send out any spoofed packets. There is, of course, the issue of multihomed networks, or networks that have satellite connectivity etc emitting spoofed source packets. Yes I know that multihoming customers must make sure packets going out to the internet over a link match the route advertised out that link .. but stupid multihoming implementations do tend to ensure that lots of people will yell loudly, and yell loudly enough for several tickets to be escalated well beyond tier 1 NOC support desks, for ISPs to kind of think twice before they put uRPF filters in .. srs
Re: BCP38 making it work, solving problems
At 07:51 PM 10/11/2004, Richard A Steenbergen wrote:
On Mon, Oct 11, 2004 at 06:03:08PM -0400, Daniel Senie wrote:
>
> I've removed the rest of your message, talking about which vendors do or
> don't have what capabilities. While I agree it'd be nice if more vendors
> offered automated tools for implementing ingress filtering, such tools are
> unnecessary in most corporate network cases, thus the lack of corporate
> customers asking for the feature. In reality every device offering access
> control lists capable of filtering on source IP address can and does have
> sufficient capability to implement BCP38.
>
> While I appreciate the desire to have a single switch solution, like was
> possible with BCP34, it's a bit more complex to do in this case. It is,
> however, disingenuous to say that devices don't support BCP38 because they
> don't have an automated widget to implement it. Keep in mind that uRPF is
> an implementation of BCP38 capability, and other implementations are
> entirely possible.
>
> This was probably obvious to you, but others reading might find the
> clarification useful.
Yes if a box has source address packet filtering capabilities you can
filter packets by source address ("Duh"). This doesn't mean that it is
going to be sane or easy to implement the filtering by manually
maintaining an acl of every prefix/host on every interface where you could
have a customer or corporate box injecting spoofed packets into the
network. I believe there are plenty of corporate networks out there that
are far too complex to maintain with manual ACLs, I believe the reason
that no one cares is simply because... no one cares. :)
One of your arguments presented was that corporate customers weren't asking
for unicast RPF, and I responded that corporate customers are not in need
of automated mechanisms to implement BCP38, since in most cases their
networks are EDGE networks, and it's quite simple to filter your egress
points to ensure you don't send out any spoofed packets.
You laid out a complaint against the equipment makers claiming they weren't
implementing automated mechanisms BECAUSE the corporate customers were not
asking for such tools, and I simply pointed out they shouldn't be expected
to do so. If network operators need features, they need to ask for them
when talking with potential vendors.
Network operators need to ensure downstreams don't advertise AS's they're
not supposed to. Last I looked, that required some custom work (whether
done by scripting or whatever, it's done off the router and pushed in). At
the same time you're building those lists, you could be building ACLs. Some
border routers will do this just fine, others won't. Next time you're
qualifying routers for possible use, maybe the ability to handle wire speed
acls might be worth testing?
If you expect people to be able to maintain these filters on any scale,
they need tools.
Why do those tools need to be built into the router? Are your tools for
maintaining AS path filteirng built into the routers? Are the tools to
archive and compare router configurations most of you use built into the
routers?
Certainly uRPF is a good tool to do this, and certainly
someone could implement some others that are different, but the complete
lack of any tool, especially on the boxes where you should be doing this
filtering, counts as a failure in my book.
I disagree that the tools must be an integral part of the router software.
Perhaps it's time to think outside the (router) box?
Re: BCP38 making it work, solving problems
On Mon, Oct 11, 2004 at 06:03:08PM -0400, Daniel Senie wrote:
>
> I've removed the rest of your message, talking about which vendors do or
> don't have what capabilities. While I agree it'd be nice if more vendors
> offered automated tools for implementing ingress filtering, such tools are
> unnecessary in most corporate network cases, thus the lack of corporate
> customers asking for the feature. In reality every device offering access
> control lists capable of filtering on source IP address can and does have
> sufficient capability to implement BCP38.
>
> While I appreciate the desire to have a single switch solution, like was
> possible with BCP34, it's a bit more complex to do in this case. It is,
> however, disingenuous to say that devices don't support BCP38 because they
> don't have an automated widget to implement it. Keep in mind that uRPF is
> an implementation of BCP38 capability, and other implementations are
> entirely possible.
>
> This was probably obvious to you, but others reading might find the
> clarification useful.
Yes if a box has source address packet filtering capabilities you can
filter packets by source address ("Duh"). This doesn't mean that it is
going to be sane or easy to implement the filtering by manually
maintaining an acl of every prefix/host on every interface where you could
have a customer or corporate box injecting spoofed packets into the
network. I believe there are plenty of corporate networks out there that
are far too complex to maintain with manual ACLs, I believe the reason
that no one cares is simply because... no one cares. :)
If you expect people to be able to maintain these filters on any scale,
they need tools. Certainly uRPF is a good tool to do this, and certainly
someone could implement some others that are different, but the complete
lack of any tool, especially on the boxes where you should be doing this
filtering, counts as a failure in my book.
--
Richard A Steenbergen <[EMAIL PROTECTED]> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: BCP38 making it work, solving problems
At 05:41 PM 10/11/2004, Richard A Steenbergen wrote: >On Mon, Oct 11, 2004 at 02:58:59AM +, Fergie (Paul Ferguson) wrote: > > It's better than a sharp stick in the eye, I'll tell ya, > lad. > > Listen to me: It's called a "best current practice" for a > reason -- people should do it. Not sit and around and endlessly > discuss it (we've already done that a thousand times). > > I wrote it, I stand beside it. I'm sick of hearing why people > haven't implemented it yet -- it's almost five years later > and there's simply no excuse. It's sickening. Tell it to the vendors of the layer 3 customer attach devices. I was just speaking about this with a major "up and coming layer 2/3 switch vendor" the other day, who had no implementation or real plans to implement uRPF in the immediate future. It apears that there are no enterprise customers asking for this feature (not a shock), despite the fact that they are probably a leading generator of hacked machines throwing bad packets down reasonably big pipes. I've removed the rest of your message, talking about which vendors do or don't have what capabilities. While I agree it'd be nice if more vendors offered automated tools for implementing ingress filtering, such tools are unnecessary in most corporate network cases, thus the lack of corporate customers asking for the feature. In reality every device offering access control lists capable of filtering on source IP address can and does have sufficient capability to implement BCP38. While I appreciate the desire to have a single switch solution, like was possible with BCP34, it's a bit more complex to do in this case. It is, however, disingenuous to say that devices don't support BCP38 because they don't have an automated widget to implement it. Keep in mind that uRPF is an implementation of BCP38 capability, and other implementations are entirely possible. This was probably obvious to you, but others reading might find the clarification useful.
Re: BCP38 making it work, solving problems
Fergie (Paul Ferguson) wrote: True, but yet another cop out. If you're not part of the solution, . - ferg -- Dan Hollis <[EMAIL PROTECTED]> wrote: On Mon, 11 Oct 2004, Fergie (Paul Ferguson) wrote: I wrote it, I stand beside it. I'm sick of hearing why people haven't implemented it yet -- it's almost five years later and there's simply no excuse. It's sickening. it's cheaper to ignore bcp38 than to implement it. Well NANOG wants to have it both ways: -Boo the providers who bill for spoofed packets -Wish it wasnt cheaper to do nothing to ensure packets leaving your network are not spoofed I vote providers should charge triple or more for ( reaction,detection and supression costs caused by) spoofed packets coming from their transit customers. Now we have incentive on both sides. The provider to identify this traffic and the customer to stop it. (Dont POTS telcos offer something like this?) The same will encourage customers to start asking for QOS and rate limiting. Now when the Provider shuts you down they have done you a nice financial favor. Toss in the the option for "spoof insurance" whereby the customer pays extra to insure that any spoofed packets from his network are not billed for and it gets a little more confusing. operators are reactive to abuse, not proactive. though this is slowly changing as abuse becomes a significant % of network traffic. -Dan -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet [EMAIL PROTECTED] or [EMAIL PROTECTED]
Re: BCP38 making it work, solving problems
>On Mon, Oct 11, 2004 at 02:58:59AM +, Fergie (Paul Ferguson) wrote: > > It's better than a sharp stick in the eye, I'll tell ya, > lad. > > Listen to me: It's called a "best current practice" for a > reason -- people should do it. Not sit and around and endlessly > discuss it (we've already done that a thousand times). > > I wrote it, I stand beside it. I'm sick of hearing why people > haven't implemented it yet -- it's almost five years later > and there's simply no excuse. It's sickening. Tell it to the vendors of the layer 3 customer attach devices. I was just speaking about this with a major "up and coming layer 2/3 switch vendor" the other day, who had no implementation or real plans to implement uRPF in the immediate future. It apears that there are no enterprise customers asking for this feature (not a shock), despite the fact that they are probably a leading generator of hacked machines throwing bad packets down reasonably big pipes. uRPF support is hardly universal outside of Cisco and Juniper, and even Juniper didn't add uRPF until semi-recently. I know Foundry still doesn't have support for it (not really anyways, they added something they call uRPF and have you activate through "ip verify unicast reverse-path" but it isn't really uRPF, just a way to prevent outside networks from spoofing local addresses, and you have to manually tag every interface as internal or external or the addresses aren't protected), and I'm pretty sure Extreme doesn't have it (or at least I've never seen anyone use it, but I don't follow 0 day Extreme code). Short of the Cisco L3 switch solutions, I'm not aware of any other vendor with a L3 switch who has uRPF functionality. I can't think of a single web hoster (or at least someone who wasn't a real network operator who got into hosting somehow) who even knows what uRPF is, let alone implements it for their customers. We're counting on their IP providers to filter the bad packets from the hundreds of FE connected servers on burstable GigE pipes out there, but some of them just aren't. I can name several major well-known "cheap" networks who don't do any uRPF filtering of their customers, and a couple more who don't do any real prefix-lists on their BGP speaking customers, only prefix-limits. There are even fewer vendors who are implementing automation for filtering basic BGP speaking customers, such as Juniper's ability (sort-of) to re-use a route filtering prefix-list for source address filtering in a firewall. It's not quite perfect, since you can't do length modifiers (upto /24, etc) in a prefix-list, and since you have to create a seperate policy-statement (with all routes listed in route-filter statements), prefix-list, AND firewall filter for each and every customer, but at least it is a start. If you don't have this, or uRPF at all, you are left trying to script ACLs based on interface configurations, static routes, and/or registered prefixes, and the bottom line is that most networks aren't going to do it if it takes that much work. If and when all the vendors who are making boxes that are supposed to be used for customer aggregation start implementing uRPF, especially considering all the enterprises, universities, and international network using these L3 switches as their sole routing equipment (think price), maybe we'll start seeing some more progress. And of course, those remaining service providers with the gear to implement it who havn't done so need to be yelled at more. Unfortunately, no customer ever complains (or takes their business elsewhere) when their provider doesn't do source address filtering, only when they are getting a DoS attack and their provider can't do anything about it. -- Richard A Steenbergen <[EMAIL PROTECTED]> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: BCP38 making it work, solving problems
RB> Date: Sun, 10 Oct 2004 20:14:01 -0700 RB> From: Randy Bush RB> when it solves critical problems, it'll grow more quickly. Maybe. * Use 25/TCP for SMTP and 587/TCP for submission * Block outbound SMTP by default, but allow for the clueful * Run SMTP authentication * Let each authenticated user have whitelisted sender addresses that they can use * Limit whitelist size * Add a delay and/or rate limit to whitelist additions. Not perfect, and certainly subject to social engineering and possible automated attack on the whitelist mechanism, but it should decrease the number of cable/DSL pipes filled with forged mail transmissions. This isn't the first time I've suggested it, and I'm sure others have, too. Not once have I received a response to the extent of "I'd love to implement this if it existed". People are worried about OPNs, not their own networks. IMNSHO, worrying about N-1 ASNs scales far more poorly than worrying about one ASN. Of course, note the parallel to BCP38; I'm sure someone will be quick to point out that, unless adopted universally, forged mail will still exist. Enter SPF as a bandaid on the receiving side, and rehash that discussion. Combine with BMF, DNSBLs, and one is well on the way to much cleaner mail. Has anyone on NANOG ever solved a jigsaw puzzle with 500+ pieces? Or are "age 2 to 4" puzzles too difficult, as even they tend to have around ten pieces per puzzle? Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _ DO NOT send mail to the following addresses: [EMAIL PROTECTED] -*- [EMAIL PROTECTED] -*- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked.
Re: BCP38 making it work, solving problems
the problem is that isp security folk doing actual measurement see very little spoofing. it's easy for the bad folk to get real bots. and tcp bad things are more popular and desirable, e.g. spam, ... and tcp does not work from spoofed addresses. isp security folk have limited resources. so why should they spend them where there is little perceived benefit when there are gaping holes elsewhere that need those resources? yes, it's a nice thing to do. but, in today's disasterous economy, so are 42 other things that won't get done. so it's growing slowly. when it solves critical problems, it'll grow more quickly. randy
Re: BCP38 making it work, solving problems
SD> Date: Sun, 10 Oct 2004 21:35:33 -0400 (EDT) SD> From: Sean Donelan SD> People think BCP38 means the packets could only originate SD> from you. Were BCP38 universal, this would be true. If one receives a packet, it's either from the supposed source or a network that allows spoofing. If no networks allow spoofing, it came from the supposed source. SD> [P]eople don't complain to the source of spoofed packet. SD> People complain to IANA about attacks coming from Net-10. They complain to the perceived source. Many Internet users are shocked at how trivial it is to forge email/packet sources; I guess they're used to services like caller ID where the end user isn't [traditionally] given the power to spoof. Then there's postal mail. At least sending spoofed packets is more costly than IP, and end-user packets frequently are tagged with an ingress label. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _ DO NOT send mail to the following addresses: [EMAIL PROTECTED] -*- [EMAIL PROTECTED] -*- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked.
Re: BCP38 making it work, solving problems
True, but yet another cop out. If you're not part of the solution, . - ferg -- Dan Hollis <[EMAIL PROTECTED]> wrote: On Mon, 11 Oct 2004, Fergie (Paul Ferguson) wrote: > I wrote it, I stand beside it. I'm sick of hearing why people > haven't implemented it yet -- it's almost five years later > and there's simply no excuse. It's sickening. it's cheaper to ignore bcp38 than to implement it. operators are reactive to abuse, not proactive. though this is slowly changing as abuse becomes a significant % of network traffic. -Dan -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet [EMAIL PROTECTED] or [EMAIL PROTECTED]
Re: BCP38 making it work, solving problems
On Mon, 11 Oct 2004, Fergie (Paul Ferguson) wrote: > I wrote it, I stand beside it. I'm sick of hearing why people > haven't implemented it yet -- it's almost five years later > and there's simply no excuse. It's sickening. it's cheaper to ignore bcp38 than to implement it. operators are reactive to abuse, not proactive. though this is slowly changing as abuse becomes a significant % of network traffic. -Dan
Re: BCP38 making it work, solving problems
It's better than a sharp stick in the eye, I'll tell ya, lad. Listen to me: It's called a "best current practice" for a reason -- people should do it. Not sit and around and endlessly discuss it (we've already done that a thousand times). I wrote it, I stand beside it. I'm sick of hearing why people haven't implemented it yet -- it's almost five years later and there's simply no excuse. It's sickening. - fergie Ref: ftp://ftp.rfc-editor.org/in-notes/bcp/bcp38.txt -- Sean Donelan <[EMAIL PROTECTED]> wrote: But BCP38 doesn't immediately help the ISP. -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet [EMAIL PROTECTED] or [EMAIL PROTECTED]
Re: BCP38 making it work, solving problems
> I have received complaints from people about NOT being able to spoof > packets. Technical Support: "This is CompanyX, how can I help you?" 31337kiddi0t: "wHy c0m3 3ye c4nt sp0of?!$!*@" With all of the different standards which tend to add confusion, too much time seems to be going to waste drafting them while networks and businesses suffer from what's currently in place. From my perspective if someone mentioned this to me via complaints their account would be cancelled immediately since there is no benefit to sending out spoofed packets. "But it's a pen test audit!" Even in situations where a security admin needed to test certain elements an aware admin would find a way to get around doing what they had to do. Blocking elements such as SMTP do have its place and I'm sure most know this is not the "definitive" solution nothing more than patch work but it still has its advantages. The same way spammers can adapt, so should an engineer be able to for those who would like to contest the notion that one would be making "smarter idiots" who send spam and create malice. I've been digging more into RFC's in hopes of learning more from a technical perspective for my own sake and to date, all I've seen is more of less patchwork. Instead of reinventing the wheel as the old saying goes, sometimes a patch can get you moving on a flat tire. Sure it is a temporary solution, but it is a solution. =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo GPG Key ID 0x51F9D78D Fingerprint 2A48 BA18 1851 4C99 CA22 0619 DB63 F2F7 51F9 D78D http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x51F9D78D sil @ politrix . orghttp://www.politrix.org sil @ infiltrated . net http://www.infiltrated.net "There is no greater mistake than the hasty conclusion that opinions are worthless because they are badly argued." -- T.H. Huxley
