Kenyan Route Hijack
[more accurate subject line] On Mar 14, 2008, at 1:33 PM, Felix Bako wrote: Hello, There is a routing loop while accesing my network 194.9.82.0/24 from some networks on the Internet. | This is a test done from lg.above.net looking glass. 1 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 4 msec 0 msec 0 msec 2 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) [MPLS: Label 78 Exp 0] 0 msec 0 msec 0 msec 3 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 8 msec 8 msec 0 msec 4 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) [MPLS: Label 80 Exp 0] 0 msec 4 msec 0 msec 5 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 4 msec 0 msec 0 msec 6 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) [MPLS: Label 78 Exp 0] 0 msec 0 msec 4 msec 7 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 64 msec 0 msec 4 msec 8 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) [MPLS: Label 80 Exp 0] 0 msec 4 msec 0 msec 9 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 4 msec 0 msec 0 msec 10 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) [MPLS: Label 78 Exp 0] 0 msec 4 msec 0 msec 11 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 4 msec 0 msec 4 msec| According to RIPE BGP play data looks to me like AS 6461 (Abovenet) began announcing 194.9.82.0/24 about 10 hours ago, pulling traffic away from AS 39615 and triggering your reachability problems (Note times are UTC): # 1/361 2008-03-15 03:05:27 Path Change from 29636 6461 2914 8513 25228 36915 rrc01 195.66.224.132 to 29636 2914 6461 # 2/361 2008-03-15 03:05:27 Route Announcement 20485 2914 6461 rrc01 195.66.224.212 About 17 minutes later AS 6461 they withdrew the route announcement: # 41/361 2008-03-15 03:22:56 Route Withdrawal ( 4777 2497 2914 6461 ) rrc06 202.249.2.20 And another 12 minutes or so later they began announcing it again: # 42/361 2008-03-15 03:35:26 Path Change from 29636 6461 2914 8513 25228 36915 rrc01 195.66.224.132 to 29636 2914 6461 ... Seemed to be a bunch more instability with this prefix around 5:53: # 66/361 2008-03-15 05:53:40 Route Announcement 25462 6461 rrc07 194.68.123.157 ... And then some withdraws around 7:43: # 183/361 2008-03-15 07:43:48 Path Change from 8468 6453 6461 rrc01 195.66.224.151 to 8468 3491 25228 25228 25228 25228 25228 36915 ... With considerable oscillation for around 40 minutes between the legit path via AS 36915 and the path via AS 6461. And the latest was this transition from AS 6461 back to the 36915 path about 2 hours ago, but only by a few ASNs, I suspect because those ASNs explicitly modified policy (either preference or filtering) to de_prefer the AS 6461 path. This is illustrated pretty nicely with BGP play: # 335/361 2008-03-15 14:59:43 Route Withdrawal ( 1916 3549 6461 ) rrc15 200.219.130.4 # 361/361 2008-03-15 15:00:27 Path Change from 13645 3356 6461 rrc11 198.32.160.150 to 13645 3491 25228 25228 25228 25228 25228 36915 BGP Play applet here: http://www.ris.ripe.net/bgplay/applet.html? Although most folks are definitely still preferring the AS 6461 path. An interesting bit is that the current announcement on routeviews directly from AS 6461 has Community 6461:5999 attached: ... 6461 64.125.0.137 from 64.125.0.137 (64.125.0.137) Origin IGP, metric 0, localpref 100, valid, external, best Community: 6461:5999 ... According to this, that community is used for internal prefixes: http://onesc.net/communities/as6461/ 6461:5999 internal prefix A sh ip bgp community 6461:5999 currently yields 130 prefixes with Origin AS of 6461 and that community. Nothing more specific than a /24, although many many adjacent prefixes that would presumably be aggregated normally are announced as well. The closest adjacent prefix to 194.9.82/24 they're announcing is 194.9.40/24, which is one of their prefixes: * 194.9.40.0 64.125.0.137 0 0 6461 i * 194.9.82.0 64.125.0.137 0 0 6461 i Unfortunately, the AS6461 forwarding loops still exists, and most ASNs still appear to be preferring their path over yours per BGP AS path route selection rules: --- [EMAIL PROTECTED] date Sat Mar 15 11:55:27 MDT 2008 ... 14 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 188.278 ms 172.714 ms 174.984 ms 15 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 176.234 ms 174.013 ms 174.109 ms 16 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 173.230 ms 172.892 ms 174.765 ms 17 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 174.721 ms 175.256 ms 174.738 ms 18 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 174.437 ms 220.815 ms 180.961 ms 19 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 177.564 ms 181.966 ms 174.771 ms 20 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 176.028 ms 174.269 ms
Re: Kenyan Route Hijack
A bit more analysis of this at the moment, and a few recommendations and related pointers is available here: http://tinyurl.com/2nqg2a -danny
Re: Routing Loop
On Mar 15, 2008, at 4:49 PM, Florian Weimer wrote: There's also somewhat odd data in RADB (look at the changed: line): route: 194.9.64.0/19 descr: SES-Newskies Customer Prefix origin:AS16422 remarks: SES-Newskies Customer Prefix notify:[EMAIL PROTECTED] mnt-by:MNT-NWSK changed: [EMAIL PROTECTED] 20080314 source:ARIN This is in the middle of RIPE-managed swamp space, a /19 definitely doesn't belong there. Yeah, I saw that a bit earlier and it did seem incredibly suspicious given the timing. Had I seen anything in the routing system itself for that explicit /19 related to this I would mentioned it, but nothing there. Amazingly, a query to the NOC at SES Newskies yielded a near- immediate response, which said they added it a few days back because they were updating some policies and noticed it existed in a prefix list for one of their upstreams, that it appeared to be legacy, and that they'd get it removed. All the more reason RIR allocation authentication used to seed IRR information would be of value for routing policy specification, let alone informational purposes. -danny
Re: Customer-facing ACLs
On Mar 7, 2008, at 12:55 PM, Justin Shore wrote: This question will probably get lost in the Friday afternoon lull but we'll give it a try anyway. What kind of customer-facing filtering do you do (ingress and egress)? This of course is dependent on the type of customer, so lets assume we're talking about an average residential customer. We did ask some of these questions in the latest Infrastructure Security Survey, available here: http://www.arbornetworks.com/report Or here if you'd prefer not to register: http://www.tcb.net/wisr_2007_v3.pdf We're in the process of putting the next round of questions together, so if there are things you'd like added please let us know. -danny
Re: RIPE NCC publishes case study of youtube.com hijack
On Feb 29, 2008, at 7:46 AM, David Ulevitch wrote: The report states: Sunday, 24 February 2008, 20:07 (UTC): AS36561 (YouTube) starts announcing 208.65.153.0/24. With two identical prefixes in the routing system, BGP policy rules, such as preferring the shortest AS path, determine which route is chosen. This means that AS17557 (Pakistan Telecom) continues to attract some of YouTube's traffic. It's worth noting that from where I sit, it appears as though none of Youtube's transit providers accepted this announcement. Only their peers. A simple artifact of shortest AS path route selection. In addition, many providers employ policies that apply preference for prefixes learned from customers over those learned from peers, assuming they're of the same length. Had those same providers explicitly not accepted the /24 announcement from AS 17557 via their peers you wouldn't have been affected at all. The point is -- Restrictive customer filtering can also bite you in the butt. Trying to require your providers to do a ge 19 le 25 (or whatever your largest supernet is), rather than filters for specific prefix sizes seems a worthwhile endeavor so you can de- aggregate on the fly, as necessary. Deaggregation in order to mitigate less specific route hijacking is a hack that in most cases only half fixes the problem, if that. If providers didn't have those policies in place it'd be /32s that were being hijacked and route table growth and churn would be far worse than it already is. You prevent this by ubiquitous deployment of explicit customer and inter- provider prefix filters, you don't open things up more so that when problems occur, folks can try to hack around them. -danny
Re: RIPE NCC publishes case study of youtube.com hijack
On Feb 29, 2008, at 11:49 AM, David Ulevitch wrote: Of course... In fact, wouldn't it even providers benefit from having some logic that says don't ever accept a more specific of a customer-announced prefix? Sure, that'd suck less, I guess, although then you have to punch holes for multi-homed customers, etc.., which is actually trivial if policy is generated automatically based on RPSL or the like and the policy is registered accordingly. But I still prefer explicit route policy where what an AS is permitted to announce is all that's permitted and any other prefixes are discarded. Any policy that allows lots of more specifics to be announced in case of route hijacking to me is like putting a band-aid on a headache. Customers might not like that though... :-) Right, it's breaks fail-over with multi-homing, in particular. As far as other more specifics for TE and the like, well, they're certainly welcome to register those prefixes such that they can be reflected in routing policies, but this would also ease announcements of unintentional more-specifics. I don't consider this one of those 'YMMV' things. Today, if providers explicitly filter at all they filter customer routes based on some IRR data or other internal database. They may put a few safety nets in place for bogon prefixes and certain prefix length policies or ASNs, or perhaps not accept their own aggregate or more specifics from peers. However, they accept everything else from peers, which means tomorrow, when this happens again, all they can do is get pissed because some monkey on the other side of the world fat-fingered a 2 instead of a 3, or forget to attached a no_advertise, no_export or other explicit non-transit community to a blackhole route .. and now some other site that presumably matters is offline, or half reachable, or whatever... Further, we can keep experiencing more extraneous route table bloat because of folks advertising more specifics of their own aggregates in order to minimize any impact a potential hijacking might have to their own space.. Or, we could start implementing explicit inter-provider filtering. Explicit policy on all inter-domain peers, customer or provider, based on RIR allocations, IRR objects and RPSLish language, and work on removing deployment barriers (e.g., stale IRR data, allocation authentication, IRR update vulnerabilities, router configuration scale and load issues, TTM for newly announced prefixes, etc..), with real deployment likely in an incremental bi-lateral manner between ISPs that employ IRR data for customer route policy today and already have tools to manage and deploy new policy. I challenge providers to step up here, the onus is on you and nothing else is going to make this problem go away.There's tangible incremental benefit to any provider that institutes such a policy, and by it's very nature, the right ISPs will encourage other sites on the Internet to begin employing IRRs and similar mechanisms, if for no reason other than to enable propagation of their own legitimate routes more quickly. -danny
Re: BGP prefix filtering, how exactly? [Re: YouTube IP Hijacking]
On Feb 25, 2008, at 6:08 AM, Pekka Savola wrote: In a lot of this dialogue, many say, you should prefix filter. However, I'm not seeing how an ISP could easily adopt such filtering. So, this is no excuse for not doing prefix filtering if you only do business in the RIPE region, but anywhere else the IRR data is pretty much useless, incorrect, or both. Agreed. (Yeah, we prefix filter all our customers. Our IPv6 peers are also prefix filtered, based on RIPE IRR data (with one exception). IPv4 peers' advertisements seem to be too big a mess, and too long filters, to fix this way.) Do you explicitly filter routes from your upstream or transit providers? E.g., if one were to announce, say, a more specific of one of your customer's routes to you would you accept it? What about someone else's address space? The only full set of prefix filtering I've ever seen implemented (i.e., BGP customers AND peers) was b y ANS during my days at iMCI ~95. It was extremely painful at times, even for us, if we wanted to advertise new address space we had to update IRR objects and wait on their nightly push of updated routing policies at ANS. We generated our own routing policies automatically off our IRR, which mirrored others as well, and explicitly prefix filtered customers with some fixed prefix and AS path-based policies applied to peers. If it became really urgent, then we'd call ANS and have them manually update their policy, and subsequently 'bounce' the route announcement to trigger transmission of a new update. This was long before incrementally updated filters and things like BGP route refresh ever existed. Prefixes and AS-MACROs had to be right in the IRRs or the policies wouldn't be updated. It's to bad other folks didn't follow suit. As for this event, a slightly different spin here: http://tinyurl.com/3y3pzl -danny
Re: [admin] [summary] RE: YouTube IP Hijacking
On Feb 25, 2008, at 12:51 PM, Alex Pilosov wrote: ** Nobody brought up the important point - the BGP announcement filtering are only as secure as the weakest link. No [few?] peers or transits are filtering large ISPs (ones announcing few hundred routes and up). There are a great many of them, and it takes only one of them to mess up filtering a downstream customer for the route to be propagated. Yes, that was my implicit point to Pekka. Even if you do everything feasible today (i.e., explicitly filter customers, some amount of policy for peers, and perhaps a few hacks for multi-homed customers), you're still pretty much screwed if someone announces your address space. Heck, you're as likely to accept the announcement as anyone. ** Paul Wall brought up the fact that even obviously bogus routes (1/8 and 100/7) were accepted by 99% of internet during an experiment. I'm not sure why this would surprise anyone. ** What I'd like to see discussed: Issues of filtering your transit downstream customers, who announce thousands of routes. Does *anyone* do it? Lots of folks do. The interesting bit is that even then, those same providers would accept perhaps even those customer routes from their peers implicitly. * Typos vs Malicious announcements ** Some ways of fixing the problem (such as IRR filtering) only address the typos or unintentional announcements. You mean as opposed to intentionally malice acts? Well, not completely. See Pekka's email, for example. Of course, it does vary widely across IRRs, etc.. There's full agreement that IRR is full of junk, which is not authenticated in any sort. Mostly, though not completely. ** Things like PHAS won't work if hijacker keeps the origin-AS same (by getting their upstream to establish session with different ASN) NO, that's not even necessary. Simple originate the route from the legit AS, and then transit it with the local AS as a transit AS. AS path manipulation is trivial. ** What I'd like to see discussed: Who (ICANN/RIRs/LIRs) is actively working on implementing chain of trust of IP space allocations? * Ways to address the issue without cooperation of 3491: ** Filtering anything coming out of 17557 Bad idea. ** Suggestions given: ** What I'd like to see discussed: Can an network operator, *today*, filter the possibly bogus routes from their peers, without manual intervention, and without false positives? Sure, if they want to dedicate an engineer to it, automate policy deployment and deal with brokenness by turning steam valves. * Yelling at people who don't filter That's been productive for over a decade now. ** Per above, 3491 isn't the only one who filters. In fact, claims were made that *nobody* filters large enough downstreams. (beyond aspath/maxpref) Wrong. -danny
Re: [admin] [summary] RE: YouTube IP Hijacking
On Feb 25, 2008, at 1:22 PM, Alex Pilosov wrote: Well, in this case, they *aren't* filtering! (unless I am misunderstanding what you are saying, due to repeated use of 'their'). What I'm saying is that best case today ISPs police routes advertised by their customers, yet they accept routes implicitly (including routes from address space that may belong to their customers) from peers. Seems a little hokey, eh? Oh yeah, d'oh! Thanks for correction. But that is also an important point against PHAS and IRRPT filtering - they are powerless against truly malicious hijacker (one that would register route in IRR, add the right origin-as to AS-SET, and use correct origin). Yep, pretty much. Sure, if they want to dedicate an engineer to it, automate policy deployment and deal with brokenness by turning steam valves. I'd hear to see who does it, and get them to present the operational lessons at the next nanog! Maybe Curtis V. would present what ANS was doing in 1994 :-) But now we've even got things like BGP route refresh, incrementally updatable filters, and BGP soft reconfiguration to ease the deployment burden. There have been two or three panels on this exact topic in the past, you can find them in the index of talks. Unfortunately, the problem hasn't changed at all. Perhaps we could just replay those video streams :-) -danny
Re: [admin] [summary] RE: YouTube IP Hijacking
I'd hear to see who does it, and get them to present the operational lessons at the next nanog! On second thought, I guess one thing has changed considerably since 15 years ago. Rather than ~5000 monkeys with keyboard access to manipulate global routing tables, there are likely well North of 250,000 (25k active ASNs * 10 meat computers per), which is surely well on the conservative side. The bottom line is [still] that ISPs should at least explicitly filter prefixes from customers and networks from which they provide transit services. -danny
Re: BGP TTL Security
On Feb 14, 2008, at 11:28 AM, Ben Butler wrote: =191 and the session stays down. Which is proper bizarre! Is it necessary to configure this on both side for the session to re-establish. Is this a Cisco bug? You're missing the fundamentals of what protection this mechanism is meat to provide. A remote attacker can craft a packet such that it yields a TTL of 2, 1 or 0 at the target system. However, what a remote attacker can't do is craft a packet that yields a TTL or 255 or 254, for example. You probably want both values to be 254 if you've got one intermediate hop between the peers. -danny
Re: BGP TTL Security
On Feb 14, 2008, at 11:28 AM, Ben Butler wrote: I have validated via trace in both directions as being 1 hop. I have read another article that implies the default behaviour at the other end will to be send TTL 1 not 255 and consequently I need to configure both ends to get the session to come back up. An access list reveals all the packets I am receiving have a TTL of 0. The session re-establishes if I configure: neighbor 212.121.34.1 ttl-security hops =192 =191 and the session stays down. Ben, After a prodding offlist I reread your message and understand what point you're making now. Indeed as you suggest above the normal configuration should be 'ttl-security hops 2' or 'ttl security hops 1'. Not for sure, but I'd have to speculate that if this is only working for you with 'ttl-security hops = 192' perhaps your peer is setting the TTL in it's packet to 64? I believe that's the default TTL for Linux, Foundry and a couple others. Juniper's default TTL is 1 eBGP (though configurable), and 64 for iBGP, multihop, etc. IIRC. In order to implement this effectively the peer would need to be setting the transmitted TTL to 255. And my apologies if my previous message seemed a bit negative, that was certainly not my intention. -danny
Re: Route Reflector full mesh
On Feb 5, 2008, at 11:56 PM, Adhy wrote: 1. If the update is propagated from RR1 to RR2 then RR3, will the ORIGINATOR_ID on the update mesg still RR1 ? Yes, the originator ID is preserved, while the cluster list would be additive. 2. and will the CLUSTER_LIST being used ? the cisco specs only said that the CLUSTER_LIST being used if the update mesg is reflected from clients to non clients. how about from clients to other client (which this client also a RR for another cluster) ? There are typically two models for deployment within a cluster. Either fully-meshed clients with no client-to-client reflection, or clients that only peer with the RRs, with client-to-client reflection. If client-to-client reflection is employed then the RR will attach the RR attributes, including the originator ID set to the router ID of the client, and the local cluster ID value will be added to the cluster list. We did see some routing information loops on early deployments where client-to-client reflection is employed AND the RR would reflect a route back to a client that it learned from the client. The issue here is that the client is now required to know it's a client and poison updates it receives if the originator ID value equals the local router ID. The strict don't reflect a route learned from a client back to that client because if you do a client would need to know about RR and know it's a client behavior changed from RFC 1996 to RFC 2796 (perhaps so that some implementations could optimize message copy across client groups). I've assumed that both ORIGINATOR_ID and CLUSTER_LIST is always used when route is being reflected and come to conclusion that the loops will not occur. Is it correct ? Yes, when acting on behalf of a client, or if received and already attached to a prefix, unless sent to an eBGP peer. The originator ID should be preserved if it already exists. The real offshoot with RRs employing unique cluster IDs sharing common client sets is that you trigger unnecessary replication of updates, additional churn, require additional BGP RIB space in RRs, and require that routing information originated by a client be poisoned by that client potentially multiple times, rather than avoided through routing topology. Unique cluster IDs within topological clusters can be especially problematic hierarchical route reflection. Specifically to your question, see this text in Section 8 of RFC 4456: A BGP speaker SHOULD NOT create an ORIGINATOR_ID attribute if one already exists. I'd consider modification/replacement of an existing originator ID value brokenness, and the topology you illustrate broken anyway. -danny
Re: Blackholes and IXs and Completing the Attack.
On Feb 2, 2008, at 1:16 PM, Ben Butler wrote: So, given we all now understand each other - why is no one doing the above? Some folks are doing this, just not via some third-party route servers. For example, either via customer peering sessions, or other BGP interconnections between peers. E.g.: http://www.nanog.org/mtg-0402/morrow.html I'm not sure it's ideal to employ third-party route servers for this purpose, as it only increases the attacks/error surface. I suppose if folks rely on it for native peering then it might be reasonable. At the end of the day if an IX member doesn't want the announcements don't peer with the blackhole reflector, simple, and it will get Null routed as soon as it hits my edge router at the IX - it would just seem more sensible to enable people to block the traffic before it traverses the IX and further back in their own networks. Yes, helping to ease effects of collateral damage from large-scale attacks by conveying drop policies to upstream ASes for prefixes which you originate. And perhaps as significant, being able to allow the target AS to remove those policies dynamically, rather than having to contact each upstream AS that appears to have imposed them manually out-of-band. I think Paul's comments were more regarding the fact that destination-based blackhole routing for mitigation *effectively completes the attack*, which is often times undesirable. Inter-domain source-based blackhole routing is pretty much a non-option. One other offshoot is that advertised more-specifics are going to further contribute to routing AND forwarding table bloat, and a single new prefixes might result in 10+ new paths in the iBGP RIBs. If you do implement something like this you may wish to scope advertisement only to adjacent ASes via NO_EXPORT or the like, to scope both more-specific propagation, and to ensure that some lack of consistent drop community semantic interpretation doesn't hose something. Also, if you impose this as a standard attack response mechanism recall that you lose visibility of attack scales, and knowing just when to resume normal forwarding policies is a bit more complex. As such, your policy sets may want to provide hooks that enable selective prefix advertise/withdraw drop policies so that it can be applied or removed incrementally. -danny
Re: European ISP enables IPv6 for all?
On Dec 17, 2007, at 10:37 PM, Steven M. Bellovin wrote: On Mon, 17 Dec 2007 15:29:21 -0800 Christopher Morrow [EMAIL PROTECTED] wrote: how does it improve data security exactly? Back in 1994, it was expected to be true because v6 would mandate IPsec, and v6 would be deployed long before the installed base of v4 machines would be upgraded to IPsec. Obviously, that's not what happened; while IPsec was indeed late in coming, v6 was even later, so the original belief has been OBE. The mythos, however, hasn't caught up. Similar statements can be made about stateless autoconfig vs. v4 DHCP. Perhaps the concept also holds true because there's a smaller target market for the moment, and attackers are all about ROI. We've certainly seen this at other layers of the stack. However, not sure I'd posit as such. In a slightly more realistic vein, a huge address space makes life harder for scanning worms. As Angelos Keromytis, Bill Cheswick, and I have pointed out, harder is by no means equivalent to impossible, but the myth, new as it is, still propagates. As will the worms and malware, I suppose, though perhaps with more thought-out propagation vectors that employ not only local prefix scanning, but nifty things like walking ip6.arpa or the like for presumable denser host existence. Then again, who needs self propagation, when client-side attacks seem to be more than sufficient. -danny
Re: Slate Podcast on Estonian DOS atatck
On May 24, 2007, at 4:58 PM, Bill Woodcock wrote: First of it's kind that it targeted a country. No, at the very least, Moonlight Maze and Titan Rain came before. But by today's standards, Moonlight Maze would have been trivially small. I don't have any numbers for Titan Rain. Anyone know how it compared to the 4mpps of this attack? A data point based on some information we have from looking at inter-domain traffic and attack attributes across ~40 ISPs (~1 Tbps) over ~250 days now (and rolling): Days seeing at least one attack exceeding a given threshold: 6 Mpps 1 5 Mpps 12 4 Mpps 33 3 Mpps 53 2 Mpps 91 1 Mpps 149 Total attacks exceeding a given threshold: 6 Mpps 1 5 Mpps 17 4 Mpps 82 3 Mpps 135 2 Mpps 352 1 Mpps 813 The above is from the perspective of *a single ISP*, so the aggregate of the attack is likely to be far greater (cross-ISP correlation of targets are NOT reflected in _this dataset). Mpps and greater attacks make up far less than 1% of the attacks we see (we've have data for ~142k known attacks over this period). More on this in the near future and note that none of the above is meant to marginalize the Estonian attacks in any way, 4 Mpps is a lot depending on where it's directed and how it's mitigated - it's ALL about perspective. -danny
OT: 2007 Infrastructure Security Survey
Folks, It's time for the Infrastructure Security Survey again, figured I'd include all of nanog@ in the query for respondents this time around (per request[s]). If you're willing to complete the survey please go here to receive a token (which will be mailed to you with a URL for response input): https://sec-survey.tcb.net/index.php?sid=1 Many of you have seen some of the data from the survey at previous NANOG sessions and ISP Security BOFs. You can find the previous editions of the survey report here (if you'd prefer a non-marketing polished version just drop me a line): http://www.tcb.net/ISP_Security_Report-2H2005.pdf http://www.tcb.net/2006-WISR.pdf If you have any questions, comments or concerns please let me know. Thanks in advance! -danny
ISP Security BOF @NANOG 40
Folks, Can you please try to slot the ISP security BOF into the first day (Monday) of the agenda? Something has come up and I have to leave late Monday night. Thanks for your consideration! -danny
OT: ISP Security BOF @NANOG 40
Folks, Kevin Lanning (ATT) and I will be moderating the ISP Security BOF at NANOG 40 in Bellevue. As usual, if there are security-related topics you'd like to hear about, would rather not hear about, or would like like to present, please drop us a line ASAP. Thanks! -danny kevin = NANOG 39 ISP Security BOF Agenda: - The root of a log: Extracting Intelligence from the Woods - Botnet CC: Extirpate or Infiltrate? - netdisco introduction - Bill Fenner - Defending the NANOG Network: How the local net is geared for security (Randy B. moderator) - Open Microphone
Re: BOGON Announcement question
On Apr 30, 2007, at 9:15 AM, Patrick W. Gilmore wrote: On Apr 30, 2007, at 11:11 AM, Randy Bush wrote: Collector: CIXP Prefix: 128.0.0.0/2 Last update time: 2007-04-27 07:36:30Z Peer: 192.65.185.140 Origin: 29222 My question is, why am I not seeing more issues because of the announcement? because everyone with enough clue to watch what they receive has filters in place to prevent their hearing it? And even if they didn't, what important IP space in that /2 is not covered by more specifics? A whole lot if any of those more specific were withdrawn and the /2 were to pick up for them... -danny
Re: Do routers prioritize control traffic?
On Feb 12, 2007, at 9:10 AM, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: They are used for BUSINESS traffic. Also, since these controls make routers work harder, there is no point in using them where there are no traffic problems. I concur, it only matters when it matters (i.e., when there's resource contention). Most providers build their core networks with enough headroom so that there are no traffic problems. It's not a matter of just forwarding capacity, it's a matter of control plane processing capacity, a variable typically orders of magnitude less than the the former. And the fundamental problem of QOS means that you only use it where you have to. QOS works by delaying or discarding packets. It is hard to sell that as a valuable service to ordinary users. I believe Christos's query wasn't about ordinary users or transit traffic, it was regarding control (e.g., routing) traffic. I wouldn't consider network operations or control traffic ordinary users and suspect that if network operators aren't limiting what and at what rate that what is permitted to impact the control plane then their ordinary users should be very concerned. A usual example of this is DDOS attacks much larger than 10 Gbps sustained, throwing bandwidth at the problem yields little or no return. -danny
Re: botnets: web servers, end-systems and Vint Cerf
On Feb 16, 2007, at 10:12 AM, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: You misunderstand. The problem of securing machines *IS* solved. It is possible. It is regularly done with servers connected to the Internet. There is no *COMPUTING* problem or technical problem. The problem of the 100 million machines is a social or business problem. We know how they can be secured, but the solution is not being implemented. So, you're saying we can secure them so long as we put them behind NAT AND humans don't use them? -danny
Re: botnets: web servers, end-systems and Vint Cerf
On Feb 16, 2007, at 11:41 AM, J. Oquendo wrote: After all these years, I'm still surprised a consortium of ISP's haven't figured out a way to do something a-la Packet Fence for their clients where - whenever an infected machine is detected after logging in, that machine is thrown into say a VLAN with instructions on how to clean their machines before they're allowed to go further and stay online. Umm, Mam, I'm sorry, but before you make that emergency call we'll need to go to www.update.nnn and update the OS on your machine, seems you've got some malware there at home somewhere and you're going to need to take care of it for me, OK? Sir, before you can continue watching the World Cup or Super Bowl you'll need to remove the spyware from your son's PC. If you ask me, traffic providers (NSP's/NAP's) and ISP's don't mind this garbage coming out of their networks, if they did they'd actually ban together and do something about it. Its obvious those charging for traffic will say little. Minimized traffic means minimized revenue. IIRC, most North America providers have fixed-rate broadband subscriber plans. All I see is No we despise that kind of traffic along with a shrug and nothing being done about it. I'm sure if some legislative body somewhere started levying fines against providers, the net would be a cleaner place. For comments on 100 million infected machines... Doubtable. Anyone can play fuzzy math games, heck I just strangely figured out that MS is costing me an arm and a leg! While I understand your frustration, lest we not forget, providers are in the business of making money, and solutions of this type today only add to churn, additional operational expense and liability. It's not quite so black and white as you make it, unfortunately. With that, as Sean points out, providers are trying to address the issues in an business-savvy manner and some do seem to have reasonable (IMO) solutions underway. But be careful what you ask for, some of these solutions you're mandating might very well resemble SiteFinder-style schema's (or far worse) in order to justify the investment by the providers. -danny
Re: botnets: web servers, end-systems and Vint Cerf
On Feb 17, 2007, at 6:42 PM, Sean Donelan wrote: Is there a significant difference between the many ISPs implementing walled gardens and other ISPs as far as infection rates? One might presuppose infection rates are exactly the same, at least until that ISPs user base upgrades, patches, auto-updates, AVs, anti-spywares, whatever.. or finds a new ISP. I wonder how long it'd take for such a policy institution to impact an entire 100% user base? I'd likewise be quite keen on seeing empirical evidence on trends in cleanliness and/or churn from any of those ISPs in question, the 3 huge ones in particular - any of those folks *NOG-types? Likely my last message on this tread, as I foresee the OT curmudgeon mounting up (hint to them: delete). -danny
Re: Route Reflector architecture and how to get small customer blocks in to BGP?
On Jan 28, 2007, at 9:06 AM, Joe Provo wrote: Select the latter. Modifying networks statements for move/add/changes invites trouble. Carefully constructed policies to redistribute your connected or static routes into iBGP and tagged appropriately are a win. At the very least, you can limit to subnets of my network's prefixes; If possible, leverage the nice aggregation and limit to my network's local prefixes and you scope potential future havoc. I'm not a big fan of redistribution as I've been bitten by it a few times. One of the biggest issues is that if a policy is being updated and some periodic redistribution process runs the policy at that instant is applied and things not in the policy at that snapshot are not applied (intuitive enough - now). For example, if you're redistributing routes into BGP and coloring with a community based on a route match policy and some of those routes aren't in the policy snapshot then they won't be colored with communities or the like and may be leaked or not advertised otherwise. This is particularly ugly when you've employed implicit permit external advertisement policies where routes that aren't tagged with some value are passed by default. Two lessons learned for me: o If you're going to use redistribution - or not - ensure that all external advertisement policies require explicit match of advertise communities and default is to deny o Don't unnecessarily touch policies or blindly overwrite them periodically, utilize incrementally updated prefix lists as much as possible Given the two conditions above I'm not as wary of redistribution and it may ease configuration managed as Joe suggests. -danny
ISP Security BOF @NANOG 39
Folks, For some crazy reason I've agreed again to chair the ISP Security BOF at NANOG 39 in Toronto. If you've got items you'd like to discuss, like to see discussed, or would prefer not be presented, please let me know ASAP. The two agenda items for the moment are meant to be, in the very spirit of a BOF, loose panel discussions on the following topics: The root of a log: Extracting Intelligence from the Woods Botnet CC: Extirpate or Infiltrate? If you'd be interested in sharing your views, please let me know, although RSVP is not necessary. Slides are welcome but not required. Thanks in advance, see you in Toronto! -danny
Re: BGP unsupported capability code
On Aug 17, 2006, at 10:57 PM, Joe Maimon wrote: If A tries to peer with B, and B sends a BGP capability 64 to A, if A does not support that capability what would proper and/or reasonable behavior for A be? Speaker A MAY send a NOTIFICATION message with Open Message Error/Error Subcode Unsupported Capability and a data field listing capability 64 as the trigger for the NOTIFICATION to speaker B (thereby terminating the session). However, RFC 3392 does NOT require that the local BGP speaker terminate the connection, which has particular utility when considering the implications such a hard requirement may otherwise impose on functions such as dynamic BGP capabilities. (a published source for it, if you could possibly do so.) a) send unsupported capability code 64 lengh 6 ## 2006-08-17 19:17:05 : [bgp/stack]: send NOTIFICATION msg code (Open-Error) subcode(Open Message Error: Unsupported Capability) to peer w9.x4.y7.z9 via socket 415 b) ignore it c) admin defined rfc3392.txt only appears to address the case where if A tries to peer with B, and B sends a BGP capability 64 to A, if A does not support that capability, B MAY terminate the connection with the above notification. (section 3 paragraph 5) If the peer doesn't support the capability and this is conveyed from A to B via a NOTIFICATION message (as illustrated in the log message above), then given the scenario you provide B SHOULD NOT include the capability (64) in any subsequent OPEN message sent to A when attempting to reestablish a BGP connection -- IF B so chooses to attempt to automatically reestablish a connection. Note that the above says SHOULD NOT, not MUST NOT. I'm not quite sure what your point above is, as I think the document sufficiently outlines the required behavior. Although BGP is perhaps (?) still of interest to the NANOG community, I suspect such protocol intricacies are not and would submit that queries of nature are best directed to [EMAIL PROTECTED] -danny
Re: mitigating botnet CCs has become useless
On Aug 9, 2006, at 4:04 AM, Arjan Hulsebos wrote: Maybe so, but that argument doesn't buy me more helpdesk folks. The same holds true for the bandwidth argument, especially now that bandwidth is dirt cheap. On the other hand, it shouldn't be too difficult to come up with a walled garden profile for subs that have infected PCs, basically allowing only access to a filtering proxy, so these subs can download their patches and antivirus updates through it. In addition to they still need to be able to download patches and attempt to fix their system you may not be able to shut off all services for the subscriber regardless - e.g., they've got voice services and you're killing their emergency dialing capabilities? As importantly, broadband SPs are trying to move to triple (quad) play services, how tolerant do you think your average subscriber is to losing cable television services because their kid downloaded some malware? Minimizing subscriber churn and targeting profitable services are critical, most of these solutions today only make the problem worse - when something breaks with vanilla Internet access the first person the subscriber calls is the SP, and the resources cost for fielding those calls exceeds even that of the amortized capital costs for the service - tearing deeper into losses. I half believe that Net Neutrality itself wouldn't be an issue if operators were able to run profitable businesses in broadband service markets. Adding security to the mix only compounds the problem. -danny
Re: mitigating botnet CCs has become useless
On Aug 13, 2006, at 8:35 AM, Laurence F. Sheldon, Jr. wrote: Danny McPherson wrote: As importantly, broadband SPs are trying to move to triple (quad) play services, how tolerant do you think your average subscriber is to losing cable television services because their kid downloaded some malware? At least one of us would applaud an effort to hold people accountable for what they and their kids do. Oops, I see how you could spin it that way... Let me spin it back.. What if the malware your kid's PC (or better yet, your PC) was just infected with came through a virus received in email for which no fix was currently available and the resident AV solution was unaware? Now you can't watch the game tonight, or your favorite show, or use skype to chat with your daughter in Europe, or check your email, [or call 911?] all because the malware triggered something on the network side that resulted in you being walled gardened? My position here is aligned with Sean's and Arjan's. IF you were able to offer any such walled-garden services it's not simply a binary thing, there's a large array of variables that need to be accounted for technically - entirely independent of the economic ones surrounding services that are hardly profitable already. I believe there exists a significant opportunity here for such value- adds for broadband and other services alike, but it's at least initially going to be a rather complicated one. -danny
Re: i am not a list moderator, but i do have a request
On Aug 13, 2006, at 1:02 PM, Paul Vixie wrote: which is, please move these threads to a non-SP mailing list. R [ 41: Danny McPherson ] Re: mitigating botnet CCs has become useless R [ 22: Laurence F. Sheldon] R45: Danny McPherson R [ 62: Laurence F. Sheldon] R [ 162: J. Oquendo] Re: [Full-disclosure] what can be done with botnet CC's? R 211: Payam Tarverdyan Ch R [ 66: Michael Nicks ] i already apologized to the moderators for participating in a non- ops thread here. there are plenty of mailing lists for which botnets are on- topic. nanog is not one and should not become one. nanog has other useful purposes. Interestingly enough, I lurk here 99.999% of the time. I comment on this thread and folks ask to move it to a non-SP mailing list? Perhaps non-operational, but this certainly has direct implications on SPs and I'm of the opinion it's quite relevant - well, certainly as relevant as the past recent threads: SORBS Contact New Latop Policies Fingerprinting and SPAM ID MPLS Gear for Outside Plant [perhaps] Fedex Contact Citrix Load-balancing Detecting Parked Domains I suppose it's more what I feel like reading and sending email about, as opposed to whether/what's on topic or not. I'm done with this thread on NANOG - else the slew of me too responses on this list moderator thread will divert attention from alternative cruft... Wondering if I should send a message to NANOG every time I see a thread of questionable NANOG relevance, -danny
Re: mitigating botnet CCs has become useless
On Aug 4, 2006, at 12:00 AM, [EMAIL PROTECTED] wrote: useless... perhaps. i'm partly of the mind that botnets, p2p networks, manets, and other self-organizing systems are the wave of the future (or even the present) and the technologies, per se, are not inherently evil or even bad. Well, that clearly depends on your prescription for self-organizing. I certainly wouldn't categorize the botnets I'm referring to as self- organizing, in particular when they're being employed in a _very organized manner - most always unbeknownst to each systems ultimate owner, and more and more often in such a way that allows A botherder to employ them for an ever-expanding array of malicious activities. imho, it is short sighted to try and curtail, mitigate, and eradicate these types of technologies - its kind of like trying to kill off SMTP because it only sends spam, FTP because its only used to distribute PR0N... and HTTP because its only used by peadophiles stalking my daughters on MySpace... better to understand how these things are used and figure out how to determine INTENT and then filter on that instead of technological eradication. Right, hence my point. By and large, SPs don't have the time or resources to police the greater Internet, and therefore, they respond in a very reactive fashion when some malicious activity *that* warrants action dictates. Taking out known botnet CC infrastructure is more proactive and at least from my perspective, continues to yield a discernible impact. It's all about ROI - and anything more than reactionary measures only moves them further from profitability. Putting solutions in place that allow the SPs to recoup costs associated with playing sysadmin for customers are the only way they'll be able to give dedicated focus to the problem. just my contrarian 0.02 rupias. I'd expect no less Bill :-) -danny
Re: mitigating botnet CCs has become useless
On Aug 5, 2006, at 3:17 PM, Sean Donelan wrote: Hopefully, by their nature SPs will always be a bit reactive. Unless I want them to, I don't want SPs messing with my traffic. Its my right to connect anything I want, send anything I want, do anything I want with my Internet connection. On the other hand, when I do complain I want the SP to instantly be able to stop anything I don't want, even when I don't know what it is, and be able to track every bad thing that every happened even before I knew it was bad but not keep records of what anyone has done. And of course, I don't think I should pay extra for it. I think I touched on this lightly in one of my previous posts on this topic - but yes, I completely agree.. -danny
Re: mitigating botnet CCs has become useless
On Jul 30, 2006, at 10:37 AM, Gadi Evron wrote: The few hundred *new* IRC-based CCs a month (and change), have been around and static (somewhat) for a while now. At a steady rate of change which maintains the status quo, plus a bit of new blood. In this post I ask the community about what you see, against what we have observed, and try and test my conclusions and numbers against your findings. Gadi, *SPs* today deal with command and control infrastructure on a very tactical basis, and as for specific bots themselves, even more tactically (i.e., usually when some incident requires that they take some response action). They're very incident driven from that respect, and with an attempt to focus on revenue and services profitability, it just amplifies the problem. That is, they're busy turning the steam valves and putting out fires - who has the time for strategizing and waging a global war on organized crime and it's employment of botnets that yields a negligible return on a considerable investment, just cutting deeper into their losses? [disclaimer: the above is a gross oversimplification and many SPs do far more, but it's largely applicable across a broad spectrum of SPs] Heck, they rarely have time to chase DOS attack sources past their network perimeter and today report less than 2% of *actionable* attacks to LEOs. It's an ROI game... While you could spin botnet resurrection a hundred ways, taking out the bots themselves, even if it's often times only as temporal function, is the low hanging fruit and something SPs can understand and instrument. I agree that the root of the problem is the miscreants perpetrating these crimes, and they need to be prosecuted, but the responsibility falls far wider than the SPs. I also accept the references provided by Paul and others, but what's the near-term alternative? -danny
Re: mitigating botnet CCs has become useless
On Aug 3, 2006, at 4:22 PM, Scott Weeks wrote: But shutting them down, that's like the police arresting all the informants. It doesn't stop the crime, it just eradicates all your easy leads. What're folk's thoughts on that? I'm not sure I'd liken shutting CC infrastructure down to arresting the informants. I think that's quite a bad analogy, actually, as informants are [often] third parties while CC infrastructure is used to convey actual execution instructions - which are very often much more than DoS, as John pointed out. -danny
Re: Interesting new spam technique - getting a lot more popular.
On Jun 15, 2006, at 7:06 AM, Kristal, Jeremiah wrote: I don't think it was Extreme that filed it, or at least they didn't write it. It was the good folks at Qwest engineering who came up with the idea, which was implemented (for some low value of implemented) by Extreme. The authors had moved on by the time the RFC was published, but they were certainly Qwesties (and probably CSN before that). Nope, not at all CSN.. I *think* the same idea was floated to Cisco at the same time; their PVLAN was offered in beta not long after Extreme moved super/sub-VLANs into public release. Yep, pointer here, for folks interested in the current state of that work within the IETF: http://www.ietf.org/internet-drafts/draft-sanjib-private-vlan-05.txt Unfortunately for those of us who had to actually implement said abomination, it didn't quite work as well as promised. In fact I was just trying to decide which was more painful, taking over a hosting network with 90% of their hosts in one VLAN (VLAN2, they asked for free advice when they first started to attempt to migrate), or supporting super/sub-VLANs in an operational environment. Customers hated both, but at least they saw better performance once the hosting network was broken up per-customer VLANs. Indeed, as there's a discernible gap there from concept to implementation, implementation to network engineering, beta/EFT, and from network engineering beta/EFT to deployment operationalizing any such technology. If you disregard any of these phases, for whatever reason, it'll likely to come back to bite you. Customers hated it because of some very serious operational flaws. Some stuff was to be expected, like seeing broadcast traffic in all subs under a super-VLAN. As with any new technology, some amount of education is often required when change occurs. In this case, a reasonable response would be to first ask if anything was broken as a result, and if not, then to explain the benefits such a service model provides from a billing, Internet address conservation and security perspective. Customers hating something simply because they liked and are no longer seeing broadcast traffic seems a bit intractable to me. This is the perfect example where a small amount of education can go a long way. Now, if something is broken, OTOH, that's different. Some stuff was truly flawed, like having some small percentage of packets leaking across sub-VLANs. Residential customers don't mind, and probably would never notice. Large corporate clients who are putting important servers in a hosting environment get rather concerned when you start seeing traffic (including cleartext login info) from their neighbors on their interfaces. Indeed, and this is clearly an implementation bug, and potentially, a security vulnerability, if an attacker could determine how to elicit such a behavior. Trying to convince your vendor that this (and other) flaw exists when you're the only client using it in production, and you're pushing several orders of magnitude more traffic than their labs, can be slightly frustrating. Again, this is why it's important to be able to clearly articulate such a problem, and why phases 2 3 above are so critical, and why each new application of such a mechanism requires revisiting the entire process. I personally felt that this was a solution in search of a problem. The enterprise hosting division on an RBOC was probably not the best place to deploy it. IIRC, the enterprise hosting solution of an RBOC wasn't the intended initial application, though I do suspect such a solution would provide considerable advantages in a deployment such as that - again, assuming it was engineered and operationalized appropriately. The current low-end hosting environment is a problem that fits pretty well, but based on my experience in that segment, there is a much bigger return on investment in paying a couple of engineers well enough to manage your VLAN allocations correctly and use existing (generally secondary market) hardware and tools. While I'm not sure about any of the deployment details beyond the initial problem set, which falls pretty much squarely within your that fits pretty well category, I would be interested in hearing what your solution to such a problem is? Certainly, some amount of engineering needs to be applied, and customers that justify their own IP subnets should be handled as such, but in this day and age, I'd hope that most folks separate customers into different Link layer broadcast domains, and employ some bit of cognition WRT Internet address space conversation. -danny
NANOG 37: Security BOF Agenda
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 FYI - -danny - -- Security BOF - NANOG 37 Moderators: Danny Roland - --- Probing Open Recursive Name Servers John Kristoff Analyzing the results of remote open recursive name server probes. We look at the effectiveness of different probing techniques against different sets of data including reflectors used in recent attacks, other known open recursives and a large set of DNS server queriers. Some of the who and what are open will be briefly examined as as well as some unexpected responses to our probes that may invite further analysis. - --- Infrastructure Security Survey Results Craig Labovitz - --- Does Web 2.0 = Security 0.0? Roland Dobbins 'Web 2.0' hosted applications are going mainstream; recent events have highlighted the fact that not only enterprises, but millions of small businesses, SOHO users, and individuals who depend upon these applications are adversely impacted when disruptive network events occur. However, there has to date been little or no engagement between the traditional computer security community, the operational security community, and the developers/providers of these applications. What can be done - and what *should* be done, and by whom - to help integrate 'Web 2.0' application providers into the operational security community? What role, if any, should nsp-sec play? - - Email question for discussion from Monika Machado What tools are used by network operators for event correlation and aggregation and how effective are these tools for trending, analysis and reacting to incidents? - --- Open MIC/Discussion - - -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEhMvhVbIg27AGOxoRAjCeAJ9hltsKqcX0pZenonFc1PwtnCUWlQCgnNpp pGZlxdieQ5kqvTUmG9ljicE= =UCcR -END PGP SIGNATURE-
Re: metric 0 vs 'no metric at all'
On Jan 3, 2006, at 1:03 AM, Daniel Roesen wrote: So the spec is fuzzy about how no MED vs. MED=0 should be treated, but vendors seem to largely agree to no MED == MED 0. I know of no deviation, except the old ERX bug which got fixed (ERX treated no MED as best, even better than MED=0 - contrary to documentation). I recall some earlier implementations from well known vendors that had varying behavior for MED processing as well. Fortunately, the update to RFC 1771: http://www.ietf.org/internet-drafts/draft-ietf-idr-bgp4-26.txt is considerably more explicit about this behavior, as well as a slew of other previously-left-to-the-implementation items ironed out through a great deal of implementation and deployment experience. The BGP Experience and BGP MED Considerations Internet- drafts provide a good bit of additional insight into some of these behaviors. -danny
Re: BGP AS Sets in the wild
On Aug 23, 2005, at 4:36 AM, Abhishek Verma wrote:
I was looking at route-views.routeviews.org for the BGP routes and i
dont see any AS-Sets whatsoever. Are BGP routes with AS-SETs not
generally leaked into the wild?
Is this the case?
Not quite, see below..
I am under the impression that AS_SETs are generated whenever there
are some routes that are aggregated. Is there any other way also to
generate AS_SETs in BGP?
Well, yes, but mostly via proxy aggregation type stuff, which isn't
real common these days (i.e., dynamic proxy aggregation on behalf
of downstream ASes).
Yep, quite a few (I was just looking at this a while back for something
else). As of a few hours ago, route-views had 45 unique prefixes that
contained AS_SET path attributes. Interestingly enough, ~60% of those
AS_SETs only contain a single AS -- hrmm. It's also interesting to see
some of these AS_SETs, albeit only a couple, contain private AS space,
I'm wondering if some naive implementations remote-private-as-type
knob is letting them slip by inside AS_SETs on egress.
[EMAIL PROTECTED] grep { oix-full-snapshot-latest.dat | wc -l
1717
[EMAIL PROTECTED] grep { oix-full-snapshot-latest.dat | grep \ | wc -l
45
[EMAIL PROTECTED] grep { oix-full-snapshot-latest.dat | grep ^...[0-9]
* 24.223.128.0/17 129.250.0.11 1 0 2914
1668 10796 {11060,12262} i
* 24.223.160.0/19 129.250.0.11 1 0 2914
1668 10796 {11060,12262} i
* 64.132.88.0/22 209.161.175.4 0 14608
4323 {33127} i
* 64.132.102.0/24 209.161.175.4 0 14608
4323 {32510} i
* 64.132.220.0/23 129.250.0.1117 0 2914
4323 {18664} i
* 65.17.160.0/19 129.250.0.11 1 0 2914
1668 10796 {11060,12262} i
* 65.189.0.0/16129.250.0.1117 0 2914
3356 10796 {11060,12262} i
* 65.205.32.0/24 129.250.0.1155 0 2914
10913 10913 26134 {30060} i
* 65.221.0.0/19129.250.0.11 0 0 2914 701
11486 {15297} i
* 65.254.96.0/19 129.250.0.11 0 0 2914
7911 2552 {25662,25887} ?
* 66.162.17.0/24 209.161.175.4 0 14608
4323 {29704,65102} i
* 66.163.160.0/19 129.250.0.11 7 0 2914
3561 26085 {14776} i
* 82.135.152.0/21 129.250.0.11 0 0 2914
1299 8764 8764 8764 {31006,34037} i
* 128.116.0.0 129.250.0.11 0 0 2914
7911 14041 {16519} i
* 129.19.0.0 129.250.0.11 0 0 2914
7911 14041 {12145,16519} i
* 129.165.192.0/18 134.55.200.1 0 293
10886 1701 {270,65200} ?
* 140.31.0.0 129.250.0.11 0 0 2914 701
668 {132,3381} i
* 140.32.0.0 129.250.0.11 0 0 2914 701
668 {22,5957,14077} ?
* 147.241.0.0 129.250.0.11 0 0 2914 701
668 {13} i
* 147.249.0.0 129.250.0.11 0 0 2914 701
7381 {3561,6419,12178} i
* 168.215.137.0/24 209.161.175.4 0 14608
4323 {29704} i
* 200.62.40.0/22 129.250.0.11 6 0 2914
5511 18747 {12180,23383,23520} i
* 200.85.0.0/20129.250.0.11 0 0 2914 701
26617 {17079} i
* 201.217.192.0/19 129.250.0.11 6 0 2914
5511 18747 {26596} i
* 202.30.88.0/23 129.250.0.11 5 0 2914
4766 3608 3608 3608 17832 {9494} i
* 202.224.128.0/18 129.250.0.11 258 0 2914
4688 4688 4688 {18071} i
* 206.169.96.0/22 129.250.0.1117 0 2914
4323 {11636} i
* 206.169.208.0/22 129.250.0.1117 0 2914
4323 {21593} i
* 207.19.96.0/21 129.250.0.11 0 0 2914 701
7381 {14455} i
* 207.133.7.0 129.250.0.11 0 0 2914 209
721 27064 6026 {65535} i
* 207.235.48.0/20 209.161.175.4 0 14608
4323 {32258,32418,32434} i
* 207.250.94.0/23 209.161.175.4 0 14608
4323 {33395} i
* 208.16.208.0/21 129.250.0.11 0 0 2914 701
7381 {14455} i
* 208.142.248.0/21 129.250.0.11 0 0 2914
4436 14390 {27481} ?
* 208.254.0.0/19 129.250.0.11 0 0 2914 701
11486 {12156} i
* 209.159.192.0/18 129.250.0.1147 0 2914
30167 20412 {14507} i
* 209.213.209.0129.250.0.11 0 0 2914
7911 6517 {21953} i
* 209.234.168.0/22 209.161.175.4 0 14608
4323 {5710} i
* 210.23.192.0/19 129.250.0.11 4 0 2914
9299 9299 7491 {23862} i
* 210.23.208.0/21 207.246.129.6 0 11608
2914 6453 7491 {23862} i
* 210.23.208.0/20 207.246.129.6 0 11608
2914 6453 7491
Re: Real-Time Mitigation of Denial of Service Attacks Now Available With ATT
On Jun 2, 2004, at 9:25 AM, Jon R. Kibler wrote: The sad fact is that simple ingress and egress filtering would eliminate the majority of bogus traffic on the Internet -- including (D)DoS attacks. If all ISPs would simply drop all outbound packets whose source address is not a valid IP for the subnet of origin, and all inbound packets that do not have valid source IP addresses, the DDoS problem would be (for all intents and purposes) fixed. If proper filtering was done, then any DoS attacks would have to have either valid source IP addresses, or IP addresses that spoofed IPs within their network of origin. In either case, identifying and shutting down the attackers would become a greatly simplified task compared to the mess it is today. Why no filtering by ISPs? Because it takes resources and only benefits the other guy -- unless your network is the one under attack. Maintenance of the ACLs should not be the issue. A single ACL for each subnet would be all that would be required for egress filtering. About 30 ACLs on an inbound border router would be required for ingress filtering. Keeping the ingress ACLs current is a brain-dead task -- just subscribe to the bogon mailing list at cymru.com. ACLs have had a bad reputation for greatly slowing down routers. That may have been true in the past, but properly written ACLs do not seem to have a significant impact on most new routers. Yes, they may cut peak through-put a few percent -- but if you are running that close to the edge, it is time to upgrade anyway. IMHO, there is absolutely no excuse for not doing ingress and egress filtering. In fact, if you are an ISP, I would argue that you are negligent in your fiduciary responsibilities to your customers and shareholders if you are not filtering source IP addresses. Fancy solutions may make great marketing, but simple proper router filtering is a very workable lower-cost solution. (Step down from soap box.) At least, that's my $0.02 worth. While I mostly agree with your sentiment, one minor detail.. Based on recent observations of many folks, spoofing is out of vogue. So much so that some recent discussions I've had with several folks lead me to believe that less than 1% of DDOS attacks today employ source address spoofing. As such, the value of techniques such as backscatter analysis and traceback decrease as well. I suspect that [at least] the perception of wide-scale BCP 38/uRPF and the sheer size and firepower of botnets today has resulted in a very significant decline in source-spoofed attacks. Clever folks actually spoof within the local (sometimes classful) subnet, making it slightly more difficult to identify the concerned host (IF your traceback functions ever make it to the true Internet ingress segment where a host resides, which is more often than not unlikely). I suspect this is largely because we do such a poor job fixing compromised hosts that miscreants needn't worry much about losing significant portions of their botnets to traceback and cleanup - as Rob suggests, they're more concerned with losing them to other miscreants. This is also representative of the inversion in attack methods over the past several years (i.e., the inversion from TCP-SYN type stuff to raw UDP-fill-the-pipe style attacks). Nonetheless, ingress filtering certainly helps significantly. -danny
Re: Real-Time Mitigation of Denial of Service Attacks Now Available With ATT
On Jun 2, 2004, at 10:56 AM, Richard A Steenbergen wrote: What people may being seeing is that poorly randomized source attacks are being automatically filtered by uRPF loose or other means before they ever reach the target. I keep track of my network border filter counters, and believe me spoofed attacks are not going out of style, How do you discriminate *DDOS attacks employing source address spoofing* from broken NATs, rampant worms, PMTU and other related misconfiguration resulting in backscatter and similar garbage - with filter counters? Given, tactically deployed filters in order to mitigate a specific attack to a particular destination would likely glean some value WRT the validity of the source distribution for a given attack, but not generally deployed filters for any destination. And exactly what represents spoofed by your definition? Note again that I explicitly called out **DDOS attacks employing source address spoofing**, which is non-inclusive of spoofing in general employed by worms and the like, or common misconfigurations and brokenness that results in the slew of random garbage floating about. especially from foreign and certain smaller networks. I'd be extremely interested in any empirical evidence you have to support this, and in better understanding exactly how you determined foreign and certain smaller networks were indeed the source of many of these spoofed packets. As a customer of someone who does this kind of filtering and maintains sufficient border capacity, you may never see the gigabits of src bogons, protocol 0 or 255, port 0, 40 byte syns w/no MSS option, etc, and assume that these attacks are out of style because the only ones that get through are the WinXP MSS+SACK unforged drone SYNs. I agree, if it's filtered before someone observes it, it won't be observed :-) However, distinguishing between coordinated DDOS attacks that employ source address spoofing and run of the mill spoofing (by worms and the like) or simple misconfiguration of some sort resulting in backscatter is key. -danny
Re: Real-Time Mitigation of Denial of Service Attacks Now Available With ATT
On Jun 2, 2004, at 12:36 PM, Richard A Steenbergen wrote: If it walks like a duck, and it sounds like a duck, it is probably a duck. RFC1918 sourced space, most likely from misconfigured NATs and such, account for only a very small amount of the bogon-source packets which go splat. But worms, OTOH, seems to be much more persistent. Most of the DoS attempts by volume don't fall into the category of questionable. When you see a 100Mbps stream (from a single ingress interface, with consistant TTL's) of IP proto 0 or 255, or tcp port 0, or classic SYN flooders (SYN w/no MSS) or stream (randomized seq# and fixed ack# on a packet w/TH_ACK flag only) targetting a specific IP/port with a source address of iph.ip_src.s_addr = random(), it is pretty easy to tell those apart from the usual background noise of a worm. Sure.. Some days it helps to actually have an operational network, instead of being a researcher. Even without interesting tools it isn't terribly hard to look at your PNI graphs, match up the hundreds-of-meg spikes with specific DoS incidents, and go from there. Not to point fingers at anyone in particular, but it seems to be the same foreign networks who tend to have little control over their spammers. Heh.. I certainly don't consider myself a researcher, or an operator (any longer) for that matter (though I do have access to a significant amount of both research and operational data and tend not to call a duck a goose simply because I heard a quack :-) -danny
Re: handling ddos attacks
On May 20, 2004, at 8:10 PM, Tim Wilde wrote: Call your local branch of the US Secret Service, if you're in the states, and ask for their electronic crimes division. If you're not in the states, contact your comprable local authority. They can work with you to coordinate with other jurisdictions, etc. You may have some luck directly with the local police at the point of origin, but it generally helps to have a broader agency involved to coordinate matters. I'd love to hear from anyone who has actually been successful prosecuting an attacker for launching a distributed DOS attack. I suspect it occurs very INfrequently (with the recent trend in extortion aside, as it often results in paper trails) -- unfortunately. Any pointers? -danny
Re: BGP Exploit
On May 12, 2004, at 2:41 PM, Mark Johnson wrote: What if sessions were attacked without MD5 in place. We would just see session resets. As these happen anyway frequently at peering points is there any straightforward way to determine if the vulnerability caused the reset? Depends on why it happens frequently. If it happens because you've got Network/Transport Layer or underlying connection problems then there's some other brokenness you should probably be more concerned with. If you're referring to session resets because of a peer or user action then something akin to Last reset due to FOO can likely be gleaned from show bgp neighbor output, especially since BGP performs graceful shutdown via notification messages under normal conditions I.e., you should probably be very concerned with any session reset for which no valid explanation is available via CLI or other means. -danny
NSP-SEC BOF CFP
Folks, Merike I are chairing the NSP-SEC BOF at NANOG in SF in a few weeks. Anyone with topics you'd like to see discussed, or topics you'd like to discuss, please let us know ASAP. Thanks! -danny merike
Re: UUNet Offer New Protection Against DDoS
On Mar 3, 2004, at 11:24 AM, Stephen Perciballi wrote: To the best of my knowledge, MCI/UUNET ~was~ the first to implement this. I've been using it for well over a year now. Indeed. One could even get fancy and set of different community sets to allow customers to drop traffic only on peering routers (as opposed to customer or all routers, etc..). The Customer-Triggered Real Time Blackhole tutorial that Chris, Tim and I gave in Miami talks about how to go about doing this. One step further is uRPF coupling with blackhole routing for sourced- based drops, though I suspect you probably won't want to do this with customers :-) Finally, the BGP Flow Specification stuff provides a start at a more granular BGP-based method by employing new AFI/SAFI. If you've got feedback please pass it along. http://www.tcb.net/draft-marques-idr-flow-spec-00.txt -danny
Re: BGP - weight
On Feb 14, 2004, at 5:23 AM, Sven Huster wrote: Dumb question: If I apply a equal weight to all our transit/peers, will that affect route announcements to iBGP or eBGP peers anyhow? Yes, given that it's a local parameter (i.e., not BGP, per se, though it does impact what's installed in the BGP RIB and what's subsequently advertised to your peers), you'll likely begin preferring more routes via eBGP learned peers per subsequent attributes in the best path selection algorithm (e.g., MED, AS_PATH, even LOCAL_PREF) won't be considered. We got a very simple setup: - 2 routers on the border to transit/peers (that's were i want to set the weight) - 1 edge router to customers - core router running BGP as well What I want to achieve is that traffic leaves through the border router it arrived, rather than have it bounced around. Perhaps you should first look at other reasons why this may be occurring (e.g., accepting MEDs from one peer and not the other, accepting MEDs from both but different policies are employed to derive their values, AS_PATH suggests a better path, etc..) -- then comes preference for eBGP over iBGP. We had some recent issues were it looks like the core got out of sync with the border (looks more like a sw issue than just convergence delay) and packets bounced back and forth between them. This could be any of a number of things.. Without more information I'd be hesitant to start tweaking knobs. I know this doesn't solve the cause but the before digging for the initial reason I want a quick workaround. Weight is a very influential parameter. I'm not a big fan of configuring routing policies that are entirely local to a system, for obvious reasons. But I do suspect it would accomplish what you're trying to achieve. -danny
Re: Hey, QWEST clean up your network
Not sure how many places you intend to post this or related messages, but if you've got a problem vote with your money. Whining to NANOG and a slew of other mailing lists and still giving money to Qwest seems silly to me... Likewise, the Qwest folks likely aren't quite as clueless as you've attempted to portray them over the last few days, silly policies (policies that are clearly in place for a reason) can be fixed -- and I assure you, above all else, money talks... -danny On Thursday, August 28, 2003, at 09:25 PM, John Brown wrote: Seems like QWEST doesn't have any edge ACL's in place to deal with this lovely worm issue. Count Source Prexix, rounded up to a /16 144 208.46.0.0 199 65.114.0.0 347 208.45.0.0 462 65.118.0.0 486 65.119.0.0 702 208.44.0.0 2340TOTAL Packets out of 2500 for 2 seconds This is ICMP and TCP MS bad traffic for a 2500 packet capture on a DS1 that is directly connected to Qwest. Ergo, Qwest is the transit provider. Capture period was about 2 seconds. ICK According to Qwest Tech/Noc people they can't leave filters up for more than 1 day. Given that this worm has lasted more than 1 day, I'd think its reasonable to leave filters up for say more than one day The other thing I learned from QWEST IP-NOC was that it seems managment decided *NOT TO* filter packets related to this worm issue at the edge.. john brown AS 10480 and others
Re: Hey, QWEST clean up your network
On Thursday, August 28, 2003, at 09:51 PM, John Brown wrote: Given general operational nature, I posted to NANOG, so that: 1. money can talk, others will see one view of this provider Don't talk with other peoples money, talk with your own. If you plan to post to NANOG, it'd be a wise assumption that a significant subset of the folks here reside on other lists you post to as well. 2. operationally maybe something will get done Perhaps. Though if/when it does, it'll be Qwest and you that will be involved, no one here. 3. policy wise maybe this provider will change its policy Perhaps, though given the discussions on this and a hundred other lists in the last three weeks, I'm not sure providers know what to do. As Sean points out, every other email contradicts the previous. If I filter, I'm responsive, clueful saving the Internet. When something breaks as a result, I'm clueless and trying to play netpolice, violating my SLA, plain suck, and need to just worry about delivering packets. 4. Qwest said their people had installed the ACL's properly my evidence is to the contrary. Hence the need to further engage with Qwest, folks here will be of little benefit at the end of the day. The customer that was impacted is certainly considering their options. I suspect they will vote with their checkbook. PS: Slew == 1 Private email list, 1, Well known public list 1 Local Public-ish list. Slew != as large as it may have sounded... Correct me if I'm wrong, but I seem to recall a strikingly similar message posted to several mailing lists regarding very similar topics and the same provider within the past .. 4 days (no, it was 2 days)? Had it not been for that I wouldn't have bothered posting. One attempt to humiliate your provider in order to trigger some action is perhaps arguable, two or more is just plain annoying. Policies are sometimes in place for good reasons, sometimes because the makers of said policy are void clue. To assume they are inplace for good reason is a leap imho. So providers should play netpolice or Internet-Firewall-provider some amount of time, depending on _your gauge of the activity of a given incident? Folks need to realize that if large networks didn't have policies of this sort in place they'd be blocking pretty much every port on every interface by now.. You can't have it both ways... -danny
Re: Lazy Engineers and Viable Excuses
On Monday, August 25, 2003, at 07:32 PM, Jared Mauch wrote: You of course are correct with the trusting of the data, but we are in a somewhat of a chicken and egg situation. If people don't trust the IRR, they don't filter on it, and then the data is allowed to get out of date. But people who maliciously add bogus (or excessive route objects for example) are easy to track down. This is what the maintainer objects are for and why the IRR software keeps logs of the messages (including headers) that are submitted. I fully agree with the cart/horse chicken/egg analogy. If SPs began employing IRRs more fully and more work went into commercialization of IRR infrastructure and tools (and perhaps some RIR feedback loop were created) they'd improve. Instead, folks are running about designing new protocols far more complex than BGP already is, that *still* require some authority. When in reality, 99% of the vulnerabilities could have been solved with what was in place 10 years ago. Folks are striving for perfect security, which is fine, but they've ignored the reasons why we don't even have crappy security. -danny
Lazy Engineers and Viable Excuses
Again... If folks were to implement explicit prefix filtering *everywhere* it wouldn't be necessary to filter only bogons and other miscellany explicitly. Something of this sort would only lower the lazy bar (is it possible?) for the clueless and/or lazy (those which Rob's list currently accommodates, which is better than nothing, BUT.. -- no offense Rob, I'm pretty sure our beliefs are aligned here :-). If folks want to filter, please, please, PLEASE, employ IRR infrastructure and filter customers *AND* peers explicitly. If your vendors have issues with this, push them to fix it. Then you don't have to worry about bogons, max-prefixes, route hijacking, de-aggregation, or... Then we can worry about IRR infrastructure hardening and accuracy and derive explicit data plane filters from the output, as well as other tangible benefits. Is it really that hard? -danny
Re: BGP route tracking.
On Thursday, August 14, 2003, at 11:34 PM, Danny McPherson wrote: Current_Total: 120,475 Max_Total: 123,814 Average_Total: 122,029 I failed to consider that today's average has been skewed by the outage data being factored for the last 10 hours or so towards the Daily Average. I looked at yesterday's daily average which was 122855, just over 800 more prefixes. Current v. Average: 98.73% (1554 prefixes) So comparing yesterday's average to today's current I get closer to the value Rob and others posted: Current v. Yesterday's Average:98.06% (2380 prefixes) -danny
Re: microsoft.com
*** ns2.nv.cox.net can't find www.windowsupdate.com: Non-existent host/domain Some news outlets are reporting this is actually Microsoft's plan, Sure it was, and it's probably the best thing MS could have done (for themselves AND the larger Internet) given the circumstances. After all, infected systems aren't going to stop scanning and DOS attacks from a huge number of compromised hosts targeting windowsupdate.com IPs is simply going to result in increased network utilization for a bunch of garbage traffic that'll either be dropped as a result of congestion on some networks, blackholed on others (from the folks that care no more about MS being DOS attacked then the next guy, but do care about their networks availability and the Internet in general), or hit some severely crippled server(s). MS has bugs, sure, and there's probably no excuse for lots of them. However, it could have been linux or any other OS. Folks give MS a hard time for the same reason they give Cisco a hard time -- because their products are nearly ubiquitous. I'm not going to dive into some huge rant here (others have articulated this point nicely already), some folks are much more passionate than I about the issue and I don't care to spend the cycles arguing something I care little about. MS isn't going away any time soon, like it or not, and the only way problems of this sort (that have been disclosed) are going to be cleanly resolved is by end users patching their systems. -danny PS: If folks are going to rant about MS products being horrible they might want to consider using non-MS products and posting to NANOG from non-MS mail clients/systems *8^).
Re: BGP route tracking.
Here's some of what I've seen at this point: [table format might be munged by some fonts] PrefixDaily Daily Length *Current Max Average /24 65,900 68,497 67,259 /239,904 10,157 10,027 /229,053 9,211 9,110 /216,035 6,106 6,045 /208,485 8,560 8,487 /198,175 8,221 8,161 /183,007 3,031 3,005 /171,693 1,705 1,690 /167,293 7,396 7,326 /15 473 473 469 /14 263 263 262 /13 98 98 97 /12 55 55 54 /11 12 12 11 /106 6 5 /9 4 4 3 /819 19 18 Current_Total: 120,475 Max_Total: 123,814 Average_Total: 122,029 Current v. Average: 98.73% (1554 prefixes) * Current Based on my Snapshot @9P MDT 8.14.2003 Note: This data came from an Arbor system on a regional network. There's a bunch more qualitative and quantitative data I'll munge through when I get a chance, could be interesting. If folks are looking for anything in particular let me know, I can likely dig it up... -danny
Re: Best Practices for Loopback addressing (Core routers VPNCPE)
On 6/6/03 10:05 AM, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: I was wondering what are the choices made by Service Providers on the loopback addressing. The context is an IP/MPLS Backbone providing both Internet and BGP-VPN services. If the BGP Identifier, which is used for connection collision resolution and path selection (among others seemingly random things) conflicts you'll have issues. In my experience even mediocre IP addressing frameworks typically don't have issues with RIRs when space is appropriately justified (i.e., there typically aren't issues with loopback and inter-router address space justification). What I'd be more concerned with is loopback IP allocation and it's effect on aggregation, stability, and other network policies (e.g., source-interface type stuff). For example, using a single contiguous block for all loopback IPs significantly simplifies filtering policies. OTOH, you may opt to provide more optimal aggregation and allocate loopback IPs from the same block as p-t-p IPs (per router, POP, region or other) such that less information needs to be carried internally in your IGP (or BGP). RFC 2519 provides some guidelines for inter-domain route aggregation. A slew of other documents and books provide IP address allocation guidelines as well. -danny
Re: Route Supression Problem
Dampening is done on the eBGP router where the route enters the AS, and, unless I'm mistaken, per route/path and not per prefix. So the flapping that ISP A sees from ISP B is a completely seperate thing from the flapping that ISP A sees from its customer's customer as far as the dampening algorithm is concerned. Nope. It's per-prefix. Have a look at the pointer Randy provided. I think one of the section of RFC 2439 alludes to this as well. -danny
Re: Route Supression Problem
Hrmm... Then care to take a stab at explaining the findings in the document that Randy referenced earlier? -danny [EMAIL PROTECTED] (Danny McPherson) writes: Dampening is done on the eBGP router where the route enters the AS, and, unless I'm mistaken, per route/path and not per prefix. So the flapping that ISP A sees from ISP B is a completely seperate thing from the flapping that ISP A sees from its customer's customer as far as the dampening algorithm is concerned. Nope. It's per-prefix. Nope. It is per-path. Pedro.
Re: Route Supression Problem
Thus the application effect being talked about. Sure, I understand that. I was making a different assumption... That the sources of traffic were likely from downstream ASs, not ISP A, or even B or C, and as such, the multiplication could happen per prefix. However, without knowing the number of penalizing events, what constitutes a penalizing event, and aggressiveness of your peers suppression policies -- or the source distribution of the traffic, it's tough to tell. -danny
Re: Who uses RADB? [was BGP to doom us all]
Yes, at iMCI (we) had our own registry, MCI-RR, but we only used it (in addition to data from the other IRRs) to generate customer prefix filters, not peers. Cable Wireless still uses the RR, now know as CW-RR. -danny As I remember and I could be wrong, its been a few years now, when I worked for iMCI we did and we moved over to CW we still did.
Re: Who uses RADB? [was BGP to doom us all]
Who actually uses RADB to build filters other than Verio? While my experience with other providers is limited Verio is the only one (of the ones we have used) who used RADB entries for BGP peers. Level3 do atleast. Most European providers do. For customers, though not inter-provider. -danny
Re: Who uses RADB? [was BGP to doom us all]
as you say for customers only. Inter-provider we have basic bogon checking plus maximum prefix. Its too unwieldy to build when you have peers exchanging thousands of routes... theres a belief that the peer should be behaving responsibly tho and this is a condition of most bilateral peering contracts. Unfortunately, contracts don't fix mis-(or malicious-) configurations on compromised routers or from a peers disgruntled worker. Going back to the original topic on this thread I would expect a deliberate attack on BGP routing to come from a customer not a provider such as Level3, if they are filtering in turn to their customers we have a reasonable amount of sanity checking going on A large provider I worked for in the past had a router maliciously configured to inject a more-specific prefix for a very popular network. Even the popular networks provider sent the traffic to us. Had explicit prefix-based inter- provider filtering been in place it would not have occurred, or at least the whole Internet wouldn't have been affected. With the IRRs and inter-provider filtering it's the whole chicken and egg thing. Inter-provider filters aren't in place because no one cares about IRRs (even though they have other operational value as well). Vendors don't support the amount of prefix filters required because they say no one uses them. Heck, lots of folks still don't ingress filter routes (or packets) from their customers. When ANS used to employ inter-provider filters the biggest problem was getting them updated and bouncing routes or sessions. That's no excuse anymore because pretty much everyone supports the ability to incrementally update filters, and BGP Route Refresh fixes the bounce the session/route thing. So, let's recap why no one uses them (as many have said already in the related thread): Laziness. The same laziness that results in the slew of other things many folks have pointed out not being addressed. -danny
Re: Who uses RADB? [was BGP to doom us all]
You forgot the other one - expense. AFAIK all of the registries have fees or require you to be a customer. If there is no operational value First problem, you see no operational value. for me why would I want to spend the money? Money changing hands no longer makes the IRR a dis-interested third party or research project, they now have a vested interested in object integrity and availability, and perhaps can afford resources to support these and other enhancements. I realize most of you work for companies that consider a million dollars chump change but that is not the case everywhere. If you can give me a convincing reason to register my routes in a RADB I will - but at this point I have yet to see it. When one of your peers starts filtering inter-provider based on IRR and your prefixes aren't permitted, or one of your peers advertises you more- specifics for your customers prefixes, or better yet, your routers are compromised and used to disrupt service to some now very unhappy multi- million dollar online enterprise that will seek reimbursement -- maybe that'll help convince you... What does a RADB tell you about a non-transit network that you can't see from BGP and WHOIS? There is no more security in RADB than there is in our current method of notifying our peers of the netblocks we are announcing. You should read up on it, there's a bit more capability there than just a prefix and POC email address. -danny
Re: Cascading Failures Could Crash the Global Internet
RFC 3439 talks of coupling amplification and it's relation to the Internet. -danny Sigh, there are differences between tightly coupled networks, such as the electric power grid and loosely couple networks like the Internet. But there are also some similarities, such as electric grids use DC interconnections to limit how far AC disturbances propagate; the Internet uses AS interconnections to limit IGP disturbances from propagating. http://sci.newsfactor.com/perl/story/20686.html The actual article requires payment to read http://ojps.aip.org/getabs/servlet/GetabsServlet?prog=normalid=PLEEE8660606510201idtype=cvipsgifs=Yes
Re: Who does source address validation? (was Re: what's that smell?)
If there is a magic solution, I would love to hear about it. I strongly doubt any of the large providers perform dataplane source address validation from peers. Heck, I doubt any perform explicit route filtering on routes learned from peers at the control plane. Ideally, one would first employ some mechanism to generate *explicit* ingress BGP route filters. With BGP Route Refresh the largest offshoot (manual session reset or bouncing the route) is no longer necessary. From there, you could either use BGP's Adj-RIBs-In in some uRPFish thing, or employ the same set of BGP route filters for source address filters. Of course, then the lack of registry route object integrity, secure update mechanism, etc.., etc... comes to question. -danny
Re: Who does source address validation? (was Re: what's that smell?)
install this on all your internal, upstream, downstream interfaces (cisco router) [cef required]: ip verify unicast source reachable-via any This will drop all packets on the interface that do not have a way to return them in your routing table. Of course, this is the IP RIB and may not include all the potential paths in the BGP Adj-RIBs-In, right? As such, you've still got the potential for asymmetric routing to break things. Juniper has a somewhat viable solution to the 100% source validation for bgp customers. they will consider non-best paths in their unicast-rpf check on the customer interface. This means that even if 35.0.0.0/8 is best returned via your peer instead of via the provider the packet came in, but they are advertizing the prefix to you, you will not drop the packet. What's a bgp customer? Can they support 500K+ uRPF entries here? -danny
Re: Who does source address validation? (was Re: what's that smell?)
reachable-via any means you're only going to drop the packet if you don't have *ANY* route back to them. What's a route? An IP RIB instance? A BGP Loc-RIB instance? An IGP LSDB IP prefix entry? A BGP Adj-RIB-In instance? I think you mean if you don't have *ANY* **FIB** entry for the source address. If I peer with two large providers on the same router and both have prefix D.1 behind them and advertise the prefix to me, it's likely that only one of those two paths is going to make it into the BGP Loc-RIB (and subsequently, the IP RIB then FIB). If I use ANY FIB entry as proof that it's a valid source then that only addresses RFC1918ish space and only suggest that I first need to generate an invalid BGP route for the prefix, then spoof the packets. This doesn't fix spoofing with global IP addresses. If I use only entries that occur in the RIB and associate them with the receiving interface and receive a packet with an SA of D.1 from the peer whose path wasn't installed in the BGP Loc-RIB then I'll drop it. (And there's nothing broken with this configuration -- it's why we have routers with 1 million BGP paths but only 150K routes/fib entries, as I'm sure you know). If you're going to do source address validation then you need to associated all potential valid paths for a given prefix with the associated ingress interface, else it's mostly useless. -danny
Re: Who does source address validation? (was Re: what's that smell?)
Yes, but if i continue in my ideal situation of people (mostly) filter their bgp customers, so they won't announce the 1918 space, or similar. even the large peers filter out each other so they don't pick up 1918 announcements. Plus people use Robs Secure IOS Template to drop extraneous bgp announcements for unregistered/unassigned space (from IANA). What you're doing makes plenty of sense, we agree on that. I just wanted to be sure folks understood it doesn't validate valid sources. -danny
Re: Routing Protocol Security
I know of several incidents where invalid routing announcements were maliciously employed in order to cause reachability problems to the destination prefix network. It still bugs me that router vendors don't provide the capability to support inter-provider filters (read: 10s or 100s of thousands of instances). But heck, some providers still don't even filter routing announcements for customer prefixes explicitly. This is a HUGE vulnerability. Likewise, employing the same set of inter-provider filters at the data plane as ingress source filters would suppress the bulk of these cheesy spoofed-source address attacks. This is another HUGE vulnerability (providing a solution in hardware is a bit more difficult -- though not impossible!). But heck, some providers still don't employ customer ingress filtering. Of course, then the vulnerability would be the registries, and subsequent components therein. The again, at least the former was done many moons ago, though wasn't real successful given the network, 24 hour turnarounds, etc.. However, things like BGP Route Refresh and the like could alleviate most of the offshoots of the time. Now, back to the router vendor support issue, if that's what you were soliciting input on...? -danny
Re: Survey on IBGP persistent route oscillation problem
We have a similar situation (RR + always-compare-MED off), and the BGP table version keeps changing at 1K/min (http://performance.cn.net:2003/). I suspect some route meet the criteria of IDR-oscillation draft. But in real world, it's very hard to pick up the pattern depicted in the draft from a huge log of debug bgp output. It's actually quite trivial to identify. Have a look at: http://www.cisco.com/warp/public/770/fn12942.html Juniper syntax would result in the same.. 1. How many time do our operator really find and affected by the problem depicted in draft-ietf-idr-route-oscillation-01.txt I know of more than a few, and would bet that if additional folks took the time to look, they'd realize it was occurring as well. 2. From my experience, most flapping seems to be oscillation route which escaped the eBGP damping protection. And for this, we need adjust damping parameters from RIPE 220 to make up. Nope. Dampening doesn't resolve this because it occurs intrA-domain. 3. Anyone see oscillation been magnified when injected into RR (cluster structure) ? Not sure I understand what you mean by magnified? Have a look at the Cisco FN, it's useful in understanding and identifying the problem. -danny
