Re: [admin] [summary] RE: YouTube IP Hijacking
On Feb 27, 2008, at 2:09 AM, Adrian Chadd wrote: (speaking as someone who has built large ACLs/prefix-lists and has 6MB+ configs that can't be loaded on my routers. without vendor support those that want to do the right thing can't, so the game is lost). I remember the days of making rtconfig work properly in various situations (heck, do people still use that? Does it even do IPv6 right?) and building router configs out of both public and private IRR data. Perhaps if the entry barrier to building dynamically generated router configurations were lowered significantly (to the point of it being free, GUI, and multi-platform) then it may be used for new network designs by people starting off. Getting Cisco/Juniper/etc to push -that- as part of their best practices for network design would be quite helpful. The problem isn't that the router config is too easy Jared, its that there's no nice and easy way of doing it right from scratch that matches the sort of newbie network operators that exist today. For examples of what new school netops are like, visit isp-* lists. There's a lot of clue there, its just different and haven't learnt from the old school experience clue, and its amusing/sad to watch people make the same mistakes you all did in the 90s. :) (Where's vijay now when science for generated network configurations is needed?) Make the public tools better. Push the tools as best practice. ADrian Well there is a slight risk that the more you will improve the automated tools, the less net engineers will actually understand the reasons why it is done... That being said, all the filtering work is a significant part of every Network Engineer's work, and is not that big of a deal. Education is the art of repeating, and has always been. I'm not saying we should systematically point the ones making mistakes and make it public, what I'm saying is that the reason why mailing lists such as nanog exist is actually mutual education. I'm sure the people involved in this incident were (or now are) Nanog readers, and that they've understood the message. Still, what should be done is, I assume, centralizing the info in one single mail, kept for the record: - list the incident chronology - analyze what technical mistake lead to that - list -all- the ways to prevent it Another mean of education is including the analysis of this incident (troubleshooting, resolution, implementing means to avoid it) next Nanog agenda, which I already think is the case :) Greg VILLAIN
Re: [admin] [summary] RE: YouTube IP Hijacking
Alex Pilosov wrote: Oh yeah, d'oh! Thanks for correction. But that is also an important point against PHAS and IRRPT filtering - they are powerless against truly malicious hijacker (one that would register route in IRR, add the right origin-as to AS-SET, and use correct origin). With a decent LIR DB (like the RIPE DB) this is only possible if an hijacker breaks the authentication of the according database objects which is a pain in the a** _if_ the objects use a proper authentication scheme like PGP. -- Arnd
Re: [admin] [summary] RE: YouTube IP Hijacking
On 26/02/2008 12:06, Arnd Vehling [EMAIL PROTECTED] wrote: [...] With a decent LIR DB (like the RIPE DB) this is only possible if an hijacker breaks the authentication of the according database objects which is a pain in the a** _if_ the objects use a proper authentication scheme like PGP. I wonder what percentage of maintainers in the RIPE database only have PGP and/or X.509 auth schemes. I'd be surprised if it was as high as 5%. Leo
Re: [admin] [summary] RE: YouTube IP Hijacking
Leo Vegoda wrote: On 26/02/2008 12:06, Arnd Vehling [EMAIL PROTECTED] wrote: [...] With a decent LIR DB (like the RIPE DB) this is only possible if an hijacker breaks the authentication of the according database objects which is a pain in the a** _if_ the objects use a proper authentication scheme like PGP. I wonder what percentage of maintainers in the RIPE database only have PGP and/or X.509 auth schemes. I'd be surprised if it was as high as 5%. True, but thats still better than having no authentication at all and its possible to require strong authentication on inetnum, route and AS objects. I just cant understand why LIR's like ARIN dont have any decent methods for this implemented in their DB. Or did this change recently? -- Arnd
Re: [admin] [summary] RE: YouTube IP Hijacking
Alex Pilosov ha scritto: Facts: * AS17557 announced more specific /24 to 3491, which propagated to wider internets I think that they should use remote triggered blackhole filtering with no-export community. In this way they do the job with no impact on the rest of internet. Regards, Gianluca
Re: [admin] [summary] RE: YouTube IP Hijacking
On Tue, Feb 26, 2008 at 10:40 AM, hjan [EMAIL PROTECTED] wrote: I think that they should use remote triggered blackhole filtering with no-export community. In this way they do the job with no impact on the rest of internet. so, certainly this isn't a bad idea, but given as an example: http://www.secsup.org/CustomerBlackHole/ (Sorry not a perfect example, but illustrates my point) instead of: ip route my.offensive.material.0 255.255.255.0 Null0 tag 12345 the operator in question (person not place) types: ip route my.offensive.material.0 255.255.255.0 Null0 tag 1234 oops, a simple cut/paste mistake means that a route didn't get tagged properly, didn't get community tagged properly, didn't get set no-export and didn't get kept internally :( There is no SINGLE fix for this, there is a belt+suspenders approach: 1) Know what you are advertising (customer side of the puzzle) 2) Know what you are expecting to recieve (provider side of the puzzle) 3) plan for failures in both parts of this puzzle. -Chris
RE: [admin] [summary] RE: YouTube IP Hijacking
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Missing a tag in the trigger is why you put the Murphy Filters in the trigger router's route-map (the point you were getting at but being even more explicit). In my route map on the trigger router, I would not allow any static route triggers which did not have an exact match. I would also set the BGP advertisement to have - by default - the no-export community, a community range for all my triggers, and limit all my triggers to be below /24 (i.e /25 - /32). I then have three things to set my egress prefix filters to all my peers and customers: - comply with the default communities (no export) - filter all communities in my trigger range - filter anything in the /25 - /32 range. BTW - Murphy Filters is my term for policy filters which expect Murphy's Law of Networking to kick in. You have to expect human error. In addition to this, I would have my upstream mirror my filters. Life sucks when you advertise big blocks of the Internet and you become one giant sink hole (until you go congestion collapse, drop the BGP session and start flapping like crazy). -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Christopher Morrow Sent: Tuesday, February 26, 2008 8:59 AM To: hjan Cc: nanog@merit.edu Subject: Re: [admin] [summary] RE: YouTube IP Hijacking On Tue, Feb 26, 2008 at 10:40 AM, hjan [EMAIL PROTECTED] wrote: I think that they should use remote triggered blackhole filtering with no-export community. In this way they do the job with no impact on the rest of internet. so, certainly this isn't a bad idea, but given as an example: http://www.secsup.org/CustomerBlackHole/ (Sorry not a perfect example, but illustrates my point) instead of: ip route my.offensive.material.0 255.255.255.0 Null0 tag 12345 the operator in question (person not place) types: ip route my.offensive.material.0 255.255.255.0 Null0 tag 1234 oops, a simple cut/paste mistake means that a route didn't get tagged properly, didn't get community tagged properly, didn't get set no-export and didn't get kept internally :( There is no SINGLE fix for this, there is a belt+suspenders approach: 1) Know what you are advertising (customer side of the puzzle) 2) Know what you are expecting to recieve (provider side of the puzzle) 3) plan for failures in both parts of this puzzle. -Chris -BEGIN PGP SIGNATURE- Version: PGP 8.1 iQA/AwUBR8RSF7/UEA/xivvmEQJUKACfZB+typ7sIJMnDS+QrO0MqGED+CYAoKFC iBmY+pq0CohSIJwtu5pgzCJt =xiog -END PGP SIGNATURE-
Re: [admin] [summary] RE: YouTube IP Hijacking
The biggest problem here is that Cisco needs to change their defaults to require more configuration than router bgp X neighbor 1.2.3.4 remote-as A When that's the bar for the complexity required for setting up BGP, bad things WILL happen. Period. Cisco has taken all these years and not stepped up in providing any sort of a reasonable change to the marketplace as others have done. This hurts the industry as a whole, and hurts the perception that we can't route. As for other problems with leadership, there's no good way to manage large configurations on the platforms, nor a reasonable size of NVRAM provided either. The list goes on and on, and I've communicated this more than once to the company. Nobody cares about this basic infrasturcture at Cisco, or at least nobody that can make something happen. Instead people care about what product is intruding on their turf and how to defend it instead of building a better product and improving things. Honestly, it's a lost cause and SP's don't account for any significant amount of revenue. If someone at Cisco cares to address these things, i'm interested in helping but it's clear that the head-in-the-sand policy by upper mgmt lives on and they'd rather fight amongst themselves and risk the industry as a whole because of their antics. - Jared (speaking as someone who has built large ACLs/prefix-lists and has 6MB+ configs that can't be loaded on my routers. without vendor support those that want to do the right thing can't, so the game is lost). -- Jared Mauch | pgp key available via finger from [EMAIL PROTECTED] clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: [admin] [summary] RE: YouTube IP Hijacking
for a list filled with network operators and engineers, the lot of you are quick to whip out lawyers and courts and international tribunals. perhaps I missed the message, but has anyone mentioned the direct economic impact of SFI? as a responsible network operator, would you peer with a network that didn't filter their customers and was consistently linked to route leakage issues? when is enough, enough (in general, not speaking to any specific network)? also, this: On Tue, Feb 26, 2008 at 10:21 AM, Jared Mauch [EMAIL PROTECTED] wrote: The biggest problem here is that Cisco needs to change their defaults to require more configuration than router bgp X neighbor 1.2.3.4 remote-as A When that's the bar for the complexity required for setting up BGP, bad things WILL happen. Period.
Re: [admin] [summary] RE: YouTube IP Hijacking
(speaking as someone who has built large ACLs/prefix-lists and has 6MB+ configs that can't be loaded on my routers. without vendor support those that want to do the right thing can't, so the game is lost). I remember the days of making rtconfig work properly in various situations (heck, do people still use that? Does it even do IPv6 right?) and building router configs out of both public and private IRR data. Perhaps if the entry barrier to building dynamically generated router configurations were lowered significantly (to the point of it being free, GUI, and multi-platform) then it may be used for new network designs by people starting off. Getting Cisco/Juniper/etc to push -that- as part of their best practices for network design would be quite helpful. The problem isn't that the router config is too easy Jared, its that there's no nice and easy way of doing it right from scratch that matches the sort of newbie network operators that exist today. For examples of what new school netops are like, visit isp-* lists. There's a lot of clue there, its just different and haven't learnt from the old school experience clue, and its amusing/sad to watch people make the same mistakes you all did in the 90s. :) (Where's vijay now when science for generated network configurations is needed?) Make the public tools better. Push the tools as best practice. ADrian
Re: [admin] [summary] RE: YouTube IP Hijacking
On 27/02/2008, at 11:39 AM, Adrian Chadd wrote: (speaking as someone who has built large ACLs/prefix-lists and has 6MB+ configs that can't be loaded on my routers. without vendor support those that want to do the right thing can't, so the game is lost). I remember the days of making rtconfig work properly in various situations (heck, do people still use that? Does it even do IPv6 right?) Yeah, we use it. And (following a bunch of patches we made a couple of months ago) we've convinced it to do IPv6 too. - mark -- Mark Newton Email: [EMAIL PROTECTED] (W) Network Engineer Email: [EMAIL PROTECTED] (H) Internode Systems Pty Ltd Desk: +61-8-82282999 Network Man - Anagram of Mark Newton Mobile: +61-416-202-223
Re: [admin] [summary] RE: YouTube IP Hijacking
On Wed, Feb 27, 2008 at 10:09:19AM +0900, Adrian Chadd wrote: (speaking as someone who has built large ACLs/prefix-lists and has 6MB+ configs that can't be loaded on my routers. without vendor support those that want to do the right thing can't, so the game is lost). Getting Cisco/Juniper/etc to push -that- as part of their best practices for network design would be quite helpful. The problem isn't that the router config is too easy Jared, its that there's no nice and easy way of doing it right from scratch that matches the sort of newbie network operators that exist today. For examples of what new school netops are like, visit isp-* lists. There's a lot of clue there, its just different and haven't learnt from the old school experience clue, and its amusing/sad to watch people make the same mistakes you all did in the 90s. :) (Where's vijay now when science for generated network configurations is needed?) Make the public tools better. Push the tools as best practice. The problem is that some of us have developed tools that are considered our companies property, so we can't just go ahead and release it to the public. Who is gonna start the project to get this going? How do you integrate it with your existing provisioning system? I've regularly heard some of the larger telecoms quote times of ~2-3 years and $10m+ for any project like this. Not sure if those timelines ever were started. Perhaps this is something that renesys or cariden could market and sell? (just to name two nanog sponsors that have some sort of dataset or tools that could apply). I'd like to see this all cleaned up and get better. I track obvious leaks that should be caught by as-path filtering and proper policy here: http://puck.nether.net/bgp/leakinfo.cgi there's a stats page one can find so you can track the number of leaks/day that are seen, including the most common as-paths. if you're smart try appending ?days=3 on the end of the statistics cgi. - jared -- Jared Mauch | pgp key available via finger from [EMAIL PROTECTED] clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: [admin] [summary] RE: YouTube IP Hijacking
On Mon, Feb 25, 2008, Alex Pilosov wrote: A bit of administrativia: This thread generated over a hundred posts, many without operational relevance or by people who do not understand how operators, well, operate, or by people who really don't have any idea what's going on but feel like posting. I'd like to briefly summarize the important things that were said. If you would like to add something to the thread, make sure you read this post in entirety. [snip] http://arstechnica.com/news.ars/post/20080225-insecure-routing-redirects-youtube-to-pakistan.html There's a discussion page with feedback from a non-NANOG audience. Science, anyone? Adrian
[admin] [summary] RE: YouTube IP Hijacking
A bit of administrativia: This thread generated over a hundred posts, many without operational relevance or by people who do not understand how operators, well, operate, or by people who really don't have any idea what's going on but feel like posting. I'd like to briefly summarize the important things that were said. If you would like to add something to the thread, make sure you read this post in entirety. Sorry if I didn't attribute every suggestion to a poster. Facts: * AS17557 announced more specific /24 to 3491, which propagated to wider internets * Chronology (by [EMAIL PROTECTED]) http://www.renesys.com/blog/2008/02/pakistan_hijacks_youtube.shtml * Things suggested to possibly address the problem: ** IRR filtering (using IRRPT http://sourceforge.net/projects/irrpt/ to generate filter lists) ** Notification when origin of a given route changes http://www.cs.ucla.edu/~mohit/cameraReady/ladSecurity06.pdf http://www.ris.ripe.net/myasn.html http://cs.unm.edu/~karlinjf/IAR/index.php (from pgBGP) ** pgBGP to depref suspicious routes http://www.nanog.org/mtg-0606/pdf/josh-karlin.pdf (unclear the number of false positives that will adversely affect connectivity) ** sbgp/sobgp - require full authentication for each IP block, and thus unlikely to be implemented until certificate chains are in place, and vendors release code that does verification, and operators are happy enough running it. Other things addressed: * Fragility of Internet: ** Nobody brought up the important point - the BGP announcement filtering are only as secure as the weakest link. No [few?] peers or transits are filtering large ISPs (ones announcing few hundred routes and up). There are a great many of them, and it takes only one of them to mess up filtering a downstream customer for the route to be propagated. ** Paul Wall brought up the fact that even obviously bogus routes (1/8 and 100/7) were accepted by 99% of internet during an experiment. Will it take someone announcing 9/11 to get us to pay attention? (ok, bad joke) ** What I'd like to see discussed: Issues of filtering your transit downstream customers, who announce thousands of routes. Does *anyone* do it? * Typos vs Malicious announcements ** Some ways of fixing the problem (such as IRR filtering) only address the typos or unintentional announcements. There's full agreement that IRR is full of junk, which is not authenticated in any sort. ** Things like PHAS won't work if hijacker keeps the origin-AS same (by getting their upstream to establish session with different ASN) ** What I'd like to see discussed: Who (ICANN/RIRs/LIRs) is actively working on implementing chain of trust of IP space allocations? * Ways to address the issue without cooperation of 3491: ** Filtering anything coming out of 17557 ** Suggestions given: ** What I'd like to see discussed: Can an network operator, *today*, filter the possibly bogus routes from their peers, without manual intervention, and without false positives? * Yelling at people who don't filter ** Per above, 3491 isn't the only one who filters. In fact, claims were made that *nobody* filters large enough downstreams. (beyond aspath/maxpref) ** *please* do not post additional comments about pccw bad, etc. * Malicious vs mistaken on part of AS17557 and 3491: ** *please* do not post speculation unless you have facts to back it up. ** Any discussions of cyber-jihad are off-topic unless you can produce the fatwa to back it up.
Re: [admin] [summary] RE: YouTube IP Hijacking
On Feb 25, 2008, at 12:51 PM, Alex Pilosov wrote: ** Nobody brought up the important point - the BGP announcement filtering are only as secure as the weakest link. No [few?] peers or transits are filtering large ISPs (ones announcing few hundred routes and up). There are a great many of them, and it takes only one of them to mess up filtering a downstream customer for the route to be propagated. Yes, that was my implicit point to Pekka. Even if you do everything feasible today (i.e., explicitly filter customers, some amount of policy for peers, and perhaps a few hacks for multi-homed customers), you're still pretty much screwed if someone announces your address space. Heck, you're as likely to accept the announcement as anyone. ** Paul Wall brought up the fact that even obviously bogus routes (1/8 and 100/7) were accepted by 99% of internet during an experiment. I'm not sure why this would surprise anyone. ** What I'd like to see discussed: Issues of filtering your transit downstream customers, who announce thousands of routes. Does *anyone* do it? Lots of folks do. The interesting bit is that even then, those same providers would accept perhaps even those customer routes from their peers implicitly. * Typos vs Malicious announcements ** Some ways of fixing the problem (such as IRR filtering) only address the typos or unintentional announcements. You mean as opposed to intentionally malice acts? Well, not completely. See Pekka's email, for example. Of course, it does vary widely across IRRs, etc.. There's full agreement that IRR is full of junk, which is not authenticated in any sort. Mostly, though not completely. ** Things like PHAS won't work if hijacker keeps the origin-AS same (by getting their upstream to establish session with different ASN) NO, that's not even necessary. Simple originate the route from the legit AS, and then transit it with the local AS as a transit AS. AS path manipulation is trivial. ** What I'd like to see discussed: Who (ICANN/RIRs/LIRs) is actively working on implementing chain of trust of IP space allocations? * Ways to address the issue without cooperation of 3491: ** Filtering anything coming out of 17557 Bad idea. ** Suggestions given: ** What I'd like to see discussed: Can an network operator, *today*, filter the possibly bogus routes from their peers, without manual intervention, and without false positives? Sure, if they want to dedicate an engineer to it, automate policy deployment and deal with brokenness by turning steam valves. * Yelling at people who don't filter That's been productive for over a decade now. ** Per above, 3491 isn't the only one who filters. In fact, claims were made that *nobody* filters large enough downstreams. (beyond aspath/maxpref) Wrong. -danny
Re: [admin] [summary] RE: YouTube IP Hijacking
On Mon, 25 Feb 2008, Danny McPherson wrote: ** Paul Wall brought up the fact that even obviously bogus routes (1/8 and 100/7) were accepted by 99% of internet during an experiment. I'm not sure why this would surprise anyone. To me and you, it's not surprising. To public, it might be. Even the majority of nanog attendees I think would be surprised. ** What I'd like to see discussed: Issues of filtering your transit downstream customers, who announce thousands of routes. Does *anyone* do it? Lots of folks do. The interesting bit is that even then, those same providers would accept perhaps even those customer routes from their peers implicitly. Well, in this case, they *aren't* filtering! (unless I am misunderstanding what you are saying, due to repeated use of 'their'). ** Things like PHAS won't work if hijacker keeps the origin-AS same (by getting their upstream to establish session with different ASN) NO, that's not even necessary. Simple originate the route from the legit AS, and then transit it with the local AS as a transit AS. AS path manipulation is trivial. Oh yeah, d'oh! Thanks for correction. But that is also an important point against PHAS and IRRPT filtering - they are powerless against truly malicious hijacker (one that would register route in IRR, add the right origin-as to AS-SET, and use correct origin). ** What I'd like to see discussed: Who (ICANN/RIRs/LIRs) is actively working on implementing chain of trust of IP space allocations? * Ways to address the issue without cooperation of 3491: ** Filtering anything coming out of 17557 Bad idea. Obviously :) ** Suggestions given: ** What I'd like to see discussed: Can an network operator, *today*, filter the possibly bogus routes from their peers, without manual intervention, and without false positives? Sure, if they want to dedicate an engineer to it, automate policy deployment and deal with brokenness by turning steam valves. I'd hear to see who does it, and get them to present the operational lessons at the next nanog! * Yelling at people who don't filter That's been productive for over a decade now. ** Per above, 3491 isn't the only one who filters. In fact, claims were made that *nobody* filters large enough downstreams. (beyond aspath/maxpref) Wrong. Likewise, I'd like to know who does this (names) and how can we get them to present best practices at the next nanog! -alex
Re: [admin] [summary] RE: YouTube IP Hijacking
On Feb 25, 2008, at 1:22 PM, Alex Pilosov wrote: Well, in this case, they *aren't* filtering! (unless I am misunderstanding what you are saying, due to repeated use of 'their'). What I'm saying is that best case today ISPs police routes advertised by their customers, yet they accept routes implicitly (including routes from address space that may belong to their customers) from peers. Seems a little hokey, eh? Oh yeah, d'oh! Thanks for correction. But that is also an important point against PHAS and IRRPT filtering - they are powerless against truly malicious hijacker (one that would register route in IRR, add the right origin-as to AS-SET, and use correct origin). Yep, pretty much. Sure, if they want to dedicate an engineer to it, automate policy deployment and deal with brokenness by turning steam valves. I'd hear to see who does it, and get them to present the operational lessons at the next nanog! Maybe Curtis V. would present what ANS was doing in 1994 :-) But now we've even got things like BGP route refresh, incrementally updatable filters, and BGP soft reconfiguration to ease the deployment burden. There have been two or three panels on this exact topic in the past, you can find them in the index of talks. Unfortunately, the problem hasn't changed at all. Perhaps we could just replay those video streams :-) -danny
Re: [admin] [summary] RE: YouTube IP Hijacking
I'd hear to see who does it, and get them to present the operational lessons at the next nanog! On second thought, I guess one thing has changed considerably since 15 years ago. Rather than ~5000 monkeys with keyboard access to manipulate global routing tables, there are likely well North of 250,000 (25k active ASNs * 10 meat computers per), which is surely well on the conservative side. The bottom line is [still] that ISPs should at least explicitly filter prefixes from customers and networks from which they provide transit services. -danny
RE: [admin] [summary] RE: YouTube IP Hijacking
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 There have been two or three panels on this exact topic in the past, you can find them in the index of talks. Unfortunately, the problem hasn't changed at all. Perhaps we could just replay those video streams :-) My $.02 - http://www.getit.org/wordpress/?p=82 The irony to one of those, is that in NANOG 25 right before my session which pointed out this continued threat vector, we had Protecting the BGP Routes to Top Level DNS Servers http://www.nanog.org/mtg-0206/bush.html. -BEGIN PGP SIGNATURE- Version: PGP 8.1 iQA/AwUBR8M5ib/UEA/xivvmEQISnwCgojEcUx7dyMBQPP59gZjdLgQeqqMAoL0p seCMm+llF8tkr+pGf7LlyXrW =6jfG -END PGP SIGNATURE-
Re: [admin] [summary] RE: YouTube IP Hijacking
On Mon, Feb 25, 2008, Alex Pilosov wrote: A bit of administrativia: This thread generated over a hundred posts, many without operational relevance or by people who do not understand how operators, well, operate, or by people who really don't have any idea what's going on but feel like posting. I'd like to briefly summarize the important things that were said. If you would like to add something to the thread, make sure you read this post in entirety. .. and, since I have 5 minutes to spare between disk array rebuilds, I bring you: http://nanog.cluepon.net/index.php/MailTopics For now it just links into Alex's summary post. I'll go trawl the archives now for more summaries. Adrian
