Re: [admin] [summary] RE: YouTube IP Hijacking
On Feb 27, 2008, at 2:09 AM, Adrian Chadd wrote: (speaking as someone who has built large ACLs/prefix-lists and has 6MB+ configs that can't be loaded on my routers. without vendor support those that want to do the right thing can't, so the game is lost). I remember the days of making rtconfig work properly in various situations (heck, do people still use that? Does it even do IPv6 right?) and building router configs out of both public and private IRR data. Perhaps if the entry barrier to building dynamically generated router configurations were lowered significantly (to the point of it being free, GUI, and multi-platform) then it may be used for new network designs by people starting off. Getting Cisco/Juniper/etc to push -that- as part of their best practices for network design would be quite helpful. The problem isn't that the router config is too easy Jared, its that there's no nice and easy way of doing it right from scratch that matches the sort of newbie network operators that exist today. For examples of what new school netops are like, visit isp-* lists. There's a lot of clue there, its just different and haven't learnt from the old school experience clue, and its amusing/sad to watch people make the same mistakes you all did in the 90s. :) (Where's vijay now when science for generated network configurations is needed?) Make the public tools better. Push the tools as best practice. ADrian Well there is a slight risk that the more you will improve the automated tools, the less net engineers will actually understand the reasons why it is done... That being said, all the filtering work is a significant part of every Network Engineer's work, and is not that big of a deal. Education is the art of repeating, and has always been. I'm not saying we should systematically point the ones making mistakes and make it public, what I'm saying is that the reason why mailing lists such as nanog exist is actually mutual education. I'm sure the people involved in this incident were (or now are) Nanog readers, and that they've understood the message. Still, what should be done is, I assume, centralizing the info in one single mail, kept for the record: - list the incident chronology - analyze what technical mistake lead to that - list -all- the ways to prevent it Another mean of education is including the analysis of this incident (troubleshooting, resolution, implementing means to avoid it) next Nanog agenda, which I already think is the case :) Greg VILLAIN
Re: YouTube IP Hijacking
On Mon, Feb 25, 2008 at 09:27:41AM +0200, Hank Nussbacher [EMAIL PROTECTED] wrote a message of 17 lines which said: - Lack of clue - Couldn't care less - No revenue Take your pick - or add your own reason. PCCW is not alone. They just happen to be the latest in a long line of ISPs that follow the same rules - their own. No, most operators do filter BGP announcements. I know it, because I have read it on Cnet: http://www.news.com/8301-10784_3-9878655-7.html That's because Hong Kong-based PCCW, which provides the Internet link to Pakistan Telecom, did not stop the misleading broadcast--which is what most large providers in the United States and Europe do.
Re: YouTube IP Hijacking
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -- Stephane Bortzmeyer [EMAIL PROTECTED] wrote: No, most operators do filter BGP announcements. I know it, because I have read it on Cnet: http://www.news.com/8301-10784_3-9878655-7.html Dunno -- looks like McCullagh got it pretty much spot on: At the moment, large network providers tend to trust that other network providers are behaving reasonably--and aren't intentionally trying to hijack someone else's Internet addresses. And errors that do arise tend to be fixed quickly by manual intervention. I may have missed it, but I didn't see any account of pervasive BGP filtering being done in the Internet... - - ferg -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFHw9wPq1pz9mNUZTMRAn86AJ9rsXAoSVKt5MZl1qJiyDwW5SuhngCfToYR pmt3PBk5LYSAazZBtVR5yfs= =Ov9W -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: BGP prefix filtering, how exactly? [Re: YouTube IP Hijacking]
Hi, In a lot of this dialogue, many say, you should prefix filter. However, I'm not seeing how an ISP could easily adopt such filtering. Let's consider the options: [..] a) only RIPE IRR uses a sensible security model [1], so if you use others, basically anyone can add route objects to the registry. How exactly would this model be useful? [..] So, this is no excuse for not doing prefix filtering if you only do business in the RIPE region, but anywhere else the IRR data is pretty much useless, incorrect, or both. this is all true and leads us to the question why ARIN, for example, DOESNT USE A SENSIBLE SECURITY MODEL? Actually i asking this myself for a couple of years. IMHO ARIN _should_ either improve their RR software or, better, use the RIPE DB software so ISPS can build prefix-filters for the ARIN region. So: Why dont they do it?!!! Arnd
Re: YouTube IP Hijacking
Iljitsch van Beijnum writes: Well, if they had problems like this in the past, then I wouldn't trust them to get it right. Which means that it's probably a good idea if EVERYONE starts filtering what they allow in their tables from PCCW. Obviously that makes it very hard for PCCW to start announcing new prefixes, but I can't muster up much sympathy for that. So basically, rather than generate routing registry filters for the entire world, generate routing registry filters for known careless ASes. This number should be small enough that this is somewhat doable. [...] Maybe, but how much would that help? So you suggest that we only need to filter against AS7007, AS9121, and AS17557. Personally, those are among the ones I least worry about - maybe I'm naive, but I'd hope they or their upstreams have learned their lessons. The problem is that nobody knows which of the other 25000+ ASes will be the next AS7007. So I guess we have to modify your suggestion somewhat and, in addition to filtering the known-careless also filter the unknown-maybe-careful class. Oops, that leaves only the known-careful class, which includes... my own AS, and then whom? -- Simon.
Re: hijack chronology: was [ YouTube IP Hijacking ]
Martin A Brown writes: Late last night, after poring through our data, I posted a detailed chronology of the hijack as seen from our many peering sessions. I would add to this that the speed of YouTube's response to this subprefix hijack impressed me. For a Sunday afternoon, yes, not bad. Here's a graphical version of the timeline: http://www.ris.ripe.net/cgi-bin/bgplay.cgi?prefix=208.65.153.0/24start=2008-02-24+18:46end=2008-02-24+21:05 As discussed earlier in this thread, this is really the same old song--it simply has a new verse now. (How many of our troubadors know all of the verses since AS 7007?) Probably someone is busy working on the new NANOG song with an extending refrain (AS7007, ..., AS17557, when will we ever learn?). -- Simon.
Re: YouTube IP Hijacking
Now if only everyone here on NANOG were to do what Matsuzaki has done, and take the time to educate those less clueless, the world would be a better place. Its time that people responsible for BGP routing need to show that they have the skills and knowledge for it. Every ISP requesting an ASN from one of the LIR's should be required to make a test covering the neccessary skillsets. -- Arnd
Re: YouTube IP Hijacking
Rick Astley writes: Anything more specific than a /24 would get blocked by many filters, so some of the high target sites may want to announce their mission critical IP space as /24 and avoid using prepends. Good idea. But only the high target sites, please. If you're an unimportant site that nobody cares about, then DON'T DO THIS, ok? ;-) -- Simon.
RE: YouTube IP Hijacking
This isn't the answer. If it were, there would be no car accidents, pilot error caused plane crashes, etc. Probably the reason you dont need to have a pilot license... Sorry, what? Dont get me wrong: I not the Policy this/that type but i think its a good idea to ensure that ppl who run basic network infrastructure have minimal clue of how to do this. Do you really believe that LIRs should be administering tests before issuing ASNs? Should vendors do the same prior to selling their gear? Take this further, electric company should require its customers to take a test before they are allowed to order service for fear they might electrocute themselves or the water company fearing customers may drown? -- Arnd Randy
Re: YouTube IP Hijacking
On Tue, Feb 26, 2008 at 11:43:10AM +0100, Arnd Vehling [EMAIL PROTECTED] wrote a message of 12 lines which said: Every ISP requesting an ASN from one of the LIR's should be required to make a test covering the neccessary skillsets. Giving the rapid turnover of people in this industry, I'm not sure it would help. Two months after the test, may be all the clueful people will be gone. And what about the organizations which rely on external consultancy? Should it be forbidden?
Re: [admin] [summary] RE: YouTube IP Hijacking
Alex Pilosov wrote: Oh yeah, d'oh! Thanks for correction. But that is also an important point against PHAS and IRRPT filtering - they are powerless against truly malicious hijacker (one that would register route in IRR, add the right origin-as to AS-SET, and use correct origin). With a decent LIR DB (like the RIPE DB) this is only possible if an hijacker breaks the authentication of the according database objects which is a pain in the a** _if_ the objects use a proper authentication scheme like PGP. -- Arnd
Re: YouTube IP Hijacking
Stephane Bortzmeyer wrote: On Tue, Feb 26, 2008 at 11:43:10AM +0100, Arnd Vehling [EMAIL PROTECTED] wrote a message of 12 lines which said: Every ISP requesting an ASN from one of the LIR's should be required to make a test covering the neccessary skillsets. Giving the rapid turnover of people in this industry, I'm not sure it would help. Two months after the test, may be all the clueful people will be gone. How about an ISP running BGB is required to have a number of LIR certified BGP people and is otherwise not allowed to peer *grin* And what about the organizations which rely on external consultancy? Should it be forbidden? No, the consultants can be LIR certified too. I just dont see why people need to have a damn license for lots of stuff but noone cares if the people running border routers have the neccessary qualifications. It just doesnt make sense and i am pretty sure that this _will_ change. -- Arnd
Re: YouTube IP Hijacking
Randy Epstein wrote: This isn't the answer. If it were, there would be no car accidents, pilot error caused plane crashes, etc. Probably the reason you dont need to have a pilot license... Sorry, what? You _need_ a license to drive a car, fly a plane etc. but until now you dont need to show that youre skilled enough to run a border router. Good idea? I dont think so. Do you really believe that LIRs should be administering tests before issuing ASNs? I believe that people who run ASNs should have the knowledge for it and that _someone_ should test this. Right now the LIRs seems to be the best institution for this. And no, i dont think the vendors should do this. -- Arnd
Re: [admin] [summary] RE: YouTube IP Hijacking
On 26/02/2008 12:06, Arnd Vehling [EMAIL PROTECTED] wrote: [...] With a decent LIR DB (like the RIPE DB) this is only possible if an hijacker breaks the authentication of the according database objects which is a pain in the a** _if_ the objects use a proper authentication scheme like PGP. I wonder what percentage of maintainers in the RIPE database only have PGP and/or X.509 auth schemes. I'd be surprised if it was as high as 5%. Leo
RE: YouTube IP Hijacking
Arnd wrote: You _need_ a license to drive a car, fly a plane etc. but until now you dont need to show that youre skilled enough to run a border router. Good idea? I dont think so. My point was that even with a license, accidents still occur. I believe that people who run ASNs should have the knowledge for it and that _someone_ should test this. Right now the LIRs seems to be the best institution for this. And no, i dont think the vendors should do this. Vendors currently do train their customers and certify them. LIRs don't and cannot know all the gear out there and configurations from network to network vary. This doesn't stop route leaks, nor would this protect us from intentional mischief. I'm not saying it can't happen, but most leaks are caused by accident, and I might add by trained personnel and untrained personnel alike. Many of the suggestions that we've been seeing regarding this subject have pros and cons, but some even solve both problems: both accidental and intentional leaks. I am not against training personnel, but your solution doesn't resolve either of the above for the most part. -- Arnd Randy
Re: [admin] [summary] RE: YouTube IP Hijacking
Leo Vegoda wrote: On 26/02/2008 12:06, Arnd Vehling [EMAIL PROTECTED] wrote: [...] With a decent LIR DB (like the RIPE DB) this is only possible if an hijacker breaks the authentication of the according database objects which is a pain in the a** _if_ the objects use a proper authentication scheme like PGP. I wonder what percentage of maintainers in the RIPE database only have PGP and/or X.509 auth schemes. I'd be surprised if it was as high as 5%. True, but thats still better than having no authentication at all and its possible to require strong authentication on inetnum, route and AS objects. I just cant understand why LIR's like ARIN dont have any decent methods for this implemented in their DB. Or did this change recently? -- Arnd
Re: YouTube IP Hijacking
Randy Epstein wrote: My point was that even with a license, accidents still occur. My point is that without a license more accidents will occur. Vendors currently do train their customers and certify them. A lot of companies dont send their personel to training lessons because of the costs. The vendor primarily trains how to _implement_ a BGP policy on their equipment and not neccessarily how to develop a good peering and filter policy. The youtube ip hijacking case _may_ be a result of route redistribution from an internal routing protocol to BGP without any route filters applied. Every decent BGP engineer knows that this is a very bad idea. LIRs don't and cannot know all the gear out there and configurations from network to network vary. They dont need to. They could/should ensure that people running ASNs have a good knowledge about how BGP works. Not how to _implement_ a BGP policy on a vendor device. This truly is up to the vendors and ISPs. This doesn't stop route leaks, nor would this protect us from intentional mischief. True, but it will help reducing incidents which will have a huge impact on the live and economy of a lot of people. The youtube IP hijacking was only a minor nuisance in relation to what can happen if other prefixes are hijacked or just leak due to clueless personal. -- Arnd
Re: YouTube IP Hijacking
Arnd Vehling wrote: Randy Epstein wrote: My point was that even with a license, accidents still occur. My point is that without a license more accidents will occur. The problem here is a problem causes in a *REMOTE* network, that you, as a decent engineer, should safeguard against in *YOUR* network. That *other* networks don't have a clue, doesn't mean that you can't, at least partially, protect against them. Or to get to the car analogy: driving a Hummer H1 or for that matter a Volvo or any other decent SUV (or a tank :), protects you from all those maniacs (wiht and without license) on the road, as when they hit you you will only have a dent, their car will be totaled. In other words: secure your network and make it watch out for the idiots maniacs, with and without a license. See the other threads on tip and tricks on how to get this working, but you as a decent engineer should know that already and have some nice monitoring in place to avoid incidents like these... Greets, Jeroen signature.asc Description: OpenPGP digital signature
RE: YouTube IP Hijacking
You _need_ a license to drive a car, fly a plane etc. but until now you dont need to show that youre skilled enough to run a border router. Good idea? I dont think so. My point was that even with a license, accidents still occur. Even with a licence and testing, airline crashes still occur, commercial airline pilots still arrive at work drunk or die of heart-attacks behind the wheel of the airplane. But, due to a lot of effort in making better educational material available for pilots, including better flight simulators and better simulator scenarios, flying is a lot safer than it was in 1958. Not to mention the great effort that is put into post-mortem studies of airline crashes, the open sharing of information around the world, and the steady incremental improvement of best-practices and educational materials. The Internet operations profession could do all of that without any need for laws, licenses, inspections, or whatever. In fact, if you look around you, the Internet ops profession actually *DOES* do a lot of the same stuff that the airline industry does and things ARE getting better when you measure the impact per user or per connected device. The net is a lot bigger than it was 10 years ago, and far fewer incidents happen that have wide impact. In fact, it is not even clear that this YouTube incident counts as having wide impact. How many people were impacted by the YouTube outage compared to the Asian Tsunami/landslide of 2005? You don't need a Rogers Commission (Challenger disaster) or a 9-11 Commission set up by the President to solve these problems. For all the moaning and complaining that hit this list and the blogosphere, lots of people actually are studying the root cause of this disaster and taking action to mitigate such events in the future. It should be no surprise that the most important such mitigation events are not related to installing more BGP filters, but in making sure that outages/anomalies are promptly detected and promptly escalated to the RIGHT people in the operations team who can fix or mitigate them. I am not against training personnel, but your solution doesn't resolve either of the above for the most part. Training is a form of education, and education is a necessary prelude to action. You would be a fool to just accept someone's advice from this list and run out to implement it RIGHT NOW. Better to study it, try it in the lab. Figure out what it does, why it does it. Think about how to monitor it and manage it. Write up a business case to see if you really can justify this action to management. Then document it and do it when everybody understands the problem and the solution. Alex wrote such a brilliant message summarizing the discussion to date that I'm thinking we should reshape the mailing list committee into a kind NTSB http://www.ntsb.gov/ for the Internet that would solicit comments, compile incident reports and produce best practice documents. That kind of thing might be valuable enough that somebody would pay NANOG to do it. --Michael Dillon
Re: [admin] [summary] RE: YouTube IP Hijacking
Alex Pilosov ha scritto: Facts: * AS17557 announced more specific /24 to 3491, which propagated to wider internets I think that they should use remote triggered blackhole filtering with no-export community. In this way they do the job with no impact on the rest of internet. Regards, Gianluca
Re: [admin] [summary] RE: YouTube IP Hijacking
On Tue, Feb 26, 2008 at 10:40 AM, hjan [EMAIL PROTECTED] wrote: I think that they should use remote triggered blackhole filtering with no-export community. In this way they do the job with no impact on the rest of internet. so, certainly this isn't a bad idea, but given as an example: http://www.secsup.org/CustomerBlackHole/ (Sorry not a perfect example, but illustrates my point) instead of: ip route my.offensive.material.0 255.255.255.0 Null0 tag 12345 the operator in question (person not place) types: ip route my.offensive.material.0 255.255.255.0 Null0 tag 1234 oops, a simple cut/paste mistake means that a route didn't get tagged properly, didn't get community tagged properly, didn't get set no-export and didn't get kept internally :( There is no SINGLE fix for this, there is a belt+suspenders approach: 1) Know what you are advertising (customer side of the puzzle) 2) Know what you are expecting to recieve (provider side of the puzzle) 3) plan for failures in both parts of this puzzle. -Chris
RE: [admin] [summary] RE: YouTube IP Hijacking
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Missing a tag in the trigger is why you put the Murphy Filters in the trigger router's route-map (the point you were getting at but being even more explicit). In my route map on the trigger router, I would not allow any static route triggers which did not have an exact match. I would also set the BGP advertisement to have - by default - the no-export community, a community range for all my triggers, and limit all my triggers to be below /24 (i.e /25 - /32). I then have three things to set my egress prefix filters to all my peers and customers: - comply with the default communities (no export) - filter all communities in my trigger range - filter anything in the /25 - /32 range. BTW - Murphy Filters is my term for policy filters which expect Murphy's Law of Networking to kick in. You have to expect human error. In addition to this, I would have my upstream mirror my filters. Life sucks when you advertise big blocks of the Internet and you become one giant sink hole (until you go congestion collapse, drop the BGP session and start flapping like crazy). -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Christopher Morrow Sent: Tuesday, February 26, 2008 8:59 AM To: hjan Cc: nanog@merit.edu Subject: Re: [admin] [summary] RE: YouTube IP Hijacking On Tue, Feb 26, 2008 at 10:40 AM, hjan [EMAIL PROTECTED] wrote: I think that they should use remote triggered blackhole filtering with no-export community. In this way they do the job with no impact on the rest of internet. so, certainly this isn't a bad idea, but given as an example: http://www.secsup.org/CustomerBlackHole/ (Sorry not a perfect example, but illustrates my point) instead of: ip route my.offensive.material.0 255.255.255.0 Null0 tag 12345 the operator in question (person not place) types: ip route my.offensive.material.0 255.255.255.0 Null0 tag 1234 oops, a simple cut/paste mistake means that a route didn't get tagged properly, didn't get community tagged properly, didn't get set no-export and didn't get kept internally :( There is no SINGLE fix for this, there is a belt+suspenders approach: 1) Know what you are advertising (customer side of the puzzle) 2) Know what you are expecting to recieve (provider side of the puzzle) 3) plan for failures in both parts of this puzzle. -Chris -BEGIN PGP SIGNATURE- Version: PGP 8.1 iQA/AwUBR8RSF7/UEA/xivvmEQJUKACfZB+typ7sIJMnDS+QrO0MqGED+CYAoKFC iBmY+pq0CohSIJwtu5pgzCJt =xiog -END PGP SIGNATURE-
Re: YouTube IP Hijacking
But, due to a lot of effort in making better educational material available for pilots, including better flight simulators and better simulator scenarios, flying is a lot safer than it was in 1958. At the risk of being a stereotypical American liberal, I'll point out two significant reasons flying is safer than it used to be in the US are Federal regulation and post-accident lawsuits. If there were an organization like the FAA that had the power to ground AS17557 until their network engineers completed a week's refresher course, there'd be significantly better change management techniques in play. If YouTube were currently suing Pakistani Telecom for eighty-seven gazillion dollars-- and were widely considered a lock to win their lawsuit-- suddenly a whole lot of other ISPs would magically find the training budget to make sure THEIR engineers didn't expose THEM to that sort of liability. Pilots don't spend dozens of hours in simulators because it's fun, they do it to get/keep their license. American Airlines doesn't spend millions of dollars on pilot (and ground crew) education because they're run by philanthropists, they do it because screwups could cost them orders of magnitude more money. The Internet lacks any such enforcement mechanisms. How many people do you think have lost their jobs for this latest incident? What are the odds that the responsible party lost a penny in revenue or in fines? When there is no financial or regulatory pressure to avoid screwups, avoiding screwups ceases to be a priority at Layer 8 or Layer 9. And then you have incidents like this, where the operational solutions are widely agreed upon and the political obstacles are widely agreed to be insurmountable. And we wait for the next incident. -- Dave Pooser, ACSA Manager of Information Services Alford Media http://www.alfordmedia.com
Re: YouTube IP Hijacking
[EMAIL PROTECTED] wrote: Haven't you noticed that the definition of widely visited site changes regularly, and often quite abruptly? How much traffic did YouTube get 3 years ago? Facebook? MySpace? There is no shortcut for eternal vigilance, i.e. manage your BGP relationships don't just configure and forget. I hear that computers can be programed to create and maintain a list of sites one's own customers visit frequently. The same computer (so I'm told) could also determine which AS has authority to announce the IPs for each site. The same computer (so I'm told) could also watch for announcements from other ASs for the IPs from this up-to-date widely visited sites list, and send out an alert or otherwise take action when such an announcement was seen. It would be wonderful if someone would write and share code to do just this. jc
Re: [admin] [summary] RE: YouTube IP Hijacking
The biggest problem here is that Cisco needs to change their defaults to require more configuration than router bgp X neighbor 1.2.3.4 remote-as A When that's the bar for the complexity required for setting up BGP, bad things WILL happen. Period. Cisco has taken all these years and not stepped up in providing any sort of a reasonable change to the marketplace as others have done. This hurts the industry as a whole, and hurts the perception that we can't route. As for other problems with leadership, there's no good way to manage large configurations on the platforms, nor a reasonable size of NVRAM provided either. The list goes on and on, and I've communicated this more than once to the company. Nobody cares about this basic infrasturcture at Cisco, or at least nobody that can make something happen. Instead people care about what product is intruding on their turf and how to defend it instead of building a better product and improving things. Honestly, it's a lost cause and SP's don't account for any significant amount of revenue. If someone at Cisco cares to address these things, i'm interested in helping but it's clear that the head-in-the-sand policy by upper mgmt lives on and they'd rather fight amongst themselves and risk the industry as a whole because of their antics. - Jared (speaking as someone who has built large ACLs/prefix-lists and has 6MB+ configs that can't be loaded on my routers. without vendor support those that want to do the right thing can't, so the game is lost). -- Jared Mauch | pgp key available via finger from [EMAIL PROTECTED] clue++; | http://puck.nether.net/~jared/ My statements are only mine.
RE: YouTube IP Hijacking
Since the US has no jurisdiction over 17557, other than for the US govt. to force ISPs to refuse to accept any advertisements with 17557 or any other AS that didn't meet some regulatory requirements in the path, how would you propose that the regulatory environment you envision work? American Airlines isn't the right straw-man here, Pakistan International Airlines is. The only reason THEY meet anyone else's standards is that they wouldn't be allowed to use the airspace or land if they didn't. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dave Pooser Sent: Tuesday, February 26, 2008 10:15 AM To: nanog@merit.edu Subject: Re: YouTube IP Hijacking But, due to a lot of effort in making better educational material available for pilots, including better flight simulators and better simulator scenarios, flying is a lot safer than it was in 1958. At the risk of being a stereotypical American liberal, I'll point out two significant reasons flying is safer than it used to be in the US are Federal regulation and post-accident lawsuits. If there were an organization like the FAA that had the power to ground AS17557 until their network engineers completed a week's refresher course, there'd be significantly better change management techniques in play. If YouTube were currently suing Pakistani Telecom for eighty-seven gazillion dollars-- and were widely considered a lock to win their lawsuit-- suddenly a whole lot of other ISPs would magically find the training budget to make sure THEIR engineers didn't expose THEM to that sort of liability. Pilots don't spend dozens of hours in simulators because it's fun, they do it to get/keep their license. American Airlines doesn't spend millions of dollars on pilot (and ground crew) education because they're run by philanthropists, they do it because screwups could cost them orders of magnitude more money. The Internet lacks any such enforcement mechanisms. How many people do you think have lost their jobs for this latest incident? What are the odds that the responsible party lost a penny in revenue or in fines? When there is no financial or regulatory pressure to avoid screwups, avoiding screwups ceases to be a priority at Layer 8 or Layer 9. And then you have incidents like this, where the operational solutions are widely agreed upon and the political obstacles are widely agreed to be insurmountable. And we wait for the next incident. -- Dave Pooser, ACSA Manager of Information Services Alford Media http://www.alfordmedia.com
Re: YouTube IP Hijacking
Since the US has no jurisdiction over 17557, other than for the US govt. to force ISPs to refuse to accept any advertisements with 17557 or any other AS that didn't meet some regulatory requirements in the path, how would you propose that the regulatory environment you envision work? I don't expect any regulation of the Internet to ever work. I expect us (or our successors) to be having exactly the same discussions about exactly the same sort of issues (botnets, route hijacking, spam) in thirty years when I'm starting to plan my retirement. The Internet is what it is; it has evolved to avoid any sort of supra-national regulatory body and the fact that its current model is basically anarchy is considered a necessary evil or a positive advantage, depending on who you talk to. That said, IANAL but if YouTube decided to sue the responsible parties at 17557 in a non-Pakistani court (jurisdiction being established on the basis that their messed up announcements propagated to the US/UK/wherever), I think it could easily win its case (collection of damages might be another issue, of course) and that might have a dramatic impact in encouraging other entities to adhere to BCP. -- Dave Pooser, ACSA Manager of Information Services Alford Media http://www.alfordmedia.com
Re: [admin] [summary] RE: YouTube IP Hijacking
for a list filled with network operators and engineers, the lot of you are quick to whip out lawyers and courts and international tribunals. perhaps I missed the message, but has anyone mentioned the direct economic impact of SFI? as a responsible network operator, would you peer with a network that didn't filter their customers and was consistently linked to route leakage issues? when is enough, enough (in general, not speaking to any specific network)? also, this: On Tue, Feb 26, 2008 at 10:21 AM, Jared Mauch [EMAIL PROTECTED] wrote: The biggest problem here is that Cisco needs to change their defaults to require more configuration than router bgp X neighbor 1.2.3.4 remote-as A When that's the bar for the complexity required for setting up BGP, bad things WILL happen. Period.
Re: YouTube IP Hijacking
On Feb 25, 2008, at 1:22 AM, Christopher Morrow wrote: except that even the 'good guys' make mistakes. Belt + suspenders please... is it really that hard for a network service provider to have a prefix-list on their customer bgp sessions?? L3 does it, ATT does it, Sprint does it, as do UUNET/vzb, NTT, GlobalCrossing ... seriously, it's not that hard. What Chris said. Why isn't this a BCP that people can actually agree on? Wait, being a BCP is probably what killed BCP38 take-up so never mind :(
Re: YouTube IP Hijacking
John Payne wrote: On Feb 25, 2008, at 1:22 AM, Christopher Morrow wrote: except that even the 'good guys' make mistakes. Belt + suspenders please... is it really that hard for a network service provider to have a prefix-list on their customer bgp sessions?? L3 does it, ATT does it, Sprint does it, as do UUNET/vzb, NTT, GlobalCrossing ... seriously, it's not that hard. What Chris said. Why isn't this a BCP that people can actually agree on? Wait, being a BCP is probably what killed BCP38 take-up so never mind :( Well... If you want to work on one, I'm willing help shepherd it through the process. We even have a working group setup for that purpose. joel
Re: [admin] [summary] RE: YouTube IP Hijacking
(speaking as someone who has built large ACLs/prefix-lists and has 6MB+ configs that can't be loaded on my routers. without vendor support those that want to do the right thing can't, so the game is lost). I remember the days of making rtconfig work properly in various situations (heck, do people still use that? Does it even do IPv6 right?) and building router configs out of both public and private IRR data. Perhaps if the entry barrier to building dynamically generated router configurations were lowered significantly (to the point of it being free, GUI, and multi-platform) then it may be used for new network designs by people starting off. Getting Cisco/Juniper/etc to push -that- as part of their best practices for network design would be quite helpful. The problem isn't that the router config is too easy Jared, its that there's no nice and easy way of doing it right from scratch that matches the sort of newbie network operators that exist today. For examples of what new school netops are like, visit isp-* lists. There's a lot of clue there, its just different and haven't learnt from the old school experience clue, and its amusing/sad to watch people make the same mistakes you all did in the 90s. :) (Where's vijay now when science for generated network configurations is needed?) Make the public tools better. Push the tools as best practice. ADrian
Re: [admin] [summary] RE: YouTube IP Hijacking
On 27/02/2008, at 11:39 AM, Adrian Chadd wrote: (speaking as someone who has built large ACLs/prefix-lists and has 6MB+ configs that can't be loaded on my routers. without vendor support those that want to do the right thing can't, so the game is lost). I remember the days of making rtconfig work properly in various situations (heck, do people still use that? Does it even do IPv6 right?) Yeah, we use it. And (following a bunch of patches we made a couple of months ago) we've convinced it to do IPv6 too. - mark -- Mark Newton Email: [EMAIL PROTECTED] (W) Network Engineer Email: [EMAIL PROTECTED] (H) Internode Systems Pty Ltd Desk: +61-8-82282999 Network Man - Anagram of Mark Newton Mobile: +61-416-202-223
Re: [admin] [summary] RE: YouTube IP Hijacking
On Wed, Feb 27, 2008 at 10:09:19AM +0900, Adrian Chadd wrote: (speaking as someone who has built large ACLs/prefix-lists and has 6MB+ configs that can't be loaded on my routers. without vendor support those that want to do the right thing can't, so the game is lost). Getting Cisco/Juniper/etc to push -that- as part of their best practices for network design would be quite helpful. The problem isn't that the router config is too easy Jared, its that there's no nice and easy way of doing it right from scratch that matches the sort of newbie network operators that exist today. For examples of what new school netops are like, visit isp-* lists. There's a lot of clue there, its just different and haven't learnt from the old school experience clue, and its amusing/sad to watch people make the same mistakes you all did in the 90s. :) (Where's vijay now when science for generated network configurations is needed?) Make the public tools better. Push the tools as best practice. The problem is that some of us have developed tools that are considered our companies property, so we can't just go ahead and release it to the public. Who is gonna start the project to get this going? How do you integrate it with your existing provisioning system? I've regularly heard some of the larger telecoms quote times of ~2-3 years and $10m+ for any project like this. Not sure if those timelines ever were started. Perhaps this is something that renesys or cariden could market and sell? (just to name two nanog sponsors that have some sort of dataset or tools that could apply). I'd like to see this all cleaned up and get better. I track obvious leaks that should be caught by as-path filtering and proper policy here: http://puck.nether.net/bgp/leakinfo.cgi there's a stats page one can find so you can track the number of leaks/day that are seen, including the most common as-paths. if you're smart try appending ?days=3 on the end of the statistics cgi. - jared -- Jared Mauch | pgp key available via finger from [EMAIL PROTECTED] clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: YouTube IP Hijacking
On Tue, Feb 26, 2008 at 7:17 PM, Joel Jaeggli [EMAIL PROTECTED] wrote: John Payne wrote: On Feb 25, 2008, at 1:22 AM, Christopher Morrow wrote: except that even the 'good guys' make mistakes. Belt + suspenders please... is it really that hard for a network service provider to have a prefix-list on their customer bgp sessions?? L3 does it, ATT does it, Sprint does it, as do UUNET/vzb, NTT, GlobalCrossing ... seriously, it's not that hard. What Chris said. Why isn't this a BCP that people can actually agree on? Wait, being a BCP is probably what killed BCP38 take-up so never mind :( Well... If you want to work on one, I'm willing help shepherd it through the process. We even have a working group setup for that purpose. proposal for work in GROW?
Re: YouTube IP Hijacking
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -- Christopher Morrow [EMAIL PROTECTED] wrote: Well... If you want to work on one, I'm willing help shepherd it through the process. We even have a working group setup for that purpose. proposal for work in GROW? Actually, that sounds reasonable -- if GROW [1] outcomes are heeded by the operations community. Historically, it seems, the Internet operations community picks and chooses the IETF RFCs/BCPs that each ISP implements on a one-off basis. Something (else) that I have long admired is the way the RIPE community has built working groups around critical issues. I'm still convinced that the NANOG community -- perhaps in collaboration with RIPE and APNIC, et al -- should work to craft ISP best current practices in these areas, since ISPs don't seem to heed IETF documents, except when it serves their own business operational practices. - - ferg [1] http://www.ietf.org/html.charters/grow-charter.html -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFHxMu8q1pz9mNUZTMRAk/wAJ9Mm3sWL9t1cCfy8sAhTd6DfxMIGwCfTauG 9lha9MMTiR4kPoPEHtBjfRQ= =atCM -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: YouTube IP Hijacking
Paul Ferguson wrote: I'm still convinced that the NANOG community -- perhaps in collaboration with RIPE and APNIC, et al -- should work to craft ISP best current practices in these areas, since ISPs don't seem to heed IETF documents, except when it serves their own business operational practices. - ferg [1] http://www.ietf.org/html.charters/grow-charter.html HELL NO(1)! Lets instead wait until the entire thing caves in, we run out of IPv4 space, and the governments have to step in and fix it for us. That's a solution! Andrew (1) This would be a vote of support for what Paul is saying.
Re: [admin] [summary] RE: YouTube IP Hijacking
On Mon, Feb 25, 2008, Alex Pilosov wrote: A bit of administrativia: This thread generated over a hundred posts, many without operational relevance or by people who do not understand how operators, well, operate, or by people who really don't have any idea what's going on but feel like posting. I'd like to briefly summarize the important things that were said. If you would like to add something to the thread, make sure you read this post in entirety. [snip] http://arstechnica.com/news.ars/post/20080225-insecure-routing-redirects-youtube-to-pakistan.html There's a discussion page with feedback from a non-NANOG audience. Science, anyone? Adrian
Re: YouTube IP Hijacking
At 05:31 AM 25-02-08 +, Steven M. Bellovin wrote: Seriously -- a number of us have been warning that this could happen. More precisely, we've been warning that this could happen *again*; we all know about many older incidents, from the barely noticed to the very noisy. (AS 7007, anyone?) Something like S-BGP will stop this cold. Yes, I know there are serious deployment and operational issues. The question is this: when is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done. we've been warning that this could happen *again* - this is happening every day - just look to: http://cs.unm.edu/~karlinjf/IAR/prefix.php?filter=most http://cs.unm.edu/~karlinjf/IAR/subprefix.php?filter=most for samples. Thing is - these prefix hijacks are not big ticket sites like Youtube or Microsoft or Cisco or even whitehouse.gov - but rather just sites that never make it onto the NANOG radar. -Hank
Re: YouTube IP Hijacking
On Mon, 25 Feb 2008 01:49:51 -0500 (EST) Sean Donelan [EMAIL PROTECTED] wrote: On Mon, 25 Feb 2008, Steven M. Bellovin wrote: How about state-of-the-art routing security? The problem is what is the actual trust model? Are you trusting some authority to not be malicious or never make a mistake? There are several answers to the malicious problem. There are fewer answers to never making a mistake problem. The state of the art routing security proposals let the trusted securely make mistakes. At one time or another, I think every router vendor, every ASN operator, every RIR, and so on has made a mistake at some time. Yeah, I know some of those mistakes may have actually been malicious, but so far the mistakes have outnumbered the malicious. If someone comes up with the anti-mistake routing protocol ... Right. Everyone makes mistakes, but not everyone is malicious.And the RIRs and the big ISPs are *generally* more clueful than the little guys and the newcomers. Note also that secured BGP limits the kinds of mistakes people can make. If I have a certificate from my RIR for 192.0.2.0/24, I can't neither announce 10.0.0.0/8 nor delegate it to you, no matter how badly I type. Secured BGP still strikes me as a net win. --Steve Bellovin, http://www.cs.columbia.edu/~smb
Re: YouTube IP Hijacking
On Feb 25, 2008, at 2:27 AM, Hank Nussbacher wrote: At 07:15 PM 24-02-08 -0500, Randy Epstein wrote: More importantly, why is PCCW not prefix filtering their downstreams? Why? - Lack of clue - Couldn't care less - No revenue Take your pick - or add your own reason. PCCW is not alone. They just happen to be the latest in a long line of ISPs that follow the same rules - their own. All good, er, bad reasons. Fixing the filter your downstreams problem is very important. It would also solve 90-something percent of the problems mentioned in this thread. E.g. as7007. :) -- TTFN, patrick
Re: YouTube IP Hijacking
On Feb 25, 2008, at 2:32 AM, Hank Nussbacher wrote: At 05:31 AM 25-02-08 +, Steven M. Bellovin wrote: Seriously -- a number of us have been warning that this could happen. More precisely, we've been warning that this could happen *again*; we all know about many older incidents, from the barely noticed to the very noisy. (AS 7007, anyone?) Something like S-BGP will stop this cold. Yes, I know there are serious deployment and operational issues. The question is this: when is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done. we've been warning that this could happen *again* - this is happening every day - just look to: http://cs.unm.edu/~karlinjf/IAR/prefix.php?filter=most http://cs.unm.edu/~karlinjf/IAR/subprefix.php?filter=most for samples. Thing is - these prefix hijacks are not big ticket sites like Youtube or Microsoft or Cisco or even whitehouse.gov - but rather just sites that never make it onto the NANOG radar. How many of those would be stopped with transit providers filtering their downstreams? Which doesn't require rolling out a new technology like SBGP. And, I would argue, if we cannot even get transit providers to filter their downstreams, there is no way in hell we can get transit providers to filter on some RR or doing authentication on individual prefixes. Let's start with the easy stuff. Walk before run and all that. -- TTFN, patrick
Re: YouTube IP Hijacking
On Sun, 24 Feb 2008, Sargun Dhillon wrote: I don't know how large Pakistani Telecom is, but it I bet its not large enough that PCCW should be allowing it to advertise anything. I think you're failing to take into account how multihoming generally works. The real fallacy here is that PCCW/BTN refuses to prefix-list filter their customers, as evidenced by this and past leaks. If something productive can come from today's outage, it would be PCCW beginning to do their part as responsible Internet citizens, given (excuse the pun) peer pressure. I'd also focus on the lessons learned from the un-official IP Hijacking BOF held in San Jose, during which engineers and researchers studied the extent to which obviously-bogus route advertisements propagated across the public Internet. At these events, prefixes such as 1/8 and 100/7 were advertised, and, by Renesys/bgplay/route-views/etc data, accepted by 99% (?) of the internet. IP blocks that were hijacked before (like 146.20/16) were announced with similar outcome. Results were planned to be presented at the next NANOG, but they shouldn't be a surprise to anyone in the industry: nobody filters. Paul Wall
Re: YouTube IP Hijacking
having built an ISP or two in pakistan, PTCL (Pakistan Telecom) is not the sole provider of bandwidth to the country, although it likely carries the bulk of traffic to the country. operationally, there are a number of jurisdictions which filter content and connectivity on a variety of basis. adjusting the BGP announcements is a fairly quick and sure way to hobble connectivity to specific content. although, it is quickly bypassed by shifting the content to other addresses and domain names. i'm sure that this was an accidental leakage, and that appropriate corrections were/are taken in due course. -- Jim Mercer[EMAIL PROTECTED]+971 55 410-5633 I'm Prime Minister of Canada, I live here and I'm going to take a leak. - Lester Pearson in 1967, during a meeting between himself and President Lyndon Johnson, whose Secret Service detail had taken over Pearson's cottage retreat. At one point, a Johnson guard asked Pearson, Who are you and where are you going?
Re: YouTube IP Hijacking
Interesting that (according to Renesys) BT reconnected about 500 networks in Pakistan after the big fibre cut. I wonder if there's any data around that would tell us who filters and who doesn't? On Mon, Feb 25, 2008 at 9:02 AM, Jim Mercer [EMAIL PROTECTED] wrote: having built an ISP or two in pakistan, PTCL (Pakistan Telecom) is not the sole provider of bandwidth to the country, although it likely carries the bulk of traffic to the country. operationally, there are a number of jurisdictions which filter content and connectivity on a variety of basis. adjusting the BGP announcements is a fairly quick and sure way to hobble connectivity to specific content. although, it is quickly bypassed by shifting the content to other addresses and domain names. i'm sure that this was an accidental leakage, and that appropriate corrections were/are taken in due course. -- Jim Mercer[EMAIL PROTECTED]+971 55 410-5633 I'm Prime Minister of Canada, I live here and I'm going to take a leak. - Lester Pearson in 1967, during a meeting between himself and President Lyndon Johnson, whose Secret Service detail had taken over Pearson's cottage retreat. At one point, a Johnson guard asked Pearson, Who are you and where are you going?
Re: YouTube IP Hijacking
On Mon, Feb 25, 2008 at 09:13:23AM +, Alexander Harrowell wrote: Interesting that (according to Renesys) BT reconnected about 500 networks in Pakistan after the big fibre cut. I wonder if there's any data around that would tell us who filters and who doesn't? based on my experience of routing (and de-routing) my own legacy space as well as some RIPE space through PTCL, i know they have procedures in place to restrict what their customers can send to them, so it makes sense that they have a clue as to how to control what they send out. probably fat fingers, and probably fat wobbly fingers in a rush to comply with a government directive. On Mon, Feb 25, 2008 at 9:02 AM, Jim Mercer [EMAIL PROTECTED] wrote: having built an ISP or two in pakistan, PTCL (Pakistan Telecom) is not the sole provider of bandwidth to the country, although it likely carries the bulk of traffic to the country. operationally, there are a number of jurisdictions which filter content and connectivity on a variety of basis. adjusting the BGP announcements is a fairly quick and sure way to hobble connectivity to specific content. although, it is quickly bypassed by shifting the content to other addresses and domain names. i'm sure that this was an accidental leakage, and that appropriate corrections were/are taken in due course. -- Jim Mercer[EMAIL PROTECTED]+971 55 410-5633 I'm Prime Minister of Canada, I live here and I'm going to take a leak. - Lester Pearson in 1967, during a meeting between himself and President Lyndon Johnson, whose Secret Service detail had taken over Pearson's cottage retreat. At one point, a Johnson guard asked Pearson, Who are you and where are you going? -- Jim Mercer[EMAIL PROTECTED]+971 55 410-5633 I'm Prime Minister of Canada, I live here and I'm going to take a leak. - Lester Pearson in 1967, during a meeting between himself and President Lyndon Johnson, whose Secret Service detail had taken over Pearson's cottage retreat. At one point, a Johnson guard asked Pearson, Who are you and where are you going?
Re: YouTube IP Hijacking
Patrick W. Gilmore [EMAIL PROTECTED] wrote On Feb 25, 2008, at 2:27 AM, Hank Nussbacher wrote: At 07:15 PM 24-02-08 -0500, Randy Epstein wrote: More importantly, why is PCCW not prefix filtering their downstreams? Why? - Lack of clue - Couldn't care less - No revenue Take your pick - or add your own reason. PCCW is not alone. They just happen to be the latest in a long line of ISPs that follow the same rules - their own. All good, er, bad reasons. Fixing the filter your downstreams problem is very important. It would also solve 90-something percent of the problems mentioned in this thread. E.g. as7007. :) I am in the APRICOT meeting in Taipei now, and met a guy from PCCW/AS3491. I have showed him this thread, and have suggested 1) validating prefixes from downstreams before accept, and 2) setting an inbound prefix-filter to their downstreams. regards, - Matsuzaki Yoshinobu [EMAIL PROTECTED] - IIJ/AS2497 INOC-DBA: 2497*629
Re: YouTube IP Hijacking
On 25 feb 2008, at 9:14, Paul Wall wrote: I don't know how large Pakistani Telecom is, but it I bet its not large enough that PCCW should be allowing it to advertise anything. I think you're failing to take into account how multihoming generally works. The real fallacy here is that PCCW/BTN refuses to prefix-list filter their customers, as evidenced by this and past leaks. If something productive can come from today's outage, it would be PCCW beginning to do their part as responsible Internet citizens, given (excuse the pun) peer pressure. Well, if they had problems like this in the past, then I wouldn't trust them to get it right. Which means that it's probably a good idea if EVERYONE starts filtering what they allow in their tables from PCCW. Obviously that makes it very hard for PCCW to start announcing new prefixes, but I can't muster up much sympathy for that. So basically, rather than generate routing registry filters for the entire world, generate routing registry filters for known careless ASes. This number should be small enough that this is somewhat doable. (Although better tools for generating the filters would help, I tried generating filters from the RIPE db a few times but I couldn't figure it out using available tools and I didn't have the time to write my own.)
RE: YouTube IP Hijacking
This candidate list of requirements is for route sources that North American Operators should trust to propagate long prefix routes, nothing more, nothing less. All operators already have some kind of criteria which they use to decide whether or not to trust a particular source of routes whether long prefixes or short. You are suggesting that these operators should give up this role to a trusted third party so that al North American network operators share fate in terms of BGP trust relationships. It seems that you feel this is an improvement since some network operators have very lax policies and trust people that they shouldn't. In that case, I wonder whether these operators would even bother joining such a shared-fate arrangement. But the big downside is for the operators who have carefully honed filtering policies and who are careful about who they trust. For them there is no upside to joining a shared-fate arrangement, and a potential downside if management decides that their internal BGP filtering practices can now be made more lax. And, of course, the shared fate arrangement is now a single point of failure and a very juicy target for bad guys to attack. The real solution to the YouTube issue is for people to pressure other network operators to raise their game and pay attention to how they manage their BGP trust relationships and filter announcements. In addition, more people need to get involved in information sharing arrangements like Routing Registries, MyASN, alert services and so on. None of these things create a single point of failure and some of them would be useful even if your Super AS is created. Because all of this activity is done by humans, even your Super AS can make mistakes so it would be bad for people to trust it just because it is big. Alert services, RRs, MyASN, etc., can protect against human failures whether in the Super AS or Pakistan Telecom. Perhaps you might like to propose criteria you would find useful in setting a level of trust, or some alternative method to avoid a recurrence of a site that is widely visited being black holed through another ISP advertising a more specific route? Haven't you noticed that the definition of widely visited site changes regularly, and often quite abruptly? How much traffic did YouTube get 3 years ago? Facebook? MySpace? There is no shortcut for eternal vigilance, i.e. manage your BGP relationships don't just configure and forget. Item 2: in this context, is specific to the needs of North American Network Operators accepting long prefix routes. I am not advocating not accepting routes from the ROW, just not very specific ones. It's entirely possible for North American Operators to rely on law enforcement in say, the EU and Australia. In case you hadn't noticed, there is no North American law enforcement agency and no North American courts and no North American laws outside of NAFTA. So I'm not sure what you are getting at here. Do you want to reopen NAFTA negotiations to include Internet peering? I think it would be better to propose some constructive ideas as to how we can avoid what happened today from recurring, and also deal with the issue of hijacked IP space in general. The tools and techniques are out there. All that is needed is for people to follow best practices. Seems to me that educational activity would be more productive than building castles in the sky. --Michael Dillon
Re: Secure BGP (Was: YouTube IP Hijacking)
[EMAIL PROTECTED] wrote: [..] Pushing this task off to a server that does not have packet-forwarding duties also allows for flexible interfaces to network management systems including the possibility of asking for human confirmation before announcing a new route. There is no (direct) requirement for most of these solutions to do it in the router that forwards actual packets, just add a special BGP box for this. This box then 'verifies' if the update looks OK. When the update looks fishy, it can either, depending on what you want either notify your favourite $nocmonkey to look at it and/or at least instruct the real routers to not use that path. You can take (S-)BGP(-S) for verification, but you can also use IRR data or whatever source you have for stating 'this prefix from there over this path is trusted', compare against that and voila, you got a report when the assumed vectors don't match and you can at least react to them. These kind of systems already exist, see previous emails, but clearly not too many actually make use of them, now that is too bad for your customers who couldn't see their lolcats or worse who couldn't reach their stock house for quickly selling their shares before that company went down the drain completely... Greets, Jeroen signature.asc Description: OpenPGP digital signature
Secure BGP (Was: YouTube IP Hijacking)
Right. Everyone makes mistakes, but not everyone is malicious.And the RIRs and the big ISPs are *generally* more clueful than the little guys and the newcomers. Note also that secured BGP limits the kinds of mistakes people can make. If I have a certificate from my RIR for 192.0.2.0/24, I can't neither announce 10.0.0.0/8 nor delegate it to you, no matter how badly I type. Secured BGP still strikes me as a net win. I suspect that a major part of the problem with implementing Secured BGP is that it is put forth as a solution that you implement in your routers. Network Operators are very careful about the stuff that goes into routers, even to the extent that many of them do not use SSH to manage them. Instead, they run SSH on trusted and secured servers inside their PoPs and configure their routers to only accept telnet sessions from those trusted and secured servers. Is there some way of deploying a solution like Secure BGP without actually requiring that it go into the routers? Perhaps something that allows the routers to still maintain BGP sessions that can withdraw routes, or announce routes which were recently withdrawn, but require a separate encrypted session between two servers, each one in a trust relationship with one of the BGP speaking routers, to handle announcements of new routes? Pushing this task off to a server that does not have packet-forwarding duties also allows for flexible interfaces to network management systems including the possibility of asking for human confirmation before announcing a new route. --Michael Dillon
Re: YouTube IP Hijacking
On Mon, Feb 25, 2008 at 10:12:47AM -, [EMAIL PROTECTED] wrote: In case you hadn't noticed, there is no North American law enforcement agency and no North American courts and no North American laws outside of NAFTA. So I'm not sure what you are getting at here. Do you want to reopen NAFTA negotiations to include Internet peering? the law process within NAFTA is no more than a delay tactic used by various big business and big government to defer any real resolution to the problems. the laws of Canada, Mexico and the US are still largely seperate, and the laws of one do not necessarily follow in another. -- Jim Mercer[EMAIL PROTECTED]+971 55 410-5633 I'm Prime Minister of Canada, I live here and I'm going to take a leak. - Lester Pearson in 1967, during a meeting between himself and President Lyndon Johnson, whose Secret Service detail had taken over Pearson's cottage retreat. At one point, a Johnson guard asked Pearson, Who are you and where are you going?
RE: YouTube IP Hijacking
the laws of Canada, Mexico and the US are still largely seperate, and the laws of one do not necessarily follow in another. Not to mention other North American countries such as France(1), Bermuda, Cuba, Haiti, etc., etc. --Michael Dillon (1) The islands of St. Pierre and Miquelon, Martinique, Guadeloupe P.S. The point of this is that there is nothing nice and neat about the environment in which the Internet operates therefore we really can't expect to find any nice and neat solutions to the messy problems we run across. All we can do is to limit the damage, detect the aberrations, and put them right as quickly as we can. The public has voted and their opinion is that 5 nines does not matter, best effort is good enough, if it really is *BEST* effort.
Re: YouTube IP Hijacking
On Sun, Feb 24, 2008 at 10:49 PM, Sean Donelan [EMAIL PROTECTED] wrote: On Mon, 25 Feb 2008, Steven M. Bellovin wrote: How about state-of-the-art routing security? The problem is what is the actual trust model? Are you trusting some authority to not be malicious or never make a mistake? There are several answers to the malicious problem. There are fewer answers to never making a mistake problem. [snip] +5, Insightful. The focus thus far seems to have been on establishing security on the basis of trusted senders (SPF for BGP, if you'll pardon my rather crude analogy). The impact of a mistake-based failure in a trusted scenario could actually be quite a bit higher than what we've seen thus far: 1) if data (e.g. routes) from a trusted source is allowed into a network (or used as a basis for decision-making) with minimal screening, attackers will immediately shift focus to spoofing trusted sources, just as they currently do in other scenarios; 2) the impact of a typo or other operator error when operating in trusted mode is much higher than that of mistakes from non-trusted sources (if 17557's upstream had trusted a little less - by not automatically propagating any new routes that showed up at the front door - the current incident could very well have amounted to little more than a log entry somewhere, and perhaps an email). I think what you and Steve Bellovin had to say about anti-mistake protocol and belt-and-suspenders should be garnering at least as much consideration as prevention of malicious attacks/forgeries/etc., considering the percentage of outages caused by operator error vs those caused by malice ... -- [EMAIL PROTECTED],darkuncle.net} || 0x5537F527 http://darkuncle.net/pubkey.asc for public key
Re: YouTube IP Hijacking
At 06:17 PM 25-02-08 +0900, Matsuzaki Yoshinobu wrote: All good, er, bad reasons. Fixing the filter your downstreams problem is very important. It would also solve 90-something percent of the problems mentioned in this thread. E.g. as7007. :) I am in the APRICOT meeting in Taipei now, and met a guy from PCCW/AS3491. I have showed him this thread, and have suggested 1) validating prefixes from downstreams before accept, and 2) setting an inbound prefix-filter to their downstreams. Now if only everyone here on NANOG were to do what Matsuzaki has done, and take the time to educate those less clueless, the world would be a better place. -Hank
Re: YouTube IP Hijacking
At 03:14 AM 25-02-08 -0500, Paul Wall wrote: Results were planned to be presented at the next NANOG, but they shouldn't be a surprise to anyone in the industry: nobody filters. Incorrect. Some do filter and do it well. Problem is that it is in general a minority - many of which can be found here on NANOG. -Hank
BGP prefix filtering, how exactly? [Re: YouTube IP Hijacking]
Changed the subject line a little... On Mon, 25 Feb 2008, Hank Nussbacher wrote: At 03:14 AM 25-02-08 -0500, Paul Wall wrote: Results were planned to be presented at the next NANOG, but they shouldn't be a surprise to anyone in the industry: nobody filters. Incorrect. Some do filter and do it well. Problem is that it is in general a minority - many of which can be found here on NANOG. In a lot of this dialogue, many say, you should prefix filter. However, I'm not seeing how an ISP could easily adopt such filtering. Let's consider the options: 1) manually maintained prefix-filters. OK for small ISPs or small users where the prefix churn is minimal. 2) build the filters based on IRR data. But which IRRs to use? some points here: a) only RIPE IRR uses a sensible security model [1], so if you use others, basically anyone can add route objects to the registry. How exactly would this model be useful? b) use your own IRR where you require your customers to add the route objects and verify that they're OK. This means a lot of work for you and even more for your customers. So, this is no excuse for not doing prefix filtering if you only do business in the RIPE region, but anywhere else the IRR data is pretty much useless, incorrect, or both. (Yeah, we prefix filter all our customers. Our IPv6 peers are also prefix filtered, based on RIPE IRR data (with one exception). IPv4 peers' advertisements seem to be too big a mess, and too long filters, to fix this way.) [1] Joe Abley's explanation on SIDR list on 20 Jun 2007: http://www.ietf.org/mail-archive/web/sidr/current/msg00201.html -- Pekka Savola You each name yourselves king, yet the Netcore Oykingdom bleeds. Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
Re: ISP's who where affected by the misconfiguration: start using IRR and checking your BGP updates (Was: YouTube IP Hijacking)
On Mon, 25 Feb 2008, Hank Nussbacher wrote: For us who actually have customers we care about, we probably find it better for business to try to make sure our own customers can't announce prefixes they don't own, but accept basically anything from the world that isn't ours. You are a distinct minority. My experience has shown that most ISPs don't give a sh*t about filtering what their customers can announce so what has happened, will continue to happen. I've only dealt with a handful of the bigger networks, but every transit BGP session I've ever been the customer role on has been filtered by the provider. From memory and in no particular order, that's UUNet, Level3, Digex, Intermedia, Global Crossing, Genuity, Sprint, Above.net, Time Warner, CW, MCI, XO, Broadwing, and a few smaller ones nobody's likely to have heard of. As an ISP providing transit, all of our customers get prefix-filtered. -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: YouTube IP Hijacking
Christopher Morrow wrote: On Sun, Feb 24, 2008 at 8:42 PM, Patrick W. Gilmore [EMAIL PROTECTED] wrote: except that even the 'good guys' make mistakes. Belt + suspenders please... is it really that hard for a network service provider to have a prefix-list on their customer bgp sessions?? L3 does it, ATT does it, Sprint does it, as do UUNET/vzb, NTT, GlobalCrossing ... seriously, it's not that hard. I know of one tier-1 on that list that made a filtering mistake on one of my own circuits no more than a few months ago. Even some of the biggest players make mistakes. Justin
RE: YouTube IP Hijacking
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Steven M. Bellovin How about state-of-the-art routing security? Seriously -- a number of us have been warning that this could happen. More precisely, we've been warning that this could happen *again*; we all know about many older incidents, from the barely noticed to the very noisy. (AS 7007, anyone?) Something like S-BGP will stop this cold. Yes, I know there are serious deployment and operational issues. The question is this: when is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done. BCPs stops this problem. soBGP may make it easier.
hijack chronology: was [ YouTube IP Hijacking ]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Greetings, Late last night, after poring through our data, I posted a detailed chronology of the hijack as seen from our many peering sessions. I would add to this that the speed of YouTube's response to this subprefix hijack impressed me. As discussed earlier in this thread, this is really the same old song--it simply has a new verse now. (How many of our troubadors know all of the verses since AS 7007?) Best, - -Martin [0] http://www.renesys.com/blog/2008/02/pakistan_hijacks_youtube.shtml - -- Martin A. Brown --- Renesys Corporation --- [EMAIL PROTECTED] -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (GNU/Linux) Comment: pgf-0.72 (http://linux-ip.net/sw/pine-gpg-filter/) iD8DBQFHwtYodXQGngQsWbkRAv1PAKDB0w1OjRmKjF7027ccZJzuzyboKQCgwxFt V8CNr0Nzw2lqx0O5vk3uHKo= =GUic -END PGP SIGNATURE-
Re: YouTube IP Hijacking
y'all, On Mon, Feb 25, 2008 at 06:49:35AM -0800, Barry Greene (bgreene) wrote: Seriously -- a number of us have been warning that this could happen. More precisely, we've been warning that this could happen *again*; we all know about many older incidents, from the barely noticed to the very noisy. (AS 7007, anyone?) Something like S-BGP will stop this cold. Yes, I know there are serious deployment and operational issues. The question is this: when is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done. BCPs stops this problem. soBGP may make it easier. except in the most meaningless of interpretations, i don't see how BCPs solve this problem. Barry: did you mean: If only everyone on the Internet would perfectly filter their customers prefix announcements on every connection, then everything would be fine? Or perhaps: If everyone would register all of their prefixes in some applicable routing registry with such a degree of accuracy that we could build customer and peer filters out of it then this problem would go away. ? both are true statements, but neither is ever going to happen (i'm not sure that sBGP or soBGP is going to happen ever either). in the mean time all we can do is watch and try to respond to events as quickly as they occur. In fact, we would all be better off if more people were just watching their prefix announcement progapations. Although this hijacking/blackholing was caught quickly, i have seen many other cases where the event lasted for hours (or days). by the way: my colleague Martin A. Brown (who spoke at NANOG on the Taiwan Earthquakes), has written a fairly detailed blog post on the this event. It includes a detailed timeline and some information for people who might find that useful. http://www.renesys.com/blog/2008/02/pakistan_hijacks_youtube_1.shtml Clearly this is not the first interesting accidental hijacking and certainly won't be the last. t. -- _ todd underwood +1 603 643 9300 x101 renesys corporationgeneral manager babbledog [EMAIL PROTECTED] http://www.renesys.com/blog
Rép : YouTube IP Hijacking
Le 25 févr. 08 à 02:42, Patrick W. Gilmore a écrit : On Feb 24, 2008, at 7:36 PM, Tomas L. Byrnes wrote: 1: Hosted at a Tier 1 provider. That is a silly requirement. (I am sorry, I tried hard to find a nicer way to say this, but I really feel strongly about this.) 2: Within a jurisdiction where North American operators have a good chance of having the law on their side in case of any network outage caused by the entity. This is also a bit strange. Do your users never attach to a host outside the USofA? Wasn't it humour / joke / troll :-) ? If not, maybe we've to follow discussion on nnsquad (Network Neutrality Squad) mailing list ? :-)) - Jean-Michel Planche blog: http://www.jmp.net Chairman and co-founder Witbe web : http://www.witbe.net Follow me http://www.twitter.com/jmplanche --- 2.0 Monitoring : relevant End to End monitoring for critical app. and carrier class services
Rep : YouTube IP Hijacking
If someone comes up with the anti-mistake routing protocol ... We could try to invent more idiot proof protocols, but the more control (and centralization), the more it will be a kind of Internet. Not sure the founding principles and factors that made the Internet successful would resist anymore. Anyhow, you can think all you want about security, such as Apple with its concept of his locking iPhone, but you can’t anticipate the unexpected, like the ‘jailbreaker’ success ... and peoples acting on their routers with their feet and not with their brain and their hands. It's the nature of Internet technology: something could always fail and the ability to prepare for the unexpected is one of the reason why it works and is so scalable. It's also why best effort / real knowledge / education are better approaches than searching for yet another killer secure protocol. But maybe I'm a dreamer :-)) Anyhow, I’m not saying that nothing can be done. I can see at least two possibilities: 1/ What measures can be taken to prevent such things from happening and great discussion about it on the list. 2/ How can we take a more proactive approach and be informed of such incidents as soon as they occur and not after the first customer complaint On first issue,a lot to do ... If ‘best effort’ is something that always exist in the today business world, I think we’ll arrive at an equilibrium without waiting for geni.net (certainly good) answers. On second issue, there are plenty of possibilities and it's not difficult now to be informed to the minute when big destination / AS seems to be in trouble. FYI, just see: a very interesting TCP traceroute yesterday, during the mistake on Youtube (seen from our system / AS15436 on US East coast, Europe (Paris and London) http://www.flickr.com/photos/jmplanche/2291442426/ http://www.flickr.com/photos/jmplanche/2290636351/ : routing changed http://www.flickr.com/photos/jmplanche/2290636277/ : Youtube.com unreachable http://www.flickr.com/photos/jmplanche/2290636131/ : very interesting, there is an abnormally high response time, just before severe breakdown. Is Pakistan trying to announce more and more Youtube address and less and less Youtube servers available to answer ? http://www.flickr.com/photos/jmplanche/2291429544/ : not the same overload just before crash as seen from FR and UK : http://www.flickr.com/photos/jmplanche/2291429414/ - Jean-Michel Planche blog: http://www.jmp.net Chairman and co-founder Witbe web : http://www.witbe.net Follow me http://www.twitter.com/jmplanche --- 2.0 Monitoring : relevant End to End monitoring for critical app. and carrier class services
Re: ISP's who where affected by the misconfiguration: start using IRR and checking your BGP updates (Was: YouTube IP Hijacking)
On Mon, Feb 25, 2008 at 09:28:47AM -0500, Jon Lewis wrote: I've only dealt with a handful of the bigger networks, but every transit BGP session I've ever been the customer role on has been filtered by the provider. From memory and in no particular order, that's UUNet, Level3, Digex, Intermedia, Global Crossing, Genuity, Sprint, Above.net, Time Warner, CW, MCI, XO, Broadwing, and a few smaller ones nobody's likely to have heard of. We take transit from some of these providers, and I we have a slightly different experience. While it's not quite a free-for-all, some have implemented a limit on the number of announced prefixes without any restriction to specific space. We found this out after AboveNet dampened us for announcing too many routes. No one there could ever produce any substantial evidence of that, or provide us a single example of one of these routes - but we were told it was strictly the number of prefixes that mattered. I know that I provide newly assigned prefixes to our providers, which includes PCCW. If those make it into a prefix-list at PCCW though, I don't really know for sure. -- Ross Vandegrift [EMAIL PROTECTED] The good Christian should beware of mathematicians, and all those who make empty prophecies. The danger already exists that the mathematicians have made a covenant with the devil to darken the spirit and to confine man in the bonds of Hell. --St. Augustine, De Genesi ad Litteram, Book II, xviii, 37
RE: YouTube IP Hijacking
DO NOT sign up at that site until the site admin fixes a major issue - I thought it looked interesting but now I'm in an embarrassing situation. I signed up like anyone would do and the moment I validated my email address, postings started to showup under my account that are weeks old - these postings are of sexual nature ... lovely Not impressed to say the least emailed the site admin asking something be done... Paul Stewart Senior Network Administrator Nexicom 5 King St. E., Millbrook, ON, LOA 1GO Phone: 705-932-4127 Web: http://www.nexicom.net Nexicom - Connected. Naturally. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jason Sent: Sunday, February 24, 2008 8:13 PM To: nanog@merit.edu Cc: Paul Ferguson Subject: Re: YouTube IP Hijacking This is similar, and available for all regions/ASNs. http://cs.unm.edu/~karlinjf/IAR/index.php -- Jason Paul Stewart wrote: Very nice.. is there an ARIN equal that anyone knows of OR can you use the RIPE one for ARIN registered space? Just curious.. thanks.. Paul -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Paul Ferguson Sent: Sunday, February 24, 2008 7:07 PM To: [EMAIL PROTECTED] Cc: nanog@merit.edu Subject: Re: YouTube IP Hijacking -- Daniel Roesen [EMAIL PROTECTED] wrote: On Sun, Feb 24, 2008 at 10:41:26PM +, Paul Ferguson wrote: The best you can _probably_ hope for is a opt-in mechanism in which you are alerted that prefixes you have registered with the aforementioned system are being originated by an ASN which is not authorized to originate them. http://www.ris.ripe.net/myasn.html Nice. :-) - ferg -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/ The information transmitted is intended only for the person or entity to which it is addressed and contains confidential and/or privileged material. If you received this in error, please contact the sender immediately and then destroy this transmission, including all attachments, without copying, distributing or disclosing same. Thank you.
RE: YouTube IP Hijacking
The Site admin got back to me right away I jumped the gun slightly.. Anyways, a spammer had signed into that site previously with the same username and posted lots of crap - when I signed up, those posts came back online hence my panic Should be fine now - interesting site ;) Paul -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Paul Stewart Sent: Monday, February 25, 2008 11:48 AM To: [EMAIL PROTECTED]; nanog@merit.edu Cc: Paul Ferguson Subject: RE: YouTube IP Hijacking DO NOT sign up at that site until the site admin fixes a major issue - I thought it looked interesting but now I'm in an embarrassing situation. I signed up like anyone would do and the moment I validated my email address, postings started to showup under my account that are weeks old - these postings are of sexual nature ... lovely Not impressed to say the least emailed the site admin asking something be done... Paul Stewart Senior Network Administrator Nexicom 5 King St. E., Millbrook, ON, LOA 1GO Phone: 705-932-4127 Web: http://www.nexicom.net Nexicom - Connected. Naturally. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jason Sent: Sunday, February 24, 2008 8:13 PM To: nanog@merit.edu Cc: Paul Ferguson Subject: Re: YouTube IP Hijacking This is similar, and available for all regions/ASNs. http://cs.unm.edu/~karlinjf/IAR/index.php -- Jason Paul Stewart wrote: Very nice.. is there an ARIN equal that anyone knows of OR can you use the RIPE one for ARIN registered space? Just curious.. thanks.. Paul -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Paul Ferguson Sent: Sunday, February 24, 2008 7:07 PM To: [EMAIL PROTECTED] Cc: nanog@merit.edu Subject: Re: YouTube IP Hijacking -- Daniel Roesen [EMAIL PROTECTED] wrote: On Sun, Feb 24, 2008 at 10:41:26PM +, Paul Ferguson wrote: The best you can _probably_ hope for is a opt-in mechanism in which you are alerted that prefixes you have registered with the aforementioned system are being originated by an ASN which is not authorized to originate them. http://www.ris.ripe.net/myasn.html Nice. :-) - ferg -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/ The information transmitted is intended only for the person or entity to which it is addressed and contains confidential and/or privileged material. If you received this in error, please contact the sender immediately and then destroy this transmission, including all attachments, without copying, distributing or disclosing same. Thank you.
RE: YouTube IP Hijacking
This is a very interesting site. However, I notice that, in the all in the last 24 hours it doesn't show the YouTube hijack. It does have a lot of entries for 17557, most recently on 2/17. How reliable is this system? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Hank Nussbacher Sent: Sunday, February 24, 2008 11:33 PM To: Steven M. Bellovin; nanog@merit.edu Subject: Re: YouTube IP Hijacking At 05:31 AM 25-02-08 +, Steven M. Bellovin wrote: Seriously -- a number of us have been warning that this could happen. More precisely, we've been warning that this could happen *again*; we all know about many older incidents, from the barely noticed to the very noisy. (AS 7007, anyone?) Something like S-BGP will stop this cold. Yes, I know there are serious deployment and operational issues. The question is this: when is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done. we've been warning that this could happen *again* - this is happening every day - just look to: http://cs.unm.edu/~karlinjf/IAR/prefix.php?filter=most http://cs.unm.edu/~karlinjf/IAR/subprefix.php?filter=most for samples. Thing is - these prefix hijacks are not big ticket sites like Youtube or Microsoft or Cisco or even whitehouse.gov - but rather just sites that never make it onto the NANOG radar. -Hank
Re: YouTube IP Hijacking
Tomas: It's primarily a proof of concept site, to show that such an idea would be useful, but it has been running for over a year now and discovered many interesting hijacks (such as eBay/google/etc..). You're right that there is a glaring ommission, which is yesterday's youtube hijack. This is due to a bug in the sub-prefix lookup code (which can cause the IAR to miss some sub-prefix hijacks), which I'm currently fixing. Once that is done I'll rerun the IAR over yesterday's logs and it will show up. Josh On Mon, Feb 25, 2008 at 10:37 AM, Tomas L. Byrnes [EMAIL PROTECTED] wrote: This is a very interesting site. However, I notice that, in the all in the last 24 hours it doesn't show the YouTube hijack. It does have a lot of entries for 17557, most recently on 2/17. How reliable is this system? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Hank Nussbacher Sent: Sunday, February 24, 2008 11:33 PM To: Steven M. Bellovin; nanog@merit.edu Subject: Re: YouTube IP Hijacking At 05:31 AM 25-02-08 +, Steven M. Bellovin wrote: Seriously -- a number of us have been warning that this could happen. More precisely, we've been warning that this could happen *again*; we all know about many older incidents, from the barely noticed to the very noisy. (AS 7007, anyone?) Something like S-BGP will stop this cold. Yes, I know there are serious deployment and operational issues. The question is this: when is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done. we've been warning that this could happen *again* - this is happening every day - just look to: http://cs.unm.edu/~karlinjf/IAR/prefix.php?filter=mosthttp://cs.unm.edu/%7Ekarlinjf/IAR/prefix.php?filter=most http://cs.unm.edu/~karlinjf/IAR/subprefix.php?filter=mosthttp://cs.unm.edu/%7Ekarlinjf/IAR/subprefix.php?filter=most for samples. Thing is - these prefix hijacks are not big ticket sites like Youtube or Microsoft or Cisco or even whitehouse.gov - but rather just sites that never make it onto the NANOG radar. -Hank
Re: BGP prefix filtering, how exactly? [Re: YouTube IP Hijacking]
On Feb 25, 2008, at 6:08 AM, Pekka Savola wrote: In a lot of this dialogue, many say, you should prefix filter. However, I'm not seeing how an ISP could easily adopt such filtering. So, this is no excuse for not doing prefix filtering if you only do business in the RIPE region, but anywhere else the IRR data is pretty much useless, incorrect, or both. Agreed. (Yeah, we prefix filter all our customers. Our IPv6 peers are also prefix filtered, based on RIPE IRR data (with one exception). IPv4 peers' advertisements seem to be too big a mess, and too long filters, to fix this way.) Do you explicitly filter routes from your upstream or transit providers? E.g., if one were to announce, say, a more specific of one of your customer's routes to you would you accept it? What about someone else's address space? The only full set of prefix filtering I've ever seen implemented (i.e., BGP customers AND peers) was b y ANS during my days at iMCI ~95. It was extremely painful at times, even for us, if we wanted to advertise new address space we had to update IRR objects and wait on their nightly push of updated routing policies at ANS. We generated our own routing policies automatically off our IRR, which mirrored others as well, and explicitly prefix filtered customers with some fixed prefix and AS path-based policies applied to peers. If it became really urgent, then we'd call ANS and have them manually update their policy, and subsequently 'bounce' the route announcement to trigger transmission of a new update. This was long before incrementally updated filters and things like BGP route refresh ever existed. Prefixes and AS-MACROs had to be right in the IRRs or the policies wouldn't be updated. It's to bad other folks didn't follow suit. As for this event, a slightly different spin here: http://tinyurl.com/3y3pzl -danny
Re: BGP prefix filtering, how exactly? [Re: YouTube IP Hijacking]
On Mon, 25 Feb 2008, Danny McPherson wrote: (Yeah, we prefix filter all our customers. Our IPv6 peers are also prefix filtered, based on RIPE IRR data (with one exception). IPv4 peers' advertisements seem to be too big a mess, and too long filters, to fix this way.) Do you explicitly filter routes from your upstream or transit providers? E.g., if one were to announce, say, a more specific of one of your customer's routes to you would you accept it? What about someone else's address space? Our own or our singlehomed customers' address space -- we would reject such an advertisement. The same inbound consistency check applies to peers and upstreams/transits. If it's someone else's or a more specific or the same prefix as our multihomed customers -- we accept it. There isn't anything else we can do in practise which would not hurt legitimate routing.. It was extremely painful at times, even for us, if we wanted to advertise new address space we had to update IRR objects and wait on their nightly push of updated routing policies at ANS. We generated our own routing policies automatically off our IRR, which mirrored others as well, and explicitly prefix filtered customers with some fixed prefix and AS path-based policies applied to peers. If it became really urgent, then we'd call ANS and have them manually update their policy, and subsequently 'bounce' the route announcement to trigger transmission of a new update. Sounds like a procedure that should be applied today (whether or not you want to use IRR and/or autogenerated configs is a matter of taste) but the principle seems sound. -- Pekka Savola You each name yourselves king, yet the Netcore Oykingdom bleeds. Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
[admin] [summary] RE: YouTube IP Hijacking
A bit of administrativia: This thread generated over a hundred posts, many without operational relevance or by people who do not understand how operators, well, operate, or by people who really don't have any idea what's going on but feel like posting. I'd like to briefly summarize the important things that were said. If you would like to add something to the thread, make sure you read this post in entirety. Sorry if I didn't attribute every suggestion to a poster. Facts: * AS17557 announced more specific /24 to 3491, which propagated to wider internets * Chronology (by [EMAIL PROTECTED]) http://www.renesys.com/blog/2008/02/pakistan_hijacks_youtube.shtml * Things suggested to possibly address the problem: ** IRR filtering (using IRRPT http://sourceforge.net/projects/irrpt/ to generate filter lists) ** Notification when origin of a given route changes http://www.cs.ucla.edu/~mohit/cameraReady/ladSecurity06.pdf http://www.ris.ripe.net/myasn.html http://cs.unm.edu/~karlinjf/IAR/index.php (from pgBGP) ** pgBGP to depref suspicious routes http://www.nanog.org/mtg-0606/pdf/josh-karlin.pdf (unclear the number of false positives that will adversely affect connectivity) ** sbgp/sobgp - require full authentication for each IP block, and thus unlikely to be implemented until certificate chains are in place, and vendors release code that does verification, and operators are happy enough running it. Other things addressed: * Fragility of Internet: ** Nobody brought up the important point - the BGP announcement filtering are only as secure as the weakest link. No [few?] peers or transits are filtering large ISPs (ones announcing few hundred routes and up). There are a great many of them, and it takes only one of them to mess up filtering a downstream customer for the route to be propagated. ** Paul Wall brought up the fact that even obviously bogus routes (1/8 and 100/7) were accepted by 99% of internet during an experiment. Will it take someone announcing 9/11 to get us to pay attention? (ok, bad joke) ** What I'd like to see discussed: Issues of filtering your transit downstream customers, who announce thousands of routes. Does *anyone* do it? * Typos vs Malicious announcements ** Some ways of fixing the problem (such as IRR filtering) only address the typos or unintentional announcements. There's full agreement that IRR is full of junk, which is not authenticated in any sort. ** Things like PHAS won't work if hijacker keeps the origin-AS same (by getting their upstream to establish session with different ASN) ** What I'd like to see discussed: Who (ICANN/RIRs/LIRs) is actively working on implementing chain of trust of IP space allocations? * Ways to address the issue without cooperation of 3491: ** Filtering anything coming out of 17557 ** Suggestions given: ** What I'd like to see discussed: Can an network operator, *today*, filter the possibly bogus routes from their peers, without manual intervention, and without false positives? * Yelling at people who don't filter ** Per above, 3491 isn't the only one who filters. In fact, claims were made that *nobody* filters large enough downstreams. (beyond aspath/maxpref) ** *please* do not post additional comments about pccw bad, etc. * Malicious vs mistaken on part of AS17557 and 3491: ** *please* do not post speculation unless you have facts to back it up. ** Any discussions of cyber-jihad are off-topic unless you can produce the fatwa to back it up.
Re: [admin] [summary] RE: YouTube IP Hijacking
On Feb 25, 2008, at 12:51 PM, Alex Pilosov wrote: ** Nobody brought up the important point - the BGP announcement filtering are only as secure as the weakest link. No [few?] peers or transits are filtering large ISPs (ones announcing few hundred routes and up). There are a great many of them, and it takes only one of them to mess up filtering a downstream customer for the route to be propagated. Yes, that was my implicit point to Pekka. Even if you do everything feasible today (i.e., explicitly filter customers, some amount of policy for peers, and perhaps a few hacks for multi-homed customers), you're still pretty much screwed if someone announces your address space. Heck, you're as likely to accept the announcement as anyone. ** Paul Wall brought up the fact that even obviously bogus routes (1/8 and 100/7) were accepted by 99% of internet during an experiment. I'm not sure why this would surprise anyone. ** What I'd like to see discussed: Issues of filtering your transit downstream customers, who announce thousands of routes. Does *anyone* do it? Lots of folks do. The interesting bit is that even then, those same providers would accept perhaps even those customer routes from their peers implicitly. * Typos vs Malicious announcements ** Some ways of fixing the problem (such as IRR filtering) only address the typos or unintentional announcements. You mean as opposed to intentionally malice acts? Well, not completely. See Pekka's email, for example. Of course, it does vary widely across IRRs, etc.. There's full agreement that IRR is full of junk, which is not authenticated in any sort. Mostly, though not completely. ** Things like PHAS won't work if hijacker keeps the origin-AS same (by getting their upstream to establish session with different ASN) NO, that's not even necessary. Simple originate the route from the legit AS, and then transit it with the local AS as a transit AS. AS path manipulation is trivial. ** What I'd like to see discussed: Who (ICANN/RIRs/LIRs) is actively working on implementing chain of trust of IP space allocations? * Ways to address the issue without cooperation of 3491: ** Filtering anything coming out of 17557 Bad idea. ** Suggestions given: ** What I'd like to see discussed: Can an network operator, *today*, filter the possibly bogus routes from their peers, without manual intervention, and without false positives? Sure, if they want to dedicate an engineer to it, automate policy deployment and deal with brokenness by turning steam valves. * Yelling at people who don't filter That's been productive for over a decade now. ** Per above, 3491 isn't the only one who filters. In fact, claims were made that *nobody* filters large enough downstreams. (beyond aspath/maxpref) Wrong. -danny
Re: [admin] [summary] RE: YouTube IP Hijacking
On Mon, 25 Feb 2008, Danny McPherson wrote: ** Paul Wall brought up the fact that even obviously bogus routes (1/8 and 100/7) were accepted by 99% of internet during an experiment. I'm not sure why this would surprise anyone. To me and you, it's not surprising. To public, it might be. Even the majority of nanog attendees I think would be surprised. ** What I'd like to see discussed: Issues of filtering your transit downstream customers, who announce thousands of routes. Does *anyone* do it? Lots of folks do. The interesting bit is that even then, those same providers would accept perhaps even those customer routes from their peers implicitly. Well, in this case, they *aren't* filtering! (unless I am misunderstanding what you are saying, due to repeated use of 'their'). ** Things like PHAS won't work if hijacker keeps the origin-AS same (by getting their upstream to establish session with different ASN) NO, that's not even necessary. Simple originate the route from the legit AS, and then transit it with the local AS as a transit AS. AS path manipulation is trivial. Oh yeah, d'oh! Thanks for correction. But that is also an important point against PHAS and IRRPT filtering - they are powerless against truly malicious hijacker (one that would register route in IRR, add the right origin-as to AS-SET, and use correct origin). ** What I'd like to see discussed: Who (ICANN/RIRs/LIRs) is actively working on implementing chain of trust of IP space allocations? * Ways to address the issue without cooperation of 3491: ** Filtering anything coming out of 17557 Bad idea. Obviously :) ** Suggestions given: ** What I'd like to see discussed: Can an network operator, *today*, filter the possibly bogus routes from their peers, without manual intervention, and without false positives? Sure, if they want to dedicate an engineer to it, automate policy deployment and deal with brokenness by turning steam valves. I'd hear to see who does it, and get them to present the operational lessons at the next nanog! * Yelling at people who don't filter That's been productive for over a decade now. ** Per above, 3491 isn't the only one who filters. In fact, claims were made that *nobody* filters large enough downstreams. (beyond aspath/maxpref) Wrong. Likewise, I'd like to know who does this (names) and how can we get them to present best practices at the next nanog! -alex
RE: BGP prefix filtering, how exactly? [Re: YouTube IP Hijacking]
clip Our own or our singlehomed customers' address space -- we would reject such an advertisement. The same inbound consistency check applies to peers and upstreams/transits. If it's someone else's or a more specific or the same prefix as our multihomed customers -- we accept it. There isn't anything else we can do in practise which would not hurt legitimate routing.. clip What do you do when one of your multi-homed customers on your IP space has an outage on their connection to your network? How would your customers then reach that customer? Although this wouldn't be THAT BIG of a deal for small networks, if say a larger or a Tier-1 provider practiced this (AFAIK, the only somewhat large network to do this is, believe it or not, PCCW), your customer would experience a major outage. There must be a better way. :) Pekka Savola Regards, Randy
Re: [admin] [summary] RE: YouTube IP Hijacking
On Feb 25, 2008, at 1:22 PM, Alex Pilosov wrote: Well, in this case, they *aren't* filtering! (unless I am misunderstanding what you are saying, due to repeated use of 'their'). What I'm saying is that best case today ISPs police routes advertised by their customers, yet they accept routes implicitly (including routes from address space that may belong to their customers) from peers. Seems a little hokey, eh? Oh yeah, d'oh! Thanks for correction. But that is also an important point against PHAS and IRRPT filtering - they are powerless against truly malicious hijacker (one that would register route in IRR, add the right origin-as to AS-SET, and use correct origin). Yep, pretty much. Sure, if they want to dedicate an engineer to it, automate policy deployment and deal with brokenness by turning steam valves. I'd hear to see who does it, and get them to present the operational lessons at the next nanog! Maybe Curtis V. would present what ANS was doing in 1994 :-) But now we've even got things like BGP route refresh, incrementally updatable filters, and BGP soft reconfiguration to ease the deployment burden. There have been two or three panels on this exact topic in the past, you can find them in the index of talks. Unfortunately, the problem hasn't changed at all. Perhaps we could just replay those video streams :-) -danny
Re: [admin] [summary] RE: YouTube IP Hijacking
I'd hear to see who does it, and get them to present the operational lessons at the next nanog! On second thought, I guess one thing has changed considerably since 15 years ago. Rather than ~5000 monkeys with keyboard access to manipulate global routing tables, there are likely well North of 250,000 (25k active ASNs * 10 meat computers per), which is surely well on the conservative side. The bottom line is [still] that ISPs should at least explicitly filter prefixes from customers and networks from which they provide transit services. -danny
Re: BGP prefix filtering, how exactly? [Re: YouTube IP Hijacking]
On Mon, 25 Feb 2008 15:29:01 EST, Randy Epstein said: Our own or our singlehomed customers' address space -- we would reject ^^^ such an advertisement. The same inbound consistency check applies to peers and upstreams/transits. What do you do when one of your multi-homed customers on your IP space has an outage on their connection to your network? How would your customers then reach that customer? He explicitly said single-homed. Of course, multi-homed requires different handling, because you may hear their other home announce them (although again, you probably shouldn't listen to *THAT* announcement either if *your* link to them is up). And I posit that if you don't know if your customer is single or multi-homed, you have *bigger* issues to deal with. pgpReSJzdoydy.pgp Description: PGP signature
RE: [admin] [summary] RE: YouTube IP Hijacking
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 There have been two or three panels on this exact topic in the past, you can find them in the index of talks. Unfortunately, the problem hasn't changed at all. Perhaps we could just replay those video streams :-) My $.02 - http://www.getit.org/wordpress/?p=82 The irony to one of those, is that in NANOG 25 right before my session which pointed out this continued threat vector, we had Protecting the BGP Routes to Top Level DNS Servers http://www.nanog.org/mtg-0206/bush.html. -BEGIN PGP SIGNATURE- Version: PGP 8.1 iQA/AwUBR8M5ib/UEA/xivvmEQISnwCgojEcUx7dyMBQPP59gZjdLgQeqqMAoL0p seCMm+llF8tkr+pGf7LlyXrW =6jfG -END PGP SIGNATURE-
Re: Secure BGP (Was: YouTube IP Hijacking)
Is there some way of deploying a solution like Secure BGP without actually requiring that it go into the routers? The IETF SIDR wg (shameless plug as I'm wg co-chair) is working on a way to say with strong assurance who holds what prefixes, and therefore who can authorize the origination of what prefixes. This could be used in creating filter lists, answering customer request (please announce this for me...), checking the RIB out-of-band, etc. Such info is also the foundation of any yet proposed mechanism for doing in-band bgp security (S-BGP, soBGP, psBGP, SPV, etc., etc.), but the sidr work by itself does not need to be done in the router. Maybe some of you could take a look and comment. Look for the drafts at http://www.ietf.org/html.charters/sidr-charter.html --Sandy
RE: BGP prefix filtering, how exactly? [Re: YouTube IP Hijacking]
Valdis wrote: He explicitly said single-homed. Of course, multi-homed requires different handling, because you may hear their other home announce them (although again, you probably shouldn't listen to *THAT* announcement either if *your* link to them is up). And I posit that if you don't know if your customer is single or multi-homed, you have *bigger* issues to deal with. My bad, I misread his multi-homed comment. From what I understand (and have seen in practice) PCCW does not listen to their address space from their peers no matter what the status of the connection to their customer is. I find this policy flat out flawed. Randy
Re: [admin] [summary] RE: YouTube IP Hijacking
On Mon, Feb 25, 2008, Alex Pilosov wrote: A bit of administrativia: This thread generated over a hundred posts, many without operational relevance or by people who do not understand how operators, well, operate, or by people who really don't have any idea what's going on but feel like posting. I'd like to briefly summarize the important things that were said. If you would like to add something to the thread, make sure you read this post in entirety. .. and, since I have 5 minutes to spare between disk array rebuilds, I bring you: http://nanog.cluepon.net/index.php/MailTopics For now it just links into Alex's summary post. I'll go trawl the archives now for more summaries. Adrian
Re: YouTube IP Hijacking
On Mon, Feb 25, 2008 at 2:32 AM, Hank Nussbacher [EMAIL PROTECTED] wrote: we've been warning that this could happen *again* - this is happening every day - just look to: http://cs.unm.edu/~karlinjf/IAR/prefix.php?filter=most http://cs.unm.edu/~karlinjf/IAR/subprefix.php?filter=most So, as someone that once subscribed to josh's alerts... for a large ISP, or an ISP with lots of customers using PA/PI space I got loads and loads of alerts about customer allocations popping out in other ASN's, it turns out those were almost always intended by the customer even... So I'm not sure that the above links are the best judge of this problem space. I agree though that the problem is there to some extent... much more often seen than the 1/2yr events we've noted on nanog-l over the last 8 years. -Chris
RE: YouTube IP Hijacking
Pakistan is deliberately blocking Youtube. http://politics.slashdot.org/article.pl?sid=08/02/24/1628213 Maybe we should all block Pakistan. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Will Hargrave Sent: Sunday, February 24, 2008 12:39 PM To: [EMAIL PROTECTED] Subject: Re: YouTube IP Hijacking Sargun Dhillon wrote: So, it seems that youtube's ip block has been hijacked by a more specific prefix being advertised. This is a case of IP hijacking, not case of DNS poisoning, youtube engineers doing something stupid, etc. For people that don't know. The router will try to get the most specific prefix. This is by design, not by accident. You are making the assumption of malice when the more likely cause is one of accident on the part of probably stressed NOC staff at 17557. They probably have that /24 going to a gateway walled garden box which replies with a site saying 'we have banned this', and that /24 route is leaking outside of their AS via PCCW due to dodgy filters/communities. Will
Re: YouTube IP Hijacking
Tomas L. Byrnes wrote: Clearly, they are incensed by youtube content, so what makes anyone think that they would not be trying to engage in a case of Cyber-Jihad? Because this usually doesn't work very well, is very evident, and easily fixed? Even on a sleepy Sunday, it took 3491 about two hours to filter/turn down 17557 and remove the problem. I bet most of their peers say that's too slow, however :-) I generally don't assume malice when mere incompetence will suffice, but in the case of the Islamic world, they've proved themselves malicious towards the non-Islamic world often, and violently, enough, that I don't believe they deserve that presumption of innocence any more. I think your perspective is a little off.
Re: YouTube IP Hijacking
On Sun, Feb 24, 2008 at 4:06 PM, Tomas L. Byrnes [EMAIL PROTECTED] wrote: Clearly, they are incensed by youtube content, so what makes anyone think that they would not be trying to engage in a case of Cyber-Jihad? Let's avoid speculation as to the why and reserve this thread for global restoration activity. -M
Re: YouTube IP Hijacking
While they are deliberately blocking Youtube nationally, I suspect the wider issue has no malice, and is a case of poorly constructed/ implemented outbound policies on their part, and poorly constructed/ implemented inbound polices on their upstreams part. On 25/02/2008, at 9:49 AM, Tomas L. Byrnes wrote: Pakistan is deliberately blocking Youtube. http://politics.slashdot.org/article.pl?sid=08/02/24/1628213 Maybe we should all block Pakistan. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Will Hargrave Sent: Sunday, February 24, 2008 12:39 PM To: [EMAIL PROTECTED] Subject: Re: YouTube IP Hijacking Sargun Dhillon wrote: So, it seems that youtube's ip block has been hijacked by a more specific prefix being advertised. This is a case of IP hijacking, not case of DNS poisoning, youtube engineers doing something stupid, etc. For people that don't know. The router will try to get the most specific prefix. This is by design, not by accident. You are making the assumption of malice when the more likely cause is one of accident on the part of probably stressed NOC staff at 17557. They probably have that /24 going to a gateway walled garden box which replies with a site saying 'we have banned this', and that /24 route is leaking outside of their AS via PCCW due to dodgy filters/communities. Will Neil Fenemor FX Networks
RE: YouTube IP Hijacking
Clearly, they are incensed by youtube content, so what makes anyone think that they would not be trying to engage in a case of Cyber-Jihad? I hosted the site that was rated #1 on Google for the Jyllands Posten (di2.nu) cartoons when it was a current issue, and I STILL get lots of script kiddie DOS from the Islamic world. I generally don't assume malice when mere incompetence will suffice, but in the case of the Islamic world, they've proved themselves malicious towards the non-Islamic world often, and violently, enough, that I don't believe they deserve that presumption of innocence any more. In either case, the correct COA is to filter all advertisements with AS 17557 in the path, until they fix the routes they are advertising, and let us know how they plan on making sure this doesn't happen again. -Original Message- From: Neil Fenemor [mailto:[EMAIL PROTECTED] Sent: Sunday, February 24, 2008 1:01 PM To: Tomas L. Byrnes Cc: Will Hargrave; nanog@merit.edu Subject: Re: YouTube IP Hijacking While they are deliberately blocking Youtube nationally, I suspect the wider issue has no malice, and is a case of poorly constructed/ implemented outbound policies on their part, and poorly constructed/ implemented inbound polices on their upstreams part. On 25/02/2008, at 9:49 AM, Tomas L. Byrnes wrote: Pakistan is deliberately blocking Youtube. http://politics.slashdot.org/article.pl?sid=08/02/24/1628213 Maybe we should all block Pakistan. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Will Hargrave Sent: Sunday, February 24, 2008 12:39 PM To: [EMAIL PROTECTED] Subject: Re: YouTube IP Hijacking Sargun Dhillon wrote: So, it seems that youtube's ip block has been hijacked by a more specific prefix being advertised. This is a case of IP hijacking, not case of DNS poisoning, youtube engineers doing something stupid, etc. For people that don't know. The router will try to get the most specific prefix. This is by design, not by accident. You are making the assumption of malice when the more likely cause is one of accident on the part of probably stressed NOC staff at 17557. They probably have that /24 going to a gateway walled garden box which replies with a site saying 'we have banned this', and that /24 route is leaking outside of their AS via PCCW due to dodgy filters/communities. Will Neil Fenemor FX Networks
RE: YouTube IP Hijacking
Looks like it just went back to normal: cr1-sea-Ashow ip bgp 208.65.153.253 BGP routing table entry for 208.65.153.0/24, version 41150187 Paths: (3 available, best #3) Flag: 0x8E0 Advertised to update-groups: 1 3 4 6 13 14 16 3356 3549 36561, (Received from a RR-client) 208.76.153.126 (metric 110) from 208.76.153.126 (208.76.153.126) Origin IGP, metric 0, localpref 50, valid, internal Community: 3356:3 3356:22 3356:86 3356:575 3356:666 3356:2011 3549:4142 3549:30840 11404:1000 11404:1030 2914 3549 36561, (Received from a RR-client) 208.76.153.125 (metric 310) from 208.76.153.125 (208.76.153.125) Origin IGP, metric 0, localpref 49, valid, internal Community: 2914:420 2914:2000 2914:3000 11404:1000 11404:1010 3491 3549 36561 63.216.14.137 from 63.216.14.137 (63.216.14.9) Origin IGP, localpref 51, valid, external, best Community: 3491:2000 3491:2003 3491:3549 11404:1000 11404:1020 cr1-sea-A Probably worth noting that the performace at least from our perspective (via PCCW) is abysmal.As a side note, I know PCCW allows unfiltered route-announcement capability to a large number of their customers, our feed appears to be that way (or they apply RADB filters instantly which would be a bit impressive). John van Oppen Spectrum Networks LLC 206.973.8302 (Direct) 206.973.8300 (main office) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tomas L. Byrnes Sent: Sunday, February 24, 2008 12:50 PM To: Will Hargrave; nanog@merit.edu Subject: RE: YouTube IP Hijacking Pakistan is deliberately blocking Youtube. http://politics.slashdot.org/article.pl?sid=08/02/24/1628213 Maybe we should all block Pakistan. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Will Hargrave Sent: Sunday, February 24, 2008 12:39 PM To: [EMAIL PROTECTED] Subject: Re: YouTube IP Hijacking Sargun Dhillon wrote: So, it seems that youtube's ip block has been hijacked by a more specific prefix being advertised. This is a case of IP hijacking, not case of DNS poisoning, youtube engineers doing something stupid, etc. For people that don't know. The router will try to get the most specific prefix. This is by design, not by accident. You are making the assumption of malice when the more likely cause is one of accident on the part of probably stressed NOC staff at 17557. They probably have that /24 going to a gateway walled garden box which replies with a site saying 'we have banned this', and that /24 route is leaking outside of their AS via PCCW due to dodgy filters/communities. Will
Re: YouTube IP Hijacking
On Sun Feb 24, 2008 at 04:32:45PM -0500, Martin Hannigan wrote: Let's avoid speculation as to the why and reserve this thread for global restoration activity. So, from the tit-bits I've picked up from IRC and first-hand knowledge, it would appear that 17557 leaked an announcement of 208.65.153.0/24 to 3491 (PCCW/BTN). After several calls to PCCW NOC, including from Youtube themselves, PCCW claimed to be shutting down the links to 17557. Initially I saw the announcement change from 3491 17557 to 3491 17557 17557, so I speculate that they shut down the primary link (or filtered the announcement on that link), and the prefix was still coming in over a secondary link (hence the prepend). After more prodding, that route vanished too. Various mitigations were talked about and tried, including Youtube announcing the /24 as 2*/25, but these announcements did not seem to make it out to the world at large. Currently Youtube are announcing the /24 themselves - I assume this will drop at some time once it's safe. It was noticed that all the youtube.com DNS servers were in the affected /24. Youtube have subsequently added a DNS server in another prefix. Simon -- Simon Lockhart | * Sun Server Colocation * ADSL * Domain Registration * Director|* Domain Web Hosting * Internet Consultancy * Bogons Ltd | * http://www.bogons.net/ * Email: [EMAIL PROTECTED] *
RE: YouTube IP Hijacking
Which means that, by advertising routes more specific than the ones they are poisoning, it may well be possible to restore universal connectivity to YouTube. -Original Message- From: Michael Smith [mailto:[EMAIL PROTECTED] Sent: Sunday, February 24, 2008 1:23 PM To: [EMAIL PROTECTED]; Tomas L. Byrnes Cc: [EMAIL PROTECTED]; nanog@merit.edu Subject: Re: YouTube IP Hijacking Exactly... They inadvertently made the details of their oppression more readily apparent... - Original Message - From: [EMAIL PROTECTED] [EMAIL PROTECTED] To: Tomas L. Byrnes [EMAIL PROTECTED] Cc: Will Hargrave [EMAIL PROTECTED]; nanog@merit.edu nanog@merit.edu Sent: Sun Feb 24 16:00:35 2008 Subject: Re: YouTube IP Hijacking While they are deliberately blocking Youtube nationally, I suspect the wider issue has no malice, and is a case of poorly constructed/ implemented outbound policies on their part, and poorly constructed/ implemented inbound polices on their upstreams part. On 25/02/2008, at 9:49 AM, Tomas L. Byrnes wrote: Pakistan is deliberately blocking Youtube. http://politics.slashdot.org/article.pl?sid=08/02/24/1628213 Maybe we should all block Pakistan. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Will Hargrave Sent: Sunday, February 24, 2008 12:39 PM To: [EMAIL PROTECTED] Subject: Re: YouTube IP Hijacking Sargun Dhillon wrote: So, it seems that youtube's ip block has been hijacked by a more specific prefix being advertised. This is a case of IP hijacking, not case of DNS poisoning, youtube engineers doing something stupid, etc. For people that don't know. The router will try to get the most specific prefix. This is by design, not by accident. You are making the assumption of malice when the more likely cause is one of accident on the part of probably stressed NOC staff at 17557. They probably have that /24 going to a gateway walled garden box which replies with a site saying 'we have banned this', and that /24 route is leaking outside of their AS via PCCW due to dodgy filters/communities. Will Neil Fenemor FX Networks
Re: YouTube IP Hijacking
Jake Blues mode I hate Cyber Jihads! /Jake Blues mode - Original Message - From: [EMAIL PROTECTED] [EMAIL PROTECTED] To: Neil Fenemor [EMAIL PROTECTED] Cc: Will Hargrave [EMAIL PROTECTED]; nanog@merit.edu nanog@merit.edu Sent: Sun Feb 24 16:06:50 2008 Subject: RE: YouTube IP Hijacking Clearly, they are incensed by youtube content, so what makes anyone think that they would not be trying to engage in a case of Cyber-Jihad? I hosted the site that was rated #1 on Google for the Jyllands Posten (di2.nu) cartoons when it was a current issue, and I STILL get lots of script kiddie DOS from the Islamic world. I generally don't assume malice when mere incompetence will suffice, but in the case of the Islamic world, they've proved themselves malicious towards the non-Islamic world often, and violently, enough, that I don't believe they deserve that presumption of innocence any more. In either case, the correct COA is to filter all advertisements with AS 17557 in the path, until they fix the routes they are advertising, and let us know how they plan on making sure this doesn't happen again. -Original Message- From: Neil Fenemor [mailto:[EMAIL PROTECTED] Sent: Sunday, February 24, 2008 1:01 PM To: Tomas L. Byrnes Cc: Will Hargrave; nanog@merit.edu Subject: Re: YouTube IP Hijacking While they are deliberately blocking Youtube nationally, I suspect the wider issue has no malice, and is a case of poorly constructed/ implemented outbound policies on their part, and poorly constructed/ implemented inbound polices on their upstreams part. On 25/02/2008, at 9:49 AM, Tomas L. Byrnes wrote: Pakistan is deliberately blocking Youtube. http://politics.slashdot.org/article.pl?sid=08/02/24/1628213 Maybe we should all block Pakistan. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Will Hargrave Sent: Sunday, February 24, 2008 12:39 PM To: [EMAIL PROTECTED] Subject: Re: YouTube IP Hijacking Sargun Dhillon wrote: So, it seems that youtube's ip block has been hijacked by a more specific prefix being advertised. This is a case of IP hijacking, not case of DNS poisoning, youtube engineers doing something stupid, etc. For people that don't know. The router will try to get the most specific prefix. This is by design, not by accident. You are making the assumption of malice when the more likely cause is one of accident on the part of probably stressed NOC staff at 17557. They probably have that /24 going to a gateway walled garden box which replies with a site saying 'we have banned this', and that /24 route is leaking outside of their AS via PCCW due to dodgy filters/communities. Will Neil Fenemor FX Networks
Re: YouTube IP Hijacking
I think it was NOT a typo. This was a test, much more important test for this world than last american anti-satellite missile. And if they do it again with more mind, site will became down for a weeks at least... More of that, if big national telecom operator did it and have neighbors to filter them out - it can lead to global split of the network. Of course, it should be happened early or late with THIS design of the Network. Ravi Pina wrote: Sounds more like a typo on a filter over at AS17557 than anything else. http://ca.news.yahoo.com/s/afp/080224/world/denmark_media_islam_pakistan_internet_youtube -r On Sun, Feb 24, 2008 at 12:27:29PM -0800, Sargun Dhillon wrote: As you guys probably know Youtube's IP's are being hijacked. Trace: ~ $ host youtube.com youtube.com has address 208.65.153.253 youtube.com has address 208.65.153.238 youtube.com has address 208.65.153.251 [Same /24] 701 3491 17557 64.74.137.253 (metric 1) from 66.151.144.148 (66.151.144.148) Origin IGP, metric 100, localpref 100, valid, external Community: 65010:300 Last update: Sun Feb 24 11:33:05 2008 [PST8PDT] 3491 17557 216.218.135.205 from 216.218.135.205 (216.218.252.164) Origin IGP, metric 100, localpref 100, valid, external, best Last update: Sun Feb 24 10:47:57 2008 [PST8PDT] So, it seems that youtube's ip block has been hijacked by a more specific prefix being advertised. This is a case of IP hijacking, not case of DNS poisoning, youtube engineers doing something stupid, etc. For people that don't know. The router will try to get the most specific prefix. This is by design, not by accident. This is a case of censorship on the internet. Anyways, I hope this doesn't get into a political situation, and someone stops this. What action are you going to take? Are you going to filter announcements from AS17557, or just filter that specific announcement? Considering youtube is a fairly high-traffic website I think that other operators are just going to start filtering that AS. This is a great example of global politics getting in the way of honest corporatism. This is also an example of how vulnerable the internet is, and how lax providers are in their filtering policies. I don't know how large Pakistani Telecom is, but it I bet its not large enough that PCCW should be allowing it to advertise anything. -- WBR, Max Tulyev (MT6561-RIPE, 2:463/[EMAIL PROTECTED])
RE: YouTube IP Hijacking
-- Tomas L. Byrnes [EMAIL PROTECTED] wrote: It seems to me that a more immediately germane matter regarding BGP route propagation is prevention of hijacking of critical routes. The best you can _probably_ hope for is a opt-in mechanism in which you are alerted that prefixes you have registered with the aforementioned system are being originated by an ASN which is not authorized to originate them. A lot of smart folks have given some thought to this exact issue, and perhaps one of the best examples of this is: PHAS: A Prefix Hijack Alert System Mohit Lad, Dan Massey, Dan Pei, Yiguo Wu, Beichuan Zhang, and Lixia Zhang Proceedings of 15th USENIX Security Symposium 2006 http://www.cs.ucla.edu/~mohit/cameraReady/ladSecurity06.pdf - ferg -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
RE: YouTube IP Hijacking
I figured as much, but it was worth a try. Which touches on the earlier discussion of the null routing of /32s advertised by a special AS (as a means of black-holing DDOS traffic). It seems to me that a more immediately germane matter regarding BGP route propagation is prevention of hijacking of critical routes. Perhaps certain ASes that are considered high priority, like Google, YouTube, Yahoo, MS (at least their update servers), can be trusted to propagate routes that are not aggregated/filtered, so as to give them control over their reachability and immunity to longer-prefix hijacking (especially problematic with things like MS update sites). -Original Message- From: Simon Lockhart [mailto:[EMAIL PROTECTED] Sent: Sunday, February 24, 2008 2:07 PM To: Tomas L. Byrnes Cc: Michael Smith; [EMAIL PROTECTED]; [EMAIL PROTECTED]; nanog@merit.edu Subject: Re: YouTube IP Hijacking On Sun Feb 24, 2008 at 01:49:00PM -0800, Tomas L. Byrnes wrote: Which means that, by advertising routes more specific than the ones they are poisoning, it may well be possible to restore universal connectivity to YouTube. Well, if you can get them in there Youtube tried that, to restore service to the rest of the world, and the announcements didn't propogate. Simon
Re: YouTube IP Hijacking
Which means that, by advertising routes more specific than the ones they are poisoning, it may well be possible to restore universal connectivity to YouTube. Well, if you can get them in there Youtube tried that, to restore service to the rest of the world, and the announcements didn't propogate. Some of us block prefixes longer than /24 at our borders (even if our transit providers don't). Steinar Haug, Nethelp consulting, [EMAIL PROTECTED]
RE: YouTube IP Hijacking
Not if the hijackers have advertised a /24. Anything you advertise more specific than /24 will be lost on many networks' filters. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tomas L. Byrnes Sent: Monday, 25 February 2008 8:49 AM To: Michael Smith; [EMAIL PROTECTED] Cc: [EMAIL PROTECTED]; nanog@merit.edu Subject: RE: YouTube IP Hijacking Which means that, by advertising routes more specific than the ones they are poisoning, it may well be possible to restore universal connectivity to YouTube. -Original Message- From: Michael Smith [mailto:[EMAIL PROTECTED] Sent: Sunday, February 24, 2008 1:23 PM To: [EMAIL PROTECTED]; Tomas L. Byrnes Cc: [EMAIL PROTECTED]; nanog@merit.edu Subject: Re: YouTube IP Hijacking Exactly... They inadvertently made the details of their oppression more readily apparent... - Original Message - From: [EMAIL PROTECTED] [EMAIL PROTECTED] To: Tomas L. Byrnes [EMAIL PROTECTED] Cc: Will Hargrave [EMAIL PROTECTED]; nanog@merit.edu nanog@merit.edu Sent: Sun Feb 24 16:00:35 2008 Subject: Re: YouTube IP Hijacking While they are deliberately blocking Youtube nationally, I suspect the wider issue has no malice, and is a case of poorly constructed/ implemented outbound policies on their part, and poorly constructed/ implemented inbound polices on their upstreams part. On 25/02/2008, at 9:49 AM, Tomas L. Byrnes wrote: Pakistan is deliberately blocking Youtube. http://politics.slashdot.org/article.pl?sid=08/02/24/1628213 Maybe we should all block Pakistan. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Will Hargrave Sent: Sunday, February 24, 2008 12:39 PM To: [EMAIL PROTECTED] Subject: Re: YouTube IP Hijacking Sargun Dhillon wrote: So, it seems that youtube's ip block has been hijacked by a more specific prefix being advertised. This is a case of IP hijacking, not case of DNS poisoning, youtube engineers doing something stupid, etc. For people that don't know. The router will try to get the most specific prefix. This is by design, not by accident. You are making the assumption of malice when the more likely cause is one of accident on the part of probably stressed NOC staff at 17557. They probably have that /24 going to a gateway walled garden box which replies with a site saying 'we have banned this', and that /24 route is leaking outside of their AS via PCCW due to dodgy filters/communities. Will Neil Fenemor FX Networks
Re: YouTube IP Hijacking
On Sun Feb 24, 2008 at 01:49:00PM -0800, Tomas L. Byrnes wrote: Which means that, by advertising routes more specific than the ones they are poisoning, it may well be possible to restore universal connectivity to YouTube. Well, if you can get them in there Youtube tried that, to restore service to the rest of the world, and the announcements didn't propogate. Simon
ISP's who where affected by the misconfiguration: start using IRR and checking your BGP updates (Was: YouTube IP Hijacking)
First the operational portion: For all the affected network owners, please read and start using/implement one of the following excellent ideas: * Pretty Good BGP and the Internet Alert Registry http://www.nanog.org/mtg-0606/pdf/josh-karlin.pdf * PHAS: A Prefix Hijack Alert System http://irl.cs.ucla.edu/papers/originChange.pdf (A live/direct BGP-feed version of this would be neat) * Routing Registry checking, as per the above two rr.arin.net whois.ripe.net contains all the data you need Networks who are not in there are simply not important enough to exist on the internet as clearly those ops folks don't care about their network... Of course there is also (S-)BGP(-S), but that will apparently never happen, and actually, with the a system like PGBGP or PHAS one already covers quite a bit of the issue, until a real hijacker just uses the original ASN. IRR data helps there partially though as it tends to have upstream/downstream information, but it doesn't cover all cases. For the rest google(bgp monitor hijack) for a list of other things. Now for the sillynesss non-ops political blabla FUD Max Tulyev wrote: I think it was NOT a typo. This was a test, much more important test for this world than last american anti-satellite missile. And if they do it again with more mind, site will became down for a weeks at least... More of that, if big national telecom operator did it and have neighbors to filter them out - it can lead to global split of the network. Of course, it should be happened early or late with THIS design of the Network. Oh boy oh boy, I just have to comment on this :) Wow, somebody with an email address like yours, especially the president and the .su bit are amusing, is commenting on another country doing 'tests'!? You might actually try keeping your bombers closer to the shores instead of trying to play chicken with the USS Nimitz :) http://www.upi.com/NewsTrack/Top_News/2008/02/11/russian_bomber_buzzes_nimitz/5914/ In Soviet Russia the Internet hijacks you? Please folks, keep the posts operational :) /non-ops political blabla FUD Greets, Jeroen signature.asc Description: OpenPGP digital signature
Re: ISP's who where affected by the misconfiguration: start using IRR and checking your BGP updates (Was: YouTube IP Hijacking)
On Sun, 24 Feb 2008, Jeroen Massar wrote: * Routing Registry checking, as per the above two rr.arin.net whois.ripe.net contains all the data you need Networks who are not in there are simply not important enough to exist on the internet as clearly those ops folks don't care about their network... For us who actually have customers we care about, we probably find it better for business to try to make sure our own customers can't announce prefixes they don't own, but accept basically anything from the world that isn't ours. Using pure RR based filtering just isn't cost efficient today, as these borks (unintentional mostly) we see sometimes are few and fairly far between, but problems due to wrong or missing information in the RRs is plentyful and constant. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: YouTube IP Hijacking
http://www.google.com/reader/m/view/?source=mobilepackv=2.1.4rlz=1H2GGLE_eni=-3701578819353178822c=CMOjuszq3ZECn=1 On 2/24/08, Max Tulyev [EMAIL PROTECTED] wrote: I think it was NOT a typo. This was a test, much more important test for this world than last american anti-satellite missile. And if they do it again with more mind, site will became down for a weeks at least... More of that, if big national telecom operator did it and have neighbors to filter them out - it can lead to global split of the network. Of course, it should be happened early or late with THIS design of the Network. Ravi Pina wrote: Sounds more like a typo on a filter over at AS17557 than anything else. http://ca.news.yahoo.com/s/afp/080224/world/denmark_media_islam_pakistan_internet_youtube -r On Sun, Feb 24, 2008 at 12:27:29PM -0800, Sargun Dhillon wrote: As you guys probably know Youtube's IP's are being hijacked. Trace: ~ $ host youtube.com youtube.com has address 208.65.153.253 youtube.com has address 208.65.153.238 youtube.com has address 208.65.153.251 [Same /24] 701 3491 17557 64.74.137.253 (metric 1) from 66.151.144.148 (66.151.144.148) Origin IGP, metric 100, localpref 100, valid, external Community: 65010:300 Last update: Sun Feb 24 11:33:05 2008 [PST8PDT] 3491 17557 216.218.135.205 from 216.218.135.205 (216.218.252.164) Origin IGP, metric 100, localpref 100, valid, external, best Last update: Sun Feb 24 10:47:57 2008 [PST8PDT] So, it seems that youtube's ip block has been hijacked by a more specific prefix being advertised. This is a case of IP hijacking, not case of DNS poisoning, youtube engineers doing something stupid, etc. For people that don't know. The router will try to get the most specific prefix. This is by design, not by accident. This is a case of censorship on the internet. Anyways, I hope this doesn't get into a political situation, and someone stops this. What action are you going to take? Are you going to filter announcements from AS17557, or just filter that specific announcement? Considering youtube is a fairly high-traffic website I think that other operators are just going to start filtering that AS. This is a great example of global politics getting in the way of honest corporatism. This is also an example of how vulnerable the internet is, and how lax providers are in their filtering policies. I don't know how large Pakistani Telecom is, but it I bet its not large enough that PCCW should be allowing it to advertise anything. -- WBR, Max Tulyev (MT6561-RIPE, 2:463/[EMAIL PROTECTED])
