Re: oof. panix sidelined by incompetence... again.
in case some people want to look at routeviews data for themselves, I have archived a couple of pdf file at: http://bgpinspect.merit.edu/reports.php -manish - Re: oof. panix sidelined by incompetence... again. * From: william(at)elan.net * Date: Sun Jan 22 13:34:47 2006 Can there be a confirmation of this? I see no such MOTD at http://www.panix.com/panix/help/Announcements/ and my connection to panix is fine and route I see is 166.84.0.0/17 with origin in 2033. I also checked at routeviews.org and similarly all their peers see origin in in 2033. Is there some other route that has been hijacked then or has it now ben resolved? BTW - Its interesting to note that its almost exactly one year after their domain hijacking which happened on weekend of Jan 15 16, 2005 (friday jan 14th to be more precise). -
Re: oof. panix sidelined by incompetence... again. (fwd)
I forgot to mention that you can get the RIPE data perspective on this as well by running a prefix-exact query for 166.84.0.0/16 at bgp-inspect-RIPE: http://bgpinspect.merit.edu:9090 If there are enough requests I will probably archive those as well. thanks -manish -- Forwarded message -- Date: Mon, 23 Jan 2006 11:25:58 -0500 (EST) From: Manish Karir [EMAIL PROTECTED] To: NANOG nanog@merit.edu Subject: Re: oof. panix sidelined by incompetence... again. in case some people want to look at routeviews data for themselves, I have archived a couple of pdf file at: http://bgpinspect.merit.edu/reports.php -manish - Re: oof. panix sidelined by incompetence... again. * From: william(at)elan.net * Date: Sun Jan 22 13:34:47 2006 Can there be a confirmation of this? I see no such MOTD at http://www.panix.com/panix/help/Announcements/ and my connection to panix is fine and route I see is 166.84.0.0/17 with origin in 2033. I also checked at routeviews.org and similarly all their peers see origin in in 2033. Is there some other route that has been hijacked then or has it now ben resolved? BTW - Its interesting to note that its almost exactly one year after their domain hijacking which happened on weekend of Jan 15 16, 2005 (friday jan 14th to be more precise). -
Re: oof. panix sidelined by incompetence... again.
You can easily repeat the queries on the bgpinspect website to generate the same results in html files. I just bundled them up into a single pdf for convenience. thanks manish On Mon, 23 Jan 2006, Fergie wrote: Out of curiousity, why must these be in .pdf format? I mean, what's wrong with .html? Just curious, - ferg
Re: oof. panix sidelined by incompetence... again.
Can there be a confirmation of this? I see no such MOTD at http://www.panix.com/panix/help/Announcements/ and my connection to panix is fine and route I see is 166.84.0.0/17 with origin in 2033. I also checked at routeviews.org and similarly all their peers see origin in in 2033. Is there some other route that has been hijacked then or has it now ben resolved? BTW - Its interesting to note that its almost exactly one year after their domain hijacking which happened on weekend of Jan 15 16, 2005 (friday jan 14th to be more precise). On Sun, 22 Jan 2006, Thor Lancelot Simon wrote: This is hardly as serious as the last incident -- but, well, some people do seem to have all the luck, eh? Of course, there are measures one can take against this sort of thing; but it's hard to deploy some of them effectively when the party stealing your routes was in fact once authorized to offer them, and its own peers may be explicitly allowing them in filter lists (which, I think, is the case here). Sometimes budget network connectivity isn't -- even when you've already realized that and turned off the tap! The text below is what's currently in the MOTD on Panix's NetBSD hosts: == Con Ed 'stealing' Panix routes (alexis) Sun Jan 22 12:38:16 2006 All Panix services are currently unreachable from large portions of the Internet (though not all of it). This is because Con Ed Communications, a competence-challenged ISP in New York, is announcing our routes to the Internet. In English, that means that they are claiming that all our traffic should be passing through them, when of course it should not. Those portions of the net that are closer (in network topology terms) to Con Ed will send them our traffic, which makes us unreachable. We are taking several steps to deal with this: 1) We are announcing more specific routes to our peers. More specific routes are always preferred. However, we have to contact network admins at those peers to get them to change their route filters, before this workaround will be effective. 2) We are attempting to reach Con Ed Communications. Unfortunately, so far we've been unable to do so. They don't seem to answer their phones on Sunday. 3) We are attempting to reach Verio, which is upstream from Con Ed, because they could (and should!!) choose to ignore the rogue routes from Con Ed. Since all of these depend on humans outside of Panix, we can't give a specific time at which we expect this problem to be worked around (I don't expect a real resolution for a while, because Con Ed is hopeless, but the workaround will be perfect until then). But we do expect to be able to reach responsible parties at our peers within a few hours at most. We don't know how long it will take for them to change their filters, but that's not a challenging job technically, so we hope it won't take long. I'll post another MOTD as soon as we know anything more.
Re: oof. panix sidelined by incompetence... again.
On Sun, Jan 22, 2006 at 10:33:04AM -0800, william(at)elan.net wrote: Can there be a confirmation of this? I see no such MOTD at http://www.panix.com/panix/help/Announcements/ Verio was just extremely helpful and filtered out the bogus Panix routes ConED was sending them quite rapidly upon request from Panix's staff. AFAICT ConED is still sending the bogus routes, and since they evidently don't believe in staffing their NOC on the weekend, or responding to reports of their own misconduct, heaven only knows if they'll ever stop. Thanks to Verio's quick intervention the problem, thank goodness, seems to be solved. The current Panix MOTD is below: Connectivity restored (alexis) Sun Jan 22 13:31:28 2006 At around 1:10PM, all of the Internet can now reach Panix again. We accomplished this by getting our peers to accept more-specific routes from us. We also, nearly simultaneously, got Con Ed's rogue route announcements pulled by Verio, their upstream. I'm surprised and pleased that Verio, which we don't have a business relationship with, was so easy to contact and so ready to do what they should. No mail was lost during this outage. Some was delayed, of course, and everything should be caught up again in an hour or two. Please let us know if you have network problems *after* 1:10PM EST.
Re: oof. panix sidelined by incompetence... again.
On Sun, Jan 22, 2006 at 10:33:04AM -0800, william(at)elan.net wrote: Can there be a confirmation of this? I see no such MOTD at http://www.panix.com/panix/help/Announcements/ I don't know how realtime that is ... but Panix (including their web site) was unreachable from several points earlier. It's back now. and my connection to panix is fine and route I see is 166.84.0.0/17 with origin in 2033. I also checked at routeviews.org and similarly all their peers see origin in in 2033. Is there some other route that has been hijacked then or has it now ben resolved? As noted in the MOTD posting, Panix is announcing more specifics. 166.84/16 is what they would normally accounce (and they are still announcing that), 166.84.0/17 and 166.84.128/17 are there to overcome the /16 Con Ed is advertising. As of the now (according to Panix; I haven't independantly verified it), Verio is (at Panix's request) rejecting the route from ConEd, and Panix's upstreams are accepting the /17s, so connectivity should be OK from everywhere except possibly ConEd. -- Brett
Re: oof. panix sidelined by incompetence... again.
Folx, On Sun, Jan 22, 2006 at 06:09:08PM +, Thor Lancelot Simon wrote: This is hardly as serious as the last incident -- but, well, some people do seem to have all the luck, eh? From where I'm standing this situation looks much more serious than the last one. It looks like Con Edison (AS27506) hijacked several prefixes other than just Panix's, and I'm not sure that they're done announcing them yet. I see ~70 new prefixes, many of whom are customers of Con Edison, but about 25 of these appear to have no previous relationship to 27506). I won't bore people with to many details here (unless there is great interest) but a quick, rough-n-ready analysis is up at http://www.renesys.com/blog/2006/01/coned_steals_the_net.shtml for those that are curious. (For those that don't want to read it, I think one main punchline is that Con Edison spewed a bunch of routes they didn't own and UUNet and Verio believed them). Please let me know if you see errors in there. t. -- _ todd underwood chief of operations security renesys - internet intelligence [EMAIL PROTECTED] www.renesys.com
Re: oof. panix sidelined by incompetence... again.
As of the now (according to Panix; I haven't independantly verified it), Verio is (at Panix's request) rejecting the route from ConEd, and Panix's upstreams are accepting the /17s, so connectivity should be OK from everywhere except possibly ConEd. are the following two statements true? o verio did have irr filter applied o con ed seems to have registered others' prefixes in irr randy
