Re: Fw: GLBX ICMP rate limiting (was RE: Tier-1 without their ownbackbone?)
On Fri, 29 Aug 2003, Sean Donelan wrote: On Fri, 29 Aug 2003, Christopher L. Morrow wrote: That was a ccourt order, not much any US based corporation can do about that, eh? Oh, yeah, and it didn't help stop any child pornographers, all it did was hide their tracks from the authorities :( I suspect most ISPs in the US will follow lawful orders issued by authorities with jurisdiction. Some may try to also point out how stupid or ineffective those orders are. Yes, this is true, and atleast for the cited PA article that was the case for ALOT of the affected ISP's. (the pointing out of a poor choice of solutions) In the last month there have been several worms, viruses and activites by law enforcement and other authorities related to those. I think some folks are confusing the various different requests, orders, subpoenaes, etc. This is also true, and often the front-line technical service folks are told: We were told to do this by the gum'ent, that's our story and we're stickin' to it! Which often gets abbreviated to: Yeah, we were ordered by the stormtroopers to do this, sorry! :( I have no idea if UUNET cooperated with the FBI, NICP, DHS or other AHJ concerning any of the worms or viruses over the last month. Our lawyers tell me we always cooperate when asked with a court order...
Re: Fw: GLBX ICMP rate limiting (was RE: Tier-1 without their ownbackbone?)
On Thu, 28 Aug 2003, Christopher L. Morrow wrote: perhaps a change in vendors is in order? I can't see why people would lie about this, or why they'd listen to the 'request' from DHS in the first place ;( Oh well. http://www.wired.com/news/technology/0,1282,57804,00.html Mike Fisher, Pennsylvania's attorney general, has sent letters to an unknown number of ISPs over the past few months demanding that the ISPs block Pennsylvania subscribers' access to at least 423 websites or face a $5,000 fine, according to news reports. [..] How the blocks will affect law enforcement across North America would depend on which ISP their departments are using, among other factors. But Morris pointed out that WorldCom was ordered by a judge to comply with the Pennsylvania law last September. WorldCom owns UUNet, and the U.S. government is one of UUNet's biggest customers.
Re: Fw: GLBX ICMP rate limiting (was RE: Tier-1 without their ownbackbone?)
On Thu, 28 Aug 2003, Sean Donelan wrote: On Thu, 28 Aug 2003, Christopher L. Morrow wrote: perhaps a change in vendors is in order? I can't see why people would lie about this, or why they'd listen to the 'request' from DHS in the first place ;( Oh well. http://www.wired.com/news/technology/0,1282,57804,00.html Mike Fisher, Pennsylvania's attorney general, has sent letters to an unknown number of ISPs over the past few months demanding that the ISPs block Pennsylvania subscribers' access to at least 423 websites or face a $5,000 fine, according to news reports. this is a very old article... [..] How the blocks will affect law enforcement across North America would depend on which ISP their departments are using, among other factors. But Morris pointed out that WorldCom was ordered by a judge to comply with the Pennsylvania law last September. WorldCom owns UUNet, and the U.S. government is one of UUNet's biggest customers. That was a ccourt order, not much any US based corporation can do about that, eh? Oh, yeah, and it didn't help stop any child pornographers, all it did was hide their tracks from the authorities :(
Re: GLBX ICMP rate limiting (was RE: Tier-1 without their ownbackbone?)
NAC is not a global intercontinental super-duper backbone, but we do the same. It takes some education to the customers, but after they understand why, most are receptive. Especially when they get DOS'ed. On Thu, 28 Aug 2003 [EMAIL PROTECTED] wrote: On Wed, 27 Aug 2003, [EMAIL PROTECTED] wrote: We have a similarly sized connection to MFN/AboveNet, which I won't recommend at this time due to some very questionable null routing they're doing (propogating routes to destinations, then bitbucketing traffic sent to them) which is causing complaints from some of our customers and forcing us to make routing adjustments as the customers notice MFN/AboveNet has broken our connectivity to these destinations. We've noticed that one of our upstreams (Global Crossing) has introduced ICMP rate limiting 4/5 days ago. This means that any traceroutes/pings through them look awful (up to 60% apparent packet loss). After contacting their NOC, they said that the directive to install the ICMP rate limiting was from the Homeland Security folks and that they would not remove them or change the rate at which they limit in the foreseeable future. What are other transit providers doing about this or is it just GLBX? Cheers, Rich
Re: Fw: GLBX ICMP rate limiting (was RE: Tier-1 without their ownbackbone?)
On Thu, 28 Aug 2003, Gordon wrote: Of the DDOS attacks I have had to deal with in the past year I have seen none which were icmp based. As attacks evolve and transform are we really to believe that rate limiting icmp will have some value in the attacks of tomorrow? The folks doing the attacking aren't 100% stupid... If their tcp flooder fails they will attempt udp then icmp or some other serial list of flooding tools. A large number of the 'bot' programs today have multiple flooding tools on them, so attempt proto X, if !success then attempt proto Y and so on :( Rate-limiting ICMP is 'ok' if you, as the provider, think its worthwhile and you, as the provider, want to deal with the headache phone calls... It might not stop everything, but in reality nothing really can :( If someone really wants your site/system/server off the network its as good as gone. -Chris
Re: Fw: GLBX ICMP rate limiting (was RE: Tier-1 without their ownbackbone?)
On Thu, 28 Aug 2003, Christopher L. Morrow wrote: Rate-limiting ICMP is 'ok' if you, as the provider, think its worthwhile and you, as the provider, want to deal with the headache phone calls... Would it be fair to say that UUNET haven't been asked by Homeland Security to do the rate limiting that GLBX claim they have been asked to do? Has anyone else been asked to rate limit by the U.S. Department of Homeland Security? Rich
Re: GLBX ICMP rate limiting (was RE: Tier-1 without their ownbackbone?)
On Thu, 28 Aug 2003, Wayne E. Bouchard wrote: While rate limiting ICMP can be a good thing, it has to be done carefully and probably can't be uniform across the backbone. (think of a common site that gets pinged whenever someone wants to test to see if their connection went down or if it's just loaded.. Limit ICMP into them impropperly and lots of folks notice.) Such limiting also has to undergo periodic tuning as traffic levels increase, traffic patterns shift, and so forth. Along these lines, how does this limiting affect akamai or other 'ping for distance' type localization services? I'd think their data would get somewhat skewed, right?
Re: Fw: GLBX ICMP rate limiting (was RE: Tier-1 without their ownbackbone?)
On Thu, 28 Aug 2003, [EMAIL PROTECTED] wrote: On Thu, 28 Aug 2003, Christopher L. Morrow wrote: Rate-limiting ICMP is 'ok' if you, as the provider, think its worthwhile and you, as the provider, want to deal with the headache phone calls... Would it be fair to say that UUNET haven't been asked by Homeland Security to do the rate limiting that GLBX claim they have been asked to do? Has That is not fair at all :) DHS asked 'all ISPs' to filter 'all relevant traffic' for this latest set of MS worm events. Some ISPs did the filtering in part or in whole, others didn't... I would think that any ISP should have made the decision to take action not based on DHS's decree, but on the requirements of their network. So, if the ISP's network was adversely impacted by this even, or any other, they should take the action that is appropriate for their situation. That action might be to filter some or all of the items in DHS's decree, it might be to drop prefixes on the floor or turn down customers, or a whole host of other options. Doing things for the govt 'because they asked nicely' is not really the best of plans, certianly they don't know the mechanics of your network, mine, GBLX's, CW's or anyone elses... they should not dictate a solution. They really should work with their industry reps to 'get the word out' about a problem and 'make people aware' that there could be a crisis. Dictating solutions to 'problems' that might not exist is hardly a way to get people to help you out in your cause :) Oh, and why didn't they beat on the original software vendor about this?? Ok, no more rant for me :) anyone else been asked to rate limit by the U.S. Department of Homeland Security? Just about everyone with a large enough US office was asked by DHS, in a public statement...
Re: GLBX ICMP rate limiting (was RE: Tier-1 without their ownbackbone?)
On Thu, 2003-08-28 at 17:37, Steve Carter wrote: I speak for Global Crossing when I say that ICMP rate limiting has existed on the Global Crossing network, inbound from peers, for a long time ... we learned our lesson from the Yahoo DDoS attack (when they were one of our customers) back in the day and it was shortly thereafter that we implemented the rate limiters. Over the past 24 hours we've performed some experimentation that shows outbound rate limiters being also of value and we're looking at the specifics of differentiating between happy ICMP and naughty 92 byte packet ICMP and treating the latter with very strict rules ... like we would dump it on the floor. This, I believe, will stomp on the bad traffic but allow the happy traffic to pass unmolested. I think I can safely say that GBLX is beyond looking at the specifics of dropping 92-byte ICMP's, and are in fact doing it. And have not really bothered telling their customers about it either. We happen to use GBLX as one of our upstreams, and have a GigE pipe towards them. Since MS in their infinite wisdom seem to use 92-byte ICMP Echos in the Windows tracert.exe without having any option to use another protocol and/or packetsize, this certainly has generated several calls to OUR support desk today, by customers of ours claiming your routing is broken, traceroutes aren't getting anywhere!. Although I obviously understand the reasons, it WOULD be nice if if a supplier would at least take the trouble to inform us when they start applying filters to customer traffic, so our helpdesk would be prepared to answer questions about it. We are not a peer, but a paying customer after all. Oh, and it is not rate-limiting causing this, it is most definitely 92-byte filters. traceroute -P icmp www.gblx.net 92 from a decent OS will drop, any other packetsize works like a charm. /leg
Re: Fw: GLBX ICMP rate limiting (was RE: Tier-1 without their ownbackbone?)
On Thu, 28 Aug 2003, Christopher L. Morrow wrote: Rate-limiting ICMP is 'ok' if you, as the provider, think its worthwhile and you, as the provider, want to deal with the headache phone calls... Would it be fair to say that UUNET haven't been asked by Homeland Security to do the rate limiting that GLBX claim they have been asked to do? Has anyone else been asked to rate limit by the U.S. Department of Homeland Security? I have a different question, mostly directed to the likes of ATT and GlobalCrossing that came out with this fabulous explanation - (1) Did you get an order from DHS to do that or were you just asked? (2) How did DHS managed to not know about such order? (3) Are you going to bend over and do everything DHS politely asks you to do? Thanks, Alex
Re: Fw: GLBX ICMP rate limiting (was RE: Tier-1 without their ownbackbone?)
anyone else been asked to rate limit by the U.S. Department of Homeland Security? Just about everyone with a large enough US office was asked by DHS, in a public statement... Isnt there a difference between we have been asked and we have been ordered to? Alex
Re: Fw: GLBX ICMP rate limiting (was RE: Tier-1 without their ownbackbone?)
On Thu, 28 Aug 2003 [EMAIL PROTECTED] wrote: anyone else been asked to rate limit by the U.S. Department of Homeland Security? Just about everyone with a large enough US office was asked by DHS, in a public statement... Isnt there a difference between we have been asked and we have been ordered to? I suppose there is, but DHS's request (order/asking whatever) was NOT in the form of a court order... its: http://www.dhs.gov/dhspublic/verify_redirect.jsp?url=http%3A%2F%2Fwww.dhs.gov%2Fdhspublic%2Finterweb%2Fassetlibrary%2FAdvisory_Attack_MS.PDFtitle=Advisory+-+Potential+Internet+Attack+Targeting+Microsoft+Beginning+August+16%2C+2003+-+August+14%2C+2003 (ouch, how about: http://tinyurl.com/li0i ) and/or http://tinyurl.com/li0s Neither is really an 'order' so much as a 'suggestion'.. either way, its kind of inappropriate to make this suggestion without knowing how each operator can or could apply a fix... that is my opinion atleast.
Re: Fw: GLBX ICMP rate limiting (was RE: Tier-1 without their ownbackbone?)
http://tinyurl.com/li0s Neither is really an 'order' so much as a 'suggestion'.. either way, its kind of inappropriate to make this suggestion without knowing how each operator can or could apply a fix... that is my opinion atleast. The thing is - DHS told us so is the new favourite excuse for operators to refuse to fix anything that is/or could be broken. Over last two weeks I have heard the We have implemented the DHS order as the excuse from - Transport company whose gige transport went from 5ms to 700ms rtt. - Enterprise IP provider who filtered everything but ICMP/TCP/UDP while offering multicast services. - Two different IP backbones as the explanation of ICMP echo-requests being dropped (the issue was that in reality they were selling multiple 100Mbit/sec connections from 155 link). Of course, the moment one hears the DHS told us line, nothing else can be done. Alex
Re: Fw: GLBX ICMP rate limiting (was RE: Tier-1 without their ownbackbone?)
On Thu, 28 Aug 2003 [EMAIL PROTECTED] wrote: http://tinyurl.com/li0s Neither is really an 'order' so much as a 'suggestion'.. either way, its kind of inappropriate to make this suggestion without knowing how each operator can or could apply a fix... that is my opinion atleast. The thing is - DHS told us so is the new favourite excuse for operators to refuse to fix anything that is/or could be broken. Over last two weeks I have heard the We have implemented the DHS order as the excuse from -- snip excuses -- Of course, the moment one hears the DHS told us line, nothing else can be done. perhaps a change in vendors is in order? I can't see why people would lie about this, or why they'd listen to the 'request' from DHS in the first place ;( Oh well.
