Re: DoS Attacks
Can SLA's be used to cover this sort of thing. (starts to dig out his own contracts). Surely you should be able to bounce it to your upstream provider who should deal with it for you?? Just a thought. -- Martin Hepworth Senior Systems Administrator Solid State Logic Ltd tel: +44 (0)1865 842300 ** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com **
Re: DoS Attacks
On Tue, 7 Oct 2003, Brian Bruns wrote: So, now for the fun part. Being offsite, I wasn't the one to place the calls, but my admin on site started with FSU's abuse desk. No help whatsoever. Claimed that because the abuse desk was gone, they had no authority to deal with the problem. Frustrated, annoyed, and pissed off, he tried again, and got hung up on twice. Nice people eh? You were speaking with one of our 2nd shift operators. Unfortunately, our 2nd and 3rd operators are not hired on the basis of cluefulness, but for their willingness to work these shifts and do so at minimal pay. They tend to take things very literally, so despite the fact that we have network admins on-call 7/24, unless it's reported as a network problem the operators won't call. This morning instructions have been issued (again) to treat security incidents as network problem and act accordingly. Additionally, your E-Mail has been hardcopied and posted in the operator's area, along with some not-for-public-consumption comments of my own. After hanging up with Qwest, we got a call back from FSU. After a good 20 minutes or so of talking with the net admin from FSU, things were finally set in motion. After another good 10 minutes or so, connectivity was restored and everything was back to normal. According to my guy, they yanked the whole subnet at FSU. Problem solved. The computer's MAC was blocked at the switch. Our apologies for this incident. We will continue trying to improve the responsiveness of our 2nd and 3rd shift operations staff. If all else fails, call me at home: 850-385-4725 - SLS Scott L. Stursa 850/644-2591 Network Security Officer [EMAIL PROTECTED] Academic Computing and Network Services Florida State University - No good deed goes unpunished -
Re: DoS Attacks
On Wed, 08 Oct 2003 08:44:26 +0100 Martin Hepworth [EMAIL PROTECTED] wrote: Can SLA's be used to cover this sort of thing. (starts to dig out his own contracts). Surely you should be able to bounce it to your upstream provider who should deal with it for you?? Just a thought. -- Martin Hepworth Senior Systems Administrator Solid State Logic Ltd tel: +44 (0)1865 842300 ** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com ** Due to the efficiency of our upstream provider's abuse department, opening efficiently at 8 am and closing just as efficiently at 5 pm (because we all know network abuse only occurs between 8 and 5), the ISP wasn't going to be of much help with an attack that started at 6:30pm localtime. Andrew D Kirch Security Admin - Summit Open Source Development Group http://www.sosdg.org
Re: DoS Attacks
Due to the efficiency of our upstream provider's abuse department, opening efficiently at 8 am and closing just as efficiently at 5 pm (because we all know network abuse only occurs between 8 and 5), the ISP wasn't going to be of much help with an attack that started at 6:30pm localtime. Andrew D Kirch Security Admin - Summit Open Source Development Group http://www.sosdg.org * A lot of Abuse Departments are going to close at around 5pm, because the advanced staff that would know how to deal with such things goes home around that time. But most of them should have an escalation procedure, which means that there is someone(s) over the staff in the NOC or Call (Support) Center which can be called if necessary. You should insist or demand the issue be escalated if it warrants this. If they won't escalate it ask for the phone number of the supervisor... If they escalate it and you don't get a call back in 30-60 minutes ... call again (repeat until you get a call back). Be prepared to justify why you have had your issue escalated complete with emailable detailed information. Even if the guy that calls you back (semi-technical manager type) doesn't have the proper clue, chances are he has someone else on call that does. If they don't have this escalation capability ... (IMHO) get a different ISP. --- Alan Spicer ([EMAIL PROTECTED]) http://aspicer.homelinux.net/ Systems and Network Administration, and Telecommunications (954) 977-5245 The wonderful thing about the Internet is that you're connected to everyone else. The terrible thing about the Internet is that you're connected to everyone else. -- Vincent Cerf (Father of the Internet) Customer: The Internet is running too slow. Could you reboot it please? Customer: So that'll get me connected to the Internet, right? Tech Support: Yeah. Customer: And that's the latest version of the Internet, right? Tech Support: Uhh...uh...uh...yeah.
Re: DoS Attacks
rant style=moaning and useless context=naive I really don't give a ^H^H^H^H!H * !X *!X about what timeframe abuse departments operate. I just want more upstreams (or specifically my upstreams) to have a community that I can announce a /32 to null. /rant -hc -- Haesu C. TowardEX Technologies, Inc. Consulting, colocation, web hosting, network design and implementation http://www.towardex.com | [EMAIL PROTECTED] Cell: (978)394-2867 | Office: (978)263-3399 Ext. 170 Fax: (978)263-0033 | POC: HAESU-ARIN On Wed, Oct 08, 2003 at 07:19:39PM -0400, Alan Spicer wrote: Due to the efficiency of our upstream provider's abuse department, opening efficiently at 8 am and closing just as efficiently at 5 pm (because we all know network abuse only occurs between 8 and 5), the ISP wasn't going to be of much help with an attack that started at 6:30pm localtime. Andrew D Kirch Security Admin - Summit Open Source Development Group http://www.sosdg.org * A lot of Abuse Departments are going to close at around 5pm, because the advanced staff that would know how to deal with such things goes home around that time. But most of them should have an escalation procedure, which means that there is someone(s) over the staff in the NOC or Call (Support) Center which can be called if necessary. You should insist or demand the issue be escalated if it warrants this. If they won't escalate it ask for the phone number of the supervisor... If they escalate it and you don't get a call back in 30-60 minutes ... call again (repeat until you get a call back). Be prepared to justify why you have had your issue escalated complete with emailable detailed information. Even if the guy that calls you back (semi-technical manager type) doesn't have the proper clue, chances are he has someone else on call that does. If they don't have this escalation capability ... (IMHO) get a different ISP. --- Alan Spicer ([EMAIL PROTECTED]) http://aspicer.homelinux.net/ Systems and Network Administration, and Telecommunications (954) 977-5245 The wonderful thing about the Internet is that you're connected to everyone else. The terrible thing about the Internet is that you're connected to everyone else. -- Vincent Cerf (Father of the Internet) Customer: The Internet is running too slow. Could you reboot it please? Customer: So that'll get me connected to the Internet, right? Tech Support: Yeah. Customer: And that's the latest version of the Internet, right? Tech Support: Uhh...uh...uh...yeah.
Re: DoS Attacks
Haesu wrote: rant style=moaning and useless context=naive I really don't give a ^H^H^H^H!H * !X *!X about what timeframe abuse departments operate. I just want more upstreams (or specifically my upstreams) to have a community that I can announce a /32 to null. /rant Seems like they ought to be made to offer a 1/3-the-price offering if the connectivity is only offered for 1/3 the time. No, really, I'll be alright, just me sit down here a minute.
[nanog@Overkill.EnterZone.Net: Extensions to RFC1998 - WAS: Re: DoS Attacks]
Forwarding to NANOG on behalf of Mr. Fraizer. Please don't shoot the messenger for any arguable/discussions. -hc -- Haesu C. TowardEX Technologies, Inc. Consulting, colocation, web hosting, network design and implementation http://www.towardex.com | [EMAIL PROTECTED] Cell: (978)394-2867 | Office: (978)263-3399 Ext. 170 Fax: (978)263-0033 | POC: HAESU-ARIN - Forwarded message from John Fraizer [EMAIL PROTECTED] - X-Original-To: [EMAIL PROTECTED] Delivered-To: [EMAIL PROTECTED] Date: Wed, 8 Oct 2003 21:58:26 -0400 (EDT) From: John Fraizer [EMAIL PROTECTED] To: Haesu [EMAIL PROTECTED] Cc: [EMAIL PROTECTED], [EMAIL PROTECTED] Subject: Extensions to RFC1998 - WAS: Re: DoS Attacks In-Reply-To: [EMAIL PROTECTED] X-Spam-Status: No, hits=-2.0 required=5.0 tests=EMAIL_ATTRIBUTION,IN_REP_TO,QUOTED_EMAIL_TEXT, REPLY_WITH_QUOTES,USER_AGENT_PINE version=2.55 X-Spam-Level: X-Spam-Checker-Version: SpamAssassin 2.55 (1.174.2.19-2003-05-19-exp) On Wed, 8 Oct 2003, Haesu wrote: H? What did I miss? When did RFC1998 get updated to include Null community? Feel free to let me know if they updated RFC on that lately.. b/c I havent checked it in a while. As far as I know, my upstreams are fully RFC1998 compliant and I use them well. -hc Note: please echo this to the list. I don't have post access. Ahem... Sue...Ahem... The RFC itself hasn't been updated to include a Null community but if you think about it, providing a NULL community is fully within the concept of allowing customers to influence routing policy with the use of community strings. For example: ! router bgp 65534 neighbor a.a.a.a remote-as 65530 neighbor a.a.a.a description Customer AS65530 neighbor a.a.a.a prefix-list AS-65530 in neighbor a.a.a.a route-map CUSTOMERS in ! ip prefix-list AS-65530 seq 5 permit x.x.x.x/y le 32 ! ip community-list standard POISON permit 65534:666 ! route-map CUSTOMERS permit 10 match community POISON set local-preference 500 set ip next-hop [ip address of your sink-hole] ! Of course, the rest of the route-map CUSTOMERS is going to need to do some sanity checks on the announcements you accept from the customers OTHER than blackhole requests. In our case, we pass them through a prefix-list match that includes: ip prefix-list CUSTOMERS seq 10 deny 0.0.0.0/0 ge 25 As you can see, we're doing a prefix-list check against the announcements that the customer sends us in the neighbor statement. Each customer gets their own prefix-list that covers the networks that we have LOA to accept from that customer. (Keeps boneheads from blackholing OTHER people!) The first stanza in the CUSTOMERS route-map checks for the POISON community. Any prefix that the customer sends us that includes this community will be routed to our sink-hole. The rest of the stanzas in the CUSTOMERS route-map look for other communities from the customer. One stanza looks to see if the customer is requesting us to pass their announcements of our address space on as de-aggregated announcements. If we don't see that community, they're aggregated. Other stanzas in the route-map are pretty cut and dry RFC1998. Our customers can do the following: Community Action - 13944:0 Don't announce to any peer 13944:1 Don't announce to PEERS 13944:2 Don't announce to TRANSIT 13944:3 Don't announce to CUSTOMERS 13944:20Announce specific from EnterZone aggregate for customers who are running on our IPs. 13944:90Set preference to 90 13944:100 Set preference to 100 13944:110 Set preference to 110 13944:120 Set preference to 120 13944:666 Poison a Route 13944:NNN0 don't announce to Peer NNN 13944:NNN1 prepend once towards Peer NNN 13944:NNN2 prepend twice towards Peer NNN 13944:NNN3 prepend thrice towards Peer NNN Any time I do any consulting on another network, I recommend that they at MINIMUM implement the Poisoned Route ability. It is not terribly difficult to do as you can see above. -- John Fraizer EnterZone, Inc (13944+$|13944+_14813+$|13944+_17266+$) PGP Key = 6C5903C4 Fingerprint = 2AA6 6614 1B5E EDD2 38AD C417 3E61 F975 6C59 03C4 - End forwarded message -
Re: DoS Attacks
So here I am, asking if anyone here has any advice on dealing with these issues in the future? Its painfully apparent noone takes these situations seriously enough. What should we do when we are put in a position like this? Just sit back and hope it goes away itself? Also, any ideas on how to deal with these attacks on lower bandwidth connections? Right now, 2mbit.com / sosdg.org is sitting on a 1.5/256 business DSL line. I really can't afford to be buying T1s or T3s just to hold up to attacks like this. As always, thanks. -- Brian Bruns The Summit Open Source Development Group I think I would follow two avenues next time - the direct approach with FSU (or wherever the traffic is coming from) as well as with your DSL provider. Your upstream should be able to assist in at least keeping the traffic off of your dedicated line. Whether your DSL provider has the resources to sink the traffic may be another matter -- but they are at least in a position to help you and (since you are paying them) have an interest in dealing with you. Mark Radabaugh Amplex (419) 720-3635
Re: DoS Attacks
On Tue, Oct 07, 2003 at 11:45:35PM -0400, Brian Bruns wrote: So here I am, asking if anyone here has any advice on dealing with these issues in the future? Its painfully apparent noone takes these situations seriously enough. What should we do when we are put in a position like this? Just sit back and hope it goes away itself? You were lucky to know where it was coming from. What you depends on what you know. You knew the sources are small and you knew where they were. You did the right thing by contacting FSU, and then their upstream. If either was unresponsive, they are being extremely neglegent. I'm one of the few who still believes that immediate 24/7 response to security problems should be a requirement for permanent internet connectivity. If you don't know the source, talking to your own upstream to see if there is anything they can do is a good first move. Most decent places will at least TRY and help you out. But realise that sometimes you just need to ride it out.
Re: DoS Attacks
- Original Message - From: Mark Radabaugh [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Tuesday, October 07, 2003 11:56 PM Subject: Re: DoS Attacks I think I would follow two avenues next time - the direct approach with FSU (or wherever the traffic is coming from) as well as with your DSL provider. Your upstream should be able to assist in at least keeping the traffic off of your dedicated line. Whether your DSL provider has the resources to sink the traffic may be another matter -- but they are at least in a position to help you and (since you are paying them) have an interest in dealing with you. I hate to say this, but Ameritech/SBC is utterly useless in matters like this. I mean, at one point their redback was being nailed, and they didn't seem to care one bit. After 5pm, everyone with a clue seems to leave, and we are left with useless low level help desk techs. Our DSL service isn't bad - in fact it rarely goes down. The problem is that when we need their help with something out of our league, they are completely useless. Anyone know of a contact number for SBC/Ameritech that would be useful in a case like this? -- Brian Bruns The Summit Open Source Development Group Open Solutions For A Closed World / Anti-Spam Resources http://www.2mbit.com ICQ: 8077511
Re: DoS Attacks
On Tue, 7 Oct 2003, Avleen Vig wrote: You knew the sources are small and you knew where they were. You did the right thing by contacting FSU, and then their upstream. If either was unresponsive, they are being extremely neglegent. Its generally a better idea to contact your own upstream provider first. Your own upstream knows you, and is supposedly paid to help you. Your upstream should have contacts with its BGP peers and eventually the source. A problem with calling random NOCs is they don't know you from someone trying to social engineer something. So you end up delaying effective response while they try to figure out who you are, and how your report is related to them. If you have a problem with your credit card, your own bank is in a better position to help you than calling random other banks in the world even if you did have their phone number. Other banks may care about security, but they still don't know who you are. If your own upstream won't help you, you may have no choice but to beg other NOCs to help you.
Re: DoS Attacks
First of all, have your tools ready so that whenever DoS pounds on you, you can immediately activate them and they will give you an overview of the DoS attack such as size of the attack, source/dest (random or one/two or spoofed?), et al. Then you need to contact your upstream first to hve them deal with it, and yes I understand, most SDSL providers do not like to cooperate. Considering it takes me 1 hour of buerocracy to get an ACL put up during a DoS to my current providers, getting an ACL activated by SDSL team is.psh utterly hopeless unless you have people connections :( If you can't afford T1/T3 type of circuits where you can at least call up your upstream (doesnt matter how long it takes them to put up the ACL, the point is, will they?), then I hate to say... I don't think there is much you can do :-( -hc -- Haesu C. TowardEX Technologies, Inc. Consulting, colocation, web hosting, network design and implementation http://www.towardex.com | [EMAIL PROTECTED] Cell: (978)394-2867 | Office: (978)263-3399 Ext. 170 Fax: (978)263-0033 | POC: HAESU-ARIN On Wed, Oct 08, 2003 at 12:03:19AM -0400, Brian Bruns wrote: - Original Message - From: Mark Radabaugh [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Tuesday, October 07, 2003 11:56 PM Subject: Re: DoS Attacks I think I would follow two avenues next time - the direct approach with FSU (or wherever the traffic is coming from) as well as with your DSL provider. Your upstream should be able to assist in at least keeping the traffic off of your dedicated line. Whether your DSL provider has the resources to sink the traffic may be another matter -- but they are at least in a position to help you and (since you are paying them) have an interest in dealing with you. I hate to say this, but Ameritech/SBC is utterly useless in matters like this. I mean, at one point their redback was being nailed, and they didn't seem to care one bit. After 5pm, everyone with a clue seems to leave, and we are left with useless low level help desk techs. Our DSL service isn't bad - in fact it rarely goes down. The problem is that when we need their help with something out of our league, they are completely useless. Anyone know of a contact number for SBC/Ameritech that would be useful in a case like this? -- Brian Bruns The Summit Open Source Development Group Open Solutions For A Closed World / Anti-Spam Resources http://www.2mbit.com ICQ: 8077511
