Re: DOS attack tracing

2005-05-11 Thread Elmar K. Bins

[EMAIL PROTECTED] (Richard) wrote:

 Ethernet to the primary upstream. I think that the lesson is _always_ use a
 router powerful enough to handle all ingress traffic at wire rate. Without
 access to the router, there is nothing you can do. So we are going to switch
 out the router.

If you are mostly concerned about not being able to use the router console
during attacks, you may change the CPU scheduling a bit. A brief
scheduler allocate 6 2000 has helped me a lot there. The box
stays manageable.

This does of course not help you with the router going dead in regard to
packet forwarding...

Yours,
Elmi.

--

Begehe nur nicht den Fehler, Meinung durch Sachverstand zu substituieren.
  (PLemken, [EMAIL PROTECTED])

--[ ELMI-RIPE ]---



Re: DOS attack tracing

2005-05-10 Thread Kim Onnel

1) Get 'Cisco guard' , too expensive ?
2) Get Arbor, Stealthflow, Esphion, too expensive ?
3) Use flow-tools, ntop, Silktools and open-source Netflow collectors
 analyzers
4) Apply Ingress/Egress Filtering : RFC 2827 , uRPF, Team cymru IOS template 
5) Monitor CPU/Netflow table size using SNMP
6) Request a blackholing BGP community from your upsream provider.

On 5/10/05, Scott Weeks [EMAIL PROTECTED] wrote:
 
 On Mon, 9 May 2005, Steve Gibbard wrote:
 : On Mon, 9 May 2005, Scott Weeks wrote:
 :  On Mon, 9 May 2005, Richard wrote:
 : 
 :  : type of routers. Our routers normally run at 35% CPU. What sucks is 
 that the
 :  : traffic volume doesn't have to be very high to bring down the router.
 : 
 :  That's because it's the number of packets per time period that it can't
 :  handle, not the traffic level.  At this point it seems most likely that
 :  it's a simple UDP flood.  If your CPU usually runs at 35% you definitely
 :  don't need a bigger router unless you're expecting a growth spurt.  You
 :  might want to put an RRDTool or MRTG graph on the CPU usage to be sure.
 :
 : I'll disagree here.
 
 Cool!  Good 'ol operations discussion...  :-)
 
 I took things out of order from your email, but kept the context.
 
 : www.stevegibbard.com/ddos-talk.htm
 
 Nice paper.   However, you still say what I was saying, just in a
 different sort of way.  Instead of NTop and RRDTool/MRTG, you use Cricket.
 RRDTool/MRTG alerts you to the problem and NTop directs you to the source
 of the problem.  Once you get the procedure down pat, it can go pretty
 fast.
 
 As far as puttimg something in front of the core router(s) (such as
 Riverhead), I assumed there was nothing there for Richard; just raw
 router interface(s) to the upstream and not enough budget to afford those
 nice-but-expensive boxes.  I was going to mention things like Riverhead or
 Packeteer later in the posts if appropriate.
 
 : When you're engineering a network, what you generally need to care about
 : is peak traffic, not average traffic.  While DOS attack traffic is
 : presumably traffic you'd rather not have, it tends to be part of the
 : environment.
 :
 : This is somewhat of an arms race, and no router will protect you from all
 : conceivable DOS attacks.  That said, designing your network around the
 : size of attack you typically see (plus some room for growth) raises the
 : bar, and turns attacks of the size you've designed for into non-events
 : that you don't need to wake up in the middle of the night for.
 
 This is what I was getting at.  Engineering the network.  That's more
 than buying a Bigger Badder Router and Fatter Pipes(BBRFP).  If your
 router is running at 35% during the normal peak traffic flow, you don't
 need a BBRFP.  All you need to do is design the network (and train the
 monkeys, as randy terms it... :-) to deal with extraordinary peaks.
 
 : Remember, the real goal in dealing with DOS attacks is to get to the point
 : where you don't notice them, rather than just being able to explain why
 : your network is down.
 
 Yes, but a BBRFP isn't the way to deal with this unless you've got the
 big budget.  I know that a bigger hammer is better if you've got the
 money, but if you don't engineering finesse can work well.
 
 scott
 



Re: DOS attack tracing

2005-05-10 Thread Suresh Ramasubramanian

Quite decent suggestions

On 5/10/05, Kim Onnel [EMAIL PROTECTED] wrote:
 3) Use flow-tools, ntop, Silktools and open-source Netflow collectors
  analyzers
 4) Apply Ingress/Egress Filtering : RFC 2827 , uRPF, Team cymru IOS template
 5) Monitor CPU/Netflow table size using SNMP
 6) Request a blackholing BGP community from your upsream provider.

You start with #4, first of all.  Then get #6.  Then put #2 and #5 in place.

After that, you get one or the other of these, if you can push through
a budget for expensive kit.

 1) Get 'Cisco guard' , too expensive ?
 2) Get Arbor, Stealthflow, Esphion, too expensive ?

--srs
-- 
Suresh Ramasubramanian ([EMAIL PROTECTED])


Re: DOS attack tracing

2005-05-10 Thread Scott Weeks



On Tue, 10 May 2005, Kim Onnel wrote:

: 1) Get 'Cisco guard' , too expensive ?
: 2) Get Arbor, Stealthflow, Esphion, too expensive ?
: 3) Use flow-tools, ntop, Silktools and open-source Netflow collectors
:  analyzers
: 4) Apply Ingress/Egress Filtering : RFC 2827 , uRPF, Team cymru IOS template
: 5) Monitor CPU/Netflow table size using SNMP
: 6) Request a blackholing BGP community from your upsream provider.


Yep, those are some of the things I meant by:

:  Yes, but a BBRFP isn't the way to deal with this unless you've got the
:  big budget.  I know that a bigger hammer is better if you've got the
:  money, but if you don't engineering finesse can work well.

scott






: On 5/10/05, Scott Weeks [EMAIL PROTECTED] wrote:
: 
:  On Mon, 9 May 2005, Steve Gibbard wrote:
:  : On Mon, 9 May 2005, Scott Weeks wrote:
:  :  On Mon, 9 May 2005, Richard wrote:
:  : 
:  :  : type of routers. Our routers normally run at 35% CPU. What sucks is 
that the
:  :  : traffic volume doesn't have to be very high to bring down the router.
:  : 
:  :  That's because it's the number of packets per time period that it can't
:  :  handle, not the traffic level.  At this point it seems most likely that
:  :  it's a simple UDP flood.  If your CPU usually runs at 35% you definitely
:  :  don't need a bigger router unless you're expecting a growth spurt.  You
:  :  might want to put an RRDTool or MRTG graph on the CPU usage to be sure.
:  :
:  : I'll disagree here.
: 
:  Cool!  Good 'ol operations discussion...  :-)
: 
:  I took things out of order from your email, but kept the context.
: 
:  : www.stevegibbard.com/ddos-talk.htm
: 
:  Nice paper.   However, you still say what I was saying, just in a
:  different sort of way.  Instead of NTop and RRDTool/MRTG, you use Cricket.
:  RRDTool/MRTG alerts you to the problem and NTop directs you to the source
:  of the problem.  Once you get the procedure down pat, it can go pretty
:  fast.
: 
:  As far as puttimg something in front of the core router(s) (such as
:  Riverhead), I assumed there was nothing there for Richard; just raw
:  router interface(s) to the upstream and not enough budget to afford those
:  nice-but-expensive boxes.  I was going to mention things like Riverhead or
:  Packeteer later in the posts if appropriate.
: 
:  : When you're engineering a network, what you generally need to care about
:  : is peak traffic, not average traffic.  While DOS attack traffic is
:  : presumably traffic you'd rather not have, it tends to be part of the
:  : environment.
:  :
:  : This is somewhat of an arms race, and no router will protect you from all
:  : conceivable DOS attacks.  That said, designing your network around the
:  : size of attack you typically see (plus some room for growth) raises the
:  : bar, and turns attacks of the size you've designed for into non-events
:  : that you don't need to wake up in the middle of the night for.
: 
:  This is what I was getting at.  Engineering the network.  That's more
:  than buying a Bigger Badder Router and Fatter Pipes(BBRFP).  If your
:  router is running at 35% during the normal peak traffic flow, you don't
:  need a BBRFP.  All you need to do is design the network (and train the
:  monkeys, as randy terms it... :-) to deal with extraordinary peaks.
: 
:  : Remember, the real goal in dealing with DOS attacks is to get to the point
:  : where you don't notice them, rather than just being able to explain why
:  : your network is down.
: 
:  Yes, but a BBRFP isn't the way to deal with this unless you've got the
:  big budget.  I know that a bigger hammer is better if you've got the
:  money, but if you don't engineering finesse can work well.
: 
:  scott
: 
: 
:
:



RE: DOS attack tracing

2005-05-10 Thread Hannigan, Martin


 -Original Message-
 From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of
 Kim Onnel
 Sent: Tuesday, May 10, 2005 4:19 AM
 To: Scott Weeks
 Cc: nanog@merit.edu
 Subject: Re: DOS attack tracing
 
 
 
 1) Get 'Cisco guard' , too expensive ?
 2) Get Arbor, Stealthflow, Esphion, too expensive ?

Good advice when DDOS' are constant. If this was a first and possibly
last for awhile, it may make sense to rely on the software tools
and a good 'SOP' with the provider instead. It really depends on
the scope of the problem in particular.

DDOS' is rather infrequent to zero for most enterprises. That DDOS
golden banana is rather yummy with sprinkles on top. Don't get me wrong,
the DDOS problem is real, but not for everyone, and not as frequently as
it's being hyped up to be. A managed service is a better way
to go if they're worried, IMO.


[ SNIP ]


Re: DOS attack tracing

2005-05-10 Thread Suresh Ramasubramanian

On 5/10/05, Hannigan, Martin [EMAIL PROTECTED] wrote:
 DDOS' is rather infrequent to zero for most enterprises. That DDOS
 golden banana is rather yummy with sprinkles on top. Don't get me wrong,
 the DDOS problem is real, but not for everyone, and not as frequently as
 it's being hyped up to be. A managed service is a better way
 to go if they're worried, IMO.

There's also the minimze risk thing .. take a conscious business
decision not to host one of the typical DDoS magnets (dont allow
people to run IRC bots on your colo farm, for example)

-- 
Suresh Ramasubramanian ([EMAIL PROTECTED])


Re: DOS attack tracing

2005-05-10 Thread Gadi Evron
Hannigan, Martin wrote:

Well, this is no longer about tracing DDoS I suppose..
Good advice when DDOS' are constant. If this was a first and possibly
last for awhile, it may make sense to rely on the software tools
and a good 'SOP' with the provider instead. It really depends on
the scope of the problem in particular.
DDOS' is rather infrequent to zero for most enterprises. That DDOS
golden banana is rather yummy with sprinkles on top. Don't get me wrong,
the DDOS problem is real, but not for everyone, and not as frequently as
it's being hyped up to be. A managed service is a better way
to go if they're worried, IMO.
Two things, planning for disaster and mitigation on-going DDoS attacks.
Planning...
Sound advice, but I'd phrase it a little differently.
All depending on how big they are, how much they have to invest, how 
worried they are and how much they stand to lose by such an attack, 
short or prolonged (which after their last experience they should be 
able to answer), they are more than capable to decide how much they want 
to invest.

If they are generally concerned but not truly able to pay so much for 
an.. infrequent serious risk, they can indeed get better (more 
organized) relations with their uplink, as well as perhaps check if 
their uplink can use their own.. say Cisco Guard for them or whatever 
other mitigation service they can offer. That or get a better uplink.

They could combine tactics, such as for example get the Guard but direct 
it using netflow data rather than the Detector.

It all depends on how much they are willing to invest - but knowing what 
they need is entirely up to them and after such an attack I bet they 
have a fairly good idea.

Mitigating...
As to the infrequency of the attacks, it really depends on who you ask. 
We (at Tehila) get attacked quite often, and we see others get attacked 
quite often. Others yet, get attacked on such a scale once a year or so. 
How much do you stand to lose from just ONE devastating attack?

Underplaying DDoS though is something I do not agree with you on, 
though. The scale of the problem is much bigger than most believe.

Unrelated to my own experience and that of my employer, at the drone 
armies research and mitigation mailing list we have been able to 
actively mitigate DDoS attacks in real time, what we need is a log of 
the attacking IP's with timestamps and we do our best to help.

In our last success we mitigated a 400 mega packets attack into just 
about 20, crippling the ability of the attacker to strike for a few 
weeks. After his second attempt he never went back to that target again 
(so far, anyway).

	Gadi.


RE: DOS attack tracing

2005-05-10 Thread Hannigan, Martin

 -Original Message-
 From: Suresh Ramasubramanian [mailto:[EMAIL PROTECTED]
 Sent: Tuesday, May 10, 2005 8:06 AM
 To: Hannigan, Martin
 Cc: Kim Onnel; Scott Weeks; nanog@merit.edu
 Subject: Re: DOS attack tracing
 
 
 On 5/10/05, Hannigan, Martin [EMAIL PROTECTED] wrote:
  DDOS' is rather infrequent to zero for most enterprises. That DDOS
  golden banana is rather yummy with sprinkles on top. Don't 
 get me wrong,
  the DDOS problem is real, but not for everyone, and not as 
 frequently as
  it's being hyped up to be. A managed service is a better way
  to go if they're worried, IMO.
 
 There's also the minimze risk thing .. take a conscious business
 decision not to host one of the typical DDoS magnets (dont allow
 people to run IRC bots on your colo farm, for example)

There's two classes of discussion here. One for service providers
who should have DDOS defense, and one for enterprises who should have
risk mitigation in mind. I think that operators should have DDOS defense
capabilities for themselves and their customers, and I think that enterprises
should seriously evaluate their need for a full blown implementation of a 
DDOS solution based on a solid risk analysis. 

As far as DOS tracing goes, using the freeware tools locally, and either
buying and/or subscribing to a ddos defense service make sense as much
as it makes sense to analyze the cost and your own capability as well as
your providers capability to quickly and successfully defend against a DDOS.

-M



 


RE: DOS attack tracing

2005-05-10 Thread Chris Ranch

On Monday, May 09, 2005 5:49 PM, Richard wrote:
  
  On Mon, May 09, 2005 at 01:35:06PM -1000, Richard wrote:
  
   We recently experienced several DOS attacks which drove 
   our backbone routers CPU to 100%. The routers are not 
   under attack, but the router just couldn't handle the 
   traffic. There is a plan to upgrade these routers.
  
  What kind of routers? We had problems like this with Cisco 
  7206VXRs with NPE-300s at my last job because they just 
  couldn't handle the high volume of packets-per-second from 
  certain types of attack.

 Oh... I guess that it would a known issue then... we have the 
 exactly same type of routers. Our routers normally run at 35% 
 CPU. What sucks is that the traffic volume doesn't have to be 
 very high to bring down the router.

Yes, the 7206vxr with whatever processor really checks out when under
any kind of real flood through it.  It's big brother, the 7304-NSE100
does as well.  But the 7304-NPE100 with the PXF can forward that (d)DoS
very well.  Even with fairly extensive ingress filters.  The kick in the
head is that the processors are the same price.  I don't know why they
even sell the NPE100...

Then you can take whatever measures you like to characterize and
mitigate. A combination of upstream null routing (poisoning
communities), ingress filters, core null routing, and your favorite ddos
mitigation equipment filtering has been very effective for us.  

Chris

Chris Ranch
Director of Network Architecture
Affinity Internet, Inc.


RE: DOS attack tracing

2005-05-10 Thread Chris Ranch

On Tuesday, May 10, 2005 5:06 AM, Suresh wrote:
 On 5/10/05, Hannigan, Martin [EMAIL PROTECTED] wrote:
  DDOS' is rather infrequent to zero for most enterprises. That DDOS 
  golden banana is rather yummy with sprinkles on top. Don't get me 
  wrong, the DDOS problem is real, but not for everyone, and not as 
  frequently as it's being hyped up to be. A managed service 
 is a better 
  way to go if they're worried, IMO.
 
 There's also the minimze risk thing .. take a conscious 
 business decision not to host one of the typical DDoS magnets 
 (dont allow people to run IRC bots on your colo farm, for example)

Dumping our IRC customers dropped our DDoS frequency by an order of
magnitude.  We've certainly slept better as a result.

Chris

Chris Ranch
Director of Network Architecture
Affinity Internet, Inc.


RE: DOS attack tracing

2005-05-10 Thread Chris Ranch

Correcting a typo... 

 Yes, the 7206vxr with whatever processor really checks out 
 when under any kind of real flood through it.  It's big 
 brother, the 7304-NSE100 does as well.  But the 7304-NPE100 
 with the PXF can forward that (d)DoS very well.  Even with 
 fairly extensive ingress filters.  The kick in the head is 
 that the processors are the same price.  I don't know why 
 they even sell the NPE100...

I don't know why they even sell the NSE100.  You want the NPE with the
PXF.

Chris


RE: DOS attack tracing

2005-05-10 Thread Chris Ranch

  I don't know why they even sell the NSE100.  You want the 
 NPE with the 
  PXF.
  
  Chris
 
 No, that's backward.
 
 The NSE100 has the PXF processor.
 
 The NPE-G100 is a software router.

Correct, of course.  Thanks.

Chris


RE: DOS attack tracing

2005-05-10 Thread Richard

 Right... I did mention that further down in my message. And yeah -
 almost impossible to get much done when the CPU is pegged. I remember
 a DOS attack demo where they used 7200s for the examples - almost
 wanted to yell out try pegging the CPU with lots of traffic and THEN
 try to identify / null0 the destination or source.
That's the problem in our case. One of our downstream customers got the
attack. Once we disconnected them, everything became fine. I tried pretty
much everything under our control to divert the traffic, including ingress
acl to block all incoming traffic to their subnets. But every time I turn
the downstream ISP back on, our router was dead. We got a 7206VXR and 100M
Ethernet to the primary upstream. I think that the lesson is _always_ use a
router powerful enough to handle all ingress traffic at wire rate. Without
access to the router, there is nothing you can do. So we are going to switch
out the router.

Richard




DOS attack tracing

2005-05-09 Thread Richard

Hi,

We recently experienced several DOS attacks which drove our backbone routers
CPU to 100%. The routers are not under attack, but the router just couldn't
handle the traffic. There is a plan to upgrade these routers. One criteria
is the ability to track which IP address is under attack and blackhole the
traffic quickly. Anyone can share your experience of what kind of router is
capable of doing this?

Also besides having a powerful router which can handle large volume of
traffic, is there any other things that we need to consider in selecting the
routers?

Thanks,
Richard




Re: DOS attack tracing

2005-05-09 Thread Richard A Steenbergen

On Mon, May 09, 2005 at 01:35:06PM -1000, Richard wrote:
 
 Hi,
 
 We recently experienced several DOS attacks which drove our backbone routers
 CPU to 100%. The routers are not under attack, but the router just couldn't
 handle the traffic. There is a plan to upgrade these routers. One criteria
 is the ability to track which IP address is under attack and blackhole the
 traffic quickly. Anyone can share your experience of what kind of router is
 capable of doing this?
 
 Also besides having a powerful router which can handle large volume of
 traffic, is there any other things that we need to consider in selecting the
 routers?

I recently wrecked my car, totaling it and running down several small 
children on their way to sunday school in the process. I plan to upgrade 
my car, and one of the criteria is that it not crash and kill people. Can 
you share advice on what car is capable of doing this?

This example is about as descriptive and useful at solving the problem as 
your original post. Without any details it is impossible to make any 
useful recommendation even if we wanted to. What type and scale of DoS are 
you trying to protect against, what type and scale of traffic are you 
routing, what kind of interfaces and how many, basic things like that. 
Without details, the best that you're likely to get (now that Dean is gone 
:P) is something akin to go buy a volvo, namely go buy a Juniper.

-- 
Richard A Steenbergen [EMAIL PROTECTED]   http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)


Re: DOS attack tracing

2005-05-09 Thread Scott Weeks



On Mon, 9 May 2005, Richard wrote:

: We recently experienced several DOS attacks which drove our backbone routers
: CPU to 100%. The routers are not under attack, but the router just couldn't
: handle the traffic. There is a plan to upgrade these routers. One criteria
: is the ability to track which IP address is under attack and blackhole the
: traffic quickly. Anyone can share your experience of what kind of router is
: capable of doing this?
:
: Also besides having a powerful router which can handle large volume of
: traffic, is there any other things that we need to consider in selecting the
: routers?


You shouldn't buy a bigger router just to handle DOS attacks.  THere're
many ways to address these types of issues using routers and/or servers.
What is your normal CPU usage when there is no DOS attack?  What does your
capacity look like on the router interface where the DOS is coming in on?
We need more info.

scott




Re: DOS attack tracing

2005-05-09 Thread Will Yardley

On Mon, May 09, 2005 at 01:35:06PM -1000, Richard wrote:

 We recently experienced several DOS attacks which drove our backbone
 routers CPU to 100%. The routers are not under attack, but the
 router just couldn't handle the traffic. There is a plan to upgrade
 these routers.

What kind of routers? We had problems like this with Cisco 7206VXRs
with NPE-300s at my last job because they just couldn't handle the
high volume of packets-per-second from certain types of attack.

 One criteria is the ability to track which IP address is under
 attack and blackhole the traffic quickly. Anyone can share your
 experience of what kind of router is capable of doing this?

Disclaimer: I'm not an expert on this stuff, and it's possible
(likely) that others on the list may have some other and / or better
suggestions.

Generally, I've seen this done by exporting flow data to another box,
and then analyzing this data. I've used ehnt (extremely happy netflow
tool) (http://ehnt.sourceforge.net/) to capture the flow data and
export it to an easily machine-parsable feed, then used a Perl script
to capture information on the top source / destination addresses. If
there's interest, I could see whether it's possible to get this code
and put it up somewhere (on an as-is basis) - the code was written by
Kenytt Avery at Willing Minds (willingminds.com).

We were keeping an ongoing log of such data, in case the router itself
took a crap.

On a Cisco router, you can also look at the raw cache flow data (sh ip
cache flow), which has some summary data at the top, and then data on
each flow. By rshing into the device and capturing this output, you have
access to some other data to futz around with in some sort of script.

So I'm not sure if there are any vendors which make it easy to figure
this out while logged into the device itself (or whether this is a
practical thing to do at all or something vendors are working on
implementing), but it is possible to do using tools like netflow.

w



RE: DOS attack tracing

2005-05-09 Thread Richard

 
 On Mon, May 09, 2005 at 01:35:06PM -1000, Richard wrote:
 
  We recently experienced several DOS attacks which drove our backbone
  routers CPU to 100%. The routers are not under attack, but the
  router just couldn't handle the traffic. There is a plan to upgrade
  these routers.
 
 What kind of routers? We had problems like this with Cisco 7206VXRs
 with NPE-300s at my last job because they just couldn't handle the
 high volume of packets-per-second from certain types of attack.
Oh... I guess that it would a known issue then... we have the exactly same
type of routers. Our routers normally run at 35% CPU. What sucks is that the
traffic volume doesn't have to be very high to bring down the router.

 On a Cisco router, you can also look at the raw cache flow data (sh ip
 cache flow), which has some summary data at the top, and then data on
 each flow. By rshing into the device and capturing this output, you have
 access to some other data to futz around with in some sort of script.
 
 So I'm not sure if there are any vendors which make it easy to figure
 this out while logged into the device itself (or whether this is a
 practical thing to do at all or something vendors are working on
 implementing), but it is possible to do using tools like netflow.
So far we manually login to the router and use 'sh ip cache flow' on the
router. It is ok, but not very effective. First when the router is slow to a
halt, it is not even possible to the run the command most of the time.
Secondly reading through the output and figuring out what's going on is not
an easy task. I will definitely look into the tools to automate this
process. Appreciate your suggestion. Just wonder if any router vendor has
any built-in tools.

Thanks,
Richard





RE: DOS attack tracing

2005-05-09 Thread Scott Weeks



On Mon, 9 May 2005, Richard wrote:

:   We recently experienced several DOS attacks which drove our backbone
:   routers CPU to 100%. The routers are not under attack, but the
:   router just couldn't handle the traffic. There is a plan to upgrade

: type of routers. Our routers normally run at 35% CPU. What sucks is that the
: traffic volume doesn't have to be very high to bring down the router.


That's because it's the number of packets per time period that it can't
handle, not the traffic level.  At this point it seems most likely that
it's a simple UDP flood.  If your CPU usually runs at 35% you definitely
don't need a bigger router unless you're expecting a growth spurt.  You
might want to put an RRDTool or MRTG graph on the CPU usage to be sure.

Depending on the size of your network you also might put a server at a
good place where you can mirror the traffic to it and use NTop on the
server.  The software is free and will show a huge amount of detail if the
server has the brawn to handle the load.  More detail means more server
brawn.  You'll definitely see where the DOS is going.

scott



RE: DOS attack tracing

2005-05-09 Thread Steve Gibbard
On Mon, 9 May 2005, Scott Weeks wrote:
On Mon, 9 May 2005, Richard wrote:
: type of routers. Our routers normally run at 35% CPU. What sucks is that the
: traffic volume doesn't have to be very high to bring down the router.
That's because it's the number of packets per time period that it can't
handle, not the traffic level.  At this point it seems most likely that
it's a simple UDP flood.  If your CPU usually runs at 35% you definitely
don't need a bigger router unless you're expecting a growth spurt.  You
might want to put an RRDTool or MRTG graph on the CPU usage to be sure.
I'll disagree here.
When you're engineering a network, what you generally need to care about 
is peak traffic, not average traffic.  While DOS attack traffic is 
presumably traffic you'd rather not have, it tends to be part of the 
environment.

This is somewhat of an arms race, and no router will protect you from all 
conceivable DOS attacks.  That said, designing your network around the 
size of attack you typically see (plus some room for growth) raises the 
bar, and turns attacks of the size you've designed for into non-events 
that you don't need to wake up in the middle of the night for.

Remember, the real goal in dealing with DOS attacks is to get to the point 
where you don't notice them, rather than just being able to explain why 
your network is down.

For those attacks that go beyond the capacity you can afford, being able 
to divert the traffic is a good thing.  The Riverhead system (now known as 
Cisco Guard, I think) does reasonably well at protecting networks 
downstream from it without being a big point of failure, but the network 
upstream from it still needs to be able to take the load.  And being 
better able to characterize the attack traffic may help you ask your 
upstreams to block it for you.  This can be done with some of the tools 
others have mentioned, including your router's flow cache *if your router 
hasn't already fallen over and died*.

A rather dated paper on my experiences dealing with this sort of thing is 
at http://www.stevegibbard.com/ddos-talk.htm.

-Steve


RE: DOS attack tracing

2005-05-09 Thread Scott Weeks

On Mon, 9 May 2005, Steve Gibbard wrote:
: On Mon, 9 May 2005, Scott Weeks wrote:
:  On Mon, 9 May 2005, Richard wrote:
: 
:  : type of routers. Our routers normally run at 35% CPU. What sucks is that 
the
:  : traffic volume doesn't have to be very high to bring down the router.
: 
:  That's because it's the number of packets per time period that it can't
:  handle, not the traffic level.  At this point it seems most likely that
:  it's a simple UDP flood.  If your CPU usually runs at 35% you definitely
:  don't need a bigger router unless you're expecting a growth spurt.  You
:  might want to put an RRDTool or MRTG graph on the CPU usage to be sure.
:
: I'll disagree here.

Cool!  Good 'ol operations discussion...  :-)


I took things out of order from your email, but kept the context.

: www.stevegibbard.com/ddos-talk.htm

Nice paper.   However, you still say what I was saying, just in a
different sort of way.  Instead of NTop and RRDTool/MRTG, you use Cricket.
RRDTool/MRTG alerts you to the problem and NTop directs you to the source
of the problem.  Once you get the procedure down pat, it can go pretty
fast.

As far as puttimg something in front of the core router(s) (such as
Riverhead), I assumed there was nothing there for Richard; just raw
router interface(s) to the upstream and not enough budget to afford those
nice-but-expensive boxes.  I was going to mention things like Riverhead or
Packeteer later in the posts if appropriate.


: When you're engineering a network, what you generally need to care about
: is peak traffic, not average traffic.  While DOS attack traffic is
: presumably traffic you'd rather not have, it tends to be part of the
: environment.
:
: This is somewhat of an arms race, and no router will protect you from all
: conceivable DOS attacks.  That said, designing your network around the
: size of attack you typically see (plus some room for growth) raises the
: bar, and turns attacks of the size you've designed for into non-events
: that you don't need to wake up in the middle of the night for.

This is what I was getting at.  Engineering the network.  That's more
than buying a Bigger Badder Router and Fatter Pipes(BBRFP).  If your
router is running at 35% during the normal peak traffic flow, you don't
need a BBRFP.  All you need to do is design the network (and train the
monkeys, as randy terms it... :-) to deal with extraordinary peaks.


: Remember, the real goal in dealing with DOS attacks is to get to the point
: where you don't notice them, rather than just being able to explain why
: your network is down.

Yes, but a BBRFP isn't the way to deal with this unless you've got the
big budget.  I know that a bigger hammer is better if you've got the
money, but if you don't engineering finesse can work well.

scott