Is TDM going the way of dial-up?
I've noticed over the last 3 years or so that TDM, specifically T-1, access and transport has been in a steady decline. Customers are moving to FTTH and cable, or going WiMAX and Metro-Ethernet. Ethernet seems to have taken an even bigger bite out of DS-3. The bigger pipes seem to favor ethernet. A recent upgrade from OC-3 to GigE transport actually saved us a large chunk of money. I'm wondering if others are seeing the same behavior, if it's market-dependant, or if I'm just imagining things. I'm working on building new infrastructure and my current thoughts are to minimize my TDM footprint. It would be useful to get a better feel if this is an overall trend or something local. Thoughts? Thanks,
Re: IPv6, multihoming, and customer allocations
Regurgitating the original e-mail for context and follow-up. General responses (some that didn't make it to the list): - There really is that much space, don't worry about it. - /48s for those that ask for it is fine, ARIN won't ask unless it's a bigger assignment - /52 (or /56) on smaller assignments for conservation if it makes you feel better - Open question on whether byte/octet-boundary assignment (/56 vs /52) is better for some reason I haven't seen anything on the general feel for prefix filtering. I've seen discussions from /48 down to /54. Any feel for what the standard (widely deployed) IPv6 prefix filter size will be? Thanks, On Sat, Mar 13, 2010 at 10:49 PM, Rick Ernst na...@shreddedmail.com wrote: A couple of different incantations searching the archive didn't enlighten me, and I find it hard to believe this hasn't been discussed. Apologies and a request for pointers if I'm rehashing an old question. As a small/regional ISP, we got our /32 assigned and it's time to start moving forward (customers are asking for it). New hardware, updated IOS, etc. are in the pipe. Discussions are beginning with our upstream providers for peering. Now, what do we do? A /48 seems to be the standard end-user/multi-homed customer allocation and is the minimum allocation size from ARIN. A /32 provides 65K /48s so, in theory, we could give each of our customers a /48 and still have room for growth. A /48 also appears to be generally accepted as the the longest prefix allowed through filters (although /49 through /54 are also discussed). Most customers, however, won't be multi-homed. Partly from an IPv4 scarcity perspective, and partly from general efficiency and thrift, it seems awfully silly to hand out /48s to somebody that may have a handful of servers or a couple of home machines, especially with special addressing like link-local if the hosts are not expected to be internet reachable (back-end servervs, etc). Based on the above, I'm looking to establish some initial policies to save grief in the future: - /52 allocations to end-users (residential, soho, etc.) - /48 allocations to those that request it - If you are going to multi-home, get your own space This is obviously a very broad brush and takes an insanely large addressing model and makes it even larger (assigning /52s instead of /48s) but, to me at least, it seems reasonable for a first-pass. For context/scope, we currently have the equivalent of a bit more than the equivalent of a /16 (IPv4) in use. Thanks,
IPv6, multihoming, and customer allocations
A couple of different incantations searching the archive didn't enlighten me, and I find it hard to believe this hasn't been discussed. Apologies and a request for pointers if I'm rehashing an old question. As a small/regional ISP, we got our /32 assigned and it's time to start moving forward (customers are asking for it). New hardware, updated IOS, etc. are in the pipe. Discussions are beginning with our upstream providers for peering. Now, what do we do? A /48 seems to be the standard end-user/multi-homed customer allocation and is the minimum allocation size from ARIN. A /32 provides 65K /48s so, in theory, we could give each of our customers a /48 and still have room for growth. A /48 also appears to be generally accepted as the the longest prefix allowed through filters (although /49 through /54 are also discussed). Most customers, however, won't be multi-homed. Partly from an IPv4 scarcity perspective, and partly from general efficiency and thrift, it seems awfully silly to hand out /48s to somebody that may have a handful of servers or a couple of home machines, especially with special addressing like link-local if the hosts are not expected to be internet reachable (back-end servervs, etc). Based on the above, I'm looking to establish some initial policies to save grief in the future: - /52 allocations to end-users (residential, soho, etc.) - /48 allocations to those that request it - If you are going to multi-home, get your own space This is obviously a very broad brush and takes an insanely large addressing model and makes it even larger (assigning /52s instead of /48s) but, to me at least, it seems reasonable for a first-pass. For context/scope, we currently have the equivalent of a bit more than the equivalent of a /16 (IPv4) in use. Thanks,
Re: D/DoS mitigation hardware/software needed.
I thought I had mentioned outsourcing earlier, but I don't see it in the thread... The two mechanisms I've seen for outsources D/DoS are DNS manipulation, or essentially remote BGP peering with an tunnel back to the local presence. Even if we are purely hosting, DNS manipulation doesn't do anything for attacks against an IP. For remote BGP peering/tunneling, you are are adding additional complexity and moving control outside your network. As a service-provider/data-center, it seems like outsourcing would be either ineffective and/or removes the big red button in case of trouble. Am I missing something, overly paranoid, or are there other mechanisms for outsourced protection? Rick On Mon, Jan 11, 2010 at 6:33 AM, Stefan Fouant sfou...@shortestpathfirst.net wrote: -Original Message- From: Christopher Morrow [mailto:morrowc.li...@gmail.com] Sent: Monday, January 11, 2010 2:05 AM On Mon, Jan 11, 2010 at 12:26 AM, jul jul_...@yahoo.fr wrote: Martin Hannigan wrote on 05/01/10 16:50: Outsourced services have higher cost than Arbor but can handled more. Do they? VerizonBusiness's solution was $3250US/month so ~$90USk over 2yrs. Arbor, I think, for a TMS + collectors was +100k. Don't forget to factor in OpEx. This can often tilt the scales in favor of one vs. the other. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: D/DoS mitigation hardware/software needed.
Right. Some providers allow you to BGP community trigger RTBH. There was a separate mention of D/DoS-mitigation-providers using DNS and BGP tunneling. Rick On Mon, Jan 11, 2010 at 8:14 AM, Stefan Fouant sfou...@shortestpathfirst.net wrote: -Original Message- From: Rick Ernst [mailto:na...@shreddedmail.com] Sent: Monday, January 11, 2010 10:39 AM To: NANOG Subject: Re: D/DoS mitigation hardware/software needed. As a service-provider/data-center, it seems like outsourcing would be either ineffective and/or removes the big red button in case of trouble. Am I missing something, overly paranoid, or are there other mechanisms for outsourced protection? In fact, quite the opposite. Those providers who do offer DDoS mitigation services usually allow the customer to trigger the redirect in a manner similar to RTBHs by substituting the blackhole community for some type of mitigation community. This causes the Provider's edge router (or Route Server) to advertise the affected route within the Service Provider's network with a next-hop of the scrubbers. There are some providers who do auto-mitigation on behalf of the customer, but IMO this approach is asking for trouble. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: D/DoS mitigation hardware/software needed.
I looked at one of the suggested out-sourced providers. Based on a sample size of 1, the mitigating mechanisms are DNS redirection and BGP/tunneling. While both of these solutions may be useful for an end-user (even large ones), I don't see them fitting in an SP environment. If something goes wrong, I want my own, local, big-red button. Rick On Tue, Jan 5, 2010 at 7:50 AM, Martin Hannigan mar...@theicelandguy.comwrote: On Mon, Jan 4, 2010 at 4:19 PM, Rick Ernst na...@shreddedmail.com wrote: Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device - Outsource to service provider Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. How often are you getting DDoS'd? The financials of using a managed service provider vs. buy-all-your-own-grrovy-stuff can be fairly compelling especially if the amount of DDoS you experience is almost nil. Re: Arbor. I don't have any recent experience, but they've been around for a long time, have a very experienced team that understands ISP and enterprise and the product is mature. Hard to go wrong if you can justify the costs. YMMV. Best, -M -- Martin Hannigan mar...@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants
D/DoS mitigation hardware/software needed.
Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. My current system is a home-grown NetFlow parser that spits out syslog to our NOC to investigate potential attacks and manually enter them into our RTBH. Any suggestions other than Arbor? Any other mechanisms being used? My idea is to quash the immediate problem and work additional mitigation with upstreams if needed. I could probably add some automation to my NetFlow/RTBH setup, but I still need to worry about false-positives. I'd rather somebody else do the hard work of finding the various edge-cases. Thanks, Rick
Re: D/DoS mitigation hardware/software needed.
Several responses already, and Arbor has poked their head up. I'm going to start there and keep the other suggestions at-hand. Thanks, On Mon, Jan 4, 2010 at 1:19 PM, Rick Ernst na...@shreddedmail.com wrote: Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. My current system is a home-grown NetFlow parser that spits out syslog to our NOC to investigate potential attacks and manually enter them into our RTBH. Any suggestions other than Arbor? Any other mechanisms being used? My idea is to quash the immediate problem and work additional mitigation with upstreams if needed. I could probably add some automation to my NetFlow/RTBH setup, but I still need to worry about false-positives. I'd rather somebody else do the hard work of finding the various edge-cases. Thanks, Rick
Re: D/DoS mitigation hardware/software needed.
On Mon, Jan 4, 2010 at 9:08 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 12:05 PM, Rick Ernst wrote: A solution preferably that integrates with NetFlow and RTBH. An in-line solution obviously requires an appliance, or at least special/additional hardware. The key is to not be inline all the time, but only inline *when needed*. This removes operational complexity, provides the ability to oversubscribe, and simplifies the routine troubleshooting matrix. I'd argue just the opposite. If your monitoring/mitigation system changes dependent on the situation (normal vs under attack), you are adding complexity to the system. What mode is the system in right now? Is this customer having connectivity issues because of a state change in the network? etc. I'm looking at taking the first whack at immediate mitigation at the border/edge (upstream) via uRPF and RTBH. Good plan. Additional mitigation would be via manual or automatic RTBH or security/abuse@ involvement with upstreams. Automagic is generally bad, as it can be gamed. I know you said generally, but if I'm seeing 200Kpps from a.b.c.d, I don't care if a.b.c.d is spoofed. I want the traffic blocked from the guts of my network. Note that my original question was in the context of a D/DoS composed of lots of itty-bitty packets. Other attack mechanisms do not necessarily lend themselves to chop 'em off at the knees. Rick
Re: D/DoS mitigation hardware/software needed.
I think you, Roland, and I are all agreeing on the same argument. The (my) confusion entered with the statement of, The key is to not be inline all the time, but only inline *when needed*. I inferred a topological or physical path change from that. Redirecting traffic (which is really just an extension of RTBH; a scrubber destination rather than Null0) is an understandable state. Rick On Mon, Jan 4, 2010 at 9:34 PM, Stefan Fouant sfou...@shortestpathfirst.net wrote: -Original Message- From: Rick Ernst [mailto:na...@shreddedmail.com] Sent: Tuesday, January 05, 2010 12:19 AM I'd argue just the opposite. If your monitoring/mitigation system changes dependent on the situation (normal vs under attack), you are adding complexity to the system. What mode is the system in right now? Is this customer having connectivity issues because of a state change in the network? etc. Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: Consistent asymetric latency on monitoring?
Lots of good info, and a nice mind-dump that gives me a whole host of other things that need to be looked at... Umm. thanks :) On Wed, Oct 21, 2009 at 11:10 PM, Perry Lorier pe...@coders.net wrote: Rick Ernst wrote: Resent, since I responded from the wrong address: --- The basic operation of IP SLA is as surmised; payload with timestamps and other telemetry data is sent to a 'responder' which manipulates the payload, including adding its own timestamps, and returns the altered payload. Yup :) It's the obvious way to do it :) I had to do a mental walk-through, but I think I see how drift can cause this. I'm going to generate some artificial data, graph it, and see if it matches the general waveshape I'm seeing. I purposefully have the traffic generators ntp syncing against the responders. I thought that would keep the clocks more closely in sync. I don't necessarily care if the time is 'right', just that it's the same. This causes major problems. What you're actually measuring here is how well ntp can keep the clock sync'd under assymetric latency. ntp is trying to do it's own measurements of one way delay, without the help of clocks to measure clock drift as well. As you can see from your graphs ntp is not coping[1]. You are far better to have each end sync to a local stratum 1 or stratum 2 ntp source, preferably one over a different link to the one under test. If you don't have a local stratum 1/2 time source at each end, you might be able find one over a local exchange or other less congested link. If this is very important to you then you should consider looking at running your own stratum 1 clocks at each end syncronised off something like GPS, CDMA or a T1 clock. What kind of difference should I expect if I sync both generators and responders against the same source, or not sync the responder? I'm thinking that having one source with constant drift may be better than both devices trying to walk/correct the time. Most hardware clocks in PC's/routers/switches etc have pretty atrocious amounts of drift if left to free run[2], sometimes in the order of seconds or occasionally minutes per week. To get useful numbers you really do need to syncronise them to /something/. Synchronising them to each other causes problems as ntp I think (I could be wrong) assumes mostly symmetrical latency, and if the latency isn't symmetric assumes it's because one clock is running fast/slow and will alter the clock's speed to account for it. The great thing about ntp stratum 1 servers is that by definition they have more or less the same time no matter where they are, so synchronising each against a local ntp server will be a much much better solution. If possible you should consider peering with at least 3 upstreams, preferably 4(!)[3] other ntp servers. [1]: To be fair it's a hard problem. Anything that involves time just gets more and more complicated the more you look at it, ntp is extremely clever and probably knows more about time than I'd ever want to know, but you're making it's job hard. [2]: http://vancouver-webpages.com/time/ / http://vancouver-webpages.com/time/ltmhist.png [3]: http://twiki.ntp.org/bin/view/Support/SelectingOffsiteNTPServers#Section_5.3.3 .
Consistent asymetric latency on monitoring?
Although the implementation is Cisco-specific, this feels more appropriate for NANOG. We've started rolling out a state-wide monitoring system based on Cisco's IP SLA feature set. Out of 5 sites deployed so far (different locations, different providers), we are consistently seeing one-way latency mirror the opposite direction. As source-destination latency goes up, destination-source latency goes down and vice versa. Myself and the monitoring team have ripped apart the OIDs, IP SLA configuration, and monitoring system. We've also built an ad-hoc system to compare the results. It's still consistent behavior. It's not a true mirror; there is definitely variation between the data collection, but at the 10,000 foot level, there is an obvious and consistent mirror to the data. The network topology is independant service providers all providing backhaul to a local ethernet exchange. Has anybody seen this type of behavior? We are solidly convinced that we are using the proper OIDs and making the proper transformations of the data. The two remaining causes appear to be either natural behavior of the links and/or artifact in the IP SLA mechanism. Any ideas? Thanks!
Re: Consistent asymetric latency on monitoring?
Resent, since I responded from the wrong address: --- The basic operation of IP SLA is as surmised; payload with timestamps and other telemetry data is sent to a 'responder' which manipulates the payload, including adding its own timestamps, and returns the altered payload. I had to do a mental walk-through, but I think I see how drift can cause this. I'm going to generate some artificial data, graph it, and see if it matches the general waveshape I'm seeing. I purposefully have the traffic generators ntp syncing against the responders. I thought that would keep the clocks more closely in sync. I don't necessarily care if the time is 'right', just that it's the same. What kind of difference should I expect if I sync both generators and responders against the same source, or not sync the responder? I'm thinking that having one source with constant drift may be better than both devices trying to walk/correct the time. Thanks for the input! On Wed, Oct 21, 2009 at 8:01 PM, Rick Ernst er...@shreddedmail.com wrote: Resent, since I responded from the wrong address: --- The basic operation of IP SLA is as surmised; payload with timestamps and other telemetry data is sent to a 'responder' which manipulates the payload, including adding its own timestamps, and returns the altered payload. I had to do a mental walk-through, but I think I see how drift can cause this. I'm going to generate some artificial data, graph it, and see if it matches the general waveshape I'm seeing. I purposefully have the traffic generators ntp syncing against the responders. I thought that would keep the clocks more closely in sync. I don't necessarily care if the time is 'right', just that it's the same. What kind of difference should I expect if I sync both generators and responders against the same source, or not sync the responder? I'm thinking that having one source with constant drift may be better than both devices trying to walk/correct the time. Thanks for the input! On Wed, Oct 21, 2009 at 7:55 PM, Rick Ernst er...@shreddedmail.comwrote: The basic operation of IP SLA is as surmised; payload with timestamps and other telemetry data is sent to a 'responder' which manipulates the payload, including adding its own timestamps, and returns the altered payload. I had to do a mental walk-through, but I think I see how drift can cause this. I'm going to generate some artificial data, graph it, and see if it matches the general waveshape I'm seeing. I purposefully have the traffic generators ntp syncing against the responders. I thought that would keep the clocks more closely in sync. I don't necessarily care if the time is 'right', just that it's the same. What kind of difference should I expect if I sync both generators and responders against the same source, or not sync the responder? I'm thinking that having one source with constant drift may be better than both devices trying to walk/correct the time. Thanks for the input! On Wednesday, October 21, 2009, Nathan Ward na...@daork.net wrote: On 22/10/2009, at 2:31 PM, Perry Lorier wrote: I assume this product works by having a packet with a timestamp sent from the source to the destination where it is timestamped again and either sent back, or another packet is sent in the other direction. The difference between the two timestamps gives you the latency in that direction. I believe a packet is sent, and the target router responds with a timestamp. But yeah, timestamps are being compared. I'm with Perry though - sounds like your clocks are drifting. -- Nathan Ward
Need help with performance troubleshooting
Starting about a week ago, I've had sporadic reports of slow uploads (hundreds of kbs, has been 10s of mbs) born out by multiple speed test sites and application results and also duplicated internally. Downloads are 50Mbs as expected (OC-3 and GigE uplinks to ATT/UUNET/Level3/Sprint/Qwest, etc). It feels like the commonality is Seattle, but I haven't been able to find anything conclusive. I'm also not seeing anything interesting in my network as far as CPU, utiliation, interface errors, etc. Hopefully not DoSing myself; I'd like to get some external visibility from the other direction; can I get results against our speedtest server ( http://speedtest.easystreet.com) along with traceroute results and geographic origin of the test? Note that traceroute won't make it all the way through due to some RFC addressing and firewall rules. Thanks, Rick
Re: Public/testing 4to6 gateway?
Pedantry is not necessarily a bad thing, especially when the student doesn't know the right questions to ask. :) 6in4 is what I was looking for. Thanks, On Mon, Jul 13, 2009 at 6:05 PM, Nathan Ward na...@daork.net wrote: On 14/07/2009, at 4:23 AM, Rick Ernst wrote: Either they don't exist, or my Google-fu is particularly bad this morning. I'm trying to get my toes wet with IPv6. I've established an internal 6to4/4to6 tunnel. I'd also like to have a testbed for access to public v6 sites. I'm also trying to find some clue at my upstreams, but figured I'd ask here as well. Are there any 4to6 gateway available? I have assigned v6 space. Because I'm pedantic, 6to4 and 6in4 are two different things. It sounds like you want 6in4. They use the same encapsulation, but 6to4 has specific magic in how the outer IPv4 destination is built, taken from the inner IPv6 destination address. 6over4 is different again. I think someone wrote a draft explaining this a while back.. not sure where or what it was called. -- Nathan Ward
Public/testing 4to6 gateway?
Either they don't exist, or my Google-fu is particularly bad this morning. I'm trying to get my toes wet with IPv6. I've established an internal 6to4/4to6 tunnel. I'd also like to have a testbed for access to public v6 sites. I'm also trying to find some clue at my upstreams, but figured I'd ask here as well. Are there any 4to6 gateway available? I have assigned v6 space. Thanks,
Re: Public/testing 4to6 gateway?
Multiple responses of tunnelbroker.net. Couldn't have been any easier to setup and get going. Thanks! On Mon, Jul 13, 2009 at 9:31 AM, Chad Burnham cburn...@du.edu wrote: Rick, I use this one: http://www.tunnelbroker.net/ Free! Chad -Original Message- From: Rick Ernst [mailto:na...@shreddedmail.com] Sent: Monday, July 13, 2009 10:24 AM To: NANOG Subject: Public/testing 4to6 gateway? Either they don't exist, or my Google-fu is particularly bad this morning. I'm trying to get my toes wet with IPv6. I've established an internal 6to4/4to6 tunnel. I'd also like to have a testbed for access to public v6 sites. I'm also trying to find some clue at my upstreams, but figured I'd ask here as well. Are there any 4to6 gateway available? I have assigned v6 space. Thanks,
Re: Gigabit speed test anybody?
Thanks to multiple private/public responses. I was able to get an iperf test and also a close mirror for a DVD iso. Time to put live traffic on it and see what happens. On Wed, March 25, 2009 11:05, Rick Ernst wrote: Resent from my subscribed address. Hopefully this isn't a dupe to anybody. --- I'm working on turning up our first GigE connection (400mbs CIR) and the various online speedtests I'm aware of choke after about 100Mbs or so. Does anybody know of testing sites that can handle higher bandwidth, or have an ftp host or similar to test against? I'm connected to Level3, backhauled to Seattle, WA. Thanks, Rick
Gigabit speed test anybody?
Resent from my subscribed address. Hopefully this isn't a dupe to anybody. --- I'm working on turning up our first GigE connection (400mbs CIR) and the various online speedtests I'm aware of choke after about 100Mbs or so. Does anybody know of testing sites that can handle higher bandwidth, or have an ftp host or similar to test against? I'm connected to Level3, backhauled to Seattle, WA. Thanks, Rick
Re: Gigabit speed test anybody?
Azher, Thanks for the link. I don't currently have a Linux box I can stick on the network, but I'm trying to get one built. I'm also working with somebody in Seattle for file transfer testing. Thanks, Rick On Wed, March 25, 2009 12:10, Azher Mughal wrote: You can try: http://www.measurementlab.net/measurement-lab-tools#ndt -Azher Rick Ernst wrote: Resent from my subscribed address. Hopefully this isn't a dupe to anybody. --- I'm working on turning up our first GigE connection (400mbs CIR) and the various online speedtests I'm aware of choke after about 100Mbs or so. Does anybody know of testing sites that can handle higher bandwidth, or have an ftp host or similar to test against? I'm connected to Level3, backhauled to Seattle, WA. Thanks, Rick
RE: Gigabit speed test anybody?
Yup. I use iperf for point-to-point testing, but this is an access connection which is why I'm looking more for some kind of test host on Level3 in Seattle rather than a speed test site per se. Rick On Wed, March 25, 2009 12:35, Bill Blackford wrote: Rick. The speedtests are only as good as the hosts they're hosted on and the path by which you reach them. I use iperf on each end of a link that I'm turning up. I put Linux hosts at both endpoints, but I believe iperf comes in a windows flavor too. -b From: Rick Ernst [er...@easystreet.com] Sent: Wednesday, March 25, 2009 11:05 AM To: nanog@nanog.org Subject: Gigabit speed test anybody? Resent from my subscribed address. Hopefully this isn't a dupe to anybody. --- I'm working on turning up our first GigE connection (400mbs CIR) and the various online speedtests I'm aware of choke after about 100Mbs or so. Does anybody know of testing sites that can handle higher bandwidth, or have an ftp host or similar to test against? I'm connected to Level3, backhauled to Seattle, WA. Thanks, Rick
UDP DoS mitigation?
We've had an increasing rate of DoS attacks that spew tens-of-thousands of small UDP packets to a destination on our network. We are getting roughly 2x our entire normal pps across all providers through one interface, or about 4x normal through the individual interface. The Cisco 7206VXR/NPE-G1 CPU melts (95% load vs 15% average, 20% normal peak) when this hits. I'm using CEF and ip-route-cache flow on the outside interface. Unicast RPF is also enabled on the interface. Unicast RPF in conjunction with a BGP black-hole generator handles TCP attacks fairly well. Two questions: - Are there any knobs I should be turning in the Cisco config to help with mitigate this? - Are there any platforms that deal with high PPS/small packet more gracefully? We are looking at a network refresh and aren't locked into Cisco as a vendor (although our current IP network consists entirely of Cisco gear). Our current aggregate (all providers, in- plus out-bound) bandwidth is ~500Mbs, but projected growth is 1Gbs within the year. Thanks, Rick
Re: UDP DoS mitigation?
Replying to my own since there are currently about a dozen responses.
- Hardware/ASIC routers are a consistent response. We are currently
evaluating Juniper for other reasons, but I'll add DoS mitigation to
mix.
- Upstream involvement: We get transit from 701, 1239, etc. I've had
mixed results getting timely responses from our upstreams. It's useful
for long-term issues, but I need as much local and timely control as I
can get.
- I'm not having a problem with pipe bandwidth, but high pps.
- uRPF and RTBH helped internally, but anything passing through that
upstream connection was impacted.
- This instance was a DoS, not DDoS. Single source and destination, but
the source (assuming no spoofing) was in Italy. Turning off netflow
seemed to help, but the attack itself stopped at about the same time.
Also, thanks for the offers of individual help in mitigation, although I'd
be concerned that Hey, can somebody block traffic {from} or {to}? would
be an interesting experiment in a socially-engineered DoS.
Finally, there were some suggestions S/RTBH. RTBH I get, but my
Google-fu is weak on S/RTBH. Details?
Thanks,
Rick
On Fri, December 12, 2008 10:15, Rick Ernst wrote:
We've had an increasing rate of DoS attacks that spew tens-of-thousands of
small UDP packets to a destination on our network. We are getting roughly
2x our entire normal pps across all providers through one interface, or
about 4x normal through the individual interface. The Cisco
7206VXR/NPE-G1 CPU melts (95% load vs 15% average, 20% normal peak) when
this hits.
I'm using CEF and ip-route-cache flow on the outside interface. Unicast
RPF is also enabled on the interface. Unicast RPF in conjunction with a
BGP black-hole generator handles TCP attacks fairly well.
Two questions:
- Are there any knobs I should be turning in the Cisco config to help with
mitigate this?
- Are there any platforms that deal with high PPS/small packet more
gracefully?
We are looking at a network refresh and aren't locked into Cisco as a
vendor (although our current IP network consists entirely of Cisco gear).
Our current aggregate (all providers, in- plus out-bound) bandwidth is
~500Mbs, but projected growth is 1Gbs within the year.
Thanks,
Rick
