RE: dark fiber and sfp distance limitations
If you only want 1gig, then if the SP provides it, won't it be cheaper to simply get a 1gig circuit from them that hands off to you on a GigE port rather than pay for all the various higher spec equipment that you'd otherwise require? Paul. -Original Message- From: Kevin Hodle [mailto:kevin.ho...@gmail.com] Sent: 02 January 2010 23:36 To: Mike Cc: nanog@nanog.org Subject: Re: dark fiber and sfp distance limitations On Fri, Jan 1, 2010 at 4:52 PM, Mike mike-na...@tiedyenetworks.com wrote: I am looking at the possibility of leasing a ~70 mile run of fiber. I don't have access to any mid point section for regeneration purposes, and so I am wondering what the chances that a 120km rated SFP would be able to light the path and provide stable connectivity. There are a lot of unknowns including # of splices, condition of the cable, or the actual dispersion index or other properties (until we actually get closer to leasing it). Its spare telco fibers in the same cable binder they are using interoffice transport, but there are regen huts along the way so it works for them but may not for us, and 'finding out' is potentially expensive. How would someone experienced go about determining the feasibillity of this concept and what options might there be? Replies online or off would be appreciated. I second the recommendation that you request OTDR traces from whomever you are leasing the glass from, and further request traces for each strand in *both* directions (a end to z end, z end to a end) at multiple wavelengths, say 1530nm-1640nm at a maximum of 200GHz wavelength spacing to properly identify potential problem locations in the future when you want to build out a 10GE metro DWDM solution (You really do want to know about that old mechanical splice 20km into your run, etc). An OTDR will provide you with granular loss/gain event details for your entire span, while a power meter/light source will only tell you your overall span loss. While your fiber provider may not pony up OTDR results until after you've executed the contract, they should be able to give you a rough estimate of the total loss (in dB for a 1550nm signal) for the span you are looking at leasing, and you can build provisions into your contract that enforce an absolute maximum loss on the span, in which case the provider will be forced to take necessary actions to replace old poorly executed splices with fusion splices, isolate and correct bends, etc. As most have pointed out - EDFA should not be required for a 1GE single channel solution, and probably would not be required for a simple 1GE CWDM setup either. Once you graduate to an active 10GE DWDM solution EDFA's will be more of a necessity (possibly with dispersion compensation, depending on your vendor this may be an entirely separate shelf module or may be build into the amp card). The addition of EDFA's in a multi-channel solution also adds complexity (if you want to build a scalable/cost effective solution). Most EDFA's have a maximum and minimum per-channel input power, and ideally you would want to have each channel near the same power level before hitting the EDFA. Depending on your gear, topological complexity, etc this may require the use of an optical spectrum analyzer to verify individual channel power levels so the correct amount of attenuation can be added to each channel before it hits the EDFA, however for a single point to point span this will probably not be a concern. -- Cheers, Kevin Kevin Hodle | http://www.linkedin.com/in/kevinhodle PGP Key ID | fingerprint 0x803F24BE | 1094 FB06 837F 2FAB C86B E4BE 4680 3679 803F 24BE Elegance is not a dispensable luxury but a factor that decides between success and failure. -Edsgar Dijkstra For more information about the Viatel Group, please visit www.viatel.com VTL (UK) Limited Registered in England and Wales Registered Address: Inbucon House, Wick Road, Egham, Surrey TW20 0HR Company Registration No: 04287100 VAT Registration Number: 781 4991 88 THIS MESSAGE IS INTENDED ONLY FOR THE USE OF THE INTENDED RECIPIENT TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering the message to the intended recipient, you are notified that any dissemination, distribution or copying of this e-mail is prohibited, and you should delete this e-mail from your system. This message has been scanned for viruses and spam by Viatel MailControl - www.viatel.com
Re: Does anyone have the Cyber Storm II Exercise Report?
On Jan 3, 2010, at 4:11 PM, Eric Brunner-Williams wrote: I'm sure someone must, but google as I have I only find fact sheets (marcom collateral) and reports to Congress. Thanks in advance! Eric These are restricted to those that have signed a Trusted Agent Agreement for participation in the current Cyberstorm event. Ie: if you did CS1, you get access to stuff from CS1. If you did a CS2 TAA, you get access to stuff from CS2. This is to help keep the private data private. If you are interested in participating, you can email me the following info which I will forward to the team that runs the event: Company Name: Contact Name/Info: It's an imperfect system (as are many things) but CS3 is scheduled for fall 2010. The mid-term planning conference is coming up later this month. It's not too late to get involved! - Jared
Re: InterNAP FCP (again?)
I feel your pain, as I'm a little frustrated by the stance that Internap is taking on the FCP. We've used them for many years, back when there were numerous competitors to choose from. However, now that they are pretty much the only major player they seem to care less about the results and the customer. Like Ric, you can contact me off-line for details. Tom Sands Chief Network Engineer Rackspace Hosting (210)312-4391 Ric Moseley wrote: Call me offline. Ric. 214-442-0555 -Original Message- From: Michael J McCafferty [mailto:m...@m5computersecurity.com] Sent: Wednesday, December 30, 2009 2:59 PM To: nanog Subject: InterNAP FCP (again?) All, I know this has been discussed to some degree before and I have searched the archives. However is it seems in my previous posts to this list about anything, the truly useful replies are the private replies ones that don't make it to this list. We are considering the InterNAP Flow Control Platform. Currently we have 3 transit providers but are adding one or two more and will also be adding a connection to the Any2 exchange at One Wilshire. The price is manageable, the reporting seems quite useful, but I haven't seen it actually in action on my network. If it works as claimed for managing commit levels, performance, etc. then I expect we'd be very happy. The problem is that InterNAP does not want to do any acceptance testing... all sales are final... and in my research on the web, I see a few companies that have implemented the FCP and have either removed it or switched to Avaya CNA (yes, I know it's going away). Since InterNAP has pulled way from any kind of happiness guarantee, I'd very much like to hear from actual users of the FCP, happy and unhappy, to help me feel better about signing the PO. Does it do what it says it does for performance and managing commit levels? Do you feel it was worth the integration and money? Are you happy with it? What size and shape is the network you used it on? Do you have any additional thoughts to share regarding the FCP? Thanks! Mike Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated.
Re: Consumer-grade dual-homed connectivity options?
Most of the SOHO router vendors (Netgear, Linksys, etc) have a model targeted at this application. When this class of dual homed router first came out several years ago, they were notoriously unreliable, but I would hope they work better by now. A search on the term ping based routing should give you insight into the current state of affairs, although it will probably take some work because there is no standard terminology to describe the facility, and most implementations no longer rely on ping to do the job of detecting link status. A few limitations to keep in mind: 1 - These routers are targeted at home users, are cheap, and you don't get what you don't pay for. 2 - The job can be done using real routers (Cisco, Juniper, etc), but setup requires work and getting a solution that actually works can be tricky. 3 - Be wary of any advice that you get from anyone who has not actually done it on the box in question! There are many ways a solution which should work will fail miserably. For example, when I looked at this problem a few years ago for a client, the SOHO routers tended to lock up and require a power cycle every few days while Cisco IOS routers would not clear the NAT table when a link failed soft and tended to stop testing a link once it failed, requiring manual recovery. Good luck and have fun! -- Vincent C Jones Networking Unlimited, Inc. www.networkingunlimited.com On Sat, 2010-01-02 at 18:14 -0500, Steven King wrote: You would need at least one router for this. Personally I would connect both DSL modems into a small Cisco router or multi-layer switch. Use that router as the default gateways for each LAN and have two static routes as the default gateway on the router to specify each DSL line. This would allow for load balancing each connection. Although, you run into the issue of needing PAT on both lines. This wouldn't be complex, but would need to be handled by the router as well. I am not sure about asymmetric paths though. Depending on the device, it may handle this differently, and there is no guarantee that the source of your traffic will be from the same connection all the time to the destination. This would cause connectivity issues. There really is no elegant solution to that without having a full routing table of the Internet and 2 separate providers. Others on this list may have a solution to that issue off the top of their heads, or have done this themselves. On 1/2/10 5:48 PM, Scott Weeks wrote: --- paul.w.benn...@gmail.com wrote: From: Paul Bennett paul.w.benn...@gmail.com At home, I currently run two DSL lines. Right now, we just have two separate LANs, one connected to each line, with my wife's devices attached to one, and my devices attached to the other. For a while now, I've been thinking about setting up a load-balancing routing solution to give both of us access to both lines. --- Maybe www.xincom.com/products.php will work? scott
Re: More ASN collissions
In a message written on Thu, Dec 10, 2009 at 01:35:16PM -0500, Jared Mauch wrote: As always, good research by renesys. http://www.renesys.com/blog/2009/12/bonjour-yall-asn-split-persona.shtml http://blog.isc.org/2010/01/asn-collisions-and-human-error.html -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ pgplRS0fENoWR.pgp Description: PGP signature
RE: Experiences with Comcast Ethernet/Transit service
On Mon, 4 Jan 2010, Holmes,David A wrote: I do not know of Comcast's Ethernet services specifically, but a general problem with carrier Ethernet services that are based upon the Metro Ethernet Forum (MEF) is that PIM-snooping is not implemented for multicast traffic. The absence of PIM-snooping results in the carrier's Ethernet service operating like a 1990's style Ethernet hub in which (S,G) multicast packets are incorrectly flooded out all user ports. Not implemented because it's not in the MEF specs or not implemented because of carrier operational practice? Antonio Querubin 808-545-5282 x3003 e-mail/xmpp: t...@lava.net
geolocation comparison
CAIDA plans to conduct a comparison of geolcocation tools for determining the location of Internet Protocal (IP) address (and other identifiers) in the summer of 2010. At this time we wish to receive feedback from interested parties on input to the comparison survey, whether they wish to participate and in what capacity. * What is your primary interest in geolocation services? Legal, academic, content localization, disaster preparedness, regulation, ... * Can you provide ground truth geolocation information for your organization? * Do you have suggested metrics, questions, or tests to include in the comparisons? * If you provide a geolocation service, under what conditions would you be willing to participate? * If you use a Geolocation service(s), which do you use and in what way? * Which companies do you consider to be the primary providers of geolocation services at this time? Please send any comments to: geocompare-i...@caida.org Details can be found: http://www.caida.org/projects/cybersecurity/geolocation/ -- Be always at war with your vices, at peace with your neighbors, and let each new year find you a better man. - Benjamin Franklin
D/DoS mitigation hardware/software needed.
Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. My current system is a home-grown NetFlow parser that spits out syslog to our NOC to investigate potential attacks and manually enter them into our RTBH. Any suggestions other than Arbor? Any other mechanisms being used? My idea is to quash the immediate problem and work additional mitigation with upstreams if needed. I could probably add some automation to my NetFlow/RTBH setup, but I still need to worry about false-positives. I'd rather somebody else do the hard work of finding the various edge-cases. Thanks, Rick
Re: D/DoS mitigation hardware/software needed.
We have substantial direct experience with both RioRey and IntruGuard. RR is more plug and play while IG is more robust but both are great. Use a robust firewall such as a Netscreen in front of your mitigation tool. Best regards, Jeff On Mon, Jan 4, 2010 at 4:19 PM, Rick Ernst na...@shreddedmail.com wrote: Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. My current system is a home-grown NetFlow parser that spits out syslog to our NOC to investigate potential attacks and manually enter them into our RTBH. Any suggestions other than Arbor? Any other mechanisms being used? My idea is to quash the immediate problem and work additional mitigation with upstreams if needed. I could probably add some automation to my NetFlow/RTBH setup, but I still need to worry about false-positives. I'd rather somebody else do the hard work of finding the various edge-cases. Thanks, Rick -- Jeffrey Lyon, Leadership Team jeffrey.l...@blacklotus.net | http://www.blacklotus.net Black Lotus Communications of The IRC Company, Inc. Follow us on Twitter at http://twitter.com/ddosprotection to find out about news, promotions, and (gasp!) system outages which are updated in real time. Platinum sponsor of HostingCon 2010. Come to Austin, TX on July 19 - 21 to find out how to protect your booty.
RE: D/DoS mitigation hardware/software needed.
Rick, If you pass me your contact info I can forward it to our Arbor Sales guy who can get in touch with you. I been pretty impressed by Arbor so far. Thanks, Raj Singh -Original Message- From: Rick Ernst [mailto:na...@shreddedmail.com] Sent: Monday, January 04, 2010 1:20 PM To: NANOG Subject: D/DoS mitigation hardware/software needed. Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. My current system is a home-grown NetFlow parser that spits out syslog to our NOC to investigate potential attacks and manually enter them into our RTBH. Any suggestions other than Arbor? Any other mechanisms being used? My idea is to quash the immediate problem and work additional mitigation with upstreams if needed. I could probably add some automation to my NetFlow/RTBH setup, but I still need to worry about false-positives. I'd rather somebody else do the hard work of finding the various edge-cases. Thanks, Rick
Re: D/DoS mitigation hardware/software needed.
Several responses already, and Arbor has poked their head up. I'm going to start there and keep the other suggestions at-hand. Thanks, On Mon, Jan 4, 2010 at 1:19 PM, Rick Ernst na...@shreddedmail.com wrote: Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. My current system is a home-grown NetFlow parser that spits out syslog to our NOC to investigate potential attacks and manually enter them into our RTBH. Any suggestions other than Arbor? Any other mechanisms being used? My idea is to quash the immediate problem and work additional mitigation with upstreams if needed. I could probably add some automation to my NetFlow/RTBH setup, but I still need to worry about false-positives. I'd rather somebody else do the hard work of finding the various edge-cases. Thanks, Rick
ITU G.992.5 Annex M - ADSL2+M Questions
I've been looking up information on the Annex M Standard today and am unable to find any ISPs in the US offering this. Can anyone tell me if there are providers in the US using the Annex M standards and increased upstream with it, or if not is there a good reason why its not being done yet? Thanks! :Luke Marrott
RE: Experiences with Comcast Ethernet/Transit service
PIM-snooping is not in the MEF specs, but should be if multicast is to work properly over a carrier's Ethernet service. Regardless of the specs, RFPs and other user requirements for Ethernet services should include a must have clause requiring PIM-snooping functionality. -Original Message- From: Antonio Querubin [mailto:t...@lava.net] Sent: Monday, January 04, 2010 12:13 PM To: Holmes,David A Cc: Brandon Galbraith; nanog@nanog.org Subject: RE: Experiences with Comcast Ethernet/Transit service On Mon, 4 Jan 2010, Holmes,David A wrote: I do not know of Comcast's Ethernet services specifically, but a general problem with carrier Ethernet services that are based upon the Metro Ethernet Forum (MEF) is that PIM-snooping is not implemented for multicast traffic. The absence of PIM-snooping results in the carrier's Ethernet service operating like a 1990's style Ethernet hub in which (S,G) multicast packets are incorrectly flooded out all user ports. Not implemented because it's not in the MEF specs or not implemented because of carrier operational practice? Antonio Querubin 808-545-5282 x3003 e-mail/xmpp: t...@lava.net
Re: D/DoS mitigation hardware/software needed.
Ask them if they'd come down to $10 - 20k for a full featured model and they might make two sales, although I doubt it unfortunately. Best regards, Jeff On Mon, Jan 4, 2010 at 4:59 PM, Rick Ernst na...@shreddedmail.com wrote: Several responses already, and Arbor has poked their head up. I'm going to start there and keep the other suggestions at-hand. Thanks, On Mon, Jan 4, 2010 at 1:19 PM, Rick Ernst na...@shreddedmail.com wrote: Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. My current system is a home-grown NetFlow parser that spits out syslog to our NOC to investigate potential attacks and manually enter them into our RTBH. Any suggestions other than Arbor? Any other mechanisms being used? My idea is to quash the immediate problem and work additional mitigation with upstreams if needed. I could probably add some automation to my NetFlow/RTBH setup, but I still need to worry about false-positives. I'd rather somebody else do the hard work of finding the various edge-cases. Thanks, Rick -- Jeffrey Lyon, Leadership Team jeffrey.l...@blacklotus.net | http://www.blacklotus.net Black Lotus Communications of The IRC Company, Inc. Follow us on Twitter at http://twitter.com/ddosprotection to find out about news, promotions, and (gasp!) system outages which are updated in real time. Platinum sponsor of HostingCon 2010. Come to Austin, TX on July 19 - 21 to find out how to protect your booty.
Re: Experiences with Comcast Ethernet/Transit service
The Deathstar opt-e-man service says they will knee-cap you at 1Mb/s of multicast. - Jared On Jan 4, 2010, at 4:56 PM, Holmes,David A wrote: PIM-snooping is not in the MEF specs, but should be if multicast is to work properly over a carrier's Ethernet service. Regardless of the specs, RFPs and other user requirements for Ethernet services should include a must have clause requiring PIM-snooping functionality. -Original Message- From: Antonio Querubin [mailto:t...@lava.net] Sent: Monday, January 04, 2010 12:13 PM To: Holmes,David A Cc: Brandon Galbraith; nanog@nanog.org Subject: RE: Experiences with Comcast Ethernet/Transit service On Mon, 4 Jan 2010, Holmes,David A wrote: I do not know of Comcast's Ethernet services specifically, but a general problem with carrier Ethernet services that are based upon the Metro Ethernet Forum (MEF) is that PIM-snooping is not implemented for multicast traffic. The absence of PIM-snooping results in the carrier's Ethernet service operating like a 1990's style Ethernet hub in which (S,G) multicast packets are incorrectly flooded out all user ports. Not implemented because it's not in the MEF specs or not implemented because of carrier operational practice? Antonio Querubin 808-545-5282 x3003 e-mail/xmpp: t...@lava.net
RE: ITU G.992.5 Annex M - ADSL2+M Questions
Hi Luke. We offer it, along with bonded ADSL. We don't do it often but it is very useful sometimes. Regards, John John Souvestre - New Orleans LA -Original Message- From: Luke Marrott [mailto:luke.marr...@gmail.com] Sent: Monday, January 04, 2010 4:03 PM To: nanog@nanog.org Subject: ITU G.992.5 Annex M - ADSL2+M Questions I've been looking up information on the Annex M Standard today and am unable to find any ISPs in the US offering this. Can anyone tell me if there are providers in the US using the Annex M standards and increased upstream with it, or if not is there a good reason why its not being done yet? Thanks! :Luke Marrott
[NANOG-announce] REMINDER: CFP for NANOG 48
Hello Fellow NANOG'ers, As you all should know by now, NANOG 48 is coming up in February. The NANOG Program Committee has been busily recruiting, collecting, and reviewing submissions for the program, however, we still need more content. We have lined up many interesting Tracks and Panels, but need more General Session submissions - to learn more about what you're up to in your networks and what is important to you! The NANOG 48 Call for Presentations is still available at http://www.nanog.org/meetings/nanog48/index.php. The PC will be having a call next Tuesday to review submissions, and would like to have more material to review. So what that means: - YOU: Go think of something you think is important to the NANOG Community - YOU: Write up some slides (PDF format prefered) - YOU: Submit them to the PC tool (https://pc.nanog.org) - WE, THE PC: Tell you how awesome your presentation looks - YOU: Present at NANOG 48 and take all the credit for your awesome talk! Yes, it really is that simple. Do it now! Shamelessly, for the NANOG PC, Tom Daly -- Tom Daly CTO, Dynamic Network Services, Inc. http://dyn.com/ ___ NANOG-announce mailing list nanog-annou...@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog-announce
Re: DNS question, null MX records
On Tue, Dec 15, 2009 at 7:46 AM, Eric J Esslinger eesslin...@fpu-tn.com wrote: So in any case, due to customer privacy concerns we feel we can't do that. If you don't want to handle email for the long-obsolete customer accounts, but just don't want to send that mail to anybody else, it's pretty easy to run a teergrube or other tarpit system to trap any mail addressed to the A-record. These systems basically accept mail v.e..ysl.o...w...l..y so that spammers can waste their time talking to your tarpit instead of to somebody who cares, and so you can trap their IP addresses and potentially block them or use them to support your other spam-blockers if you want. You don't need a high-performance machine because all the users are spammers and you're *trying* to give them bad service. (Some variants, like LaBrea, are used for connection attempts to non-existent machines - they'll send a syn-ack so the attacker thinks he has a successful 3-way handshake, which slows down scanning attacks.) If you do want to accept mail for the long-obsolete customer accounts, so you can give them a proper human-readable rejection message, you may need to customize. It looks like Exim supports that, though I haven't tried it. -- Thanks; Bill Note that this isn't my regular email account - It's still experimental so far. And Google probably logs and indexes everything you send it.
RE: ITU G.992.5 Annex M - ADSL2+M Questions
We offer it, but practically speaking we haven't gotten much higher than 1.5 Mbps on the upstream. Frank -Original Message- From: Luke Marrott [mailto:luke.marr...@gmail.com] Sent: Monday, January 04, 2010 4:03 PM To: nanog@nanog.org Subject: ITU G.992.5 Annex M - ADSL2+M Questions I've been looking up information on the Annex M Standard today and am unable to find any ISPs in the US offering this. Can anyone tell me if there are providers in the US using the Annex M standards and increased upstream with it, or if not is there a good reason why its not being done yet? Thanks! :Luke Marrott
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 4:25 AM, Jeffrey Lyon wrote: Use a robust firewall such as a Netscreen in front of your mitigation tool. Absolutely not - the firewall will fall over due to state-table exhaustion before the mitigation system will. Firewalls (which have no place in front of servers in the first place), load-balancers, and any other stateful devices should be southbound of the mitigation system. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
RE: dark fiber and sfp distance limitations
and to add, OTDR at several wavelengths, just in case you want to do xWDM in the future. Frank -Original Message- From: ML [mailto:m...@kenweb.org] Sent: Friday, January 01, 2010 6:24 PM To: Mike Cc: nanog@nanog.org Subject: Re: dark fiber and sfp distance limitations On 1/1/2010 5:52 PM, Mike wrote: I am looking at the possibility of leasing a ~70 mile run of fiber. I don't have access to any mid point section for regeneration purposes, and so I am wondering what the chances that a 120km rated SFP would be able to light the path and provide stable connectivity. There are a lot of unknowns including # of splices, condition of the cable, or the actual dispersion index or other properties (until we actually get closer to leasing it). Its spare telco fibers in the same cable binder they are using interoffice transport, but there are regen huts along the way so it works for them but may not for us, and 'finding out' is potentially expensive. How would someone experienced go about determining the feasibillity of this concept and what options might there be? Replies online or off would be appreciated. Thanks. Pardon my ignorance in this area but is too much to ask for OTDR data before signing contracts? In addition to data on the make of the fiber if you wanted to do xWDM in the future. NDAs shall be signed of course
Re: D/DoS mitigation hardware/software needed.
Kinda funny you state that Roland. I know of at least two very large carriers that uses Netscreens (and soon SRX's) for their DoS/DDoS mitigation. State table exhaustion on the netscreen platform is something that is very easy to protect against with some protections and hasn't been a problem for years. If you can fill up a session table on a higher end SRX then I would be very very impressed. I would argue that firewalls place is in fact directly infront of servers and load balancers to protect them. On Mon, Jan 4, 2010 at 8:04 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 4:25 AM, Jeffrey Lyon wrote: Use a robust firewall such as a Netscreen in front of your mitigation tool. Absolutely not - the firewall will fall over due to state-table exhaustion before the mitigation system will. Firewalls (which have no place in front of servers in the first place), load-balancers, and any other stateful devices should be southbound of the mitigation system. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
What Roland said, I've seen people do this, no rules in place, still was able to kill the box (firewall) with a single CPU server. -jim On Mon, Jan 4, 2010 at 10:04 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 4:25 AM, Jeffrey Lyon wrote: Use a robust firewall such as a Netscreen in front of your mitigation tool. Absolutely not - the firewall will fall over due to state-table exhaustion before the mitigation system will. Firewalls (which have no place in front of servers in the first place), load-balancers, and any other stateful devices should be southbound of the mitigation system. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
If you want to recreate D/DoS from captures (for testing purposes) you might want to check out: http://www.pcapr.net/dos This lets you validate how your mitigation solutions are holding up. K. On Mon, Jan 4, 2010 at 1:19 PM, Rick Ernst na...@shreddedmail.com wrote: Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. My current system is a home-grown NetFlow parser that spits out syslog to our NOC to investigate potential attacks and manually enter them into our RTBH. Any suggestions other than Arbor? Any other mechanisms being used? My idea is to quash the immediate problem and work additional mitigation with upstreams if needed. I could probably add some automation to my NetFlow/RTBH setup, but I still need to worry about false-positives. I'd rather somebody else do the hard work of finding the various edge-cases. Thanks, Rick
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 9:17 AM, Tim Eberhard wrote: I would argue that firewalls place is in fact directly infront of servers and load balancers to protect them. The very idea of placing a stateful firewall in front of a Web/DNS/email/etc. server, in which *every single incoming packet is unsolicited, and therefore, leaves no state to be inspected in the first place*, is absurd. There is simply no valid argument for doing so. Hardening the OS/apps/services, combined with stateless ACLs in hardware which can handle mpps, is the way to enforce policy. In fact, the idea is such a poor one that one of the major firewall vendors came out with a 'stateful inspection bypass' feature - the idea being that, you buy their 10gb/sec, $100K-plus stateful firewall, stick it in front of servers, and then . . . disable the stateful inspection, heh. ; None of the large, well-known Web properties on the Internet today - at least, the ones which stay up and running, heh - have stateful firewalls in front of them. Including prominent vendors of said stateful firewall solutions. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Tue, Jan 05, 2010, Dobbins, Roland wrote: None of the large, well-known Web properties on the Internet today - at least, the ones which stay up and running, heh - have stateful firewalls in front of them. Including prominent vendors of said stateful firewall solutions. But as you said, they're willing to sell them to you. Then claim that the traffic you're receiving is out of profile. :) (I'm not jaded about this, oh no..) Adrian
Re: D/DoS mitigation hardware/software needed.
We have such a configuration in progress, it works great without any of the issues you're proposing. Jeff On Jan 4, 2010 9:09 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 4:25 AM, Jeffrey Lyon wrote: Use a robust firewall such as a Netscreen in fro... Absolutely not - the firewall will fall over due to state-table exhaustion before the mitigation system will. Firewalls (which have no place in front of servers in the first place), load-balancers, and any other stateful devices should be southbound of the mitigation system. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Tue, Jan 5, 2010 at 8:36 AM, Jeffrey Lyon jeffrey.l...@blacklotus.net wrote: We have such a configuration in progress, it works great without any of the issues you're proposing. So .. this is interesting. The firewall would have to frontend your mail / web / whatever application .. and if something goes beyond the firewall's rated capacity (100k ++ - maybe nearly 150..175k connections per second for a high end firewall), the firewall falls over. And even before that, there's the risk of whatever application you're protecting getting pounded flat if your firewall passes even a small percentage of this traffic. Do you - 1. Have (say) two firewalls in HA config? 2. Back your firewall with routing based measures, S/RTBH, blackhole communities your upstream offers, etc [the standard nspsec bootcamp stuff] 3. Simply back the firewall with a netflow based device? 4. Estimate that the risk of a DDoS that exceeds your firewall's rated capacity is extremely low? [and yes, 150k ++ connections per second ddos is going to be massive, and relatively rare for most people] --srs -- Suresh Ramasubramanian (ops.li...@gmail.com)
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 10:14 AM, Dobbins, Roland wrote: If it's a stateful firewall, and state-tracking is turned on, it's quite possible to craft sufficient pathological traffic which conforms to the firewall policies and yet which leads to state-table inspection. That should read 'state-table exhaustion', apologies for the typo. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 10:06 AM, Jeffrey Lyon wrote: We have such a configuration in progress, it works great without any of the issues you're proposing. Then you aren't testing it to destruction, heh. ; If it's a stateful firewall, and state-tracking is turned on, it's quite possible to craft sufficient pathological traffic which conforms to the firewall policies and yet which leads to state-table inspection. And the stateful firewall serves no purpose in front of servers, in which *every incoming packet* is unsolicited. Far more sensible to enforce policy in stateless ACLs in ASIC-based router/switch hardware. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
Two more options. And for Netflow device - read that to mean Arbor or its competitors. 5 Ditch the stateful firewall and exclusively use a netflow device 6. Outsource to a hosted DDoS mitigation service (Prolexic etc) On Tue, Jan 5, 2010 at 8:43 AM, Suresh Ramasubramanian ops.li...@gmail.com wrote: Do you - 1. Have (say) two firewalls in HA config? 2. Back your firewall with routing based measures, S/RTBH, blackhole communities your upstream offers, etc [the standard nspsec bootcamp stuff] 3. Simply back the firewall with a netflow based device? 4. Estimate that the risk of a DDoS that exceeds your firewall's rated capacity is extremely low? [and yes, 150k ++ connections per second ddos is going to be massive, and relatively rare for most people]
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 10:18 AM, Suresh Ramasubramanian wrote: 5 Ditch the stateful firewall and exclusively use a netflow device NetFlow analysis is very useful for network visibility, and detection/classification/traceback. There are both open-source and commercial NetFlow collection and analysis systems available (full disclosure: I work for a vendor of both NetFlow analysis and DDoS mitigation solutions); however, they don't provide mitigation, which is where S/RTBH, flow-spec, and/or IDMS come into play. PCI DSS iatrogenically *requires* that a 'Web application firewall' be placed in front of Web servers which process credit card information (PCI DSS completely ignores availability, and contains a number of recommendations which are actually harmful from an opsec standpoint). Running mod_security or its equivalent on the front-end Web servers themselves fulfills this requirement without putting a stateful DDoS chokepoint in front of the servers. It's also a really good idea to front Web servers with a tier of caching-only transparent reverse proxies; Squid is a good choice for this, as well as various commercial offerings. WCCPv2 clustering (supported by Squid and several commercial caching/proxying solutions) allows this tier to be scaled horizontally in order to meet capacity demands. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Mon, Jan 4, 2010 at 9:18 PM, jim deleskie deles...@gmail.com wrote: What Roland said, I've seen people do this, no rules in place, still was able to kill the box (firewall) with a single CPU server. not to pile on, but... +1 to roland here as well. I've seen more than enough folks put in a 'firewall' in front of their 'server' (say a mail server) and then watch that die long before the rest of the system did. Now, if you have equipment capable today of doing a few million session creates/second and you feel comfortable that you can keep track of how attacks grow vs your capacity stays the same and move ahead of the curve well enough, then... by all means do as you want :) There's a cost analysis which Roland sidestepped here as well, state-tracking at the rates required is expensive, as compared to relatively simple acls in hardware with no state on the upstream router. Spend where it matters, and make sure you understand where the failure points are that you place into your network. -chris -jim On Mon, Jan 4, 2010 at 10:04 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 4:25 AM, Jeffrey Lyon wrote: Use a robust firewall such as a Netscreen in front of your mitigation tool. Absolutely not - the firewall will fall over due to state-table exhaustion before the mitigation system will. Firewalls (which have no place in front of servers in the first place), load-balancers, and any other stateful devices should be southbound of the mitigation system. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Bonded SDSL (was RE: ITU G.992.5 Annex M - ADSL2+M Questions)
Frank Bulk - iName.com frnk...@iname.com wrote: We offer it, but practically speaking we haven't gotten much higher than 1.5 Mbps on the upstream. Sorry that I'm coming into this thread late (I have just subscribed), but since I see people discussing DSL with beefy upstream, I thought I would be brave and ask: do you esteemed high-end network op folks think that there may be anyone in the world who might be interested in bonded SDSL or not? I have spent the past 5 years of my life learning everything there is to know about SDSL. Don't ask me why, I don't really know the answer to that question myself. I won't waste the bandwidth of this elite list with dirty details of just what I've done with SDSL over the past 5 y, but I'll give a link to an open source project that contains the body of SDSL knowledge amassed over those years: http://ifctfvax.Harhan.ORG/OpenSDSL/ To make the long story short, for most of those years I kept trudging on my project, treating it as an ultra-weird hobby that no one else in the world could possibly have any interest in. That persisted until 2009 when my project got noticed by two fairly major North American DSL network operators. (Well, one very major and one semi-major, but I'll spare the names.) Both of those had contacted me via my Open SDSL Connectivity Project expressing interest in SDSL bonding. Both companies were telling me how much interest they had in SDSL bonding, how much it would help their business to be able to offer bonded SDSL services at 3 or 6 Mbps, how many customers they would be able to sign up for these services, etc. But when I asked them to back their verbally-expressed interest with the tiniest amount of money or even no money at all but a letter of intent which I could show to SBA etc, they both went silent. We've been playing a game of cat-and-mouse ever since. As far as I could understand the existing situation is that the SDSL infrastructure already deployed en masse by the major North American DSL network operators already has the capability to serve out bonded SDSL circuits, bonding either in the DSLAM or somewhere upstream of it, using MLPPP, Multilink Frame Relay or whatever else one can think of, but the problem is with CPE. Apparently bonding-capable multiport SDSL CPE devices are quite scarce. Considering everything I've done with SDSL over the past 5 y, I believe I have a right to say with confidence that I am more than capable of designing and building a bonding-capable multiport SDSL CPE device for any existing SDSL flavor with any desired number of ports (2, 4 or whatever). But what I don't know, and what I'm asking this highly esteemed list for advice with, is this question: is there anyone at all in the world who might have a real serious interest in such a thing? If there is someone in the world who would truly appreciate having a bonded SDSL solution, I would be delighted to work on developing such a thing. I would see it as a service to humanity whereby more use would be made out of existing copper infrastructure in the ground instead of having to dig more ditches to bury more fiber or whatever. But if there is no one in the world who would be interested in bonded SDSL (or at least interested enough to invest one dime into development), then why bother... MS
Re: D/DoS mitigation hardware/software needed.
1. We have multiple nodes conducting DDoS scrubbing, one failing would not be catastrophic. 2. Indeed. 3. Sort of, such devices are downstream for extremely valid reasons I won't get into now. 4. Indeed, were equipped to handle substantially higher than 150kpps. I'm sure Arbor is really neat but I disagree that any DDoS appliance is a standalone solution. I don't expect an employee of the vendor themselves to attest to this though. Best regards, Jeff Best regards, Jeff On Jan 4, 2010 10:14 PM, Suresh Ramasubramanian ops.li...@gmail.com wrote: On Tue, Jan 5, 2010 at 8:36 AM, Jeffrey Lyon jeffrey.l...@blacklotus.net wrote: We have such a c... So .. this is interesting. The firewall would have to frontend your mail / web / whatever application .. and if something goes beyond the firewall's rated capacity (100k ++ - maybe nearly 150..175k connections per second for a high end firewall), the firewall falls over. And even before that, there's the risk of whatever application you're protecting getting pounded flat if your firewall passes even a small percentage of this traffic. Do you - 1. Have (say) two firewalls in HA config? 2. Back your firewall with routing based measures, S/RTBH, blackhole communities your upstream offers, etc [the standard nspsec bootcamp stuff] 3. Simply back the firewall with a netflow based device? 4. Estimate that the risk of a DDoS that exceeds your firewall's rated capacity is extremely low? [and yes, 150k ++ connections per second ddos is going to be massive, and relatively rare for most people] --srs -- Suresh Ramasubramanian (ops.li...@gmail.com)
Re: D/DoS mitigation hardware/software needed.
A lot of this has to do with scaling the environment. I've had plenty of asa's and even netscreens fall over from state-table and session limitations. I've also seen a hosts fill up the connection table prior to a firewall being affected. I'm not familiar with the specs and anyone can chime in, but the newer variety of SRX's, I believe implement more in hardware as line-rate routers do. A layered approach is useful as well. If the source can be identified via netflow and null routed before it gets to the firewall and content layer, then all the better. This is much more tricky with DDOS so having robust firewall that can eat traffic is helpful. My 3 cents -b On Mon, Jan 4, 2010 at 7:35 PM, Christopher Morrow morrowc.li...@gmail.comwrote: On Mon, Jan 4, 2010 at 9:18 PM, jim deleskie deles...@gmail.com wrote: What Roland said, I've seen people do this, no rules in place, still was able to kill the box (firewall) with a single CPU server. not to pile on, but... +1 to roland here as well. I've seen more than enough folks put in a 'firewall' in front of their 'server' (say a mail server) and then watch that die long before the rest of the system did. Now, if you have equipment capable today of doing a few million session creates/second and you feel comfortable that you can keep track of how attacks grow vs your capacity stays the same and move ahead of the curve well enough, then... by all means do as you want :) There's a cost analysis which Roland sidestepped here as well, state-tracking at the rates required is expensive, as compared to relatively simple acls in hardware with no state on the upstream router. Spend where it matters, and make sure you understand where the failure points are that you place into your network. -chris -jim On Mon, Jan 4, 2010 at 10:04 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 4:25 AM, Jeffrey Lyon wrote: Use a robust firewall such as a Netscreen in front of your mitigation tool. Absolutely not - the firewall will fall over due to state-table exhaustion before the mitigation system will. Firewalls (which have no place in front of servers in the first place), load-balancers, and any other stateful devices should be southbound of the mitigation system. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken -- Bill Blackford Network Engineer
Re: D/DoS mitigation hardware/software needed.
With these safeguards in place - and with flow devices being part of the mix somewhere .. what you propose is quite reasonable. There's still the question of whether an application that receives a lot of new / untrusted traffic - a mail or web server - would benefit from having a stateful firewall in front .. Roland seems to think not. --srs On Tue, Jan 5, 2010 at 9:35 AM, Jeffrey Lyon jeffrey.l...@blacklotus.net wrote: 1. We have multiple nodes conducting DDoS scrubbing, one failing would not be catastrophic. 2. Indeed. 3. Sort of, such devices are downstream for extremely valid reasons I won't get into now. 4. Indeed, were equipped to handle substantially higher than 150kpps. I'm sure Arbor is really neat but I disagree that any DDoS appliance is a standalone solution. I don't expect an employee of the vendor themselves to attest to this though. -- Suresh Ramasubramanian (ops.li...@gmail.com)
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 11:05 AM, Jeffrey Lyon wrote: I'm sure Arbor is really neat but I disagree that any DDoS appliance is a standalone solution. I disagree with this proposition, too. S/RTBH and/or flow-spec are great DDoS mitigation tools which don't require any further investment beyond the network infrastructure an operator has already purchased and deployed. These should be the first mitigation tools anyone deploys; in many cases, they're all that's needed. I don't expect an employee of the vendor themselves to attest to this though. Wrong again. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Mon, Jan 4, 2010 at 11:20 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 11:05 AM, Jeffrey Lyon wrote: I'm sure Arbor is really neat but I disagree that any DDoS appliance is a standalone solution. I disagree with this proposition, too. S/RTBH and/or flow-spec are great DDoS mitigation tools which don't require any further investment beyond the network infrastructure an operator has already purchased and deployed. These should be the first mitigation tools anyone deploys; in many cases, they're all that's needed. Is it fair to say that most folks in this thread believe there is not 'one size fits all', and that there are quite a few tools available in the security/networking toolbox? Some of these are outlined in past nanog tutorials: http://www.nanog.org/meetings/nanog23/presentations/greene.pdf http://www.nanog.org/meetings/nanog26/presentations/ispsecure.pdf http://www.nanog.org/meetings/nanog28/presentations/sink.pdf http://www.nanog.org/meetings/nanog36/presentations/greene.ppt Sorry to pick on barry here, but he's got a few preso's up from past NANOG's that cover this topic pretty well. All of them talk about a set of tools a network operator should be familiar with, with escalating costs (dollars and packet-loss/collateral damage), and some cut/pasteable configlets even. I don't expect an employee of the vendor themselves to attest to this though. Wrong again. eh, roland's always happy to bash on employers :) but, he's got some solid standing on this set of points. Again, if you know what you're doing then feel free to go off and do it, but at first blush there are LOTS of people putting 'servers' out on the 'public network' behind devices whoafully prepared to deal with 'real world traffic demands', so instead of making the same mistake, perhaps learning from some experience would be in order? The original poster seemed to be asking about appliance based solutions, so your pointed remarks about Roland aside he was actually answering the question. I wonder if Stefan Fouant would offer some of his experience with 'not arbor' vendor solutions to be used when other techniques come up short? (note I think Roland may have been party to some of the presenations I linked in this... I certainly was for one of them at least, in case that matters.) -Chris ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 11:41 AM, Christopher Morrow wrote: (note I think Roland may have been party to some of the presenations I linked in this... Yes, sir, and thanks for posting those links! You and Barry and Tim Battles and Sean Donelan and Danny McPherson and Don Smith and Steve Bellovin and Jared Mauch and John Kristoff and a lot of other folks too numerous to mention here have contributed greatly to advancing the state of the art and generating the body of real-world opsec material out there, which it behooves all network operators to study and take into consideration in their own contexts. I also highly recommend this book by Dave Smith and Gregg Schudel of Cisco - it's the best (and only!) book on real-world opsec out there, available in dead-tree, Kindle, and Adobe Reader formats: http://www.amazon.com/Router-Security-Strategies-Securing-Network/dp/1587053365/ref=sr_1_1?ie=UTF8s=booksqid=1262667257sr=8-1 [Full disclosure; I'm cited in the book, but received and continue to receive no renumeration of any kind due to same.] Here's another preso on this same topic which may be of interest: http://files.me.com/roland.dobbins/k54qkv --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Tue, Jan 5, 2010 at 10:35 AM, Rick Ernst na...@shreddedmail.com wrote: I'm interested in seeing products (including software) that already have the development (anomaly detection, trends/reports, etc.) work done so I can spend my cycles elsewhere. This might fit the bill - http://www.zurich.ibm.com/aurora/ Now commercially available as http://www-01.ibm.com/software/tivoli/products/netcool-performance-flow/ Full disclosure - I work for big blue - but not in any division that works on Aurora / Tivoli Netcool. -- Suresh Ramasubramanian (ops.li...@gmail.com)
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 12:05 PM, Rick Ernst wrote: A solution preferably that integrates with NetFlow and RTBH. An in-line solution obviously requires an appliance, or at least special/additional hardware. The key is to not be inline all the time, but only inline *when needed*. This removes operational complexity, provides the ability to oversubscribe, and simplifies the routine troubleshooting matrix. I'm looking at taking the first whack at immediate mitigation at the border/edge (upstream) via uRPF and RTBH. Good plan. Additional mitigation would be via manual or automatic RTBH or security/abuse@ involvement with upstreams. Automagic is generally bad, as it can be gamed. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Mon, Jan 4, 2010 at 9:08 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 12:05 PM, Rick Ernst wrote: A solution preferably that integrates with NetFlow and RTBH. An in-line solution obviously requires an appliance, or at least special/additional hardware. The key is to not be inline all the time, but only inline *when needed*. This removes operational complexity, provides the ability to oversubscribe, and simplifies the routine troubleshooting matrix. I'd argue just the opposite. If your monitoring/mitigation system changes dependent on the situation (normal vs under attack), you are adding complexity to the system. What mode is the system in right now? Is this customer having connectivity issues because of a state change in the network? etc. I'm looking at taking the first whack at immediate mitigation at the border/edge (upstream) via uRPF and RTBH. Good plan. Additional mitigation would be via manual or automatic RTBH or security/abuse@ involvement with upstreams. Automagic is generally bad, as it can be gamed. I know you said generally, but if I'm seeing 200Kpps from a.b.c.d, I don't care if a.b.c.d is spoofed. I want the traffic blocked from the guts of my network. Note that my original question was in the context of a D/DoS composed of lots of itty-bitty packets. Other attack mechanisms do not necessarily lend themselves to chop 'em off at the knees. Rick
Re: D/DoS mitigation hardware/software needed.
On Tue, Jan 5, 2010 at 10:38 AM, Dobbins, Roland rdobb...@arbor.net wrote: Additional mitigation would be via manual or automatic RTBH or security/abuse@ involvement with upstreams. Automagic is generally bad, as it can be gamed. ... and manual wont scale in ddos -- Suresh Ramasubramanian (ops.li...@gmail.com)
RE: D/DoS mitigation hardware/software needed.
-Original Message- From: Christopher Morrow [mailto:morrowc.li...@gmail.com] Sent: Monday, January 04, 2010 11:41 PM The original poster seemed to be asking about appliance based solutions, so your pointed remarks about Roland aside he was actually answering the question. I wonder if Stefan Fouant would offer some of his experience with 'not arbor' vendor solutions to be used when other techniques come up short? Interesting thread! And I'm happy to chime in - thanks Chris! I too would have to strongly agree with Roland's comments about not front-ending your mitigation solution with firewalls or load-balancers - these are precisely the types of things which topple over first under a big attack, as such having your mitigation devices behind them makes little sense. If you must use firewalls and/or LBs, put them behind the mitigation device, where the traffic has already been scrubbed and your state tables won't be exhausted. Having said that, if you've got a router capable of doing generic packet filters upstream of your mitigation device, this is certainly a good place to apply stateless filters which can pitch any traffic you are sure you will never need to receive. Flowspec and/or automated blackhole routing works very well for this application when you want to get rid of certain types of cruft, before it hits your mitigation device. Now, on to the OPs original question regarding appliance based solutions, I would say I am actually a firm believer in having multiple vendors in place if you can afford it. Arbor definitely has a corner on the market here, with the most comprehensive solution which entails everything from detection to mitigation and pretty much everything in between. Arbor can also automate the FlowSpec process and/or generate router ACLs for propagation to upstream routers... They do a lot of other stuff as well such as identification of BGP hijacking, DNS analysis, can automate a lot of the RTBH or S/RTBH stuff. Where Arbor generally suffers is with sophisticated attack traffic which requires complex mitigations - these often require a lot of tweaking in order to properly scrub the traffic... knowing your environment and which attack vectors are likely to be exploited is your best bet here, where you can configure mitigation templates in advance for rapid deployment during an attack. I've also worked with the RioRey devices and I have to say that although they don't have all the bells and whistles that some of the other vendors offer, their approach to mitigation is entirely unique and can genuinely mitigate attacks in record-time. Without disclosing too much of their intellectual property, I will say that their algorithms essentially look at the randomness and probability of address space distribution within the attack traffic, and can generally offer a high degree of certainty of scrubbing the majority of the bad traffic - and they do this WITHOUT having to do DPI as many other vendors are currently doing. Bottom line - if you're looking for something with a lot of bells and whistles and the ability to monitor/detect/analyze/etc., you're probably better off going with an Arbor solution. If you have lower OpEx and just want something that you can set it and forget it, you'd be hard pressed not to look at the RioRey. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 12:19 PM, Suresh Ramasubramanian wrote: ... and manual wont scale in ddos Actually, it can and does. ; I'm referring to the employment and selection of situationally-appropriate tools, mind. The tools themselves must of necessity perform their work in a largely automated fashion once they're employed, which is what I believe you actually meant. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 12:19 PM, Rick Ernst wrote: I'd argue just the opposite. If your monitoring/mitigation system changes dependent on the situation (normal vs under attack), you are adding complexity to the system. What mode is the system in right now? Is this customer having connectivity issues because of a state change in the network? etc. I strongly disagree with this, except for properties which are under sustained attack 24/7. If one has constructed one's detection/classification/traceback/mitigation system properly, one always knows at a glance the state of the system. Otherwise, whenever there's any issue whatsoever with the properties under protection, one must try and prove a negative - i.e., that the mitigation solution isn't causing the problem. Happens every time, heh. I know you said generally, but if I'm seeing 200Kpps from a.b.c.d, I don't care if a.b.c.d is spoofed. I want the traffic blocked from the guts of my network. Not if it's legit, you don't, or if the attacker is spoofing, say, the IPs of the root nameservers, or the TLDs, or an e-commerce/supply-chain partner . . . or if the attack is originating behind a broadband mega-proxy, or a mobile CGN. ; Also, if you've a variety of tools at your disposal, like S/RTBH and/or flow-spec, and then more sophisticated (and expensive) tools like IDMS, the freedom to choose the least intrusive/most situationally-appropriate tool to mitigate a given attack is essential for resource preservation and the ability to oversubscribe the more sophisticated tools. Note that my original question was in the context of a D/DoS composed of lots of itty-bitty packets. Other attack mechanisms do not necessarily lend themselves to chop 'em off at the knees. Absolutely, which is where the situationally-specific selection of tools/modes comes into play. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
RE: D/DoS mitigation hardware/software needed.
-Original Message- From: Rick Ernst [mailto:na...@shreddedmail.com] Sent: Tuesday, January 05, 2010 12:19 AM I'd argue just the opposite. If your monitoring/mitigation system changes dependent on the situation (normal vs under attack), you are adding complexity to the system. What mode is the system in right now? Is this customer having connectivity issues because of a state change in the network? etc. Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: D/DoS mitigation hardware/software needed.
On Tue, Jan 5, 2010 at 10:52 AM, Dobbins, Roland rdobb...@arbor.net wrote: I'm referring to the employment and selection of situationally-appropriate tools, mind. The tools themselves must of necessity perform their work in a largely automated fashion once they're employed, which is what I believe you actually meant. fair enough. -- Suresh Ramasubramanian (ops.li...@gmail.com)
Re: D/DoS mitigation hardware/software needed.
On Tue, Jan 05, 2010, Stefan Fouant wrote: Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp. Has anyone deployed a DDoS distributed enough to inject ETOOMANY routes into the hardware forwarding tables of routers? I mean, I assume that there's checks and balances in place to limit then number of routes being injected into the network so one doesn't overload the tables, but what's the behaviour if/when this limit is reached? Does mitigation cease being as effective? Adrian
Re: D/DoS mitigation hardware/software needed.
I think you, Roland, and I are all agreeing on the same argument. The (my) confusion entered with the statement of, The key is to not be inline all the time, but only inline *when needed*. I inferred a topological or physical path change from that. Redirecting traffic (which is really just an extension of RTBH; a scrubber destination rather than Null0) is an understandable state. Rick On Mon, Jan 4, 2010 at 9:34 PM, Stefan Fouant sfou...@shortestpathfirst.net wrote: -Original Message- From: Rick Ernst [mailto:na...@shreddedmail.com] Sent: Tuesday, January 05, 2010 12:19 AM I'd argue just the opposite. If your monitoring/mitigation system changes dependent on the situation (normal vs under attack), you are adding complexity to the system. What mode is the system in right now? Is this customer having connectivity issues because of a state change in the network? etc. Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
RE: D/DoS mitigation hardware/software needed.
-Original Message- From: Suresh Ramasubramanian [mailto:ops.li...@gmail.com] Sent: Tuesday, January 05, 2010 12:19 AM On Tue, Jan 5, 2010 at 10:38 AM, Dobbins, Roland rdobb...@arbor.net wrote: Additional mitigation would be via manual or automatic RTBH or security/abuse@ involvement with upstreams. Automagic is generally bad, as it can be gamed. ... and manual wont scale in ddos There are pros and cons to each approach. Certain types of things can be automated, in fact I've done this using the Auto-mitigate feature in Arbor coupled with pre-configured mitigation templates for certain types of traffic and it works very well. But generally, as Roland mentioned automagic stuff can be gamed and for the majority of the stuff you are going to want an operator to look at the alert before making the decision to offramp. The trick is to try to automate as much around the process as possible - I've worked in environments where just making little changes to incident handling response methods reduced the time to mitigate an attack from hours to minutes, all the while still requiring an operator to press the big red button to offramp and enable the mitigation. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 12:39 PM, Adrian Chadd wrote: I mean, I assume that there's checks and balances in place to limit then number of routes being injected into the network so one doesn't overload the tables, but what's the behaviour if/when this limit is reached? Does mitigation cease being as effective? For IDMS 'scrubbing' solutions, one merely injects the route of the attack targets into one's iBGP, in order to draw all traffic towards that specific target into the scrubbing center. For S/RTBH and flow-spec, modern edge routers can scale to millions of routes; also note one isn't limited to /32s. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 12:39 PM, Stefan Fouant wrote: The trick is to try to automate as much around the process as possible - I've worked in environments where just making little changes to incident handling response methods reduced the time to mitigate an attack from hours to minutes, all the while still requiring an operator to press the big red button to offramp and enable the mitigation. Concur 100% - and when the end-customer is under attack and screaming, this reduction in time to detect/classify/traceback/mitigate makes all the difference. Your very salient comments highlight the paramount importance of preparation as the key enabling phase of the six-phase security incident-handling methodology: 1. Preparation. 2. Detection/identification. 3. Classification. 4. Traceback. 5. Reaction. 6. Post-mortem (feeding lessons learned back into the Preparation phase). --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 12:39 PM, Rick Ernst wrote: I think you, Roland, and I are all agreeing on the same argument. GMTA. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 11:57 AM, Dobbins, Roland wrote: You and Barry and Tim Battles and Sean Donelan and Danny McPherson and Don Smith and Steve Bellovin and Jared Mauch and John Kristoff and a lot of other folks too numerous to mention . . . including Paul Vixie, Darrel Lewis, Ryan McDowell, Paul Quinn, Michael Behringer, Craig Huegen, Craig Labvoitz, Dave Smith, Gregg Schudel, and many others. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
RE: D/DoS mitigation hardware/software needed.
On Tue, 5 Jan 2010, Stefan Fouant wrote: Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp. That said, what are all those ISPs doing now that Cisco has stopped developing the Guard? -Hank
RE: D/DoS mitigation hardware/software needed.
-Original Message- From: Hank Nussbacher [mailto:h...@efes.iucc.ac.il] Sent: Tuesday, January 05, 2010 1:02 AM On Tue, 5 Jan 2010, Stefan Fouant wrote: Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp. That said, what are all those ISPs doing now that Cisco has stopped developing the Guard? Well of course, moving to Arbor haha... eased in part by a little Cisco initiative called Clean Pipes 2.0 :) Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: D/DoS mitigation hardware/software needed.
I actually agree with most of this, but want to correct a clearly inadvertent error below, and make a couple of points. PCI DSS does not require a Web application firewall. It requires that OR an independent audit of all code within PCI scope by a third party. If a WAF is used, it only need be deployed in such a manner as to protect devices inside of PCI scope (depending on design, this may or may not include everything that has public exposure). The powers that be specified additional methods by which PCI compliance could be satisfied other than just these two after the wailing of the masses. I don't know off the top of my head if those other methods will be acceptable in 2010. Personally, I believe a DOS detection/prevention system is typically going to be best placed between screening routers/switches and the next L3/L4 aware device- generally you will want it (or them) to be as close to your ingress edge as you can put it- why allow DOS traffic to go further downstresm? So I suspect Roland and I agree on that fwiw. Earlier, Roland mentions load-balancers and firewalls as both being susceptible to state-table overflows. Certainly this is possible and happened in the past, and it argues for a DOS prevention device being in front of them. At least one modern high-end load balancer handles this well and is in front of a large number if not the majority of major sites. It is possible to build equipment that can handle vast numbers of state entries and handle lookups into them in very attractive big O's. It is also possible to build systems that gracefully handle table exhaustion and enter into aggressive reaping modes. This has been empirically proven to me wrt load-balancers. Some device is going to have to handle the state and insert itself into the path- even if that is a lone webserver. I maintain that there is no reason to believe that it is not possible for a firewall to do that as well as a load-balancer. As for whether you should have a stateful firewall in front of a production web server farm, I understand Roland's point. However, I will say that there are many reasons why people put firewalls in front of webservers- to name some: * Defense in depth. You've never had a host that received external traffic ever accidentally have iptables or windows firewall turned off? Even when debugging a production outage or on accident? * Location for IDS/IDP. * Connection cleanup, re-assembling fragments, etc. * SYN flood protection, etc. * Single choke point to block incoming traffic deemed undesirable. * Single log point for inbound connections for analysis and auditing requirements. * Allows outbound traffic enforcement. * Allows conditional inbound traffic from specific approved external hosts- e.g. a partner. * Some firewalls allow programmatic modification of configurations with all the benefits/pain that brings. This is alongside traditional CLI and GUI interfaces. * In some/many cases a zone based firewall configuration can be much easier to work with than a large iptables config. Certainly the management tools are better. * Yeah, auditors like it. I'm not at all adverse to putting transparent proxies in front of your website. CDN vendors aren't either. I will say that I have had several bad experiences with WCCP not working as expected and failing non-gracefully. I am not saying its always a good idea to have a stateful firewall in front of your web servers, I'm saying that there are reasons why you might want to and in my experience it is common. --D On Mon, Jan 4, 2010 at 7:31 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 10:18 AM, Suresh Ramasubramanian wrote: 5 Ditch the stateful firewall and exclusively use a netflow device NetFlow analysis is very useful for network visibility, and detection/classification/traceback. There are both open-source and commercial NetFlow collection and analysis systems available (full disclosure: I work for a vendor of both NetFlow analysis and DDoS mitigation solutions); however, they don't provide mitigation, which is where S/RTBH, flow-spec, and/or IDMS come into play. PCI DSS iatrogenically *requires* that a 'Web application firewall' be placed in front of Web servers which process credit card information (PCI DSS completely ignores availability, and contains a number of recommendations which are actually harmful from an opsec standpoint). Running mod_security or its equivalent on the front-end Web servers themselves fulfills this requirement without putting a stateful DDoS chokepoint in front of the servers. It's also a really good idea to front Web servers with a tier of caching-only transparent reverse proxies; Squid is a good choice for this, as well as various commercial offerings. WCCPv2 clustering (supported by Squid and several commercial caching/proxying solutions) allows this tier to be scaled horizontally in order to meet capacity demands.
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 2:38 PM, Darren Bolding wrote: * Defense in depth. You've never had a host that received external traffic ever accidentally have iptables or windows firewall turned off? Even when debugging a production outage or on accident? Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. 'Stateful inspection' where in fact there is no useful state to inspect is pointless. * Location for IDS/IDP. Non-sequitur, as these things have nothing to do with one another (plus, these devices are useless, anyways, heh). * Connection cleanup, re-assembling fragments, etc. Far, far, far better and more scalably handled by the hosts themselves and/or load-balancers. * SYN flood protection, etc. Firewalls simply don't handle this well, marketing claims aside. They crash and burn. * Single choke point to block incoming traffic deemed undesirable. Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. * Single log point for inbound connections for analysis and auditing requirements. Contextless, arbitrary syslog from firewalls and other such devices is largely useless for this purpose. NetFlow combined with server/app/service logs is the answer to this requirement. * Allows outbound traffic enforcement. Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. * Allows conditional inbound traffic from specific approved external hosts- e.g. a partner. Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. * Some firewalls allow programmatic modification of configurations with all the benefits/pain that brings. This is alongside traditional CLI and GUI interfaces. Ugly, brittle, siloed, to be avoided at all costs. * In some/many cases a zone based firewall configuration can be much easier to work with than a large iptables config. Certainly the management tools are better. Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. * Yeah, auditors like it. Education is the answer here. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 2:38 PM, Darren Bolding wrote: PCI DSS does not require a Web application firewall. http://searchsoftwarequality.techtarget.com/news/article/0,289142,sid92_gci1313797,00.html Since no business is going to allow an external 'code review' (if it's even possible, given that they're likely using COTS products, the source code of which they simply don't have), this defaults to a requirement for the 'Web application firewall'. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
