Re: BCP38 dismissal
Jo Rhett wrote: On Sep 4, 2008, at 7:24 AM, James Jun wrote: Indeed... In today's internet, protecting your own box (cp-policer/control plane filtering) is far more important IMO than implementing BCP38 when much of attack traffic comes from legitimate IP sources anyway (see botnets). I'm sorry, but nonsense statements such as these burn the blood. Sure, yes, protecting yourself is so much more important than protecting anyone else. Anyone else want to stand up and join the I am an asshole club? normally i would have just hit delete. but your ad hominem attack on the messenger need a response. the reality of life is that he is correct in that attack traffic comes from legitimate IP sources anyway. therefore, your first duty should be to keep your hosts from joining the massive army of botnets. randy
Re: an effect of ignoring BCP38
http://www.caida.org/workshops/wide/0808/slides/measuring_reverse_paths.pdf great work on a tough problem randy
InterCage, Inc. (NOT Atrivo)
Hello Everyone, Good morning. Seeing the activity in regards to our company here at NANOG, I believe this is the most reasonable and responsible place to respond to the current issues on our network. We hope to obtain non-bias opinion's and good honest and truthful information from the users here. Being that there are much larger operators here then us, what kind of insight can you give to the issues that have arisen? We've near completely removed (completion monday 09/08/08) Hostfresh from our network. 2 of their /24's have been removed: 58.65.238.0/24 dropped 58.65.239.0/24 dropped The machine's they leased from us have been canceled. What do you suggest for the next move? Thank you for your time. Have a great day. --- Russell M. InterCage, Inc.
Re: Cisco uRPF failures
Jo Rhett wrote: That's the surprising thing -- no scenario. Very basic configuration. Enabling uRPF and then hitting it with a few gig of non-routable packets consistently caused the sup module to stop talking on the console, and various other problems to persist throughout the unit, ie no arp response. We were able to simulate this with two 2 pc's direction connected to a 6500 in a lab. If I remember right, we had to enable CEF to see the problem, but since CEF is a kitchen sink that dozens of other features require you simply couldn't disable it. Definately sounds like it could be a problem - I'd like to try and replicate this. What do you mean by non-routable traffic - traffic whose destination has no route (I assume you are running defaultless), or traffic that fails the uRPF check? And correct me if I'm wrong but I thought you can't disable CEF on the 6500 platform? hs-6513-1#conf t Enter configuration commands, one per line. End with CNTL/Z. hs-6513-1(config)#no ip cef % Incomplete command. hs-6513-1(config)#no ip cef ? accounting Enable CEF accounting distributed Distributed Cisco Express Forwarding event-log CEF event log commands interface CEF linecard commands linecardCEF linecard commands load-sharingLoad sharing nsf Set CEF non-stop forwarding (NSF) characteristics table Set CEF forwarding table characteristics traffic-statistics Enable collection of traffic statistics hs-6513-1(config)#no ip cef distributed %Cannot disable CEF on this platform hs-6513-1(config)#exit hs-6513-1#sh version | inc IOS IOS (tm) s72033_rp Software (s72033_rp-ADVENTERPRISEK9_WAN-M), Version 12.2(18)SXF11, RELEASE SOFTWARE (fc1) Sam
Re: only WV FIBER now peering with Atrivo / Intercage
Brandon Butterworth wrote: Anton's post that GX is still providing them transit is a bit curious, since I was under the impression GX had severed all ties with Atrivo. But the table does not lie, a path of 174 3549 27595 is clearly transit. GX, care to comment? After poking for a bit, it's unclear what, if anything, GX is or isn't doing here. Does it matter? The implication of this thread is the imposition of an internet death penalty on this AS. I vote yea.
Re: only WV FIBER now peering with Atrivo / Intercage
Anton's post that GX is still providing them transit is a bit curious, since I was under the impression GX had severed all ties with Atrivo. But the table does not lie, a path of 174 3549 27595 is clearly transit. GX, care to comment? After poking for a bit, it's unclear what, if anything, GX is or isn't doing here. Does it matter? The implication of this thread is the imposition of an internet death penalty on this AS. In which case people will have to hound each AS they're seen behind, whatever they're providing, and that will change as they run for cover. This seems a change of tactic from just null routing people you don't like on your own network brandon
BGP Clueful from Windstream/Alltel? (Resend-addendum)
Is there anyone hanging around here who happens to either be or can get me in touch with a BGP-clueful person at Windstream/Alltel (AS7029)??? Unicast would be great, I hate to tie up the list with this, but the standard prescribed methods haven't worked in 72 hours now... Sorry for the resend, but if you could contact me at [EMAIL PROTECTED] that will be much easier! Any emails to the subscribed address here won't work, which is the problem I'm attempting to solve with Windstream! :) Thanks in advance! Scott Morris [EMAIL PROTECTED]
Re: Force10 Gear
Paul Wall wrote: Once upon a time, vendors released products which relied on CPU-based flow setup. Certain vintages of Cisco, Extreme, Foundry, Riverstone, etc come to mind. These could forward at line rate under normal conditions. Sufficient randomization on the sources and/or destinations (DDoS, Windows worm, portscans, ...) and they'd die a spectacular death. Nowadays, this is less of a concern, as the higher-end boxes can program a full routing table (and then some) worth of prefixes in CAM. Either way, I think it's a good test metric. I'd be interested in hearing of why you think that's not the case. Back on topic, doing a couple of gigs of traffic at line rate is a walk in the park for any modern product billed as a layer 3 switch. The differentiator between, say, a Dell and a Cisco, is in the software and profoundly not the forwarding performance. Without disagreeing with your main point, there are still plenty of ways to bring even very large boxes to near-zero forwarding rates: 1. Set IP options. A pair of Cat 6509Es using VSS can forward packets without options at up to 770 mpps, but when packets have options the maximum is more like 20 kpps. And that's a high-speed example; the options forwarding rate is more like 0 pps with some other devices. Silicon that forwards packets very fast is only good when header lengths are fixed. 2. Use weak hashing algorithms. Some switches (names removed to protect the guilty) hash on the low-order three bits of MAC address to decide which of eight ASICs will forward a packet. I have heard of, but have not tested, devices that use only three bits for making OSPF ECMP decisions. Not surprisingly either technique can lead to lots of hash collisions in routed environments. 3. Offer IP multicast. Maximum forwarding rates for multicast are rather different than IP unicast with some vendors' implementations, and may be affected by group count, mroute count and amount of replication. dn
Re: only WV FIBER now peering with Atrivo / Intercage
I'm not sure where that 58.65.238.0/24 prefix with AS3549 in the path came from. I *currently* see no BGP RIB entries with AS 3549_27595 (GBLX Intercage) in the path. A query for the past 6 hours yields 32 AS 27595 originated prefixes, here are each with their associated upstream ASN(s) (26769::BANDCON, 19151::WVFIBER-1): 58.65.238.0/24 - 26769 58.65.239.0/24 - 26769 64.28.176.0/20 - 26769,19151 67.210.0.0/21 - 26769,19151 67.210.8.0/22 - 26769,19151 67.210.14.0/23 - 26769,19151 69.22.162.0/23 - 26769,19151 69.22.168.0/21 - 26769,19151 69.22.184.0/22 - 26769,19151 69.31.64.0/20 - 26769,19151 69.50.160.0/19 - 26769,19151 69.50.173.0/24 - 26769,19151 69.50.182.0/23 - 26769,19151 85.255.113.0/24 - 26769,19151 85.255.114.0/23 - 26769,19151 85.255.116.0/22 - 26769,19151 85.255.116.0/23 - 26769,19151 85.255.118.0/24 - 26769,19151 85.255.119.0/24 - 26769,19151 85.255.120.0/23 - 26769,19151 85.255.120.0/24 - 26769,19151 85.255.121.0/24 - 26769,19151 85.255.122.0/24 - 26769,19151 116.50.10.0/24 - 26769,19151 116.50.11.0/24 - 26769,19151 216.255.176.0/20 - 26769,19151 216.255.176.0/21 - 26769,19151 216.255.176.0/22 - 19151 216.255.180.0/22 - 19151 216.255.184.0/21 - 26769,19151 216.255.184.0/22 - 19151 216.255.188.0/22 - 19151 As for which of these prefixes seem to be associated with alleged nefarious activities, I'll leave that as an exercise for the operator. -danny
Re: only WV FIBER now peering with Atrivo / Intercage
On Sep 7, 2008, at 8:16 AM, Andrew D Kirch wrote: Brandon Butterworth wrote: Anton's post that GX is still providing them transit is a bit curious, since I was under the impression GX had severed all ties with Atrivo. But the table does not lie, a path of 174 3549 27595 is clearly transit. GX, care to comment? After poking for a bit, it's unclear what, if anything, GX is or isn't doing here. Does it matter? The implication of this thread is the imposition of an internet death penalty on this AS. I vote yea. Seconded. (Thirded?) -- TTFN, patrick
Re: Force10 Gear
On Sun, Sep 07, 2008, David Newman wrote: 1. Set IP options. A pair of Cat 6509Es using VSS can forward packets without options at up to 770 mpps, but when packets have options the maximum is more like 20 kpps. And that's a high-speed example; the options forwarding rate is more like 0 pps with some other devices. Silicon that forwards packets very fast is only good when header lengths are fixed. So what you're saying is send the right crafted packets and DoS the internet, right? (I think I know which options may make routers go all software-path on the packets but I haven't given it a run on a Cat6500. Hm, I wonder if this here 3750 in the lab will do..) Adrian
Re: InterCage, Inc. (NOT Atrivo)
On Sep 7, 2008, at 4:32 AM, InterCage - Russ wrote: Seeing the activity in regards to our company here at NANOG, I believe this is the most reasonable and responsible place to respond to the current issues on our network. We hope to obtain non-bias opinion's and good honest and truthful information from the users here. Being that there are much larger operators here then us, what kind of insight can you give to the issues that have arisen? We've near completely removed (completion monday 09/08/08) Hostfresh from our network. 2 of their /24's have been removed: 58.65.238.0/24 dropped 58.65.239.0/24 dropped The machine's they leased from us have been canceled. What do you suggest for the next move? A hearty pat on the back for a job well done. -- TTFN, patrick
Re: InterCage, Inc. (NOT Atrivo)
On Sep 7, 2008, at 11:58 AM, Patrick W. Gilmore wrote: On Sep 7, 2008, at 4:32 AM, InterCage - Russ wrote: Seeing the activity in regards to our company here at NANOG, I believe this is the most reasonable and responsible place to respond to the current issues on our network. We hope to obtain non- bias opinion's and good honest and truthful information from the users here. Being that there are much larger operators here then us, what kind of insight can you give to the issues that have arisen? We've near completely removed (completion monday 09/08/08) Hostfresh from our network. 2 of their /24's have been removed: 58.65.238.0/24 dropped 58.65.239.0/24 dropped The machine's they leased from us have been canceled. What do you suggest for the next move? A hearty pat on the back for a job well done. P.S. And removing you from the will _not_ buy from list of providers. -- TTFN, patrick
Re: Force10 Gear
On 9/7/08 8:49 AM, Adrian Chadd wrote: On Sun, Sep 07, 2008, David Newman wrote: 1. Set IP options. A pair of Cat 6509Es using VSS can forward packets without options at up to 770 mpps, but when packets have options the maximum is more like 20 kpps. And that's a high-speed example; the options forwarding rate is more like 0 pps with some other devices. Silicon that forwards packets very fast is only good when header lengths are fixed. So what you're saying is send the right crafted packets and DoS the internet, right? My experience *in lab testing* is that most and perhaps all switches do slow-path processing of v4 and v6 packets with IP options set, and that slow-path forwarding rates are a tiny fraction of fast-path forwarding rates. Christian Huitema made a similar observation in one of his textbooks 10 or more years ago; tests as recently as this year suggest this is still the case. I'm not making any assertions about DoS attacks on production networks. Rate controls and other mechanisms can help mitigate the effects of flooding attacks, but that's a different topic. (I think I know which options may make routers go all software-path on the packets but I haven't given it a run on a Cat6500. Hm, I wonder if this here 3750 in the lab will do..) The record route option will cause rather precipitous drops in forwarding rates on both boxes (and many others). I have not tried other option types, but other testers have told me these too will be slow-pathed. Again, from the ASIC/NP/FPGA's standpoint: Fixed-length, good. Variable-length, not so much... dn
Re: ingress SMTP
On Sep 3, 2008, at 6:52 PM, Tim Sanderson wrote: Anybody not wanting to use their ISP email would notice it. I see filtering 25 FROM the customer as something that is not likely to happen because of this. When a customer buys bandwidth, they want to be able to use it for whatever they choose. This would be just one more restriction giving competitive advantage to any ISP not doing the filtering. In my country, some ISPs block outbound SMTP for home users and they require those users to use the ISPs SMTP server for outgoing which happen to do antivirus and antispam filtering. They will unblock port 25 if you provide good and rational explanation why you need it open and that you understand that in case of problems you will be held responsible. Eugen.
Re: 10GE CWDM
On Mon, 01 Sep 2008 20:50:46 -0400 Robert Boyle [EMAIL PROTECTED] wrote: The only affordable CWDM 10G system I have seen although I haven't used it yet is a single 10G band at 1310 or 1550 with 8 additional 2.5G bands around it. I've wondered if one could shoot with DWDM 10G optics into two channels of a CWDM mux. For example, by connecting DWDM channel 359 (center 1530.33 nm) and 334 (center 1550.12 nm) to the 1530/1550 filters of a CWDM mux with 20nm spacing (+/- 6.5nm pass band). Might that support 1x10gig + 3x1gig on a single strand, or 2x10G + 6x1G on a pair? (and no, I haven't tried it). Bradley
Re: ingress SMTP
Eugeniu Patrascu wrote: On Sep 3, 2008, at 8:08 PM, Winders, Timothy A wrote: Yes, setting up a 587 submit server internally would be best, but man power is at a premium and it hasn't happened. I don't know what SMTP server you're using, but on Postfix you just need to uncomment one line in master.cf, do a reload and that's it. it takes less than a minute to do it on server. YMMV. Would that it were so easy :) You also have the more daunting task of hooking up your auth/aaa infrastructure with your MTA's, and all of the care and feeding that entails. Mike
RE: Force10 Gear
I think it would be interesting to put a table of routing devices together along with the commands it takes to knock down their forwarding rates. And to find out what platform has the higher percentage drop in forwarding rate. As mentioned elsewhere, it's not the pps, but operations per second. Frank -Original Message- From: David Newman [mailto:[EMAIL PROTECTED] Sent: Sunday, September 07, 2008 9:23 AM To: nanog@nanog.org Subject: Re: Force10 Gear Paul Wall wrote: Once upon a time, vendors released products which relied on CPU-based flow setup. Certain vintages of Cisco, Extreme, Foundry, Riverstone, etc come to mind. These could forward at line rate under normal conditions. Sufficient randomization on the sources and/or destinations (DDoS, Windows worm, portscans, ...) and they'd die a spectacular death. Nowadays, this is less of a concern, as the higher-end boxes can program a full routing table (and then some) worth of prefixes in CAM. Either way, I think it's a good test metric. I'd be interested in hearing of why you think that's not the case. Back on topic, doing a couple of gigs of traffic at line rate is a walk in the park for any modern product billed as a layer 3 switch. The differentiator between, say, a Dell and a Cisco, is in the software and profoundly not the forwarding performance. Without disagreeing with your main point, there are still plenty of ways to bring even very large boxes to near-zero forwarding rates: 1. Set IP options. A pair of Cat 6509Es using VSS can forward packets without options at up to 770 mpps, but when packets have options the maximum is more like 20 kpps. And that's a high-speed example; the options forwarding rate is more like 0 pps with some other devices. Silicon that forwards packets very fast is only good when header lengths are fixed. 2. Use weak hashing algorithms. Some switches (names removed to protect the guilty) hash on the low-order three bits of MAC address to decide which of eight ASICs will forward a packet. I have heard of, but have not tested, devices that use only three bits for making OSPF ECMP decisions. Not surprisingly either technique can lead to lots of hash collisions in routed environments. 3. Offer IP multicast. Maximum forwarding rates for multicast are rather different than IP unicast with some vendors' implementations, and may be affected by group count, mroute count and amount of replication. dn
Re: ingress SMTP
- Original Message - From: Michael Thomas [EMAIL PROTECTED] Date: Monday, September 8, 2008 7:31 am Subject: Re: ingress SMTP Would that it were so easy :) You also have the more daunting task of hooking up your auth/aaa infrastructure with your MTA's, and all of the care and feeding that entails. As a matter of interest, it took but a couple of person hours to sort this out at my last place of work, the largest time chunk in equation was the compiling of TLS and the various SASL modules into Postfix. The second from largest chunk of time was to get the script to get the information required from the various other back end mail servers on campus, including, but not limited to, Lotus Notes, M$ Exchange, and Sun/iPlanet messaging server and it's LDAP server. The only down side to the system was password changed took up to 15 minutes to get to the mail systems as there was no direct connection between the external gateways and the internal auth systems. Of course the above doesn't take into account the several weeks of political badgering and grandstanding that we endured to get the faculties to actually accept that that was the way it was going to be. They couldn't stand that there would only be incoming and outgoing mail via the central gateway. Such is life at Universities. Regards, M
Re: Force10 Gear
On 9/7/08, Frank Bulk [EMAIL PROTECTED] wrote: I think it would be interesting to put a table of routing devices together along with the commands it takes to knock down their forwarding rates. And to find out what platform has the higher percentage drop in forwarding rate. As mentioned elsewhere, it's not the pps, but operations per second. Send a 3kpps stream of multicast packets with TTL=1 towards a sup720 and you can watch it keel over and cry uncle. It really, really doesn't take much these days to kill high-end hardware; they're so optimized for a specific class of traffic that they handle well in hardware, as that's what the bulk of the normal traffic is, and that's what the marketing department needs to chase to keep up with the competition; any traffic profile outside of that doesn't get the same focus from the hardware forwarding teams because that's not where the pressure to keep up from the marketplace is coming from. *Nobody* goes out and says I have $10M to spend on routers, but to qualify they must be able to forward 10Mpps of IPv4 packets with IP options enabled, sustained rate, with no loss. That's just not a driving market force right now. I think you would find that your table simply reflects what the *bulk* of the traffic profiles from major customers represent; those areas that cause the routers pain in terms of forwarding are exactly those traffic patterns that are *not* highly represented among the majority of the customer base. Matt
Re: 10GE CWDM
CWDM filter bandpass is wide to allow for drifting optics. Anything within about 7nm of 1530/1550 should work fine. I've got some optics near 34 and 59 on order to do exactly that in a bidir single fiber arrangement. I'll report back my results. On 9/7/08, Bradley Urberg-Carlson, VISI [EMAIL PROTECTED] wrote: On Mon, 01 Sep 2008 20:50:46 -0400 Robert Boyle [EMAIL PROTECTED] wrote: The only affordable CWDM 10G system I have seen although I haven't used it yet is a single 10G band at 1310 or 1550 with 8 additional 2.5G bands around it. I've wondered if one could shoot with DWDM 10G optics into two channels of a CWDM mux. For example, by connecting DWDM channel 359 (center 1530.33 nm) and 334 (center 1550.12 nm) to the 1530/1550 filters of a CWDM mux with 20nm spacing (+/- 6.5nm pass band). Might that support 1x10gig + 3x1gig on a single strand, or 2x10G + 6x1G on a pair? (and no, I haven't tried it). Bradley -- Sent from Gmail for mobile | mobile.google.com
RE: Force10 Gear
This once again quickly reduces to a question of real-life need in my mind. What proportion of useful traffic actually carries IP options these days? Who uses them other than fooling around with the occasional source-routing or RR exercise, if their local infrastructure even permits it to be sent? Many sites just firewall off all that stuff because in real life they never use it or need it. For them it's more trouble than it would be worth trying to process correctly, especially in a security context, so that's their accepted real-life practice. Clearly, those who design the routing iron are well aware of such practice because they optimized the hardware to process the far more typical no-options day to day traffic. While they still accomodate options it's evidently done as kind of an add-on bandaid way outside of the normal path. Now you have to ask yourself where the people making the iron cheaped out -- should they have designed in the ability to handle all these corner cases at their normal wire speed, or should they have dismissed such foolishness and concentrated on what they knew the industry actually requires? How much additional cost does anyone think ASICs to deal with options and other seldom- seen conditions at the same transit rates would incur? I think the most common answer would be f'geddaboudit. The TTL=1 is an interesting one, but really, under normal conditions shouldn't happen that often. People tracerouting certainly shouldn't expect 100% reliability on getting the expired ICMP back, and automatically throttling the rate of errors from routing loops might have certain benefits so why try to handle *every* expired TTL that comes along? It seems like taking anything out of the fast-path should only be done if the slow path is good and ready to accept it, and if someone's trying to hammer the box with stupid stuff then most of it should simply be dropped. Handling it should be near the bottom of the main processor's priority list ... and rather than allow the fast iron to pile way too much crap in its inbox to be dealt with, process-switching should have a VERY short queue and be able to tell the lower levels not now and simply increment some obscure counter for stupid packets dropped without letting that impinge on the more important tasks at hand. Having the whole box fall over is the *least* acceptable result. _H*
[funsec] Security Fix: Updates on Atrivo/Intercage (fwd)
-- Forwarded message -- Date: Mon, 8 Sep 2008 04:17:29 GMT From: Paul Ferguson [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: [funsec] Security Fix: Updates on Atrivo/Intercage -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Brian Krebs add some late updates to his Security Fix article from Friday 5 September 2008: [snip] Update, Sunday, Sept. 7, 8:02 p.m.: I spoke today with Randy Epstein, president of WVFiber and co-founder of Host.net, which acquired WVFiber just six weeks ago. Epstein said after reading reports from Security Fix, Hostexploit.com, Spamhaus.org and others about cyber crime activities at Atrivo, WVFiber has decided to drop Atrivo as a customer. WVFiber plans to stop providing upstream connectivity to Atrivo by Wednesday or Thursday at the latest, Epstein said. That would leave Atrivo with just a single upstream provider -- Bandcon. Update, Sunday, Sept. 7, 9:15 p.m.: nLayer Communications, a company that owns a significant slice of the Internet addresses used by Atrivo/Intercage, is demanding that Atrivo vacate the space and return the addresses by Sept 30. Atrivo/Intercage has not been a direct customer of nLayer Communications since December 2007, but they still have some legacy reallocations from our IP space, wrote nLayer co-founder Richard A. Steenbergen, in an e-mail to Security Fix. Since they are no longer a customer, we require that they return our non-portable IP space, and have given them a deadline of September 30th to do so. If the IP space is not returned by that point, we will follow standard procedure to reclaim it, including null routing the space, and sending cease and desist letters to any network who still transits it without our permission. According to Steenbergen, Atrivo/Intercage must return roughly 7,400 IP addresses. [snip] Ref: http://voices.washingtonpost.com/securityfix/2008/09/scam-heavy_us_isp_grow s_more_i.html FYI, - - ferg -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFIxKdMq1pz9mNUZTMRAnLcAKCRgGjZrgwr5xCmFFXPV/a0xUAlVwCaAkPL nHo38nvc5azHws2QPhshAvY= =zWoJ -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/ ___ Fun and Misc security discussion for OT posts. https://linuxbox.org/cgi-bin/mailman/listinfo/funsec Note: funsec is a public and open mailing list.
Re: an effect of ignoring BCP38
On Sun, Sep 07, 2008 at 07:43:41PM +1200, Randy Bush wrote: http://www.caida.org/workshops/wide/0808/slides/measuring_reverse_paths.pdf great work on a tough problem yes, but would it work if we all did BCP38 filtering? --bill
