Cheap Juniper Gear for Lab
Hello All, I am tasked with replacing an old linux router setup with Juniper gear in the near future. Though I am a Cisco guy myself. Does anyone know of any older cheap Juniper gear I might find on Ebay so that I may build a home lab without going broke? Thanks! -- Steve King Network/Linux Engineer - AdSafe Media Cisco Certified Network Professional CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Routers in Data Centers
Cisco uses their own ASICS is their higher end flag ship devices. Devices such as the Catalyst 6500 series or the 2960 switches. You pretty much singled out all the major players, including those who have been bought out (Foundry by HP) and claimed they do not provide their own, yet 3rd party flawed ASICS. I am actually surprised you didn't mention HP, Linksys or Dell as they are the most guilty of using 3rd party ASICS and shotty software. If you are buying data center grade equipment from these vendors, it will be quality hardware backed by their support (if purchased) such as Cisco's SmartNet agreements. Moral of the story, do your research on the devices you plan to implement and ask for data sheets on how the features you need are handled (in software or hardware). I know Juniper and Cisco provide such documentation for their devices. Quality hardware, however more expensive, will give you less trouble in the long run. You truly get what you pay for in the networking industry. On 9/24/10 9:28 PM, Richard A Steenbergen wrote: On Fri, Sep 24, 2010 at 03:52:22PM +0530, Venkatesh Sriram wrote: Hi, Can somebody educate me on (or pass some pointers) what differentiates a router operating and optimized for data centers versus, say a router work in the metro ethernet space? What is it thats required for routers operating in data centers? High throughput, what else? A datacenter router is a box which falls into a particular market segment, characterized by extremely low cost, low latency, and high density ethernet-centric boxes, at the expense of advanced features typically found in more traditional routers. For example, these boxes tend to lack any support for non-ethernet interfaces, MPLS, advanced VLAN tag manipulation, advanced packet filters, and many have limited FIB sizes. These days it also tends to mean you'll be getting a box with only (or mostly) SFP+ interfaces, which are cheaper and easier to do high density 10GE with, but at the expense of long reach optic availability. A metro ethernet box also implies a particular market segment, typically a smaller box (1-2U) that has certain advanced features which are typically not found in other small boxes. Specifically, you're likely to see advanced VLAN tag manipulation and stacking capabilities, MPLS support for doing pseudowire/vpn PE termination, etc, that you might normally only expect to see on a large carrier-class router. Also, an interesting side-effect of the quest for high density 10GE at low prices is that modern datacenter routers are largely built on third party commodity silicon rather than the traditional in-house ASIC designs. Many of the major router vendors (Cisco, Juniper, Foundry, Force10, etc) are currently producing datacenter routers which are actually just their software (or worse, someone else's software with a little search and replace action on a few strings) wrapped around third party ASICs (EZchip, Marvell, Broadcom, Fulcrum, etc). These boxes can definitely offer some excellent price/performance numbers, but one unfortunate side effect is that many (actually, most) of these chips have not been fully baked by the years of experience the more traditional router vendors have developed. Many of them have some very VERY serious design flaws, causing everything from preventing them from fully implementing some of the features you would normally except from a quality rouer (multi-label stack MPLS, routed vlan interface counters, proper control-plane DoS filter/policing capabilities, etc), or worse (in some cases, much, much worse). YYMV, but the 30 second summary is that many vendors consider datacenter users and/or use cases to be unsophisticated, and they're hoping you won't notice or care about some of these serious design flaws, just the price per port. Depending on your application, that may or may not be true. :) -- Steve King Senior Linux Engineer - Advance Internet, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Routers in Data Centers
On 9/25/10 5:35 AM, Richard A Steenbergen wrote: On Sat, Sep 25, 2010 at 03:11:25AM -0400, Steven King wrote: Cisco uses their own ASICS is their higher end flag ship devices. Devices such as the Catalyst 6500 series or the 2960 switches. You pretty much singled out all the major players, including those who have been bought out (Foundry by HP) and claimed they do not provide their own, yet 3rd party flawed ASICS. I am actually surprised you didn't mention HP, Linksys or Dell as they are the most guilty of using 3rd party ASICS and shotty software. If you are buying data center grade equipment from these vendors, it will be quality hardware backed by their support (if purchased) such as Cisco's SmartNet agreements. My point was that every major vendor, even the ones who normally make their own in-house ASICs, are also actively selling third party silicon (or in some cases complete third party boxes) in order to compete in the cheap datacenter optimized space. Folks like HP and Dell were never in the business of making real routers to begin with, so them selling a Broadcom reference design with 30 seconds of search and replace action on the bundled software is not much of a shocker. The guys who do a better job of it, like Foundry (who was bought by Brocade, not HP), at least manage to use their own OS as a wrapper around the third party hardware. But my other major point was that almost all of these third party ASICs are sub-par in some way compared to the more traditional in-house hardware. Many of them have critical design flaws that will limit them greatly, and many of these design flaws are only just now being discovered by the router vendors who are selling them. BTW, Cisco is actually the exception to the datacenter optimized boxes being third party, as their Nexus 7K is an evolution of the 6500/7600 EARL ASICs, and their third party hw boxes are EZchip based ASR9k's. Of course their Nexus software roadmap looks surprisingly similar to other vendors doing it with third party hw, go figure. :) Cisco definitely is doing some interesting things with the Nexus. Have you seen the virtualized version? Moral of the story, do your research on the devices you plan to implement and ask for data sheets on how the features you need are handled (in software or hardware). I know Juniper and Cisco provide such documentation for their devices. Quality hardware, however more expensive, will give you less trouble in the long run. You truly get what you pay for in the networking industry. It takes a pretty significant amount of experience and inside knowledge to know who is producing the hardware and what the particular issues are, which is probably well beyond most people. The vendors aren't going to come out and tell you Oh woops we can't actually install a full routing table in our FIB like we said we could, or Oh btw this box can't filter control-plane traffic and any packet kiddie with a T1 can take you down, or FYI you won't be able to bill your customers 'cause the vlan counters don't work, or just so you know, this box can't load balance for shit, and L2 netflow won't work, or yeah sorry you'll never be able to do a double stack MPLS VPN. The devil is in the caveats, and the commodity silicon that's all over the datacenter space right now is certainly full of them. I agree it takes a significant amount of experience to know that informatin off the top of your head, but I am able to find block diagrams, and part information for 98% of Cisco's hardware. Old or new. One needs to do their research on the device to know if it meets their needs. The caveats are everywhere I agree, even some of the experienced network guys get tripped up with them if they aren't careful. Planning is the key to overcoming these problems. -- Steve King Senior Linux Engineer - Advance Internet, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: eBGP Multihop
The last company I worked for moved to eBGP Multi-Hop where there were two connections to the same provider (same AS). This allowed them to utilize both links in both directions vs only one link in one direction and have failover. As you have mentioned link state detection gets a bit crazy with this. If you have a MetroE connection (for example) with multiple segments, this could be problematic. If your side of the link goes down, then you stop sending traffic to the provider, but the provider still tries to send traffic to you. If a segment in the middle goes down, then neither side stop sending traffic. Due to the fact that the BGP session is still up, and the interface on your router is still up, BGP sees the link as a valid path. However there is a fix for this. If your provider supports it that is. Ethernet OAM (Ethernet Operations, Administration, and Management) will allow you to monitor the connection on Layer 2 end to end and not switch to switch. If any part of the link breaks, OAM brings your and the other side of the link down, telling BGP that the link is no longer usable, therefore avoiding the issues above. If you are using a POS, MPLS, or other similar technology, then the issues talked about above are either less of an issue, or not an issue at all. The biggest problem with multi segment Ethernet links is that you need OAM to reliably run eBGP Multihop and OAM isn't supported by a lot of providers (mainly because it requires a newer software version). Hope this helps. On 9/2/10 5:30 AM, Graham Beneke wrote: I have been asked to investigate moving an entire network to multi-hop on all the eBGP sessions. Basically all upstreams, downstreams and peers will eBGP with a route reflector located in the core. This RR will be some kind of quagga or similar box. The dev guys want to be able to poke at the BGP feeds directly and do *magic* that standard router aren't capable of. My gut feel is that this is a bad idea. Besides anything else it makes sane link state detection very challenging - especially where we have multiple sessions with a peer. Is their any BCP or operational experience that agrees or disagrees with my gut. ;-) -- Steve King Senior Linux Engineer - Advance Internet, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: 33-Bit Addressing via ONE bit or TWO bits ? does NANOG care?
I am very curious to see how this would play with networks that wouldn't support such a technology. How would you ensure communication between a network that supported 33-Bit addressing and one that doesn't? On 7/24/10 3:26 PM, IPv3.com wrote: 33-Bit Addressing via ONE bit or TWO bits ? does NANOG care? As some people (who understand IPv4) know, there is a SINGLE spare/unused bit in the IPv4 header that can be set to 0 or 1. Some religions require that it be set to 0. Adult content is marked with a 1. That single bit can be viewed as common between the Source and Destination creating a 33rd bit of addressing. Since it is a single bit, it is welded together for both Source and Destination. 0-Normal 1-Evil/Other/Adult/XXX In anticipation of expanding to 33-bit addressing, another bit was deprecated years ago. It can now be used to UNWELD the EVIL bit. That would allow EVIL to be only for the Source. The Destination would have its own EVIL bit. If two bits are used, then the potential to communicate between the previously welded address spaces arises. Some enforcement could still be used in Edge Network Elements to make sure both bits are 0 or both 1. Enforcements are hard to maintain and full 33-bit addressing may emerge. As an aside, NAT was primarily added to improve the .NET Architecture with a Flash Upgrade-able Network Element. It is a shame that IPv6 salesman do not seem to understand Architecture. They continue on the [NAT is Evil] path. NANOG can play an important role in shaping how Address Plans for North America evolve. Since Network Elements are going to be flash upgraded for the new DNS, it is easy to (unweld/change) the 33-bit addressing for .XXX The 33-bit addressing works into the 66-bit Triple-Tagged VLAN addressing with Content Rating. http://en.wikipedia.org/wiki/IEEE_802.1Q The Locator is 33-bits and the ID is 33-bits. Both are UNIQUE. Both fit in the IPv4 foot-print. The three-ring circus architecture emerges. (((Core)Edge)Fringe) does NANOG care? is NANOG now Fringe ? -- Steve King Senior Linux Engineer - Advance Internet, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Cisco CSS 11503 SSL and reverse DNS
rDNS should not affect the performance of an SSL device. On 5/18/10 11:06 AM, Bobby Mac wrote: Hi All: Will having correct reverse DNS mapping improve SSL performance on a 11503 during peak load? My guess is no but I don't want to pound my prod device to find out. -Bobby -- Steve King Senior Linux Engineer - Advance Internet, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Auto MDI/MDI-X + conference rooms + bored == loop
Along with bpduguard, Cisco switches also continue to look for loops with loopguard. They continuously look for the Keepalive packets that they send out each port. So as long as you have not turned off STP all together on the port, you will be fine. On 3/26/10 6:21 PM, Matthew Huff wrote: Bpduguard if running cisco. set all the switch ports to bpduguard or enable it globally -Original Message- From: Chuck Anderson [mailto:c...@wpi.edu] Sent: Friday, March 26, 2010 6:09 PM To: nanog@nanog.org Subject: Auto MDI/MDI-X + conference rooms + bored == loop Anyone have suggestions on Ethernet LAN loop-prevention? With the advent of Auto MDI/MDI-X ports on switches, it seems way too easy to accidentally or maliciously create loops between network jacks. We have bored or inattentive people plugging in patch cords between adjacent network jacks. STP for loop-prevention isn't working so well for us. STP edge or portfast or faststart modes are required for end-station ports (with normal STP, DHCP often times out after 30+ seconds it takes to go into Forwarding state). Since the edge STP mode goes into Forwarding state immediately, there is a period when loops will form, causing havok with upstream gear until STP blocks the port (if it ever does see below). Desktop switches. You know, those 4 or 5 port Gigabit Ethernet switches. Apparently, many of them don't do any kind of STP at all. Recommendations on ones that do STP? RSTP: is it any better than traditional STP in regards to edge ports and blocking before a loop gets out of hand? Or perhaps blocking for 5-10 seconds before going into Forwarding state, hopefully preventing loops before they happen but also allowing DHCP clients to get an address without timeouts? Recommendations on Desktop switches that do RSTP? Thanks for your suggestions/discussion. -- Steve KingSenior Senior Linux Engineer - Advance Internet, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Consumer-grade dual-homed connectivity options?
You would need at least one router for this. Personally I would connect both DSL modems into a small Cisco router or multi-layer switch. Use that router as the default gateways for each LAN and have two static routes as the default gateway on the router to specify each DSL line. This would allow for load balancing each connection. Although, you run into the issue of needing PAT on both lines. This wouldn't be complex, but would need to be handled by the router as well. I am not sure about asymmetric paths though. Depending on the device, it may handle this differently, and there is no guarantee that the source of your traffic will be from the same connection all the time to the destination. This would cause connectivity issues. There really is no elegant solution to that without having a full routing table of the Internet and 2 separate providers. Others on this list may have a solution to that issue off the top of their heads, or have done this themselves. On 1/2/10 5:48 PM, Scott Weeks wrote: --- paul.w.benn...@gmail.com wrote: From: Paul Bennett paul.w.benn...@gmail.com At home, I currently run two DSL lines. Right now, we just have two separate LANs, one connected to each line, with my wife's devices attached to one, and my devices attached to the other. For a while now, I've been thinking about setting up a load-balancing routing solution to give both of us access to both lines. --- Maybe www.xincom.com/products.php will work? scott -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Routing to multiple uplinks
Maybe I am missing something, but how does VRRP/HSRP cause latency? On 12/19/09 3:45 AM, Scott Berkman wrote: Anycast? http://www.nanog.org/meetings/nanog29/abstracts.php?pt=NjcxJm5hbm9nMjk=nm=n anog29 Might need to know a little more about the layout here for a better answer. -Scott -Original Message- From: rodrick brown [mailto:rodrick.br...@gmail.com] Sent: Friday, December 18, 2009 7:47 PM To: nanog@nanog.org list Subject: Routing to multiple uplinks This may be slightly off topic however I have a very unique situation where I need to provide two diverse paths to a major stock exchange. Each host may either use route A or B for any given reason to access this particular exchange using two distinct routers and target address. The applicatiOn running on these hosts must only see/use one target address this needs to be transparent as possible. NIC bonding/teaming on the host side isn't a viable solution because of the latency overhead same goes for vrrp/hsrp. I believe my only option here is to setup multiple default routes with a preferred path of some sort. This seems to be possible using ip route2 on Linux. This just seems wrong on many levels and I thought I would post here because I know there is something obvious I'm missing. Please clue me in. Thanks. Sent from my iPhone 3GS. -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Routing to multiple uplinks
HSRP/VRRP can be tweaked to less than 1s fail over time. Can you provide a copy of your network map for analysis? GLBP might be a viable option as fail over is not actually an issue at that point. On 12/19/09 2:48 PM, Rodrick Brown wrote: VRRP/HSRP does not cause latency the problem we faced prior was when links flapped or timed out this would be too much of a hindrance for our users to reconcile application state with various trading venues we are trading thousands upon thousands of trades a minute to various destinations. As stated before Path A and Path B are two distinct paths they do however provide identical services but application state is not preserved. A new session and state must be established if a user decides to switch between paths. Essentially we provide the ability for users either shutdown and start sending orders to Path A or Path B based on latency from our servers to these trading venues we're actively monitoring latency between both end points. The overall design is being driven by our rigorous application needs more than anything. The implementation is straight forward we receive a duplicate set of feeds from site A and site B and can also access various services coming from site A or site B however, at any given time a user will be sending/recieving data from one of those destinations. Never both simultaneously. So my question what is the best way to provide this type of redundancy at the host level? The application will only use one target address. On Sat, Dec 19, 2009 at 1:17 PM, Steven King sk...@kingrst.com mailto:sk...@kingrst.com wrote: Maybe I am missing something, but how does VRRP/HSRP cause latency? On 12/19/09 3:45 AM, Scott Berkman wrote: Anycast? http://www.nanog.org/meetings/nanog29/abstracts.php?pt=NjcxJm5hbm9nMjk=nm=n http://www.nanog.org/meetings/nanog29/abstracts.php?pt=NjcxJm5hbm9nMjk=nm=n anog29 Might need to know a little more about the layout here for a better answer. -Scott -Original Message- From: rodrick brown [mailto:rodrick.br...@gmail.com mailto:rodrick.br...@gmail.com] Sent: Friday, December 18, 2009 7:47 PM To: nanog@nanog.org mailto:nanog@nanog.org list Subject: Routing to multiple uplinks This may be slightly off topic however I have a very unique situation where I need to provide two diverse paths to a major stock exchange. Each host may either use route A or B for any given reason to access this particular exchange using two distinct routers and target address. The applicatiOn running on these hosts must only see/use one target address this needs to be transparent as possible. NIC bonding/teaming on the host side isn't a viable solution because of the latency overhead same goes for vrrp/hsrp. I believe my only option here is to setup multiple default routes with a preferred path of some sort. This seems to be possible using ip route2 on Linux. This just seems wrong on many levels and I thought I would post here because I know there is something obvious I'm missing. Please clue me in. Thanks. Sent from my iPhone 3GS. -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional -- [ Rodrick R. Brown ] http://www.rodrickbrown.com http://www.linkedin.com/in/rodrickbrown -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: about interdomain multipath routing.
We use multipath setups for our EIGRP and iBGP configurations for our internal routing as well. Although for larger networks iBGP multipath might be of use due to memory limitations on a lot of devices. Doug Lane wrote: On Tue, Nov 10, 2009 at 3:50 AM, Matthew Petach mpet...@netflight.com wrote: I've outlawed the use of multihop eBGP for load-sharing here; when we get multiple links off the same router to a peer or upstream, they are configured with multipath. We've got hundreds of BGP sessions across the network configured with multipath on them. Do you use iBGP multipath as well to load-balance between links on different routers? I know eBGP multipath is fairly common, but I wonder how many are using iBGP multipath as well. I doubt any carriers would support it, so it's probably only useful for load-balancing outbound traffic. The problem with eBGP multipath alone is that you might want to terminate circuits from a given carrier on two different routers for redundancy reasons, but that precludes any load-balancing with eBGP multipath. Obviously your network has to be designed with equal-cost paths for iBGP multipath to be of any value. -Doug -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: about interdomain multipath routing.
We use eBGP multipath where I work. We usually get two or more connections to each provider we have. Using multipath we are able to add hardware redundancy with bandwidth balancing (to an extent) with this method. There are some providers who will only allow multipath eBGP and not even let you run multihop eBGP. Bin Dai wrote: Hi: These days, in the research, the interdomain multipath routing is pretty hot but i doubt its actually use in reality. Does anyone tell me any use of interdomain multipath routing like multipath BGP in the real world? Best, Daniel -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: about interdomain multipath routing.
Those are very good points Jack. We stopped using multihop for those same reasons. Jack Bates wrote: Matthew Petach wrote: I've outlawed the use of multihop eBGP for load-sharing here; when we get multiple links off the same router to a peer or upstream, they are configured with multipath. We've got hundreds of BGP sessions across the network configured with multipath on them. Same here for my connections, though some of my customers are stuck with multihop eBGP in certain remote areas, but that's a completely different scenario (single link, but obsolete equipment) and out of my control. I much prefer multipath, especially given that the standard multihop config uses static routing and there are conditions that could cause the flap of the eBGP session during a single link outage. With Multipath, only the effected path goes down, as it should. Jack -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Cisco 7600 (7609) as a core BGP router.
We use the 7600 platform as a Customer Border device. It attaches directly to our core, and directly to our customers. This has been a solid platform. Before this we used to use the 7600 as a load balancer for a DNS cluster. Worked fairly well. We use the 6500 series for our main network infrastructure and to border/core/dist layers and they are rock solid, as long as you stay away from the SXH images. These are a bit buggy and we have had routers crash due to that image. We have deployed a few new devices with the SXI and are very happy with them currently. Jim Wininger wrote: I have an opportuniy to put two 7609s into the core of my network. Currently we have 3 upstream providers, taking full BGP routes. (2 in one router and one in another). We have 17 BGP peers/customers (peering to each router), and adding about one new BGP peer every 2-3 months. It is a modest network by most standards. We are running OSPF and BGP between the existing routers. Not rocket science, nothing special (no MPLS, no VRF etc), very simple network. Does anyone have any recommendations on the 7600's as a core BGP router? Good or bad? Have they been a stable platform in a core/BGP environment? -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Is your ISP blocking outgoing port 25?
Most MTAs don't come preconfigured with port 587 either. It is amazing how many people/organizations go with the if it isn't broke, don't fix it mentality, even though it clearly needs to be revised and something new needs to be done/supported. Email needs to be revamped on a larger scale than just adding standards. Sean Donelan wrote: On Fri, 19 Jun 2009, Jeroen Wunnink wrote: 1. Customers remember it more easily 2. Some ISP's also block 587 (hence 'SMTP ports' rather then 'SMTP port' in my previous comment ;-) Those same clueless ISPs will probably block 2525 someday too, clueless expands to fill any void. And using non-standard things like 2525 only lead to more confusion for customers later when they try someone else's non-standard choice, e.g. port 26 or 24 or 5252 and wonder why those don't work. On the other hand, why don't modern mail user agents and mail transfer agents come configured to use MSA port 587 by default for message submission instead of making customers remember anything? RFC 2476 was published over a decade ago, software developers should have caught up to it by now. Imagine if the little box in Outlook and Exchange had the MSA port already filled in, and you only needed to change it for legacy things. -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Unicast Flooding
Very true Eric. Microsoft even acknowledges the issue, and still has not fixed it. I have had a few customers use NLB and have this issue. Eric Gauthier wrote: Brian, The first is preventing it in the first place. As annoying as this might sound, this is one of the standard operating modes for load balancing within a Microsoft server cluster (see NLB). We've tried to avoid it, but it seems to come up around once a year from someone on our campus... Eric :) -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Unicast Flooding
I have had the same issue in the past. The best fix for this has been to set the Layer2/3 aging timers to be the same. Matthew Huff wrote: Unicast flooding is a common occurrence in large datacenters especially with asymmetrical paths caused by different first hop routers (via HSRP, VRRP, etc). We ran into this some time ago. Most arp sensitive systems such as clusters, HSRP, content switches etc are smart enough to send out gratuitous arps which eliminates the worries of increasing the timeouts. We haven't had any issues since we made the changes. After debugging the problem we added mac-address-table aging-time 14400 to our data center switches. That syncs the mac aging time to the same timeout value as the ARP timeout Matthew Huff | One Manhattanville Rd OTA Management LLC | Purchase, NY 10577 http://www.ox.com | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: Brian Shope [mailto:blackwolf99...@gmail.com] Sent: Wednesday, June 17, 2009 5:33 PM To: nanog@nanog.org Subject: Unicast Flooding Recently while running a packet capture I came across some unicast flooding that was happening on my network. One of our core switches didn't have the mac-address for a server, and was flooding all packets destined to that server. It wasn't learning the mac-address because the server was responding to packets out on a different network card on a different switch. The flooding I was seeing wasn't enough to cause any network issues, it was only a few megs, but it was something that I wanted to fix. I've ran into this issue before, and solved it by statically entering the mac-address into the cam tables. I want to avoid this problem in the future, and I'm looking at two different things. The first is preventing it in the first place. Along those lines, I've seen some recommendations on-line about changing the arp and cam timeouts to be the same. However, there seems to be a disagreement on which is better, making the arp timeouts match the cam table timeouts, or vice versa. Also, when talking about this, everyone seems to be only considering routers, but what about the timers on a firewall? I'm worried that I might cause other issues by changing these timers. The second thing I'm considering is monitoring. I'd like to setup something to monitor for any excessive unicast flooding in the future. I understand that a little unicast flooding is normal, as the switch has to do a little bit of flooding to find out where people are. While looking for a way to monitor this, I came across the 'mac-address-table unicast-flood' command on Cisco switches. This looked perfect for what I needed, but apparently it is currently not an option on 6500 switches with Sup720s. Since there doesn't appear to be an option on Cisco that monitors specificaly for unicast floods, I thought that maybe I could setup a server with a network card in promiscuous mode and then keep stats of all packets received that aren't destined for the server and that also aren't legitimate broadcasts or multicasts. The only problem with that is that I don't want to have to completely custom build my own solution. I was hoping that someone may have already created something like this, or that maybe there is a good reporting tool for wireshark or something that could generate the report that I want. Anyone have any suggestions on either prevention/monitoring? Thanks!! -Brian -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Unicast Flooding
I wouldn't consider this a defect. Historically L2 and L3 devices have always been separate. When you get L3 switch those functions are just combined into one device. In Cisco devices that support CEF, the CEF table is used to make all forwarding decisions. But the CEF table is dependent the ARP and Routing tables on the L3 side. When it comes to forwarding the frame of the proper interface the CAM table comes into play. If that table is timing out quicker than the L3 tables, there will be times the CAM table is incomplete. This is mostly present in redundant gateway setups. In bound traffic is usually load balanced between the two redundant devices. The gateways learn about the servers/workstations by traffic leaving the VLAN, not coming into the VLAN. In the case of HSRP/VRRP the servers/workstations are only using one of the two redundant devices to send traffic out of the VLAN. In this case, one device will end up with incomplete information every 5 minutes (default MAC aging timer). This will cause traffic coming in to the VLAN (usually load balanced with EIGRP or OSPF) to be a unknown unicast flood out all ports on the standby device. Making the L2/3 timers the same corrects this. The reason this corrects this because, for CEF to make a forwarding decision, it must have the layer 3 engine make an ARP request if the ARP entry is not present. This causes an ARP broadcast. With the ARP reply being returned the active and standby device can both keep their CAM/ARP/CEF tables up to date. As I do not consider this a defect that these are not synchronized by default, I do agree it would be very beneficial and prevent a lot of confusion and hours of troubleshooting when unsuspecting engineers are trying to figure out why they have a ton of unknown unicast packets. Just my additional 0.02 Holmes,David A wrote: In a layer 3 switch I consider unicast flooding due to an L2 cam table timeout a design defect. To test vendors' L3 switches for this defect we have used a traffic generator to send 50-100 Mbps of pings to a device that does not reply to the pings, where the L3 switch was routing from one vlan to another to forward the pings. In defective devices the L2 cam table entry expires, causing the 50-100 Mbps unicast stream to be flooded out all ports in the destination vlan. In my view the L3 and L2 forwarding state machines must be synchronized such that the L3 forwarding continues as long as there are packets entering the L3 switch on one vlan, and exiting the switch on another vlan via routing. It seems that gratuitous arps are a workaround which serves to reset the cam entry timeout interval, but not an elegant solution. -Original Message- From: Matthew Huff [mailto:mh...@ox.com] Sent: Wednesday, June 17, 2009 2:58 PM To: 'Brian Shope'; 'nanog@nanog.org' Subject: RE: Unicast Flooding Unicast flooding is a common occurrence in large datacenters especially with asymmetrical paths caused by different first hop routers (via HSRP, VRRP, etc). We ran into this some time ago. Most arp sensitive systems such as clusters, HSRP, content switches etc are smart enough to send out gratuitous arps which eliminates the worries of increasing the timeouts. We haven't had any issues since we made the changes. After debugging the problem we added mac-address-table aging-time 14400 to our data center switches. That syncs the mac aging time to the same timeout value as the ARP timeout Matthew Huff | One Manhattanville Rd OTA Management LLC | Purchase, NY 10577 http://www.ox.com | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: Brian Shope [mailto:blackwolf99...@gmail.com] Sent: Wednesday, June 17, 2009 5:33 PM To: nanog@nanog.org Subject: Unicast Flooding Recently while running a packet capture I came across some unicast flooding that was happening on my network. One of our core switches didn't have the mac-address for a server, and was flooding all packets destined to that server. It wasn't learning the mac-address because the server was responding to packets out on a different network card on a different switch. The flooding I was seeing wasn't enough to cause any network issues, it was only a few megs, but it was something that I wanted to fix. I've ran into this issue before, and solved it by statically entering the mac-address into the cam tables. I want to avoid this problem in the future, and I'm looking at two different things. The first is preventing it in the first place. Along those lines, I've seen some recommendations on-line about changing the arp and cam timeouts to be the same. However, there seems to be a disagreement on which is better, making the arp timeouts match the cam table timeouts, or vice versa. Also, when talking about this, everyone seems to be only considering routers, but what about the timers on a firewall? I'm worried
Re: comcast price check
I can't even get reliable home cable internet service from them. No way I would ever consider using them for transit. I would only consider a stub peer with them to help out the poor Comcast customers who are also trying to get to my data centers. Owen DeLong wrote: Fair warning, Comcast is totally into the bait and switch game. Talk to any 3 people at Comcast and you will receive at least 4 different answers about what is or isn't included. Having a particular offer in writing makes no difference to them. I will be contacting the Santa Clara County District Attorney about my experiences with Comcast in violation of CA BP code S17500 soon. I spent the last two months trying repeatedly to get Comcast to recognize and live up to their obligations under the offer they originally extended to me. They waffled for a very long time before I finally reached someone who flat-out told me that they were not ever going to deliver what was promised. Owen On Feb 20, 2009, at 8:26 PM, John Martinez wrote: Does any one here use comcast's ethernet services? If so, what is their price range? Thanks in advance. -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Craptastic Service! (was: Re: comcast price check)
I don't think the expectations are that high for the money spent. They are promising a service for a particular price. They either deliver on that service in a 100% working condition or its false advertising and thus is not honest. It isn't the customers fault they decided to promise a service at a price blow market value. Jim Popovitch wrote: On Sat, Feb 21, 2009 at 02:44, Sharma, Kapeel kapeel.sha...@mckesson.com wrote: This is BS how narrow minded our providers are. It is also BS how high the expectations are for the $$ spent. ;-) -Jim P. -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Cogent haiku
LMAO thats great. I am so glad we don't peer with Cogent. Steve Fischer wrote: That is too funny! -Original Message- From: neal rauhauser [mailto:nrauhau...@gmail.com] Sent: Friday, January 09, 2009 3:06 PM To: nanog@nanog.org Subject: Cogent haiku Cogent drops packets. Angry customers call. Twice. Admin writes haiku. -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Level 3 issues
We saw our bandwidth drop on our Level3 OC-48 to about half of what we were doing. We had to stop announcing our subnets to Level3 to get traffic to fail over properly throughout the world. We have a ticket open with Level3's NOC but have not received word on what happened or when to expect a resolution. Kevin Loch wrote: marco wrote: From what I heard, it was some some malfunction with a router in Washington D.C. which terminated a 100GB bundle from Paris. It was carring about 50GB at the time of the failure. Not sure why routes within the US would be effected. We connect to level3 in Ashburn/DC and saw traffic drop 50% in both directions on that port. Testing showed 100% loss on 50% of the flows. We shut that port down and now it won't come back up. I have link but no arp for their IP. This is a new link that was turned up in the past few weeks. - Kevin -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Advice/resources for setting up TACACS server
I disagree with the RADIUS suggestion. TACACS+ is a much more secure protocol. It encrypts the packet contents and has a more secure handshake procedure. Leslie wrote: The best answer actually does seem to be to use freeradius instead of tacacs, so I will probably go with that (though if anyone has any good tips on freeradius, please, let me know) Leslie On Nov 7, 2008, at 1:30 PM, Leslie wrote: Hi -- We are currently trying to set up a TACACS server for authentication to our network gear and have it run on suse linux hosts. Does anyone have any advice/good webpages or guides regarding this? Thank you very much in advance! Leslie -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Google SMTP acceptance policy?
From my experience it just takes time. As users mark your email as legitimate and not as spam your domain will build a good report Google. Also, try implementing DKIM to help Google to verify the email. Frank Bulk wrote: Have you worked through this Q/A process? http://mail.google.com/support/bin/answer.py?answer=80369 I went through it and at the end it says there's not a way to whitelist a domain. For Bulk e-mail senders: https://mail.google.com/support/bin/answer.py?answer=17205 There's this checklist, too: https://mail.google.com/support/bin/answer.py?answer=81126 And here's a form to fill out: https://mail.google.com/support/bin/request.py?ctx=bulksendnomods=1 Frank -Original Message- From: Jonathan Traylor [mailto:[EMAIL PROTECTED] Sent: Wednesday, November 05, 2008 8:08 PM To: nanog@nanog.org Subject: Google SMTP acceptance policy? Anyone have guidance on how to legimately stay out of Google/GMail's spam classifier and arrive at the inbox? We have a domain that is relatively newly registered, has proper MTA configuration and SPF records that I haven't been able to find on any blocklist, but GMail sends email from it straight to the spam folder. I haven't been able to find useful documentation for GMail in this regard around the web. Have looked at abuse.net's info, links and resources. Thanks, -- Jonathan -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Peering - Benefits?
My company will be peering with two other SPs in the area purely for business strategic purposes. It turns out that at least one of these SPs owns the fiber running to the first CO in our transit back to Chicago. So it helps to be buddies with these companies. Paul Vixie wrote: Paul Stewart [EMAIL PROTECTED] writes: ... My question was meant at a much higher level - a level where costs are equal for peering/transit and all the technical and the financial homework has been done already now I'm the stage of one last meeting with top level management to explain peering and it's magic. These are mainly non-technical people - so my question to NANOG was for viewpoints on peering of which hopefully I could reinforce some of my own thoughts with. Whether or not someone operating at scale isn't the discussion - and it's funny how many people involved with companies (that are operating at scale) have emailed me offline since this discussion started a few days ago with questions/thoughts and strategy. if the financials and technicals are similar enough to be factored out, then what you have to look at is possible variance between tactical and strategic cost/benefit ratios. basically this boils down to the cost of lock-in. if you're going to avoid lock-in then you have get your own address space and build out to at least one IXP and then, buy diverse transit. once you have done all that, the cost of also peering is in the noise, whereas the advantage of also peering is noticeable if not always easily measureable. if you're not going to avoid lock-in, then everything you'd need to spend to avoid it can be avoided, and you won't be peering unless it's for purely strategic reasons. -- Steve King Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Peering - Benefits?
It would only be a redundant connection if the AS your peering with is a transit AS. The AS that I work with is a stub AS and can not function as a fully redundant link. Just something to watch out for. Paul Stewart wrote: Thanks! That's a really good one and surprised myself I missed it..;) _ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 29, 2008 3:28 PM To: Paul Stewart Cc: nanog@nanog.org Subject: Re: Peering - Benefits? * PGP Signed by an unknown key On Wed, 29 Oct 2008 15:17:45 EDT, Paul Stewart said: I can think of some but looking to develop a concrete list of appealing reasons etc. such as: -control over routing between networks -security aspect (being able to filter/verify routes to some degree) -latency/performance I'm surprised you didn't include chance to pick up a redundant connection. * Unknown Key * 0xB4D3D7B0 The information transmitted is intended only for the person or entity to which it is addressed and contains confidential and/or privileged material. If you received this in error, please contact the sender immediately and then destroy this transmission, including all attachments, without copying, distributing or disclosing same. Thank you. -- Steve King Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Peering - Benefits?
But if that AS is a stub, you still can't use their up stream providers to get data out to the rest of the world. It still wouldn't even function as an alternative path it would only function for the subnets which that AS owns. Paul Stewart wrote: Thanks - I believe the wording meant was alternative path versus connection... in other words if an AS has issues with one or more upstream providers for whatever reason, you have good chances the peering connection will remain in better shape (not always granted, but good odds) Paul -Original Message- From: Steven King [mailto:[EMAIL PROTECTED] Sent: October 29, 2008 6:22 PM To: Paul Stewart Cc: [EMAIL PROTECTED]; nanog@nanog.org Subject: Re: Peering - Benefits? It would only be a redundant connection if the AS your peering with is a transit AS. The AS that I work with is a stub AS and can not function as a fully redundant link. Just something to watch out for. Paul Stewart wrote: Thanks! That's a really good one and surprised myself I missed it..;) _ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 29, 2008 3:28 PM To: Paul Stewart Cc: nanog@nanog.org Subject: Re: Peering - Benefits? * PGP Signed by an unknown key On Wed, 29 Oct 2008 15:17:45 EDT, Paul Stewart said: I can think of some but looking to develop a concrete list of appealing reasons etc. such as: -control over routing between networks -security aspect (being able to filter/verify routes to some degree) -latency/performance I'm surprised you didn't include chance to pick up a redundant connection. * Unknown Key * 0xB4D3D7B0 The information transmitted is intended only for the person or entity to which it is addressed and contains confidential and/or privileged material. If you received this in error, please contact the sender immediately and then destroy this transmission, including all attachments, without copying, distributing or disclosing same. Thank you. -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Another driver for v6?
Kind of a side question but we have not implemented IPv6 in our network yet, nor have we made any plans to do this in the near future. Our management does not see a need for it as our customer base is not requesting it at this time. Does anyone see any benefits to beginning a small deployment of IPv6 now even if its just for internal usage? Bruce Curtis wrote: On Oct 29, 2008, at 10:32 AM, Joe Maimon wrote: Mikael Abrahamsson wrote: On Tue, 28 Oct 2008, Steven M. Bellovin wrote: They claim they will deploy IPv6 in their worldwide enterprise network, do away with central based enterprise firewalls and do host-to-host IPv6+IPSEC, Active Directory based certificates for authentication. You know that windows 2000 was released with this functionality. Its nothing new and it is not ipv6 specific. Who is using it precisely? Microsoft, on 200,000 computers at the time of the paper below. http://technet.microsoft.com/en-us/library/bb735174.aspx We have a couple of departments using IPsec here and one more seriously looking at it. (Mainly a matter of finding time to test and implement.) Plus there are at least a couple of other Universities. http://members.microsoft.com/CustomerEvidence/Search/EvidenceDetails.aspx?EvidenceID=14258LanguageID=1 https://members.microsoft.com/customerevidence/search/EvidenceDetails.aspx?EvidenceID=14205LanguageID=1 And I see a City has been added to the list. http://www.microsoft.com/casestudies/casestudy.aspx?casestudyid=400161 http://www.cu.ipv6tf.org/pdf/v6security_6Sense_Jan2006.pdf --- Bruce Curtis [EMAIL PROTECTED] Certified NetAnalyst II701-231-8527 North Dakota State University -- Steve King Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: Another driver for v6?
I personally agree with that. Now only if I can convince our management to start work on that. isabel dias wrote: question - beginning a small deployment of IPv6 now even if its just for internal usage Sure! there are plenty of reasons .most obvious one is to feel confortable about ipv6 --- On Wed, 10/29/08, Steven King [EMAIL PROTECTED] wrote: From: Steven King [EMAIL PROTECTED] Subject: Re: Another driver for v6? To: Bruce Curtis [EMAIL PROTECTED] Cc: nanog@nanog.org Date: Wednesday, October 29, 2008, 11:32 PM Kind of a side question but we have not implemented IPv6 in our network yet, nor have we made any plans to do this in the near future. Our management does not see a need for it as our customer base is not requesting it at this time. Does anyone see any benefits to beginning a small deployment of IPv6 now even if its just for internal usage? Bruce Curtis wrote: On Oct 29, 2008, at 10:32 AM, Joe Maimon wrote: Mikael Abrahamsson wrote: On Tue, 28 Oct 2008, Steven M. Bellovin wrote: They claim they will deploy IPv6 in their worldwide enterprise network, do away with central based enterprise firewalls and do host-to-host IPv6+IPSEC, Active Directory based certificates for authentication. You know that windows 2000 was released with this functionality. Its nothing new and it is not ipv6 specific. Who is using it precisely? Microsoft, on 200,000 computers at the time of the paper below. http://technet.microsoft.com/en-us/library/bb735174.aspx We have a couple of departments using IPsec here and one more seriously looking at it. (Mainly a matter of finding time to test and implement.) Plus there are at least a couple of other Universities. http://members.microsoft.com/CustomerEvidence/Search/EvidenceDetails.aspx?EvidenceID=14258LanguageID=1 https://members.microsoft.com/customerevidence/search/EvidenceDetails.aspx?EvidenceID=14205LanguageID=1 And I see a City has been added to the list. http://www.microsoft.com/casestudies/casestudy.aspx?casestudyid=400161 http://www.cu.ipv6tf.org/pdf/v6security_6Sense_Jan2006.pdf --- Bruce Curtis [EMAIL PROTECTED] Certified NetAnalyst II701-231-8527 North Dakota State University -- Steve King Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional -- Steve King Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Re: ARP Table Timeout and Mac-Address-Table Timeout
I saw that one before. Thats what we based our current fix on. Frank Bulk wrote: Steven: This was recently discussed on cisco-nsp: http://marc.info/?l=cisco-nspm=121316151010190w=2 Frank -Original Message- From: Steven King [mailto:[EMAIL PROTECTED] Sent: Sunday, September 14, 2008 7:27 PM To: nanog@nanog.org Subject: ARP Table Timeout and Mac-Address-Table Timeout I am a network engineer for a large web hosting company. We are having an issue with our distribution routers flooding traffic in one of our VLANs. We have a customer with a routed mode ASA 5550. They have their own private VLAN that is a /23 This VLAN is 145. The outside interface of the firewall is in VLAN 132. We are routing all traffic for VLAN 145 to the IP of the outside interface of the firewall in VLAN 132. VLAN 132 is Layer3 routable and VLAN 145 is only Layer2 switchable. We have two distribution switches which are redundant with HSRP. Dist1 is the active forwarder in this case. Traffic coming into these two routers are load balanced between Dist1 and Dist2 with EIGRP routes with equal cost. We have found that traffic coming into Dist2 (the standby) is flooding traffic destined for the firewall outside interface. But Dist1 is not. We have tracked down the cause of this to the MAC-Address-Table timing out before the ARP table times out. We leave these values at the Cisco default. ARP = 4hr MAC = 5 minutes. Since Dist2 is not receiving any traffic from the firewall going out to the internet, it is not updating the MAC-Address-Table after it expires. Instead, it waits 4 hours for the ARP cache to expire for that IP, and then updates everything. But Dist2 ends up flooding traffic for that 4 hours causing latency. We have done some research on this problem and have found so far the best solution to be to make the ARP timeout less than the MAC-Address-Table aging-timer.We have set the ARP = 1hr and MAC = 2hrs in this case to correct the problem. So when the ARP entry times out before the MAC entry, the forced update of the ARP entry before the MAC timeout causes the MAC entry age to reset. Indeed this does correct the problem. Is this the best solution to the problem, or is there another preferred solution? Has anyone ran into this in their own Enterprise Networks? Please let me know if I didn't explain anything well enough. -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA Network+ Certified Professional CompTIA A+ Certified Professional -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA Network+ Certified Professional CompTIA A+ Certified Professional
ARP Table Timeout and Mac-Address-Table Timeout
I am a network engineer for a large web hosting company. We are having an issue with our distribution routers flooding traffic in one of our VLANs. We have a customer with a routed mode ASA 5550. They have their own private VLAN that is a /23 This VLAN is 145. The outside interface of the firewall is in VLAN 132. We are routing all traffic for VLAN 145 to the IP of the outside interface of the firewall in VLAN 132. VLAN 132 is Layer3 routable and VLAN 145 is only Layer2 switchable. We have two distribution switches which are redundant with HSRP. Dist1 is the active forwarder in this case. Traffic coming into these two routers are load balanced between Dist1 and Dist2 with EIGRP routes with equal cost. We have found that traffic coming into Dist2 (the standby) is flooding traffic destined for the firewall outside interface. But Dist1 is not. We have tracked down the cause of this to the MAC-Address-Table timing out before the ARP table times out. We leave these values at the Cisco default. ARP = 4hr MAC = 5 minutes. Since Dist2 is not receiving any traffic from the firewall going out to the internet, it is not updating the MAC-Address-Table after it expires. Instead, it waits 4 hours for the ARP cache to expire for that IP, and then updates everything. But Dist2 ends up flooding traffic for that 4 hours causing latency. We have done some research on this problem and have found so far the best solution to be to make the ARP timeout less than the MAC-Address-Table aging-timer.We have set the ARP = 1hr and MAC = 2hrs in this case to correct the problem. So when the ARP entry times out before the MAC entry, the forced update of the ARP entry before the MAC timeout causes the MAC entry age to reset. Indeed this does correct the problem. Is this the best solution to the problem, or is there another preferred solution? Has anyone ran into this in their own Enterprise Networks? Please let me know if I didn't explain anything well enough. -- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA Network+ Certified Professional CompTIA A+ Certified Professional
