Wireless/Free Space Enterprise ISP in Palo Alto
Apologies if this is not the most appropriate forum for this, but I am not aware of a better list to use. I recently took over responsibility for the network connectivity at an office in downtown Palo Alto (University and Emerson). Unfortunately, and perhaps ironically, the connectivity options here are not as great as I would like- currently they are using DSL, there is no cable service, and lead times for T1's and above are higher than I would like. I have already started the process of getting fiber from the city of Palo Alto between the office and Equinix/SD/PAIX/529 Bryant, which will resolve the problem. However, the city's time estimate is longer than we need, and experience implies there may be delays. So- I am investigating wireless/free space ISP's in downtown Palo Alto, and also possible options for doing a point-to-point wireless connection between our office and 529 Bryant (I am reaching out to Equinix on this). Any pointers to known good providers, or other suggestions would be very welcome- off list is fine. I am already reaching out to a telecom broker, but wanted to reach out for suggestions. To clarify- I am trying to get either a) rapidly turned up IP transit or b) rapidly turned up point-to-point (presumably wireless) between 529 Bryant/PAIX and the office (University and Emerson). Thanks very much! --D -- -- Darren Bolding -- -- dar...@bolding.org --
Experiences with advanced network taps.
We are planning on purchasing some network taps for a couple of locations in our network, and we expect to make significantly greater use of them in the next year or two. Something that is new since I last investigated taps (it has been a while) is that many of them now allow for functionality I would typically think of as far outside what a simple tap does. For example: Selective forwarding of packets based on MAC address, TCP/UDP port, IP address range etc. Selective forwarding/load balancing based on flow, so that you can distribute traffic across a cluster of devices (e.g. IDS or netflow probes) Ability to insert a device (firewall, IDS, etc) into the network flow and via software configuration bypass traffic around the device- e.g. able to quickly drop a device out of the network path. - Some have the ability to send network probes, or monitor traffic downstream of an inline device so they can automatically take the device out of line if it fails to pass traffic. - Some can filter which traffic goes through the inline device and merge it back with the traffic that was not sent to the inline device for downstream consumption. Some can be connected and automatically be managed as if one device, allowing monitor and replication ports to be used across the stack/mesh of devices. All of this is very interesting. Of course these taps cost more than your basic dumb tap. More interestingly to me is that these taps are no longer dumb, and that makes them a bit of a riskier proposition. In evaluating some we have run into issues ranging from misconfiguration/user error to what appear to be crashes (with associated loss of forwarding). I'm wondering if anyone has had significant experience deploying these more advanced taps, whether it was good or bad, general comments you might like to share regarding them, and whether you would recommend particular vendors. If people reply off-list, I will make a point of summarizing back if I get any feedback. Thanks! --D -- -- Darren Bolding -- -- dar...@bolding.org --
Re: Enterprise DNS providers
I have been quite happy with Dynect so far. They were very flexbile on a number of items and the service has been great. On Mon, Oct 18, 2010 at 12:13 AM, Shacolby Jackson shaco...@bluejeansnet.com wrote: I have used UltraDNS before. They are decent. I am however evaluating Dynect (www.dyn.com) who are very popular with social media companies like Twitter. On Sun, Oct 17, 2010 at 11:17 PM, Ken Gilmour ken.gilm...@gmail.com wrote: On 18 October 2010 06:53, Jonas Björklund jo...@bjorklund.cn wrote: I have worked for one of the biggest poker networks and we used UltraDNS. The company was first operated from Sweden and later Austria. /Jonas I would tend to agree... I have also used UltraDNS in the past for other companies, however we needed them urgently and someone else responded faster and they seem to be doing a good job so far. Regards, Ken -- -- Darren Bolding -- -- dar...@bolding.org --
Re: What must one do to avoid Gmail's retarded non-spam filtering?
Ignoring the irony, you could signup with Microsoft's spam filtering service (formerly frontbridge) or postini (now google) and use them as outbound relays. They will do outbound relay, with attendant spam filtering and increases in deliverability. That means a lot more people will accept your mail when it is being relayed by postini/front bridge. I believe postini will do this for any account, even one that has only, say, one e-mail address being filtered by them and the rest silently passed. Heck, you may not even need to point your mx records to their servers- they just need a reinjection method for outbound filtering if I recall correctly. Again, I acknowledge the irony involved in such a solution. --D On Tue, Sep 28, 2010 at 9:05 PM, Erik L erik_l...@caneris.com wrote: Google appears to have blacklisted our domain. From the edge MTA, I sent three messages, differing only in the From header: 1. valid email @klssys.com 2. valid email @caneris.com 3. abc...@caneris.com 1 not spam; 2 3 spam Return-Path of all three was still another address @caneris.com and the two SPF passed headers were still there. The body subject of all messages was simply testing5. The recipient even clicked on not spam on #2 prior to #3 being sent. At least it explains why all the other standard stuff we all looked at didn't explain the issue. The next (and last) question is: does anyone have a clueful Google mail ops contact? Thanks again for all the help and sorry for the OT noise. Erik - Original Message - From: Erik L erik_l...@caneris.com To: William Pitcock neno...@systeminplace.net Cc: nanog@nanog.org Sent: Tuesday, September 28, 2010 7:17:45 PM Subject: Re: What must one do to avoid Gmail's retarded non-spam filtering? Hi William, I do so for our entire IP space on a regular basis. The edge MTA I mentioned in the reply to Bill shows up as Neutral there. Thanks Erik - Original Message - From: William Pitcock neno...@systeminplace.net To: Erik L erik_l...@caneris.com Cc: nanog@nanog.org Sent: Tuesday, September 28, 2010 6:06:49 PM Subject: Re: What must one do to avoid Gmail's retarded non-spam filtering? Hi, Have you checked the IronPort reputation scores for your mailserver IPs? Google uses this data as part of it's spam detection method. William On Tue, 2010-09-28 at 16:15 -0400, Erik L wrote: I realize that this is somewhat OT, but I'm sure that others on the list encounter the same issues and that at least some folks might have useful comments. An increasingly large number of our customers are using Gmail or Google Apps and almost all of our OSS/BSS mail is getting spam filtered by Google. Among others, these e-mails include invoices, order confirmations, payment notifications, customer portal logins, and tickets. Almost anything we send to customers on Google ends up in their spam folder. This results in a lot of calls and makes much of our automation pointless, never mind all the lost sales. The problem is compounded by those who use mail clients and do not log in to the webmail at all, since they would never see the contents of the Google spam folder. We have proper A+PTR records on the edge MTAs, proper SPF records for the originating domain, proper Return-Path and other headers, and so on. There isn't anything that I can think of other than the content itself which would be abnormal, and obviously the content is repetitive and can't be changed much. Is there something obvious which we've missed? Aside from the following clearly impractical solutions, what can we do? 1. Asking everyone (including those we don't even know yet) to whitelist all of our addresses, to check their spam folders, and to click on this is not spam 2. Providing our own free e-mail service to everyone (including those we don't even know yet) and putting up don't use Google ads on all of our customer-facing systems At least this isn't Hotmail where mail is just silently deleted with no NDR after it's accepted by their MTAs. The call volume has been going up instead of down lately and it's gotten to the point where we're sending MTA log extracts to people to prove to them that we really did e-mail them. Would greatly appreciate any advice. Erik -- -- Darren Bolding -- -- dar...@bolding.org --
Re: SPANS Vs Taps
Tap manufactures will be sure to tell you of many issues. The main concern I would have is that it is possible for a switch to drop frames of a SPAN. Your decision might be influenced based on your application and the impact of such errors (billing, lawful intercept, forensics). A tap vendors take: http://www.networkcritical.com/What-are-Network-Taps On a somewhat related note, I will mention that TNAPI from ntop is quite handy. http://www.ntop.org/TNAPI.html http://www.networkcritical.com/What-are-Network-Taps--D On Thu, Jul 1, 2010 at 1:48 PM, Bein, Matthew mb...@iso-ne.com wrote: As I was doing a design today. I found that I had a bunch of 100 MB connections that I was going to bring into a aggregation tap. Then I was thinking, why don't I use a switch like a Cisco 3560 to gain more density. Anyone run into this? Any down falls with using a switch to aggregate instead of a true port aggregator?? Regards, Matthew -- -- Darren Bolding -- -- dar...@bolding.org --
Re: Experiences with A10 AX series Load Balancers?
Very interesting to see about A10's performance- I've heard mixed things about them. Just an FYI, the newer F5 platforms don't utilize the ASIC's- the performance curve of general-purpose CPU's has once again eclipsed what can be done with specialized silicon without aggressive (and expensive) revision cycles. The ASIC's also could only be used in simpler virtual server configurations and with certain subsets of iRules. That said, nothing else I'm aware of provides the functionality of iRules. I've used netscalers only a relatively small amount- and they are nice- particularly if your requirements are within their feature set- but my experience has been that things I take for granted using an iRule are seriously painful to implement on a netscaler. --D On Wed, Mar 24, 2010 at 3:33 PM, Justin Horstman jhorst...@adknowledge.comwrote: The boxes do alright at low load levels. They do not have an asic tech like the F5s so choke on large amounts of traffic. Management is a bit immature and you will find yourself having to use the CLI and the Gui to accomplish most advanced tasks. When we put them head to head A10 AX3200 vs F5 6400 ltm (note: 6400 was what we were looking to replace) Test: 1000 concurrent users from Gomez's Networks Loadtesting platform hitting as fast as the requests would close, going through our standard vip config on the f5, and the A10 engineering teams 3 best efforts to beat that config that balanced between two Identical Dell 1950 servers serving a php page that responded with a random number (to avoid caching). The 6400 we used was in production at the time, and was older so we were expecting to get blown away, see the results here: F5 - Peaked 160k completed transactions a minute sustained for 10 minutes, 0 errors, 112ms average transaction response time A10 - Held 60k completed transactions a minute sustained for 10 minutes, 0 errors, 360ms average transaction response time If anyone is interested in the graphs I think I can still pull them out of gomez. Though notable that this was all done a year ago, so things might be different now. ~J -Original Message- From: Welch, Bryan [mailto:bryan.we...@arrisi.com] Sent: Tuesday, March 23, 2010 8:35 PM To: nanog@nanog.org Subject: Experiences with A10 AX series Load Balancers? Does anyone have any experiences good/bad/indifferent with this company and their products? They claim 2x the performance at ½ the cost and am a bit leery as you can imagine. We are looking to replace our aging F5 BigIP LTM's and will be evaluating these along with the Netscaler and new generation F5 boxes. Regards, Bryan -- -- Darren Bolding -- -- dar...@bolding.org --
Re: log parsing tool?
SEC (Simplet Event Correlator) is a very effective tool for this, IMHO. I am by no means an expert with it, but I know several people who are, and while it is not as well known as splunk or some other tools, I have been very impressed by the results I've seen using it. As with any event correlation tool, there is a significant level of invested effort required to make use of this. http://simple-evcorr.sourceforge.net/ Below is a presentation about SEC. http://www.occam.com/sa/CentralizedLogging2009.pdf On Mon, Feb 22, 2010 at 2:15 PM, fedora fedora fedoraf...@gmail.com wrote: Greetings, Anyone has good recommendations for an open-sourced log parsing and analyzing application? It will be used to work with syslog-ng and other general syslog and application logs. I have been looking at swatch and logwatch, but would like to find out if there are other good choices, thanks FD -- -- Darren Bolding -- -- dar...@bolding.org --
Re: The Internet Revealed - A film about IXPs v2.0: now available
Look, it's a very nice video, and I think it is useful and the creators should be complimented on their work. Overall it is something I would like to use to educate less IP-savvy folk. But, as a hyper-aware viewer I did detect a tone in favor of network neutrality type arguments- and I suppose that is OK. One thing I found that didn't match with my recollection is that it depicts IXP's as a response to private peering. My recollection was that while the earliest peering may have been some private peering, rapidly MAE-EAST etc. became points of major traffic sharing and large scale private peering/interconnects were a response to the issues at the various meeting points. Perhaps my recollection is incorrect? And aren't most exchanges today effectively private interconnects across a shared L2 device? On Wed, Feb 10, 2010 at 3:30 PM, Patrick W. Gilmore patr...@ianai.netwrote: On Feb 10, 2010, at 11:50 AM, Mikael Abrahamsson wrote: On Wed, 10 Feb 2010, Patrick W. Gilmore wrote: Agree to disagree is right. The film is called The Internet Revealed: _A_film_about_IXPs_. You find it strange that the film would actually focus on IXPs. I find it strange that you couldn't figure this out before clicking play. If it would have said The internet revealed - an advertisement for IXPs I might have been expecting the thing I got. It's a matter of degree, right? However, I do believe you should know how the Internet works. And if you honestly believe packets in a single stream cannot travel over different paths, you clearly do not. And before you come back with BS about normal operation or such, realize your statement was far more factually incorrect than what the video said about private interconnects. I'm saying they don't normally do so, as one might believe when looking at the movie. Any core router ECMP algorithm that sprays L4 sessions like that will cause re-ordering which is bad, mkay. Yes, flow switching is common, but it is by no means guaranteed. Lots of people do per-packet across LAG bundles. The Internet topology changes do not wait until all TCP sessions are complete. Not everyone does flow switching. Etc. Which all means, as I said in my last sentence above, that you are doing exactly what you accuse them of doing - only worse. Your facts are not facts, the most you can accuse this video of is not explaining things fully. I guess the only question left is: What are you advertising? But I'll shut up after this, I'm obviously not jaded enough like you other people to just swallow this as advertisement. I expected a correct factual way of describing how the Internet works including IXPs, not an IXP advertisement. My expectations were obviously wrong from the response I'm seeing. I wouldn't call you jaded when you do what you accuse others of doing. And to be clear, you got a correct factual way of describing how the Internet works including IXPs. It may not have been complete, but if you honestly expected a complete description of the Internet in a film of /any/ length ... well, words fail me. -- TTFN, patrick -- -- Darren Bolding -- -- dar...@bolding.org --
Re: D/DoS mitigation hardware/software needed.
I know of several companies, with large websites, that used code reviews as at least one way they met this DSS requirement. So, erm, empirically denied. The PCI DSS does not require code review of the software running in COTS equipment, nor of underlying OS's or applications. It requires a code review of the application code that is inside PCI scope. In general, this means the code you write to run your website is the maximum scope of this requirement. Plenty of companies allow code reviews for security and other purposes, and with good reason. There exist entire practices in IT security firms dedicated to performing code reviews, and they appear to be growing. Also, the PCI security council allows people to use automated code auditing tools (such as fortify), performing a manual application assessment- which plenty of firms will let you pay them to do, or even to use an automated web application security scanners. Several vendors of Vulnerability Assessment tools that meet this spec are available. I believe their is strong evidence that the use of web application firewalls to meet this DSS requirement is smaller than you might think. I would not be surprised if it was significantly less than 50%- perhaps 20%. To make the operational content clear- if someone tells you that you need to buy a Web Application Firewall to meet PCI requirements (process credit cards), be aware that is not the only option. I'd recommend you choose the option that is most likely to genuinely improve the security of your infrastructure and business, which may well be a WAF. --D On Mon, Jan 4, 2010 at 11:54 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 2:38 PM, Darren Bolding wrote: PCI DSS does not require a Web application firewall. http://searchsoftwarequality.techtarget.com/news/article/0,289142,sid92_gci1313797,00.html Since no business is going to allow an external 'code review' (if it's even possible, given that they're likely using COTS products, the source code of which they simply don't have), this defaults to a requirement for the 'Web application firewall'. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken -- -- Darren Bolding -- -- dar...@bolding.org --
Re: D/DoS mitigation hardware/software needed.
Comments inline. To reiterate- my entire point is that stateful firewalls are at least sometimes useful in front of large websites. Something of a veer off of the original topic, but not an at all useless exercise IMHO. On Mon, Jan 4, 2010 at 11:47 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 2:38 PM, Darren Bolding wrote: * Defense in depth. You've never had a host that received external traffic ever accidentally have iptables or windows firewall turned off? Even when debugging a production outage or on accident? Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. 'Stateful inspection' where in fact there is no useful state to inspect is pointless. Stateful policies help protect against misbehaving software, operations error, malicious internal activity, external attackers and trojans/viruses/bots making outbound connections to arbitrary ports anywhere on the Internet. Seems to have a point to me. See my previous point about blocking outbound connections. If you allow internal hosts to make connections out to any port on any ip, then perhaps this point is invalid. But lots of people _don't_ allow outbound from any to any/any, and thats a good thing in my book. As for performance, you can get .9mpps stateful firewalls for reasonable prices, and gear exists that can scale up to 15mpps. * Location for IDS/IDP. Non-sequitur, as these things have nothing to do with one another (plus, these devices are useless, anyways, heh). The gear I'm talking about above supports IDP on the same platform. If I am going to deploy an IDP, why wouldn't I choose one that is flow based so it blocks attacks rather than send out panicked RST's in response to attacks, and which has a stateful firewall included (yes, any flow based IDP is essentially a stateful firewall, and I suspect that a well managed IDP with customized rules is similar in functionality to what WAF's do out of the box. In fact one vendors WAF product functionality pre-trained out of the box is snort rules) * Connection cleanup, re-assembling fragments, etc. Far, far, far better and more scalably handled by the hosts themselves and/or load-balancers. Huh, I would have sworn I've seen fragment based attacks on certain linux kernels and Windows systems. I'm sure it won't ever happen again. Load-balancers may be a good place for this, but there are good reasons that you would prefer to have this taken care of as close to ingress as possible. For example, if you want to compare packet flows of a connection on either side of a load-balancer, it is easier to do that if the incoming connections have already been cleaned up. * SYN flood protection, etc. Firewalls simply don't handle this well, marketing claims aside. They crash and burn. I believe that to have been true in the not so recent past, I don't think it is true of more recent equipment. * Single choke point to block incoming traffic deemed undesirable. Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. A firewall with easier to use CLI, web UI, management station and API is a much easier way to implement this and offers lots of control for time-of-day based ACL's, etc. etc. I like screening routers, and in a DOS situation I would recommend blocking as close to ingress as possible (preferably outside your own network!) but for typical blocking of nuisance connections, full featured firewalls that can be integrated with a configuration change tracking tool are a better way to go. And what about when the way of identifying undesirable traffic is something other than the source/dest ip/port such as a URI? (more of an argument for an IDP than a simply stateful firewall) * Single log point for inbound connections for analysis and auditing requirements. Contextless, arbitrary syslog from firewalls and other such devices is largely useless for this purpose. NetFlow combined with server/app/service logs is the answer to this requirement. Really? How does netflow show me that a packet was dropped because of rule X? Or permitted because of rule Y? It's great at showing me data about the flow that was extant, but it doesn't tell me the context of the enforcement performed on it. I can assure you that there is a use in looking over denied packet logs. Firewall logs provide _more_ context than netflow. For DOS detection I get that netflow is probably the best tool to use- not arguing that- you made the statement that stateful firewalls are always useless in front of large websites/webservers. I am saying that they are usefull, at least some of the time, and that it is common for them to be deployed in that manner. * Allows outbound traffic enforcement. Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. So you advocate allowing any packet with ACK set
Re: D/DoS mitigation hardware/software needed.
My basis for this is discussions with PCI assessors from multiple firms that perform large numbers of assessments per year. Next time I run into some, I'll ask to see if the usage has increased, its been a few months since I asked this of any of them. --D On Tue, Jan 5, 2010 at 1:02 AM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 3:58 PM, Darren Bolding wrote: I believe their is strong evidence that the use of web application firewalls to meet this DSS requirement is smaller than you might think. I would not be surprised if it was significantly less than 50%- perhaps 20%. This directly contradicts my experience working for vendor of such products, FWIW. But I hope this is indeed the case, as it will lead to higher availability for organizations which go this route! --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken -- -- Darren Bolding -- -- dar...@bolding.org --
Re: D/DoS mitigation hardware/software needed.
. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken -- -- Darren Bolding -- -- dar...@bolding.org --
Re: fight club :) richard bennett vs various nanogers, on paid peering
Whether or not Mr Bennett has any idea what he is talking about- and I have started to develop an opinion on that subject myself- I really would rather not see Nanog become a forum for partisan political discussion. There are _lots_ of places for that, which as a political junkie I read regularly. I like Nanog in part because it typically steers clear of this sort of thing (and you know the mailing list charter sez) and in some way serves as a refreshing change between reading Daily Kos and Powerline blogs. I will also say that while Mr Bennett's affiliation and paycheck have some relevance to interpreting what he says, it isn't justification for tossing everything he says out. If he seems to have no idea what he is talking about, that is reason for tossing out what he says. One final point- referring to conservadems is about as telling about perspective as certain people referring to RINO's. Bennett hasn't said anything blatantly partisan (perhaps he is to polished for that), his critics certainly have. You diminish your argument by doing so. I say all this even though some of the people getting engaged in this are people I've known for a while and respect a great deal, and others are ones I've read on Nanog for a number of years. I'm actually intersted in the substantive content, but I'd rather avoid the rest if you wouldn't mind. Thanks for listening, --D On Wed, Nov 25, 2009 at 7:13 AM, valdis.kletni...@vt.edu wrote: On Wed, 25 Nov 2009 03:32:02 PST, Richard Bennett said: ITIF is not opposed to network neutrality in principle, having released a paper on A Third Way on Network Neutrality, http://www.itif.org/index.php?id=63. All of four paragraphs, which don't in fact address what the provider is or is not providing to Joe Sixpack - point 1 says discriminatory plans are OK as long as the discriminatory are on display in the cellar of the ISP office, with no stairs, in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying Beware of the Leopard. And points 2 and 3 are saying that this should all be overseen by the same agencies that oversaw the previous decade's massive buildout of fiber to the home that was financed by massive multi-billion dollar incentives. Oh wait, those billions got pocketed - if the massive fiber buildout had happened, we'd have so much bandwidth that neutrality wouldn't be an issue... But then, the Republicans keep saying they are not opposed to health care reform in principle either... -- -- Darren Bolding -- -- dar...@bolding.org --
Re: Password repository
Pwman On 11/18/09, Jay Nakamura zeusda...@gmail.com wrote: Quick question, does anyone have software/combination of tools they recommend on centrally store various passwords securely? Thanks. -- Sent from my mobile device -- Darren Bolding -- -- dar...@bolding.org --
Re: Redundant Data Center Architectures
Also, commercial solutions from F5 (their GTM product and their old 3-DNS product). Using CDN's is also a way of handling this, but you need to be prepared for all your traffic to come from their source-ip's or do creative things with x-forwarded-for etc. Making an active/active datacenter design work (or preferably one with enough sites such that more than one can be down without seriously impacted service) is a serious challenge. Lots of people will tell you (and sell you solutions for) parts of the puzzle. My experience has been that the best case is when the architecture of the application/infrastructure have been designed with these challenges in mind from the get-go. I have seen that done on the network and server side, but never on the software side- that has always required significant effort when the time came. The drop in solutions for this (active/active database replication, middleware solutions, proxies) are always expensive in one way or another and frequently have major deployment challenges. The network side of this can frequently be the easiest to resolve, in my experience. If you are serving up content that does not require synchronized data on the backend, then that will make your life much easier, and GSLB, a CDN or similar may help a great deal. --D On Wed, Oct 28, 2009 at 10:57 AM, Roland Dobbins rdobb...@arbor.net wrote: On Oct 29, 2009, at 12:42 AM, Ray Sanders wrote: Could you elaborate on GSLB (Global Load Balancing?) ? Architectural choices, implementation scenarios, DNS tricks to ensure optimal cleaving to and availability of distributed nodes within a given tier: http://www.backhand.org/mod_backhand/ http://www.backhand.org/wackamole/ http://www.spread.org/ http://www.dsn.jhu.edu/research/group/secure_spread/ http://wiki.blitzed.org/DNS_balancing http://www.cisco.com/en/US/products/hw/contnetw/ps4162/ --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Sorry, sometimes I mistake your existential crises for technical insights. -- xkcd #625 -- -- Darren Bolding -- -- dar...@bolding.org --
Power basics- any resources?
I need to teach an operations group (Systems, Network, Security, IT folks) some basics of power as it applies to the team. Normal things they will see and should understand when they are planning on bringing equipment online etc. I have an idea of many things to talk about, particularly as they apply to our environment, but I was hoping there might be some publicly available slides or docs that I could take advantage of. I'll be happy to share anything we come up with- though this may just be me in front of a whiteboard. Here are some of the questions I plan to cover: What are amps, volts and watts? What are Volt-Amps, and why do they matter? How much can I load that circuit? How does a breaker work? Where are the breakers at our sites? What should you do if a breaker trips? How do we know how much load a device will place on a circuit? How do we monitor power usage? Basic safety. Redundant power. Thanks for any pointers! -- -- Darren Bolding -- -- dar...@bolding.org --
Re: Request for a pointer - Linux modifying DSCP on replies?
I wanted to go ahead and reply back with what I figured out. The easiest way to solve this problem turned out to be to utilize netfilters CONNMARK module, which sadly is not available in some older but still used linux kernels. Syntax for this is as follows: # Set outgoing DSCP on connections to same as incoming DSCP -A PREROUTING -m dscp --dscp 1 -j CONNMARK --set-mark 1 -A POSTROUTING -m connmark --mark 1 -j DSCP --set-dscp 1 -A PREROUTING -m dscp --dscp 2 -j CONNMARK --set-mark 2 -A POSTROUTING -m connmark --mark 2 -j DSCP --set-dscp 2 And this goes on for all 63 possible non-zero markings. This seems to have had negligible performance or memory impact on some very busy hosts, so it seems like a viable solution. --D On Mon, Aug 17, 2009 at 4:08 PM, Darren Bolding dar...@bolding.org wrote: Steve, Perhaps it is outside the DS domain, and that is the issue. It seems odd that the behavior with ICMP/Ping is different than that with TCP however. Not sure which is technically correct, but I am going to follow up on some of the pointers I've gotten to try and learn that. It just seems natural to me that connection oriented traffic would have the same markings on both sides of the conversation unless explicitly told otherwise. I would love to be able to mark the traffic at the edge of the DS domain- I do this at ingress from one location. The challenge I am trying to solve is that the DS edge switch will not reliably know how to policy-route traffic unless it has been previously marked. To clarify, as in many other environments, we have stateful devices such as firewalls and load balancers. I want to be able to route traffic that ingressed through one of these devices to egress through it as well. This is entirely solvable by splitting equipment functionally (a cluster of servers and associated network equipment, real or virtual associated with a service) or by employing SNAT solutions. However, for various reasons these solutions are not preferred in our environment, and I dare say I am not alone in that viewpoint. What I am trying to deploy now is a system where the stateful equipment (in this case a load balancer) has its traffic to the rest of the network tagged on ingress. Since I am using Cisco 6500's with sup720's, I can classify and mark the traffic with a DSCP setting via PFC/DFC hardware. I then inspect traffic at the layer-3 edge for the various pools of servers. Depending on the DSCP marking of the packet, I change the next-hop. Since this is implemented through an extended ACL for a route-map it is handled in hardware (a good thing). Research shows that I can implement similar functionality in hardware on L3 switching gear from Juniper, Foundry, etc. so I am not boxing myself into a vendor. I don't believe Cisco supports using reflexive-acl's to apply policy routing, and even if they did, that would likely swamp our sup's CPU's, so I don't believe maintaining a stateful filter on the switch is viable. This all works as expected for Ping's and the ICMP replies. It breaks down for TCP http/mysql connections. It sounds like the correct (per-spec) solution may be to have the Linux servers track the incoming connections DSCP setting and mark the outgoing packets related to those connections. I am not at all certain this will not hit the servers CPU's more than desired or require additional connection-tracking resources than the ones we currently implement via iptables. Is there some other design option I am not considering? Thanks to those of you who have replied so far, it is at least a start down some additional paths of research for me! It's been since the days of BSDI that I have been involved in system networking internals, so I have been at a loss who to even ask! --D On Mon, Aug 17, 2009 at 2:44 PM, Steve Miller stmi...@gmail.com wrote: Would not the end station be considered to be outside of the DS domain? It does not necessarily make sense (to me) for reply packets to be marked unless they are appropriate classified and marked on the return path at the point they re-enter the DS domain. I would imagine that iptables and the DSCP target would do what you wanted, yes. I'd consider classifying and marking traffic at whatever switch you would consider to be at the edge of the DS domain (connected to this server.) -Steve 2009/8/17 Darren Bolding dar...@bolding.org: I believe this is operational content, but may well be better asked somewhere else. I would love to have a pointer to another list/website. I am looking to do some policy routing based on DSCP marking, and I have this all working inside the networking equipment. I DSCP mark some packets at ingress and I policy-route others based on ACL's matching those DSCP markings. This should allow me to solve some problems in a rather elegant manner, if I do say so myself. And this works fine for some things- I have verified that Ping's
Re: F5/Cisco catalyst configuration question
What model BIG-IP? On some models I have had to set the BIG-IP's or the 6500 (can't remember which) to specified speed/duplex and the other side to auto. I believe it was auto on the BIG-IP and fixed on the 6500. Setting both sides the same did not work. On Wed, Aug 19, 2009 at 10:41 AM, Christopher Greves christopher.gre...@mindspark.com wrote: Scott, We've had issues in the past with IOS 6500's auto-negotiating uplink ports with an LTM into ISL Trunk mode. This only occurred when we had the port on the LTM configured as a tagged interface. It was easily solved by forcing the port on the 6500 into dot1q encapsulation. I'm not sure this necessarily explains why you aren't seeing a link light on the LTM though. I can't remember what the interface status was on both sides. This does correlate to why it's working on the 2950's as they don't support ISL and would likely negotiate into dot1q. Chris Christopher Greves | Senior Systems Engineer One North Lexington Ave, 9th Floor - White Plains, NY 10601 T 914-826-2067 | C 914.420.8340 | E christopher.gre...@mindspark.com Mindspark Interactive Network, Inc. is an IAC company. -Original Message- From: Scott Spencer [mailto:sc...@dwc-computer.com] Sent: Wednesday, August 19, 2009 1:13 PM To: nanog@nanog.org Subject: F5/Cisco catalyst configuration question Trying to link an F5 Local Traffic Manager with a Cisco Catalyst 6500 , have matched ports (speed,duplex ect..) but no link light at all on the F5. Does link with a Cisco 2950 switch in between but I need a direct connection with the 6500. Any suggestions what to try? Best regards, Scott Spencer Data Center Asset Recovery/Remarketing Manager Duane Whitlow Co. Inc. Nationwide Toll Free: 800.977.7473. Direct: 972.865.1395 Fax: 972.931.3340 mailto:sc...@dwc-computer.com sc...@dwc-computer.com http://www.dwc-it.com/ www.dwc-it.com Cisco/Juniper/F5/Foundry/Brocade/Sun/IBM/Dell/Liebert and more ~ -- -- Darren Bolding -- -- dar...@bolding.org --
Request for a pointer - Linux modifying DSCP on replies?
I believe this is operational content, but may well be better asked somewhere else. I would love to have a pointer to another list/website. I am looking to do some policy routing based on DSCP marking, and I have this all working inside the networking equipment. I DSCP mark some packets at ingress and I policy-route others based on ACL's matching those DSCP markings. This should allow me to solve some problems in a rather elegant manner, if I do say so myself. And this works fine for some things- I have verified that Ping's to a host work as expected- the Ping shows up at the destination host DSCP marked, and the ICMP reply leaves with the same DSCP marking. However, when I do this with apache and mysql connections (TCP 80/3306), the incoming packets are marked, but the replies are not. My research into the subject doesn't seem to suggest there is a standard for whether replies to a TCP connection are required to have the same DSCP marking, but it seems to make a lot of sense that they would. I've disabled iptables on the server host to no avail. I've looked around for an apache or Linux kernel setting and found nothing. At this point I'm looking for pointers- to a way to solve this issue, or to a better place to ask. I've started investigating writing iptables rules to match incoming connections that have DSCP marking and explicitly mark response traffic, but that seems, I don't know... wrong. Linux kernel we are using is 2.6.9-67.ELsmp. Any help or pointers would be appreciated! --D -- -- Darren Bolding -- -- dar...@bolding.org --
Re: Request for a pointer - Linux modifying DSCP on replies?
Steve, Perhaps it is outside the DS domain, and that is the issue. It seems odd that the behavior with ICMP/Ping is different than that with TCP however. Not sure which is technically correct, but I am going to follow up on some of the pointers I've gotten to try and learn that. It just seems natural to me that connection oriented traffic would have the same markings on both sides of the conversation unless explicitly told otherwise. I would love to be able to mark the traffic at the edge of the DS domain- I do this at ingress from one location. The challenge I am trying to solve is that the DS edge switch will not reliably know how to policy-route traffic unless it has been previously marked. To clarify, as in many other environments, we have stateful devices such as firewalls and load balancers. I want to be able to route traffic that ingressed through one of these devices to egress through it as well. This is entirely solvable by splitting equipment functionally (a cluster of servers and associated network equipment, real or virtual associated with a service) or by employing SNAT solutions. However, for various reasons these solutions are not preferred in our environment, and I dare say I am not alone in that viewpoint. What I am trying to deploy now is a system where the stateful equipment (in this case a load balancer) has its traffic to the rest of the network tagged on ingress. Since I am using Cisco 6500's with sup720's, I can classify and mark the traffic with a DSCP setting via PFC/DFC hardware. I then inspect traffic at the layer-3 edge for the various pools of servers. Depending on the DSCP marking of the packet, I change the next-hop. Since this is implemented through an extended ACL for a route-map it is handled in hardware (a good thing). Research shows that I can implement similar functionality in hardware on L3 switching gear from Juniper, Foundry, etc. so I am not boxing myself into a vendor. I don't believe Cisco supports using reflexive-acl's to apply policy routing, and even if they did, that would likely swamp our sup's CPU's, so I don't believe maintaining a stateful filter on the switch is viable. This all works as expected for Ping's and the ICMP replies. It breaks down for TCP http/mysql connections. It sounds like the correct (per-spec) solution may be to have the Linux servers track the incoming connections DSCP setting and mark the outgoing packets related to those connections. I am not at all certain this will not hit the servers CPU's more than desired or require additional connection-tracking resources than the ones we currently implement via iptables. Is there some other design option I am not considering? Thanks to those of you who have replied so far, it is at least a start down some additional paths of research for me! It's been since the days of BSDI that I have been involved in system networking internals, so I have been at a loss who to even ask! --D On Mon, Aug 17, 2009 at 2:44 PM, Steve Miller stmi...@gmail.com wrote: Would not the end station be considered to be outside of the DS domain? It does not necessarily make sense (to me) for reply packets to be marked unless they are appropriate classified and marked on the return path at the point they re-enter the DS domain. I would imagine that iptables and the DSCP target would do what you wanted, yes. I'd consider classifying and marking traffic at whatever switch you would consider to be at the edge of the DS domain (connected to this server.) -Steve 2009/8/17 Darren Bolding dar...@bolding.org: I believe this is operational content, but may well be better asked somewhere else. I would love to have a pointer to another list/website. I am looking to do some policy routing based on DSCP marking, and I have this all working inside the networking equipment. I DSCP mark some packets at ingress and I policy-route others based on ACL's matching those DSCP markings. This should allow me to solve some problems in a rather elegant manner, if I do say so myself. And this works fine for some things- I have verified that Ping's to a host work as expected- the Ping shows up at the destination host DSCP marked, and the ICMP reply leaves with the same DSCP marking. However, when I do this with apache and mysql connections (TCP 80/3306), the incoming packets are marked, but the replies are not. My research into the subject doesn't seem to suggest there is a standard for whether replies to a TCP connection are required to have the same DSCP marking, but it seems to make a lot of sense that they would. I've disabled iptables on the server host to no avail. I've looked around for an apache or Linux kernel setting and found nothing. At this point I'm looking for pointers- to a way to solve this issue, or to a better place to ask. I've started investigating writing iptables rules to match incoming connections that have DSCP marking
Re: Cisco 7600 (7609) as a core BGP router.
Can someone provide a link, or more detail, on the netflow issues. Particularly as they relate to 6509's and sup720's. Thanks! On 7/18/09, Roland Dobbins rdobb...@arbor.net wrote: On Jul 18, 2009, at 2:37 PM, Saku Ytti wrote: I'm guessing point Roland was making (which he likely would have not made couple moons ago:) I've made this point for years, quite publicly, actually - even when it was unpopular for me to do so in certain quarters. ; uRPF for 7600/6500 can only be in one mode for the whole box, all interfaces. This is a major problem in many cases. The NetFlow issues render flow telemetry unusable in production situations. The ACLs work very differently on this platform due to LOU issues, as you say. Most folks don't know this, and many end up overflowing their TCAMs and not realizing it until their boxes fall over, heh. If one has fairly complex ACLs covering various ranges of ports, ACLs on 7600/6500 quickly become very difficult to manage. EARL8 (Nexus7k) fixes the IPv6/uRPF and IPv6/ACL issue. And the NetFlow issues. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Unfortunately, inefficiency scales really well. -- Kevin Lawton -- Sent from my mobile device -- Darren Bolding -- -- dar...@bolding.org --
Fire, Power loss at Fisher Plaza in Seattle
Fisher Plaza, a self-styled carrier hotel in Seattle, and home to multiple datacenter and colocation providers, has had a major issue in one of its buildings late last night, early this morning. The best information I am aware of is that there was a failure in the main/generator transfer switch which resulted in a fire. The sprinkler system activated. From speaking to the fire battalion chief, I am under the impression that Seattle Fire did use water on the fire as well, but I am unsure of this. Given the failure location, generator power was not available, and cooling failed. UPS power to systems continued, and I can personally vouch that they held out for well over an hour. When we were able to access our equipment, ambient air temps were well over 100 degrees in the room our equipment is located in. At least some, if not many circuits were affected. Several large co-location providers and other datacenters are located in the facility, these facilities have no power. As this was the main/generator switch, and it is now highly damaged, the circuits in the area are damaged, and the entire area is doused in water, a rapid restoration of power does not seem likely. Fisher Plaza's phone numbers now result in fast-busy signals, so I have no recent update from them directly. Interestingly, this building is also the production studios for several Seattle TV and radio stations. There is no ETA for resolution. --D -- -- Darren Bolding -- -- dar...@bolding.org --
Re: Fire, Power loss at Fisher Plaza in Seattle
Power to some of the affected sections of the building has been restored via existing onsite generators. The central power risers cannot be connected to current generators in a timely manner due to excessive damage to the electrical switching equipment (and those generators may still be in standing water). These provide power to a number of colocated systems. Temporary generators are on order to be connected to the central risers, and the site expects that to be complete sometime late this evening. As best I can tell, there is still no utility power connected to any of the systems. The AC systems (chiller and crac) are currently not working. It is not clear to me whether these will be brought back on line when the temporary generators are available, but I am assuming so. It was pleasant to see the general positive attitude, sharing of information and offers of assistance that were made by representatives of the various tenants, customers and carriers that were on the scene. The usual suspects (companies and individuals) stepped up and took care of things, as they always seem to. --D On Fri, Jul 3, 2009 at 1:39 PM, Leo Bicknell bickn...@ufp.org wrote: In a message written on Fri, Jul 03, 2009 at 03:22:14PM -0400, Sean Donelan wrote: Are you better off with a single tier 4 data center, multiple tier 1 data centers, or something in between? It depends entirely on your dependency on connectivity. One extreme is something like a Central Office. Lots of cables from end-sites terminate in the building. Having a duplicate of the head end termination equipment on the opposite coast is darn near useless. If the building goes down, the users going through it go down. Tier 4 is probably a good idea. The other extreme is a pure content play (YouTube, Google Search). Users don't care which data center they hit (within reason), and indeed often don't know. You're better off having data centers spread out all over, both so you're more likely to only loose one at a time, but also so that the loss of one is relatively unimportant. Once you're already in this architecture, Tier 1 is generally cheaper. There are two problems though. First, most folks don't fit neatly in one of these buckets. They have some ties to local infrastructure, and some items which are not tied. Latency as a performance penality is very subjective. A backup 1000 miles away is fine for many things, and very bad for some things. Second, most folks don't have choices. It would be nice if most cities had three each Tier 1, 2, 3 and 4 data centers available so there was choice and competition but that's rare. Very few companies consider these choices rationally; often because choices are made by different groups. I am amazed how many times inside of an ISP the folks deploying the DNS and mail servers are firewalled from the folks deploying the network, to the point where you have to get to the President to reach common management. This leads to them making choices in opposite directions that end up costing extra money the company, and often resulting in a much lower uptimes than expected. Having the network group deploy a single point of failure to the Tier 4 data center the server guys required is, well, silly. However, more important than all of this is testing your infrastructure. Would you feel comfortable walking into your data center and ripping the power cable out of some bit of equipment at random _right now_? If not, you have no faith your equipment will work in an outage. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ -- -- Darren Bolding -- -- dar...@bolding.org --
Re: integrated KVMoIP and serial console terminal server
We just switched from using Avocent/Cyclades to using Raritan for our terminal servers, and I am happier with the Raritan. I have used Raritan IP KVM's in the past and been happy, and the IT folks seem to like their new one. I found the Raritan terminal server docs much more complete, it's support for direct access to serial ports via ssh much more complete, its support for remote authentication (TACACS+) better, it's console sharing features better, etc. etc. I was surprised how much better we liked it than the other products we've used. --D On Fri, Apr 24, 2009 at 11:33 AM, Luke S Crawford l...@prgmr.com wrote: Joe Abley jab...@hopcount.ca writes: What is everybody's favourite combination rack-mount VGA/USB KVM-over- IP and serial console concentrator in 2009? I'm looking for something that will accommodate 8 or so 9600bps serial devices and about 12 VGA/USB devices, all reachable over IP via sane means (ssh, https, etc). Being able to trunk everything through cat5E/ RJ45 plant is not necessary; in this application everything is in the same cabinet. I can't speak to the KVM over IP (for *NIX, they are obviously inferior to serial) but I do have some suggestions for the serial end. Personally, I use an opengear cm4148; it seems to work fairly well. If I only needed 8 ports, I'd still be using my solution from 2005, which was an 8 port rocketport serial card in a FreeBSD box. I only moved to the opengear because I need many more ports I like both the opengear and the freebsd box because I can use ssh auth, I can log, and I can lock down each user so that a given private key can only view a certain port. -- -- Darren Bolding -- -- dar...@bolding.org --
Re: shipping pre-built cabinets vs. build-on-site
I know some VAR's (CDW for example) will do racking/shipping from their facility. In the option we explored, they offered to allow us to rack gear ourselves or rack it themselves. They also were amenable to a VPN setup so we could get in via console and KVM's. They claimed to ship pre-built cabinets like this on a regular basis. On Mon, Apr 6, 2009 at 12:24 PM, Leo Bicknell bickn...@ufp.org wrote: In a message written on Mon, Apr 06, 2009 at 02:46:35PM -0400, Joe Abley wrote: Anybody here have experience shipping pre-built cabinets, with ~20U of routers and servers installed, connected and tested, to remote sites for deployment? shipping, no, moving yes. In past lives I've hired the same good folks who you might use to move your house to move entire racks. The major moving companies have teams who have experience with eletronic equipment, including full racks. Any quality 4 post rack with things well secured should be able to travel this way. I've never tried out of range of a truck, in all of my examples it was put on a truck, driven to the destination, and unloaded by the same team of folks. Still, if you don't need it overnight that can be the entire united states... -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ -- -- Darren Bolding -- -- dar...@bolding.org --
Re: do I need to maintain with RADB?
Is there a good source to explain the whole RADB system, and tools/processes people use to maintain routing policies/filters based on it? I'd like to both review and make sure my current understanding is accurate, and have a doc to send people to. Thanks for any pointers! --D On Thu, Feb 19, 2009 at 12:17 PM, Jon Lewis jle...@lewis.org wrote: On Thu, 19 Feb 2009, Zaid Ali wrote: Hi, need some advise here. Do I still need to maintain my objects (and pay) RADB? I use ARIN as source and all my route objects can be verified with a whois. If your objects are all maintained via another routing registry (ARIN's, altdb, etc.) and you don't care to maintain objects with radb.ra.net, then you do not need to pay RADB maintenance fees. -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_ -- -- Darren Bolding -- -- dar...@bolding.org --
Re: SANS: DNS Bug Now Public?
After a bit of looking around, I have not been able to find a list of firewalls/versions which are known to provide appropriate randomness in their PAT algorithms (or more importantly, those that do not). I would be very interested in such a list if anyone knows of one. As a side note, most people here realize this but, while people mention firewalls, keep in mind that if a load-balancer or other device is your egress PAT device, you might be interested in checking those systems port-translation randomness as well. --D On Wed, Jul 23, 2008 at 10:11 AM, Joe Abley [EMAIL PROTECTED] wrote: On 23 Jul 2008, at 12:16, Jorge Amodio wrote: Let me add that folks need to understand that the patch is not a fix to a problem that has been there for long time and it is just a workaround to reduce the chances for a potential attack, and it must be combined with best practices and recommendations to implent a more robust DNS setup. Having just seen some enterprise types spend time patching their nameservers, it's also perhaps worth spelling out that patch in this case might require more than upgrading resolver code -- it could also involve reconfigurations, upgrades or replacements of NAT boxes too. If your NAT reassigns source ports in a predictable fashion, then no amount of BIND9 patching is going to help. (Reconfiguring your internal resolvers to forward queries to an external, patched resolver which can see the world other than through NAT-coloured glasses may also be a way out.) Joe -- -- Darren Bolding -- -- [EMAIL PROTECTED] --
Re: Best utilizing fat long pipes and large file transfer
And while I certainly like open source solutions, there are plenty of commercial products that do things to optimize this. Depending on the type of traffic the products do different things. Many of the serial-byte caching variety (e.g. Riverbed/F5) now also do connection/flow optimization and proxying, while many of the network optimizers now are doing serial-byte caching. I also for a while was looking for multicast based file transfer tools, but couldn't find any that were stable. I'd be interested in seeing the names of some of the projects Robert is talking about- perhaps I missed a few when I looked. One thing that is a simple solution? Split the file and then send all the parts at the same time. This helps a fair bit, and is easy to implement. Few things drive home the issues with TCP window scaling better than moving a file via ftp and then via ttcp. Sure, you don't always get all the file, but it does get there fast! --D On Thu, Jun 12, 2008 at 5:02 PM, Randy Bush [EMAIL PROTECTED] wrote: The idea is to use tuned proxies that are close to the source and destination and are optimized for the delay. Local systems can move data through them without dealing with the need to tune for the delay-bandwidth product. Note that this man in the middle may not play well with many security controls which deliberately try to prevent it, so you still may need some adjustments. and for those of us who are addicted to simple rsync, or whatever over ssh, you should be aware of the really bad openssh windowing issue. randy -- -- Darren Bolding -- -- [EMAIL PROTECTED] --
