Perspectives: Diseconomies of Scale [was: Re: (latency) cooling door ...
I offer this as an afterthought without any expectations of replies to last week's latency discussion that focused on centralized vs. distributed, or one vs. many, data centers. While IMO the authors do not take into proper account the costs associated with network bandwidth and node provisioning, I thought it was nonetheless interesting due to the other factors they highlighted. There's nothing particularly correct here, merely some additional viewpoints on the subject that were not covered here earlier. Enjoy! Diseconomies of Scale - by Ken Church James Hamilton April 6, 2008 http://perspectives.mvdirona.com/2008/04/06/DiseconomiesOfScale.aspx Frank ---
RE: cooling door
Eg, I click on a web page to log in. The login process then kicks off a few authentication sessions with servers located halfway around the world. Then you do the data gathering, 2 phase locks, distributed file systems with the masters and lock servers all over the place. Your hellish user experience, let me SHOW YOU IT. I know that this kind of hellish user experience exists today but in those situations, the users are not happy with it, and I don't think anybody would consider it to be a well-architected application. But let's use two concrete examples. One is a city newspaper hosted in a local ISP data center which interacts with a database at the newspaper's premises. The other is an Internet retailer hosted in the same local ISP data center which basically runs standalone except for sending shipping orders to a fulfillment center in another city, and the nightly site updates uploaded by the store owner after returning home from the dayjob. Now, move the retailer half a mile closer to the city center in distributed pod 26 of a distributed ISP data center, and move the newspaper to pod 11 which just happens to be on their own premises because they have outsourced their data center to the ISP who runs these distributed data center pods. For the life of me, I can't see any technical issue with this. Obviously, it doesn't suit the customers who can't fit into a single pod because they risk creating a scenario like you describe. I'm not foolish enough to suggest that one size fits all. All I am suggesting is that there is room for a new type of data center that mitigates the power and cooling issues by spreading out into multiple pod locations instead of clustering everything into one big blob. Other than that minor handwaving, we are all good. Turns out that desining such distributed datacenters and setting up applications that you just handwaved away is a bit harder than it looks. I eagerly await papers on distributed database transactions with cost estimates for a distributed datacenter model vs. a traditional model. For the big applications which cannot fit inside a single pod, I expect the Amazon EC2/S3 model to influence the way in which they approach decentralized scaling. And in that case, when these people figure out the details, then the distributed pod data center architecture can support this model just as easily as the big blob architecture. I'm not holding my breath waiting for papers because in the real world, people are going with what works. The scientific world has bought into the grid architecture, or the so-called supercomputer cluster model. Everyone else is looking to Google MapReduce/BigTable, Amazon EC2/S3, Yahoo Hadoop, XEN virtualization and related technologies. See above. That right there doesn't quite solve most of the problems of traditional jobsets but its kind of hard to hear with the wind in my ears. This is NANOG. Traditional jobsets here are web servers, blogs/wikis, Internet stores, Content-management sites like CNN, on-demand video, etc. The kind of things that our customers run out of the racks that they rent. Tell me again how on-demand video works better in one big data center? I guess I can try to make it clearer by example: look at the cross-sectional bandwidth availability of a datacenter, now compare and contrast what it would take to pull it apart by a few tens of miles and conduct the cost comparison. It would be pretty dumb to try and pull apart a big blob architecture and convert it into a distributed pod architecture. But it would be very clever to build some distributed pods and put new customers in there. --Michael Dillon
RE: cooling door
I doubt we'll ever see the day when running gigabit across town becomes cost effective when compared to running gigabit to the other end of your server room/cage/whatever. You show me the ISP with the majority of their userbase located at the other end of their server room, and I'll concede the argument. Last time I looked the eyeballs were across town so I already have to deliver my gigabit feed across town. My theory is that you can achieve some scaling advantages by delivering it from multiple locations instead of concentrating one end of that gigabit feed in a big blob data center where the cooling systems will fail within an hour or two of a major power systems failure. --Michael Dillon
Re: cooling door
On Wed, Apr 2, 2008 at 3:06 AM, [EMAIL PROTECTED] wrote: I doubt we'll ever see the day when running gigabit across town becomes cost effective when compared to running gigabit to the other end of your server room/cage/whatever. You show me the ISP with the majority of their userbase located at the other end of their server room, and I'll concede the argument. Last time I looked the eyeballs were across town so I already have to deliver my gigabit feed across town. My theory is that you can achieve some scaling advantages by delivering it from multiple locations instead of concentrating one end of that gigabit feed in a big blob data center where the cooling systems will fail within an hour or two of a major power systems failure. It might be worth the effort to actually operate a business with real datacenters and customers before going off with these homilies. Experience says that for every transaction sent to the user, there are a multiplicity of transactions on the backend that need to occur. This is why the bandwidth into a datacenter is often 100x smaller than the bandwidth inside the datacenter. Communication within a rack, communication within a cluster, communication within a colo and then communication within a campus are different than communication with a user. /vijay --Michael Dillon
Re: cooling door
On Wed, Apr 2, 2008 at 6:06 AM, [EMAIL PROTECTED] wrote: I doubt we'll ever see the day when running gigabit across town becomes cost effective when compared to running gigabit to the other end of your server room/cage/whatever. You show me the ISP with the majority of their userbase located at the other end of their server room, and I'll concede the argument. Last time I looked the eyeballs were across town so I already have to deliver my gigabit feed across town. My theory is that you can achieve some scaling advantages by delivering it from multiple locations instead of concentrating one end of that gigabit feed in a big blob data center where the cooling systems will fail within an hour or two of a major power systems failure. That would be a choice for most of us. -M
RE: cooling door
Alex's point is that 5x density does not mean that the infrastructure costs are less than 5x. At a certain point in time there is a rate of return lower than 1. We're so stuck thinking that costs are primarily related to square feet, but with powering and cooling costs being the primary factors, we may be better off thinking in terms of costs in relation to amps. Frank -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Robert Boyle Sent: Saturday, March 29, 2008 10:29 PM To: Alex Pilosov; Paul Vixie Cc: nanog@merit.edu Subject: Re: cooling door snip I don't disagree with what you have written above, but if you can get 100A into all 5 racks (and cool it!), then you have five times the revenue with the same fixed infrastructure costs (with the exception of a bit more power, GenSet, UPS and cooling, but the rest of my costs stay the same.) To rephrase vijay, what is the problem being solved? For us in our datacenters, the problem being solved is getting as much return out of our investment as possible. -Robert Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 Well done is better than well said. - Benjamin Franklin
Re: cooling door
On Mon, Mar 31, 2008 at 8:24 AM, [EMAIL PROTECTED] wrote: Here is a little hint - most distributed applications in traditional jobsets, tend to work best when they are close together. Unless you can map those jobsets onto truly partitioned algorithms that work on local copy, this is a _non starter_. Let's make it simple and say it in plain English. The users of services have made the decision that it is good enough to be a user of a service hosted in a data center that is remote from the client. Remote means in another building in the same city, or in another city. Try reading for comprehension. The users of services have made the decision that it is good enough to be a user of a service hosted in a datacenter, and thanks to the wonders of AJAX and pipelining, you can even get snappy performance. What the users haven't signed up for is the massive amounts of scatter gathers that happen _behind_ the front end. Eg, I click on a web page to log in. The login process then kicks off a few authentication sessions with servers located halfway around the world. Then you do the data gathering, 2 phase locks, distributed file systems with the masters and lock servers all over the place. Your hellish user experience, let me SHOW YOU IT. Now, given that context, many of these good enough applications will run just fine if the data center is no longer in one physical location, but distributed across many. Of course, as you point out, one should not be stupid when designing such distributed data centers or when setting up the applications in them. Other than that minor handwaving, we are all good. Turns out that desining such distributed datacenters and setting up applications that you just handwaved away is a bit harder than it looks. I eagerly await papers on distributed database transactions with cost estimates for a distributed datacenter model vs. a traditional model. I would assume that every data center has local storage available using some protocol like iSCSI and probably over a separate network from the external client access. That right there solves most of your problems of traditional jobsets. And secondly, I am not suggesting that everybody should shut down big data centers or that every application should be hosted across several of these distributed data centers. See above. That right there doesn't quite solve most of the problems of traditional jobsets but its kind of hard to hear with the wind in my ears. There will always be some apps that need centralised scaling. But there are many others that can scale in a distributed manner, or at least use distributed mirrors in a failover scenario. Many many others indeed. No matter how much optical technology you have, it will tend to be more expensive to run, have higher failure rates, and use more power, than simply running fiber or copper inside your datacenter. There is a reason most people, who are backed up by sober accountants, tend to cluster stuff under one roof. Frankly I don't understand this kind of statement. It seems obvious to me that high-speed metro fibre exists and corporate IT people already have routers and switches and servers in the building, connected to the metro fiber. Also, the sober accountants do tend to agree with spending money on backup facilities to avoid the risk of single points of failure. Why should company A operate two data centers, and company B operate two data centers, when they could outsource it all to ISP X running one data center in each of the two locations (Company A and Company B). I guess I can try to make it clearer by example: look at the cross-sectional bandwidth availability of a datacenter, now compare and contrast what it would take to pull it apart by a few tens of miles and conduct the cost comparison. /vijay In addition, there is a trend to commoditize the whole data center. Amazon EC2 and S3 is not the only example of a company who does not offer any kind of colocation, but you can host your apps out of their data centers. I believe that this trend will pick up steam and that as the corporate market begins to accept running virtual servers on top of a commodity infrastructure, there is an opportunity for network providers to branch out and not only be specialists in the big consolidated data centers, but also in running many smaller data centers that are linked by fast metro fiber. --Michael Dillon
RE: cooling door
--On March 29, 2008 5:04:01 PM -0500 Frank Coluccio [EMAIL PROTECTED] wrote: Michael Dillon is spot on when he states the following (quotation below), although he could have gone another step in suggesting how the distance insensitivity of fiber could be further leveraged: The high speed fibre in Metro Area Networks will tie it all together with the result that for many applications, it won't matter where the servers are. In fact, those same servers, and a host of other storage and network elements, can be returned to the LAN rooms and closets of most commercial buildings from whence they originally came prior to the large-scale data center consolidations of the current millennium, once organizations decide to free themselves of the 100-meter constraint imposed by UTP-based LAN hardware and replace those LANs with collapsed fiber backbone designs that attach to remote switches (which could be either in-building or remote), instead of the minimum two switches on every floor that has become customary today. Yeah except in a lot of areas there is no MAN, and the ILECs want to bend you over for any data access. I've no idea how well the MAN idea is coming along in various areas, but you still have to pay for access to it somehow, and that adds to overhead. Which leads to attempt efficiency gains through centralization and increased density. We often discuss the empowerment afforded by optical technology, but we've barely scratched the surface of its ability to effect meaningful architectural changes. The earlier prospects of creating consolidated data centers were once near-universally considered timely and efficient, and they still are in many respects. However, now that the problems associated with a/c and power have entered into the calculus, some data center design strategies are beginning to look more like anachronisms that have been caught in a whip-lash of rapidly shifting conditions, and in a league with the constraints that are imposed by the now-seemingly-obligatory 100-meter UTP design. In order for the MAN scenarios to work though access has to be pretty cheap, and fairly ubiquitous. Last i checked though making a trench was a very messy very expensive process. So MANs are great once they're installed but those installing/building them will want to recoup their large investments.
Re: cooling door
On Tue, 01 Apr 2008 16:48:47 MDT, Michael Loftis said: Yeah except in a lot of areas there is no MAN, and the ILECs want to bend you over for any data access. I've no idea how well the MAN idea is coming along in various areas, but you still have to pay for access to it somehow, and that adds to overhead. Which leads to attempt efficiency gains through centralization and increased density. I doubt we'll ever see the day when running gigabit across town becomes cost effective when compared to running gigabit to the other end of your server room/cage/whatever. pgpmr49k9aqIl.pgp Description: PGP signature
Re: cooling door
On Sat, Mar 29, 2008 at 3:04 PM, Frank Coluccio [EMAIL PROTECTED] wrote: Michael Dillon is spot on when he states the following (quotation below), although he could have gone another step in suggesting how the distance insensitivity of fiber could be further leveraged: Dillon is not only not spot on, dillon is quite a bit away from being spot on. Read on. The high speed fibre in Metro Area Networks will tie it all together with the result that for many applications, it won't matter where the servers are. In fact, those same servers, and a host of other storage and network elements, can be returned to the LAN rooms and closets of most commercial buildings from whence they originally came prior to the large-scale data center consolidations of the current millennium, once organizations decide to free themselves of the 100-meter constraint imposed by UTP-based LAN hardware and replace those LANs with collapsed fiber backbone designs that attach to remote switches (which could be either in-building or remote), instead of the minimum two switches on every floor that has become customary today. Here is a little hint - most distributed applications in traditional jobsets, tend to work best when they are close together. Unless you can map those jobsets onto truly partitioned algorithms that work on local copy, this is a _non starter_. We often discuss the empowerment afforded by optical technology, but we've barely scratched the surface of its ability to effect meaningful architectural changes. No matter how much optical technology you have, it will tend to be more expensive to run, have higher failure rates, and use more power, than simply running fiber or copper inside your datacenter. There is a reason most people, who are backed up by sober accountants, tend to cluster stuff under one roof. The earlier prospects of creating consolidated data centers were once near-universally considered timely and efficient, and they still are in many respects. However, now that the problems associated with a/c and power have entered into the calculus, some data center design strategies are beginning to look more like anachronisms that have been caught in a whip-lash of rapidly shifting conditions, and in a league with the constraints that are imposed by the now-seemingly-obligatory 100-meter UTP design. Frank, lets assume we have abundant dark fiber, and a 800 strand ribbon fiber cable costs the same as a utp run. Can you get me some quotes from a few folks about terminating and patching 800 strands x2? /vijay Frank A. Coluccio DTI Consulting Inc. 212-587-8150 Office 347-526-6788 Mobile On Sat Mar 29 13:57 , sent: Can someone please, pretty please with sugar on top, explain the point behind high power density? It allows you to market your operation as a data center. If you spread it out to reduce power density, then the logical conclusion is to use multiple physical locations. At that point you are no longer centralized. In any case, a lot of people are now questioning the traditional data center model from various angles. The time is ripe for a paradigm change. My theory is that the new paradigm will be centrally managed, because there is only so much expertise to go around. But the racks will be physically distributed, in virtually every office building, because some things need to be close to local users. The high speed fibre in Metro Area Networks will tie it all together with the result that for many applications, it won't matter where the servers are. Note that the Google MapReduce, Amazon EC2, Haddoop trend will make it much easier to place an application without worrying about the exact locations of the physical servers. Back in the old days, small ISPs set up PoPs by finding a closet in the back room of a local store to set up modem banks. In the 21st century folks will be looking for corporate data centers with room for a rack or two of multicore CPUs running XEN, and Opensolaris SANs running ZFS/raidz providing iSCSI targets to the XEN VMs. --Michael Dillon
RE: cooling door
Here is a little hint - most distributed applications in traditional jobsets, tend to work best when they are close together. Unless you can map those jobsets onto truly partitioned algorithms that work on local copy, this is a _non starter_. Let's make it simple and say it in plain English. The users of services have made the decision that it is good enough to be a user of a service hosted in a data center that is remote from the client. Remote means in another building in the same city, or in another city. Now, given that context, many of these good enough applications will run just fine if the data center is no longer in one physical location, but distributed across many. Of course, as you point out, one should not be stupid when designing such distributed data centers or when setting up the applications in them. I would assume that every data center has local storage available using some protocol like iSCSI and probably over a separate network from the external client access. That right there solves most of your problems of traditional jobsets. And secondly, I am not suggesting that everybody should shut down big data centers or that every application should be hosted across several of these distributed data centers. There will always be some apps that need centralised scaling. But there are many others that can scale in a distributed manner, or at least use distributed mirrors in a failover scenario. No matter how much optical technology you have, it will tend to be more expensive to run, have higher failure rates, and use more power, than simply running fiber or copper inside your datacenter. There is a reason most people, who are backed up by sober accountants, tend to cluster stuff under one roof. Frankly I don't understand this kind of statement. It seems obvious to me that high-speed metro fibre exists and corporate IT people already have routers and switches and servers in the building, connected to the metro fiber. Also, the sober accountants do tend to agree with spending money on backup facilities to avoid the risk of single points of failure. Why should company A operate two data centers, and company B operate two data centers, when they could outsource it all to ISP X running one data center in each of the two locations (Company A and Company B). In addition, there is a trend to commoditize the whole data center. Amazon EC2 and S3 is not the only example of a company who does not offer any kind of colocation, but you can host your apps out of their data centers. I believe that this trend will pick up steam and that as the corporate market begins to accept running virtual servers on top of a commodity infrastructure, there is an opportunity for network providers to branch out and not only be specialists in the big consolidated data centers, but also in running many smaller data centers that are linked by fast metro fiber. --Michael Dillon
Re: cooling door
Matthew Petach wrote: On 3/29/08, Alex Pilosov [EMAIL PROTECTED] wrote: Can someone please, pretty please with sugar on top, explain the point behind high power density? Raw real estate is cheap (basically, nearly free). Increasing power density per sqft will *not* decrease cost, beyond 100W/sqft, the real estate costs are a tiny portion of total cost. Moving enough air to cool 400 (or, in your case, 2000) watts per square foot is *hard*. I've started to recently price things as cost per square amp. (That is, 1A power, conditioned, delivered to the customer rack and cooled). Space is really irrelevant - to me, as colo provider, whether I have 100A going into a single rack or 5 racks, is irrelevant. In fact, my *costs* (including real estate) are likely to be lower when the load is spread over 5 racks. Similarly, to a customer, all they care about is getting their gear online, and can care less whether it needs to be in 1 rack or in 5 racks. To rephrase vijay, what is the problem being solved? I have not yet found a way to split the ~10kw power/cooling demand of a T1600 across 5 racks. Yes, when I want to put a pair of them into an exchange point, I can lease 10 racks, put T1600s in two of them, and leave the other 8 empty; but that hasn't helped either me the customer or the exchange point provider; they've had to burn more real estate for empty racks that can never be filled, I'm paying for floor space in my cage that I'm probably going to end up using for storage rather than just have it go to waste, and we still have the problem of two very hot spots that need relatively 'point' cooling solutions. There are very specific cases where high density power and cooling cannot simply be spread out over more space; thus, research into areas like this is still very valuable. The problem with point heating is often that the hot point is then the *intake* for other equipment. If you spread your two T1600s into 10 racks (i.e. skip 2, drop one, skip 4, drop 1, leaving two at the end) your hot point problem is much less of a concern. If you bought 10 racks... not in a row, but SURROUNDING (in each of the rows opposite the cabinets)... Say 12 (a = vacant, b,c = T1600) abca You would be doing everyone in your datacenter a service by a) not thinking linearly and b) providing adequate sq ft space to dissipate your heat. Deepak
Re: cooling door
Here is a little hint - most distributed applications in traditional jobsets, tend to work best when they are close together. Unless you can map those jobsets onto truly partitioned algorithms that work on local copy, this is a _non starter_. I thought that I had made my view clear in this respect in my earlier postings. When moving to the kinds of optical extension techniques I outlined earlier in this thread, I can't be too emphatic in noting that one size does not fit all. And while I appreciate the hint, you probably don't realize it yet, but in some ways you are helping to make my argument. Consider for a moment my initial point (my main point, in fact) concerning moving all of the LAN gear in an enterprise building out to the cloud (customer-owned data center or colo, machs niches). My contention here is that, this not only eliminates LAN rooms and all the switches and environmentals that fill them, but it radically reduces the bulk of the gear normally found in the building's telecom center, as well, since hierarchical routing infrastructure, and severs of most types that are used for networking purposes (along with their associated power and air provisions to keep all of them going) would also no longer have a place onsite, either. Rather, these elements, too, could be centrally located in an offsite data center (hence reducing their overall number for multi-site networks) _or _in _a _colo _ OR in one of the enterprise's other sites where it makes sense. OR IN A COLO is especially relevant here, and in some ways related to the point you are arguing. There are many enterprises, in fact, who have already, over their own dark fibernets (now lit, of course) and leased optical facilities, long since taken the steps to move their server farms and major network node positions to 111-8th, 611 Wilshire, Exodus and scores of other exchange locations around the world, although most of them, up until now, have not yet taken the next precipitous step of moving their LAN gear out to the cloud, as well. Of course when they do decide to free the premises of LAN gear, so too will they obviate the requirement for many of the routers and associated networking elements in those buildings, too, thus streamlining L3 route administration within the intranet, as well. I should emphasize here that when played properly this is NOT a zero sum game. By the same token, however, what we're discussing here is really a situational call. I grant you, for instance, that some, perhaps many, jobs are best suited to having their elements sited close to one another. Many, as I've outlined above do not fit this constraint. This is a scalar type of decision process, where the smallest instance of the fractal doesn't require the same absolute level of provisioning as the largest, where each is a candidate that must meet a minimum set of criteria before making full sense. lets assume we have abundant dark fiber, and a 800 strand ribbon fiber cable costs the same as a utp run. Can you get me some quotes from a few folks about terminating and patching 800 strands x2? This is likely an rhetorical question, although it needn't be. Yes, I can get those quotes, and quotes that are many times greater in scope, and have for a number of financial trading floors and outside plant dark nets. It wasn't very long ago, however, when the same question could still be asked about UTP, since every wire of every pair during those earlier times required the use of a soldering iron. My point being, the state of the art of fiber heading, connectorization and splicing continues to improve all the time, as does the quality and costing of pre-connectorized jumpers (in the event your question had to do with jumpers and long cross-conns). There is a reason most people, who are backed up by sober accountants, tend to cluster stuff under one roof. Agreed. Sometimes, however, perhaps quite often in fact, one can attribute this behavior to a quality known more commonly as bunker mentality. -- When I closed my preceding message in this subthread I stated I would welcome a continuation of this discussion offlist with anyone who was interested. Since then I've received onlist responses, so I responded here in kind, but my earlier offer still holds. Frank A. Coluccio DTI Consulting Inc. 212-587-8150 Office 347-526-6788 Mobile -- On Mon Mar 31 9:53 , vijay gill sent: On Sat, Mar 29, 2008 at 3:04 PM, Frank Coluccio [EMAIL PROTECTED] wrote: Michael Dillon is spot on when he states the following (quotation below), although he could have gone another step in suggesting how the distance insensitivity of fiber could be further leveraged: Dillon is not only not spot on, dillon is quite a bit away from being spot on. Read on. The high speed fibre in Metro Area Networks will tie it all together with the result that for many applications, it won't matter where the servers are. In fact, those same servers, and a host of other
Re: cooling door
On Mon, Mar 31, 2008 at 11:24 AM, [EMAIL PROTECTED] wrote: Let's make it simple and say it in plain English. The users of services have made the decision that it is good enough to be a user of a service hosted in a data center that is remote from the client. Remote means in another building in the same city, or in another city. Now, given that context, many of these good enough applications will run just fine if the data center is no longer in one physical location, but distributed across many. Of course, as you point out, one should not be stupid when designing such distributed data centers or when setting up the applications in them. I think many folks have gotten used to 'my application runs in a datacenter' (perhaps even a remote datacenter) but they still want performance from their application. If the DC is 20ms away from their desktop (say they are in NYC and the DC is in IAD which is about 20ms distant on a good run) things are still 'snappy'. If the application uses bits/pieces from a farm of remote datacenters (also 20ms away or so so anywhere from NYC to ATL to CHI away from IAD) latency inside that application is now important... something like a DB heavy app will really suffer under this scenario if locality of the database's data isn't kept in mind as well. Making multiple +20ms hops around for information is really going to impact user experience of the application I think. The security model as well would be highly interesting in this sort of world.. both physical security (line/machine/cage) and information security (data over the links). This seems to require fairly quick encryption in a very distributed envorinment where physical security isn't very highly assured. I would assume that every data center has local storage available using some protocol like iSCSI and probably over a separate network from the external client access. That right there solves most of your problems of traditional jobsets. And secondly, I am not suggesting that everybody should shut down big data centers or that every application should be hosted across several of these distributed data centers. There will always be some apps that need centralised scaling. But there are many others that can scale in a distributed manner, or at least use distributed mirrors in a failover scenario. ah, like the distributed DR sites financials use? (I've heard of designs, perhaps from this list even, of distributed DC's 60+ miles apart with iscsi on fiber between the sites... pushing backup copies of transaction data to the DR facility) That doesn't help in scenarios with highly interactive data sets, or lower latency requirements for applications... I also remember a SAP (I think) installation that got horribly unhappy with the database/front-end parts a few cities apart from each other over an 'internal' network... In addition, there is a trend to commoditize the whole data center. Amazon EC2 and S3 is not the only example of a company who does not offer any kind of colocation, but you can host your apps out of their data centers. I believe that this trend will pick up asp's were a trend in the late 90's, for some reason things didn't work out then (reason not really imporant now). Today/going-forward some things make sense to outsource in this manner, I'm not sure that customer critical data or data with high change-rates are it though, certainly nothing that's critical to your business from an IP perspective, at least not without lots of security thought/controls. When working at a large networking company we found it really hard to get people to move their applications out from under their desk (yes, literally) and into a production datacenter... even with offers of mostly free hardware and management of systems (so less internal budget used). Some of that was changing when I left, but certainly not quickly. it's an interesting proposition, and the DC's in question were owned by the company in question, I'm not sure about moving off to another company's facilities though... scary security problems result. -Chris
Re: latency (was: RE: cooling door)
On Sat, 29 Mar 2008, Frank Coluccio wrote: Understandably, some applications fall into a class that requires very-short distances for the reasons you cite, although I'm still not comfortable with the setup you've outlined. Why, for example, are you showing two Ethernet switches for the fiber option (which would naturally double the switch-induced latency), but only a single switch for the UTP option? Yes, I am showing a case where you have switches in each rack so each rack is uplinked with a fiber to a central aggregation switch, as opposed to having a lot of UTP from the rack directly into the aggregation switch. Now, I'm comfortable in ceding this point. I should have made allowances for this type of exception in my introductory post, but didn't, as I also omitted mention of other considerations for the sake of brevity. For what it's worth, propagation over copper is faster propagation over fiber, as copper has a higher nominal velocity of propagation (NVP) rating than does fiber, but not significantly greater to cause the difference you've cited. The 2/3 speed of light in fiber as opposed to propagation speed in copper was not in my mind. As an aside, the manner in which o-e-o and e-o-e conversions take place when transitioning from electronic to optical states, and back, affects latency differently across differing link assembly approaches used. In cases where 10Gbps My opinion is that the major factors of added end-to-end latency in my example is that the packet has to be serialisted three times as opposed to once and there are three lookups instead of one. Lookups take time, putting the packet on the wire take time. Back in the 10 megabit/s days, there were switches that did cut-through, ie if the output port was not being used the instant the packet came in, it could start to send out the packet on the outgoing port before it was completely taken in on the incoming port (when the header was received, the forwarding decision was taken and the equipment would start to send the packet out before it was completely received from the input port). By chance, is the deserialization you cited earlier, perhaps related to this inverse muxing process? If so, then that would explain the disconnect, and if it is so, then one shouldn't despair, because there is a direct path to avoiding this. No, it's the store-and-forward architecture used in all modern equipment (that I know of). A packet has to be completely taken in over the wire into a buffer, a lookup has to be done as to where this packet should be put out, it needs to be sent over a bus or fabric, and then it has to be clocked out on the outgoing port from another buffer. This adds latency in each switch hop on the way. As Adrian Chadd mentioned in the email sent after yours, this can of course be handled by modifying or creating new protocols that handle this fact. It's just that with what is available today, this is a problem. Each directory listing or file access takes a bit longer over NFS with added latency, and this reduces performance in current protocols. Programmers who do client/server applications are starting to notice this and I know of companies that put latency-inducing applications in the development servers so that the programmer is exposed to the same conditions in the development environment as in the real world. This means for some that they have to write more advanced SQL queries to get everything done in a single query instead of asking multiple and changing the queries depending on what the first query result was. Also, protocols such as SMB and NFS that use message blocks over TCP have to be abandonded and replaced with real streaming protocols and large window sizes. Xmodem wasn't a good idea back then, it's not a good idea now (even though the blocks now are larger than the 128 bytes of 20-30 years ago). -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: latency (was: RE: cooling door)
[EMAIL PROTECTED] (Mikael Abrahamsson) writes: ... Back in the 10 megabit/s days, there were switches that did cut-through, ie if the output port was not being used the instant the packet came in, it could start to send out the packet on the outgoing port before it was completely taken in on the incoming port (when the header was received, the forwarding decision was taken and the equipment would start to send the packet out before it was completely received from the input port). had packet sizes scaled with LAN transmission speed, i would agree. but the serialization time for 1500 bytes at 10MBit was ~1.2ms, and went down by a factor of 10 for FastE (~120us), another factor of 10 for GigE (~12us) and another factor of 10 for 10GE (~1.2us). even those of us using jumbo grams are getting less serialization delay at 10GE (~7us) than we used to get on a DEC LANbridge 100 which did cutthrough after the header (~28us). ..., it's the store-and-forward architecture used in all modern equipment (that I know of). A packet has to be completely taken in over the wire into a buffer, a lookup has to be done as to where this packet should be put out, it needs to be sent over a bus or fabric, and then it has to be clocked out on the outgoing port from another buffer. This adds latency in each switch hop on the way. you may be right about the TCAM lookup times having an impact, i don't know if they've kept pace with transmission speed either. but someone's theory here yesterday that software (kernel and IP stack) architecture is more likely to be at fault, there are still plenty of queue it here, it'll go out next time the device or timer interrupt handler fires and this can be in the ~1ms or even ~10ms range. this doesn't show up on file transfer benchmarks since packet trains usually do well, but miss an ACK, or send a ping, and you'll see a shelf. As Adrian Chadd mentioned in the email sent after yours, this can of course be handled by modifying or creating new protocols that handle this fact. It's just that with what is available today, this is a problem. Each directory listing or file access takes a bit longer over NFS with added latency, and this reduces performance in current protocols. here again it's not just the protocols, it's the application design, that has to be modernized. i've written plenty of code that tries to cut down the number of bytes of RAM that get copied or searched, which ends up not going faster on modern CPUs (or sometimes going slower) because of the minimum transfer size between L2 and DRAM. similarly, a program that sped up on a VAX 780 when i taught it to match the size domain of its disk I/O to the 512-byte size of a disk sector, either fails to go faster on modern high-bandwidth I/O and log structured file systems, or actually goes slower. in other words you don't need NFS/SMB, or E-O-E, or the WAN, to erode what used to be performance gains through efficiency. there's plenty enough new latency (expressed as a factor of clock speed) in the path to DRAM, the path to SATA, and the path through ZFS, to make it necessary that any application that wants modern performance has to be re-oriented to take modern (which in this case means, streaming) approach. correspondingly, applications which take this approach, don't suffer as much when they move from SATA to NFS or iSCSI. Programmers who do client/server applications are starting to notice this and I know of companies that put latency-inducing applications in the development servers so that the programmer is exposed to the same conditions in the development environment as in the real world. This means for some that they have to write more advanced SQL queries to get everything done in a single query instead of asking multiple and changing the queries depending on what the first query result was. while i agree that turning one's SQL into transactions that are more like applets (such that, for example, you're sending over the content for a potential INSERT that may not happen depending on some SELECT, because the end-to-end delay of getting back the SELECT result is so much higher than the cost of the lost bandwidth from occasionally sending a useless INSERT) will take better advantage of modern hardware and software architecture (which means in this case, streaming), it's also necessary to teach our SQL servers that ZFS recordsize=128k means what it says, for file system reads and writes. a lot of SQL users who have moved to a streaming model using a lot of transactions have merely seen their bottleneck move from the network into the SQL server. Also, protocols such as SMB and NFS that use message blocks over TCP have to be abandonded and replaced with real streaming protocols and large window sizes. Xmodem wasn't a good idea back then, it's not a good idea now (even though the blocks now are larger than the 128 bytes of 20-30 years ago). i think xmodem and kermit moved
RE: latency (was: RE: cooling door)
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Paul Vixie Sent: Sunday, March 30, 2008 10:35 AM To: nanog@merit.edu Subject: Re: latency (was: RE: cooling door) [EMAIL PROTECTED] (Mikael Abrahamsson) writes: Programmers who do client/server applications are starting to notice this and I know of companies that put latency-inducing applications in the development servers so that the programmer is exposed to the same conditions in the development environment as in the real world. This means for some that they have to write more advanced SQL queries to get everything done in a single query instead of asking multiple and changing the queries depending on what the first query result was. while i agree that turning one's SQL into transactions that are more like applets (such that, for example, you're sending over the content for a potential INSERT that may not happen depending on some SELECT, because the end-to-end delay of getting back the SELECT result is so much higher than the cost of the lost bandwidth from occasionally sending a useless INSERT) will take better advantage of modern hardware and software architecture (which means in this case, streaming), it's also necessary to teach our SQL servers that ZFS recordsize=128k means what it says, for file system reads and writes. a lot of SQL users who have moved to a streaming model using a lot of transactions have merely seen their bottleneck move from the network into the SQL server. I have seen first hand (worked for a company and diagnosed issues with their applications from a network perspective, prompting a major re-write of the software), where developers work with their SQL servers, application servers, and clients all on the same L2 switch. They often do not duplicate the environment they are going to be deploying the application into, and therefore assume that the network is going to perform the same. So, when there are problems they blame the network. Often the root problem is the architecture of the application itself and not the network. All the servers and client workstations have Gigabit connections to the same L2 switch, and they are honestly astonished when there are issues running the same application over a typical enterprise network with clients of different speeds (10/100/1000, full and/or half duplex). Surprisingly, to me, they even expect the same performance out of a WAN. Application developers today need a network guy on their team. One who can help them understand how their proposed application architecture would perform over various customer networks, and that can make suggestions as to how the architecture can be modified to allow the performance of the application to take advantage of the networks' capabilities. Mikael (seems to) complain that developers have to put latency inducing applications into the development environment. I'd say that those developers are some of the few who actually have a clue, and are doing the right thing. Also, protocols such as SMB and NFS that use message blocks over TCP have to be abandonded and replaced with real streaming protocols and large window sizes. Xmodem wasn't a good idea back then, it's not a good idea now (even though the blocks now are larger than the 128 bytes of 20- 30 years ago). i think xmodem and kermit moved enough total data volume (expressed as a factor of transmission speed) back in their day to deserve an honourable retirement. but i'd agree, if an application is moved to a new environment where everything (DRAM timing, CPU clock, I/O bandwidth, network bandwidth, etc) is 10X faster, but the application only runs 2X faster, then it's time to rethink more. but the culprit will usually not be new network latency. -- Paul Vixie It may be difficult to switch to a streaming protocol if the underlying data sets are block-oriented. Fred Reimer, CISSP, CCNP, CQS-VPN, CQS-ISS Senior Network Engineer Coleman Technologies, Inc. 954-298-1697 smime.p7s Description: S/MIME cryptographic signature
RE: latency (was: RE: cooling door)
On Sun, 30 Mar 2008, Fred Reimer wrote: application to take advantage of the networks' capabilities. Mikael (seems to) complain that developers have to put latency inducing applications into the development environment. I'd say that those developers are some of the few who actually have a clue, and are doing the right thing. I was definately not complaining, I brought it up as an example where developers have clue and where they're doing the right thing. I've too often been involved in customer complaints which ended up being the fault of Microsoft SMB and the customers having the firm idea that it must be a network problem since MS is a world standard and that can't be changed. Even proposing to change TCP Window settings to get FTP transfers quicker is met with the same sceptisism. Even after describing to them about the propagation delay of light in fiber and the physical limitations, they're still very suspicious about it all. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: latency (was: RE: cooling door)
On Sun, 30 Mar 2008 13:03:18 +0800 Adrian Chadd [EMAIL PROTECTED] wrote: Oh, and kernel hz tickers can have similar effects on network traffic, if the application does dumb stuff. If you're (un)lucky then you may see 1 or 2ms of delay between packet input and scheduling processing. This doesn't matter so much over 250ms + latent links but matters on 0.1ms - 1ms latent links. (Can someone please apply some science to this and publish best practices please?) There's been a lot of work done on TCP throughput. Roughly speaking, and holding everything else constant, throughput is linear in the round trip time. That is, if you double the RTT -- even from .1 ms to .2 ms -- you halve the throughput on (large) file transfers. See http://www.slac.stanford.edu/comp/net/wan-mon/thru-vs-loss.html for one summary; feed tcp throughput equation into your favorite search engine for a lot more references. Another good reference is RFC 3448, which relates throughput to packet size (also a linear factor, but if serialization delay is significant then increasing the packet size will increase the RTT), packet loss rate, the TCP retransmission timeout (which can be approximated as 4x the RTT), and the number of packets acknowledged by a single TCP acknowledgement. On top of that, there are lots of application issues, as a number of people have pointed out. --Steve Bellovin, http://www.cs.columbia.edu/~smb
RE: latency (was: RE: cooling door)
Thanks for the clarification; that's why I put the seems to in the reply. Fred Reimer, CISSP, CCNP, CQS-VPN, CQS-ISS Senior Network Engineer Coleman Technologies, Inc. 954-298-1697 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mikael Abrahamsson Sent: Sunday, March 30, 2008 12:30 PM To: nanog@merit.edu Subject: RE: latency (was: RE: cooling door) On Sun, 30 Mar 2008, Fred Reimer wrote: application to take advantage of the networks' capabilities. Mikael (seems to) complain that developers have to put latency inducing applications into the development environment. I'd say that those developers are some of the few who actually have a clue, and are doing the right thing. I was definately not complaining, I brought it up as an example where developers have clue and where they're doing the right thing. I've too often been involved in customer complaints which ended up being the fault of Microsoft SMB and the customers having the firm idea that it must be a network problem since MS is a world standard and that can't be changed. Even proposing to change TCP Window settings to get FTP transfers quicker is met with the same sceptisism. Even after describing to them about the propagation delay of light in fiber and the physical limitations, they're still very suspicious about it all. -- Mikael Abrahamssonemail: [EMAIL PROTECTED] smime.p7s Description: S/MIME cryptographic signature
RE: latency (was: RE: cooling door)
... feed tcp throughput equation into your favorite search engine for a lot more references. There has been a lot of work in some OS stacks (Vista and recent linux kernels) to enable TCP auto-tuning (of one form or another), which is attempting to hide some of the worst of the TCP uglynesses from the application/end-users. I am not convinced this is always a good thing, since having the cruft exposed to the developers (in particular) means one needs to plan for errors and less than ideal cases. Gary
Re: cooling door
Perhaps this is apropos: Linkname: Slashdot | Iceland Woos Data Centers As Power Costs Soar URL: http://hardware.slashdot.org/hardware/08/03/29/2331218.shtml On Sat, Mar 29, 2008 at 23:29:18PM -0400, Robert Boyle wrote: At 02:11 PM 3/29/2008, Alex Pilosov wrote: Can someone please, pretty please with sugar on top, explain the point behind high power density? More equipment in your existing space means more revenue and more profit. Raw real estate is cheap (basically, nearly free). Increasing power density per sqft will *not* decrease cost, beyond 100W/sqft, the real estate costs are a tiny portion of total cost. Moving enough air to cool 400 (or, in your case, 2000) watts per square foot is *hard*. It depends on where you are located, but I understand what you are saying. However, the space is the cheap part. Installing the electrical power, switchgear, ATS gear, Gensets, UPS units, power distribution, cable/fiber distribution, connectivity to the datacenter, core and distribution routers/switches are all basically stepped incremental costs. If you can leverage the existing floor infrastructure then you maximize the return on your investment. I've started to recently price things as cost per square amp. (That is, 1A power, conditioned, delivered to the customer rack and cooled). Space is really irrelevant - to me, as colo provider, whether I have 100A going into a single rack or 5 racks, is irrelevant. In fact, my *costs* (including real estate) are likely to be lower when the load is spread over 5 racks. Similarly, to a customer, all they care about is getting their gear online, and can care less whether it needs to be in 1 rack or in 5 racks. I don't disagree with what you have written above, but if you can get 100A into all 5 racks (and cool it!), then you have five times the revenue with the same fixed infrastructure costs (with the exception of a bit more power, GenSet, UPS and cooling, but the rest of my costs stay the same.) -- Henry Yen Aegis Information Systems, Inc. Senior Systems Programmer Hicksville, New York
Re: cooling door
On 3/29/08, Alex Pilosov [EMAIL PROTECTED] wrote: Can someone please, pretty please with sugar on top, explain the point behind high power density? Raw real estate is cheap (basically, nearly free). Increasing power density per sqft will *not* decrease cost, beyond 100W/sqft, the real estate costs are a tiny portion of total cost. Moving enough air to cool 400 (or, in your case, 2000) watts per square foot is *hard*. I've started to recently price things as cost per square amp. (That is, 1A power, conditioned, delivered to the customer rack and cooled). Space is really irrelevant - to me, as colo provider, whether I have 100A going into a single rack or 5 racks, is irrelevant. In fact, my *costs* (including real estate) are likely to be lower when the load is spread over 5 racks. Similarly, to a customer, all they care about is getting their gear online, and can care less whether it needs to be in 1 rack or in 5 racks. To rephrase vijay, what is the problem being solved? I have not yet found a way to split the ~10kw power/cooling demand of a T1600 across 5 racks. Yes, when I want to put a pair of them into an exchange point, I can lease 10 racks, put T1600s in two of them, and leave the other 8 empty; but that hasn't helped either me the customer or the exchange point provider; they've had to burn more real estate for empty racks that can never be filled, I'm paying for floor space in my cage that I'm probably going to end up using for storage rather than just have it go to waste, and we still have the problem of two very hot spots that need relatively 'point' cooling solutions. There are very specific cases where high density power and cooling cannot simply be spread out over more space; thus, research into areas like this is still very valuable. Matt
Re: latency (was: RE: cooling door)
[EMAIL PROTECTED] (Buhrmaster, Gary) writes: ... feed tcp throughput equation into your favorite search engine for a lot more references.=20 There has been a lot of work in some OS stacks (Vista and recent linux kernels) to enable TCP auto-tuning (of one form or another), ... on http://www.onlamp.com/pub/a/bsd/2008/02/26/whats-new-in-freebsd-70.html i'd read that freebsd 7 also has some tcp auto tuning logic. -- Paul Vixie
Re: latency (was: RE: cooling door)
On 30 Mar 2008 21:00:25 + Paul Vixie [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] (Buhrmaster, Gary) writes: ... feed tcp throughput equation into your favorite search engine for a lot more references.=20 There has been a lot of work in some OS stacks (Vista and recent linux kernels) to enable TCP auto-tuning (of one form or another), ... on http://www.onlamp.com/pub/a/bsd/2008/02/26/whats-new-in-freebsd-70.html i'd read that freebsd 7 also has some tcp auto tuning logic. There are certain things that the stack can do, like auto-adjusting the window size, tuning retransmission intervals, etc. But other problem are at the application layer, as you noted a few posts ago. --Steve Bellovin, http://www.cs.columbia.edu/~smb
Re: latency (was: RE: cooling door)
Mikael, I see your points more clearly now in respect to the number of turns affecting latency. In analyzing this further, however, it becomes apparent that the collapsed backbone regimen may, in many scenarios offer far fewer opportunities for turns, and more occasions for others. To the former class of winning applications, because it eliminates local access/distribution/aggregation switches and then an entire lineage of hierarchical in-building routing elements. To the latter class of loser applications, no doubt, if a collapsed backbone design were to be dropped-shipped in place on a Friday Evening, as is, the there would surely be some losers that would require re-designing, or maybe simply some re-tuning, or they may need to be treated as one-offs entirely. BTW, in case there is any confusion concerning my earlier allusion to SMB, it had nothing to do with the size of message blocks, protocols, or anything else affecting a transaction profile's latency numbers. Instead, I was referring to the _s_mall-to-_m_edium-sized _b_usiness class of customers that the cable operator Bright House Networks was targeting with its passive optical network business-grade offering, fwiw. -- Mikael, All, I truly appreciate the comments and criticisms you've offered on this subject up until now in connection with the upstream hypothesis that began with a post by Michael Dillon. However, I shall not impose this topic on the larger audience any further. I would, however, welcome a continuation _offlist _ with anyone so inclined. If anything worthwhile results I'd be pleased to post it here at a later date. TIA. Frank A. Coluccio DTI Consulting Inc. 212-587-8150 Office 347-526-6788 Mobile On Sun Mar 30 3:17 , Mikael Abrahamsson sent: On Sat, 29 Mar 2008, Frank Coluccio wrote: Understandably, some applications fall into a class that requires very-short distances for the reasons you cite, although I'm still not comfortable with the setup you've outlined. Why, for example, are you showing two Ethernet switches for the fiber option (which would naturally double the switch-induced latency), but only a single switch for the UTP option? Yes, I am showing a case where you have switches in each rack so each rack is uplinked with a fiber to a central aggregation switch, as opposed to having a lot of UTP from the rack directly into the aggregation switch. Now, I'm comfortable in ceding this point. I should have made allowances for this type of exception in my introductory post, but didn't, as I also omitted mention of other considerations for the sake of brevity. For what it's worth, propagation over copper is faster propagation over fiber, as copper has a higher nominal velocity of propagation (NVP) rating than does fiber, but not significantly greater to cause the difference you've cited. The 2/3 speed of light in fiber as opposed to propagation speed in copper was not in my mind. As an aside, the manner in which o-e-o and e-o-e conversions take place when transitioning from electronic to optical states, and back, affects latency differently across differing link assembly approaches used. In cases where 10Gbps My opinion is that the major factors of added end-to-end latency in my example is that the packet has to be serialisted three times as opposed to once and there are three lookups instead of one. Lookups take time, putting the packet on the wire take time. Back in the 10 megabit/s days, there were switches that did cut-through, ie if the output port was not being used the instant the packet came in, it could start to send out the packet on the outgoing port before it was completely taken in on the incoming port (when the header was received, the forwarding decision was taken and the equipment would start to send the packet out before it was completely received from the input port). By chance, is the deserialization you cited earlier, perhaps related to this inverse muxing process? If so, then that would explain the disconnect, and if it is so, then one shouldn't despair, because there is a direct path to avoiding this. No, it's the store-and-forward architecture used in all modern equipment (that I know of). A packet has to be completely taken in over the wire into a buffer, a lookup has to be done as to where this packet should be put out, it needs to be sent over a bus or fabric, and then it has to be clocked out on the outgoing port from another buffer. This adds latency in each switch hop on the way. As Adrian Chadd mentioned in the email sent after yours, this can of course be handled by modifying or creating new protocols that handle this fact. It's just that with what is available today, this is a problem. Each directory listing or file access takes a bit longer over NFS with added latency, and this reduces performance in current protocols. Programmers who do client/server applications are starting to notice this and I know of companies
Re: latency (was: RE: cooling door)
Silly me. I didn't mean turns alone, but also intended to include the number of state transitions (e-o, o-e, e-e, etc.) in my preceding reply, as well. Frank A. Coluccio DTI Consulting Inc. 212-587-8150 Office 347-526-6788 Mobile On Sun Mar 30 16:47 , Frank Coluccio sent: Mikael, I see your points more clearly now in respect to the number of turns affecting latency. In analyzing this further, however, it becomes apparent that the collapsed backbone regimen may, in many scenarios offer far fewer opportunities for turns, and more occasions for others. To the former class of winning applications, because it eliminates local access/distribution/aggregation switches and then an entire lineage of hierarchical in-building routing elements. To the latter class of loser applications, no doubt, if a collapsed backbone design were to be dropped-shipped in place on a Friday Evening, as is, the there would surely be some losers that would require re-designing, or maybe simply some re-tuning, or they may need to be treated as one-offs entirely. BTW, in case there is any confusion concerning my earlier allusion to SMB, it had nothing to do with the size of message blocks, protocols, or anything else affecting a transaction profile's latency numbers. Instead, I was referring to the _s_mall-to-_m_edium-sized _b_usiness class of customers that the cable operator Bright House Networks was targeting with its passive optical network business-grade offering, fwiw. -- Mikael, All, I truly appreciate the comments and criticisms you've offered on this subject up until now in connection with the upstream hypothesis that began with a post by Michael Dillon. However, I shall not impose this topic on the larger audience any further. I would, however, welcome a continuation _offlist _ with anyone so inclined. If anything worthwhile results I'd be pleased to post it here at a later date. TIA. Frank A. Coluccio DTI Consulting Inc. 212-587-8150 Office 347-526-6788 Mobile On Sun Mar 30 3:17 , Mikael Abrahamsson sent: On Sat, 29 Mar 2008, Frank Coluccio wrote: Understandably, some applications fall into a class that requires very-short distances for the reasons you cite, although I'm still not comfortable with the setup you've outlined. Why, for example, are you showing two Ethernet switches for the fiber option (which would naturally double the switch-induced latency), but only a single switch for the UTP option? Yes, I am showing a case where you have switches in each rack so each rack is uplinked with a fiber to a central aggregation switch, as opposed to having a lot of UTP from the rack directly into the aggregation switch. Now, I'm comfortable in ceding this point. I should have made allowances for this type of exception in my introductory post, but didn't, as I also omitted mention of other considerations for the sake of brevity. For what it's worth, propagation over copper is faster propagation over fiber, as copper has a higher nominal velocity of propagation (NVP) rating than does fiber, but not significantly greater to cause the difference you've cited. The 2/3 speed of light in fiber as opposed to propagation speed in copper was not in my mind. As an aside, the manner in which o-e-o and e-o-e conversions take place when transitioning from electronic to optical states, and back, affects latency differently across differing link assembly approaches used. In cases where 10Gbps My opinion is that the major factors of added end-to-end latency in my example is that the packet has to be serialisted three times as opposed to once and there are three lookups instead of one. Lookups take time, putting the packet on the wire take time. Back in the 10 megabit/s days, there were switches that did cut-through, ie if the output port was not being used the instant the packet came in, it could start to send out the packet on the outgoing port before it was completely taken in on the incoming port (when the header was received, the forwarding decision was taken and the equipment would start to send the packet out before it was completely received from the input port). By chance, is the deserialization you cited earlier, perhaps related to this inverse muxing process? If so, then that would explain the disconnect, and if it is so, then one shouldn't despair, because there is a direct path to avoiding this. No, it's the store-and-forward architecture used in all modern equipment (that I know of). A packet has to be completely taken in over the wire into a buffer, a lookup has to be done as to where this packet should be put out, it needs to be sent over a bus or fabric, and then it has to be clocked out on the outgoing port from another buffer. This adds latency in each switch hop on the way. As Adrian Chadd mentioned in the email sent after yours, this can of course be handled by modifying or creating new protocols that handle this fact. It's just that with
Re: cooling door
I can lease 10 racks, put T1600s in two of them, and leave the other 8 empty; but that hasn't helped either me the customer or the exchange point provider; they've had to burn more real estate for empty racks that can never be filled Seems fine to me, you used your power in two racks, to get more power will cost the provider more so you will have to pay more. This is just an extreme example of blade servers causing half a rack to be empty the facts haven't changed no matter how bad seeing empty racks feels Waste implies you could use it for no additional cost, old pricing models were vulnerable to gaming on combinations they'd not thought through. It might be easier for people to understand in these cases if the provider put yellow/black stripe tape over the unused space with a big sign saying not yours I'm paying for floor space in my cage that I'm probably going to end up using for storage rather than just have it go to waste That's nice, I hate cages where you have no room for tools or to work and the kit hits the walls when you try and unrack it. I've no idea how they fit a normal engineer in some and we still have the problem of two very hot spots that need relatively 'point' cooling solutions. Accepted, big fan on the back of the rack? Plenty of empty space for such solutions. High density servers seem to be vendor driven, they can charge more and make you buy the switch and other ancillaries you'd likely choose cheaper from others. And when new models come out the whole lot gets replaced rather than just the odd few U of servers. The convenience may be worth the high price for some situations Density is just another DC design parameter to be optimised for profit brandon -- You know a nanog thread has gone on too long when I overcome inertia and post. More science please.
Re: cooling door
I have a need for a 1U that will just act as a backup (higher MX) mailserver and, occasionally, deliver some large .iso images at under 10Mbit/Sec :) And I'm sure that there are other technically saavy users just like me that could help you out with this surplus space! :) see http://www.vix.com/personalcolo/ for some places to host that backup MX. (note, i have no business affiliation with any of the entities listed there.)
cooling door
page 10 and 11 of http://www.panduit.com/products/brochures/105309.pdf says there's a way to move 20kW of heat away from a rack if your normal CRAC is moving 10kW (it depends on that basic air flow), permitting six blade servers in a rack. panduit licensed this tech from IBM a couple of years ago. i am intrigued by the possible drop in total energy cost per delivered kW, though in practice most datacenters can't get enough utility and backup power to run at this density. if cooling doors were to take off, we'd see data centers partitioned off and converted to cubicles. -- Paul Vixie
Re: cooling door
At 3:17 PM + 3/29/08, Paul Vixie wrote: page 10 and 11 of http://www.panduit.com/products/brochures/105309.pdf says there's a way to move 20kW of heat away from a rack if your normal CRAC is moving 10kW (it depends on that basic air flow), permitting six blade servers in a rack. panduit licensed this tech from IBM a couple of years ago. i am intrigued by the possible drop in total energy cost per delivered kW, though in practice most datacenters can't get enough utility and backup power to run at this density. While the chilled water door will provide higher equipment density per rack, it relies on water piping back to a Cooling Distribution Unit (CDU) which is in the corner sitting by your CRAC/CRAH units. Whether this is actually more efficient depends quite a bit on the (omitted) specifications for that unit...I know that it would have to be quite a bit before many folks would: 1) introduce another cooling system (with all the necessary redundancy), and 2) put pressurized water in the immediate vicinity of any computer equipment. /John
Re: cooling door
On Sat, 29 Mar 2008, John Curran wrote: unit...I know that it would have to be quite a bit before many folks would: 1) introduce another cooling system (with all the necessary redundancy), and 2) put pressurized water in the immediate vicinity of any computer equipment. What could possibly go wrong? :) If it leaks, you get the added benefits of conductive and evaporative cooling. -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: cooling door
On 29 Mar 2008, Paul Vixie wrote: page 10 and 11 of http://www.panduit.com/products/brochures/105309.pdf says there's a way to move 20kW of heat away from a rack if your normal CRAC is moving 10kW (it depends on that basic air flow), permitting six blade servers in a rack. panduit licensed this tech from IBM a couple of years ago. i am intrigued by the possible drop in total energy cost per delivered kW, though in practice most datacenters can't get enough utility and backup power to run at this density. if cooling doors were to take off, we'd see data centers partitioned off and converted to cubicles. Can someone please, pretty please with sugar on top, explain the point behind high power density? Raw real estate is cheap (basically, nearly free). Increasing power density per sqft will *not* decrease cost, beyond 100W/sqft, the real estate costs are a tiny portion of total cost. Moving enough air to cool 400 (or, in your case, 2000) watts per square foot is *hard*. I've started to recently price things as cost per square amp. (That is, 1A power, conditioned, delivered to the customer rack and cooled). Space is really irrelevant - to me, as colo provider, whether I have 100A going into a single rack or 5 racks, is irrelevant. In fact, my *costs* (including real estate) are likely to be lower when the load is spread over 5 racks. Similarly, to a customer, all they care about is getting their gear online, and can care less whether it needs to be in 1 rack or in 5 racks. To rephrase vijay, what is the problem being solved? [not speaking as mlc anything]
Re: cooling door
Can someone please, pretty please with sugar on top, explain the point behind high power density? maybe. Raw real estate is cheap (basically, nearly free). not in downtown palo alto. now, you could argue that downtown palo alto is a silly place for an internet exchange. or you could note that conditions giving rise to high and diverse longhaul and metro fiber density, also give rise to high real estate costs. Increasing power density per sqft will *not* decrease cost, beyond 100W/sqft, the real estate costs are a tiny portion of total cost. Moving enough air to cool 400 (or, in your case, 2000) watts per square foot is *hard*. if you do it the old way, which is like you said, moving air, that's always true. but, i'm not convinced that we're going to keep doing it the old way. I've started to recently price things as cost per square amp. (That is, 1A power, conditioned, delivered to the customer rack and cooled). Space is really irrelevant - to me, as colo provider, whether I have 100A going into a single rack or 5 racks, is irrelevant. In fact, my *costs* (including real estate) are likely to be lower when the load is spread over 5 racks. Similarly, to a customer, all they care about is getting their gear online, and can care less whether it needs to be in 1 rack or in 5 racks. To rephrase vijay, what is the problem being solved? if you find me 300Ksqft along the caltrain fiber corridor in the peninsula where i can get 10mW of power and have enough land around it for 10mW worth of genset, and the price per sqft is low enough that i can charge by the watt and floor space be damned and still come out even or ahead, then please do send me the address.
RE: cooling door
Can someone please, pretty please with sugar on top, explain the point behind high power density? It allows you to market your operation as a data center. If you spread it out to reduce power density, then the logical conclusion is to use multiple physical locations. At that point you are no longer centralized. In any case, a lot of people are now questioning the traditional data center model from various angles. The time is ripe for a paradigm change. My theory is that the new paradigm will be centrally managed, because there is only so much expertise to go around. But the racks will be physically distributed, in virtually every office building, because some things need to be close to local users. The high speed fibre in Metro Area Networks will tie it all together with the result that for many applications, it won't matter where the servers are. Note that the Google MapReduce, Amazon EC2, Haddoop trend will make it much easier to place an application without worrying about the exact locations of the physical servers. Back in the old days, small ISPs set up PoPs by finding a closet in the back room of a local store to set up modem banks. In the 21st century folks will be looking for corporate data centers with room for a rack or two of multicore CPUs running XEN, and Opensolaris SANs running ZFS/raidz providing iSCSI targets to the XEN VMs. --Michael Dillon
Re: cooling door
[EMAIL PROTECTED] (John Curran) writes: While the chilled water door will provide higher equipment density per rack, it relies on water piping back to a Cooling Distribution Unit (CDU) which is in the corner sitting by your CRAC/CRAH units. it just has to sit near the chilled water that moves the heat to the roof. that usually means CRAC-adjacency but other arrangements are possible. I know that it would have to be quite a bit before many folks would: 1) introduce another cooling system (with all the necessary redundancy), and 2) put pressurized water in the immediate vicinity of any computer equipment. the pressure differential between the pipe and atmospheric isn't that much. nowhere near steam or hydraulic pressures. if it gave me ~1500w/SF in a dense urban neighborhood i'd want to learn more. -- Paul Vixie
Re: cooling door
At 7:06 PM + 3/29/08, Paul Vixie wrote: While the chilled water door will provide higher equipment density per rack, it relies on water piping back to a Cooling Distribution Unit (CDU) which is in the corner sitting by your CRAC/CRAH units. it just has to sit near the chilled water that moves the heat to the roof. that usually means CRAC-adjacency but other arrangements are possible. When one of the many CRAC units decides to fail in an air-cooled environment, another one starts up and everything is fine. The nominal worse case leaves the failed CRAC unit as a potential air pressure leakage source for the raised-floor and/or ductwork, but that's about it. Chilled water to the rack implies multiple CDU's with a colorful hose and valve system within the computer room (effectively a miniature version of the facility chilled water loop). Trying to eliminate potential failure modes in that setup will be quite the adventure, which depending on your availability target may be a non-issue or a great reason to consider moving to new space. /John
Re: cooling door
John Curran wrote: Chilled water to the rack implies multiple CDU's with a colorful hose and valve system within the computer room (effectively a miniature version of the facility chilled water loop). Trying to eliminate potential failure modes in that setup will be quite the adventure, which depending on your availability target may be a non-issue or a great reason to consider moving to new space. Actually it wouldn't have to be pressurized at all if you located a large tank containing chilled water above and to the side, with a no-kink, straight line to the tank. N+1 chiller units could feed the tank. Thermo-siphoning would occur (though usually done with a cold line at the bottom and a return, warmed line at the top of the cooling device) as the warm water rises to the chilled tank and more chilled water flows down to the intake. You would of course have to figure out how to monitor/cut off/contain any leaks. Advantage is that cooling would continue up to the limit of the BTUs stored in the chilled water tank, even in the absence of power. Cordially Patrick Giagnocavo [EMAIL PROTECTED]
RE: cooling door
Michael Dillon is spot on when he states the following (quotation below), although he could have gone another step in suggesting how the distance insensitivity of fiber could be further leveraged: The high speed fibre in Metro Area Networks will tie it all together with the result that for many applications, it won't matter where the servers are. In fact, those same servers, and a host of other storage and network elements, can be returned to the LAN rooms and closets of most commercial buildings from whence they originally came prior to the large-scale data center consolidations of the current millennium, once organizations decide to free themselves of the 100-meter constraint imposed by UTP-based LAN hardware and replace those LANs with collapsed fiber backbone designs that attach to remote switches (which could be either in-building or remote), instead of the minimum two switches on every floor that has become customary today. We often discuss the empowerment afforded by optical technology, but we've barely scratched the surface of its ability to effect meaningful architectural changes. The earlier prospects of creating consolidated data centers were once near-universally considered timely and efficient, and they still are in many respects. However, now that the problems associated with a/c and power have entered into the calculus, some data center design strategies are beginning to look more like anachronisms that have been caught in a whip-lash of rapidly shifting conditions, and in a league with the constraints that are imposed by the now-seemingly-obligatory 100-meter UTP design. Frank A. Coluccio DTI Consulting Inc. 212-587-8150 Office 347-526-6788 Mobile On Sat Mar 29 13:57 , sent: Can someone please, pretty please with sugar on top, explain the point behind high power density? It allows you to market your operation as a data center. If you spread it out to reduce power density, then the logical conclusion is to use multiple physical locations. At that point you are no longer centralized. In any case, a lot of people are now questioning the traditional data center model from various angles. The time is ripe for a paradigm change. My theory is that the new paradigm will be centrally managed, because there is only so much expertise to go around. But the racks will be physically distributed, in virtually every office building, because some things need to be close to local users. The high speed fibre in Metro Area Networks will tie it all together with the result that for many applications, it won't matter where the servers are. Note that the Google MapReduce, Amazon EC2, Haddoop trend will make it much easier to place an application without worrying about the exact locations of the physical servers. Back in the old days, small ISPs set up PoPs by finding a closet in the back room of a local store to set up modem banks. In the 21st century folks will be looking for corporate data centers with room for a rack or two of multicore CPUs running XEN, and Opensolaris SANs running ZFS/raidz providing iSCSI targets to the XEN VMs. --Michael Dillon
RE: cooling door
On Sat, 29 Mar 2008, Frank Coluccio wrote: In fact, those same servers, and a host of other storage and network elements, can be returned to the LAN rooms and closets of most commercial buildings from whence they originally came prior to the How does that work? So now we buy a whole bunch of tiny gensets, and a whole bunch of baby UPSen and smaller cooling units to support little datacenters? Not to mention diverse paths to each point.. Didn't we (the customers) try that already and realize that it's rather unmanagable? I suppose the maintenance industry would love the surge in extra contracts to keep all the gear running ..david --- david raistrickhttp://www.netmeister.org/news/learn2quote.html [EMAIL PROTECTED] http://www.expita.com/nomime.html
RE: cooling door
I referenced LAN rooms as an expedient and to highlight an irony. The point is, smaller, less-concentrated, distributed enclosures suffice nicely for many purposes, similar to how Google's distributed containers and Sun Micro's Data Centers in a box do. And while LAN rooms that have been vacated, as a result of using collapsed fiber, might fit these needs, since they would have been already powered and conditioned in many cases, those could actually be reclaimed by tenants and landlords as usable floor space in many cases. I suppose the maintenance industry would love the surge in extra contracts to keep all the gear running Your supposition is open to wide interpretation. I'll take it to mean that you think more gear, not less, will require maintenance. Maybe in some cases, but in the vast majority not. Consider a multi-story commercial building that is entirely devoid of UTP-based switches, but instead is supported over fiber to a colo or managed service provider location. Why would this building require L2/3 aggregation switches and routers, simply to get in and out, if it hasn't any access switches inside? It wouldn't require any routers, is my point. This reduces the number of boxes reqired by a factor of two or more, since I no longer require routers onsite, and I no longer require their mates in the upstream or colos. I wouldn't require a classical in-building L3 hierarchy employing high-end routers at the distribution and core levels at all, or I'd require a lot fewer of them. Extending this rationale further, the level of logistics and LAN element administration required to keep on-prem applications humming is also reduced, ir not eliminated, and/or could easily be sourced more efficiently elsewhere. I.e., from a CLI or Web browser the LAN admin could be doing her thing from Mumbai (and in some cases this is already being done) or from home. So there's actually less gear to manage, not more. I realize this isn't a one-size-fits all model, and I didn't intend to make it appear that it was. But for the vast majority of enterprise buildings with tenants occupying large contiguous areas, I think it makes a great deal of sense, or at least would be worth evaluating to determine if it does. Frank A. Coluccio DTI Consulting Inc. 212-587-8150 Office 347-526-6788 Mobile On Sat Mar 29 18:30 , david raistrick sent: On Sat, 29 Mar 2008, Frank Coluccio wrote: In fact, those same servers, and a host of other storage and network elements, can be returned to the LAN rooms and closets of most commercial buildings from whence they originally came prior to the How does that work? So now we buy a whole bunch of tiny gensets, and a whole bunch of baby UPSen and smaller cooling units to support little datacenters? Not to mention diverse paths to each point.. Didn't we (the customers) try that already and realize that it's rather unmanagable? I suppose the maintenance industry would love the surge in extra contracts to keep all the gear running I suppose the maintenance industry would love the surge in extra contracts to keep all the gear running ..david --- david raistrickhttp://www.netmeister.org/news/learn2quote.html [EMAIL PROTECTED] http://www.expita.com/nomime.html
latency (was: RE: cooling door)
On Sat, 29 Mar 2008, Frank Coluccio wrote: We often discuss the empowerment afforded by optical technology, but we've barely scratched the surface of its ability to effect meaningful architectural changes. If you talk to the server people, they have an issue with this: Latency. I've talked to people who have collapsed layers in their LAN because they can see performance degradation for each additional switch packets have to pass in their NFS-mount. Yes, higher speeds means lower serialisation delay, but there is still a lookup time involved and 10GE is substantionally more expensive than GE. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: latency (was: RE: cooling door)
Please clarify. To which network element are you referring in connection with extended lookup times? Is it the collapsed optical backbone switch, or the upstream L3 element, or perhaps both? Certainly, some applications will demand far less latency than others. Gamers and some financial (program) traders, for instance, will not tolerate delays caused by access provisions that are extended over vast WAN, or even large Metro, distances. But in a local/intramural setting, where optical paths amount to no more than a klick or so, the impact is almost negligible, even to the class of users mentioned above. Worst case, run the enterprise over the optical model and treat those latency-sensitive users as the one-offs that they actually are by tying them into colos that are closer to their targets. That's what a growing number of financial firms from around the country have done in NY and CHI colos, in any case. As for cost, while individual ports may be significantly more expensive in one scenario than another, the architectural decision is seldom based on a single element cost. It's the TCO of all architectural considerations that must be taken into account. Going back to my original multi-story building example-- better yet, let's use one of the forty-story structures now being erected at Ground Zero as a case in point: When all is said and done it will have created a minimum of two internal data centers (main/backup/load-sharing) and a minimum of eighty (80) LAN enclosures, with each room consisting of two L2 access switches (where each of the latter possesses multiple 10Gbps uplinks, anyway), UPS/HVAC/Raised flooring, firestopping, sprinklers, and a commitment to consume power for twenty years in order to keep all this junk purring. I think you see my point. So even where cost may appear to be the issue when viewing cost comparisons of discreet elements, in most cases that qualify for this type of design, i.e. where an organization reaches critical mass beyond so many users, I submit that it really is not an issue. In fact, a pervasively-lighted environment may actually cost far less. Frank A. Coluccio DTI Consulting Inc. 212-587-8150 Office 347-526-6788 Mobile On Sat Mar 29 19:20 , Mikael Abrahamsson sent: On Sat, 29 Mar 2008, Frank Coluccio wrote: We often discuss the empowerment afforded by optical technology, but we've barely scratched the surface of its ability to effect meaningful architectural changes. If you talk to the server people, they have an issue with this: Latency. I've talked to people who have collapsed layers in their LAN because they can see performance degradation for each additional switch packets have to pass in their NFS-mount. Yes, higher speeds means lower serialisation delay, but there is still a lookup time involved and 10GE is substantionally more expensive than GE. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: latency (was: RE: cooling door)
On Sat, 29 Mar 2008, Frank Coluccio wrote: Please clarify. To which network element are you referring in connection with extended lookup times? Is it the collapsed optical backbone switch, or the upstream L3 element, or perhaps both? I am talking about the matter that the following topology: server - 5 meter UTP - switch - 20 meter fiber - switch - 20 meter fiber - switch - 5 meter UTP - server has worse NFS performance than: server - 25 meter UTP - switch - 25 meter UTP - server Imagine bringing this into metro with 1-2ms delay instead of 0.1-0.5ms. This is one of the issues that the server/storage people have to deal with. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: cooling door
At 02:11 PM 3/29/2008, Alex Pilosov wrote: Can someone please, pretty please with sugar on top, explain the point behind high power density? More equipment in your existing space means more revenue and more profit. Raw real estate is cheap (basically, nearly free). Increasing power density per sqft will *not* decrease cost, beyond 100W/sqft, the real estate costs are a tiny portion of total cost. Moving enough air to cool 400 (or, in your case, 2000) watts per square foot is *hard*. It depends on where you are located, but I understand what you are saying. However, the space is the cheap part. Installing the electrical power, switchgear, ATS gear, Gensets, UPS units, power distribution, cable/fiber distribution, connectivity to the datacenter, core and distribution routers/switches are all basically stepped incremental costs. If you can leverage the existing floor infrastructure then you maximize the return on your investment. I've started to recently price things as cost per square amp. (That is, 1A power, conditioned, delivered to the customer rack and cooled). Space is really irrelevant - to me, as colo provider, whether I have 100A going into a single rack or 5 racks, is irrelevant. In fact, my *costs* (including real estate) are likely to be lower when the load is spread over 5 racks. Similarly, to a customer, all they care about is getting their gear online, and can care less whether it needs to be in 1 rack or in 5 racks. I don't disagree with what you have written above, but if you can get 100A into all 5 racks (and cool it!), then you have five times the revenue with the same fixed infrastructure costs (with the exception of a bit more power, GenSet, UPS and cooling, but the rest of my costs stay the same.) To rephrase vijay, what is the problem being solved? For us in our datacenters, the problem being solved is getting as much return out of our investment as possible. -Robert Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 Well done is better than well said. - Benjamin Franklin
Re: latency (was: RE: cooling door)
On Sun, Mar 30, 2008, Mikael Abrahamsson wrote: On Sat, 29 Mar 2008, Frank Coluccio wrote: Please clarify. To which network element are you referring in connection with extended lookup times? Is it the collapsed optical backbone switch, or the upstream L3 element, or perhaps both? I am talking about the matter that the following topology: server - 5 meter UTP - switch - 20 meter fiber - switch - 20 meter fiber - switch - 5 meter UTP - server has worse NFS performance than: server - 25 meter UTP - switch - 25 meter UTP - server Imagine bringing this into metro with 1-2ms delay instead of 0.1-0.5ms. This is one of the issues that the server/storage people have to deal with. Thats because the LAN protocols need to be re-jiggled a little to start looking less like LAN protocols and more like WAN protocols. Similar things need to happen for applications. I helped a friend debug an NFS throughput issue between some Linux servers running Fortran-77 based numerical analysis code and a 10GE storage backend. The storage backend can push 10GE without too much trouble but the application wasn't poking the kernel in the right way (large fetches and prefetching, basically) to fully utilise the infrastructure. Oh, and kernel hz tickers can have similar effects on network traffic, if the application does dumb stuff. If you're (un)lucky then you may see 1 or 2ms of delay between packet input and scheduling processing. This doesn't matter so much over 250ms + latent links but matters on 0.1ms - 1ms latent links. (Can someone please apply some science to this and publish best practices please?) adrian
Re: latency (was: RE: cooling door)
Understandably, some applications fall into a class that requires very-short distances for the reasons you cite, although I'm still not comfortable with the setup you've outlined. Why, for example, are you showing two Ethernet switches for the fiber option (which would naturally double the switch-induced latency), but only a single switch for the UTP option? Now, I'm comfortable in ceding this point. I should have made allowances for this type of exception in my introductory post, but didn't, as I also omitted mention of other considerations for the sake of brevity. For what it's worth, propagation over copper is faster propagation over fiber, as copper has a higher nominal velocity of propagation (NVP) rating than does fiber, but not significantly greater to cause the difference you've cited. As an aside, the manner in which o-e-o and e-o-e conversions take place when transitioning from electronic to optical states, and back, affects latency differently across differing link assembly approaches used. In cases where 10Gbps or greater is being sent across a multi-mode fiber link in a data center or other in-building venue, for instance, parallel optics are most ofen used, i.e., multiple optical channels (either fibers or wavelengths) that undergo multiplexing and de-multiplexing (collectively: inverse multiplexing or channel bonding) -- as opposed to a single fiber (or a single wavelength) operating at the link's rated wire speed. By chance, is the deserialization you cited earlier, perhaps related to this inverse muxing process? If so, then that would explain the disconnect, and if it is so, then one shouldn't despair, because there is a direct path to avoiding this. In parallel optics, e-o processing and o-e processing is intensive at both ends of the 10G link, respectively. These have the effect of adding more latency than a single-channel approach would. Yet, most of the TIA activity taking place today that is geared to increasing data rates over in-building fiber links continues to favor multi-mode and the use of parallel optics, as opposed to specifying single-mode supporting a single channel. But singlemode solutions are also available to those who dare to be different. I'll look more closely at these issues and your original exception during the coming week, since they represent an important aspect in assessing the overall model. Thanks. Frank A. Coluccio DTI Consulting Inc. 212-587-8150 Office 347-526-6788 Mobile On Sat Mar 29 20:30 , Mikael Abrahamsson sent: On Sat, 29 Mar 2008, Frank Coluccio wrote: Please clarify. To which network element are you referring in connection with extended lookup times? Is it the collapsed optical backbone switch, or the upstream L3 element, or perhaps both? I am talking about the matter that the following topology: server - 5 meter UTP - switch - 20 meter fiber - switch - 20 meter fiber - switch - 5 meter UTP - server has worse NFS performance than: server - 25 meter UTP - switch - 25 meter UTP - server Imagine bringing this into metro with 1-2ms delay instead of 0.1-0.5ms. This is one of the issues that the server/storage people have to deal with. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: cooling door
On Sat, Mar 29, 2008 at 06:54:02PM +, Paul Vixie wrote: Can someone please, pretty please with sugar on top, explain the point behind high power density? Customers are being sold blade servers on the basis that it's much more efficient to put all your eggs in one basket without being told about the power or cooling requirements and how not a whole lot of datacenters really want/are able to support customers installing 15 racks of blade servers in one spot with 4x 230V/30A circuits each. (Yes, I had that request.) Customers don't want to pay for the space. They forget that they still have to pay for the power and that that charge also includes a fee for the added load on the UPS as well as the AC to get rid of the heat. While there are advantages to blade servers, a fair number of sales are to gullable users who don't know what they're getting into, not those who really know how to get the most out of them. They get sold on the idea of using blade servers, stick them into SD, Equinix, and others and suddenly find out that they can only fit 2 in a rack because of the per-rack wattage limit and end up having to buy the space anyway. (Wether it's extra racks or extra sq ft or meters, it's the same problem.) Under current rules for most 3rd party datacenters, one of the principle stated advantages, that of much greater density, is effectively canceled out. Increasing power density per sqft will *not* decrease cost, beyond 100W/sqft, the real estate costs are a tiny portion of total cost. Moving enough air to cool 400 (or, in your case, 2000) watts per square foot is *hard*. (Remind me to strap myself to the floor to keep from becoming airborne by the hurricane force winds while I'm working in your datacenter.) Not convinved of the first point but experience is limited there. For the second, I think the practical upper bound for my purposes is probably between 150 and 200 watts per sq foot. (Getting much harder once you cross the 150 watt mark.) Beyond that, it gets quite difficult to supply enough cool air to the cabinet to keep the equipment happy unless you can guarentee a static load and custom design for that specific load. (And we all know that will never happen.) And don't even talk to me about enclosed cabinets at that point. if you do it the old way, which is like you said, moving air, that's always true. but, i'm not convinced that we're going to keep doing it the old way. One thing I've learned over the various succession of datacenter / computer room builds and expansions that I've been involved in is that if you ask the same engineer about the right way to do cooling in medium and large scale datacenters (15k sq ft and up), you'll probably get a different oppinion every time you ask the question. There are several theories of how best to hand this and *none* of them are right. No one has figured out an ideal solution and I'm not convinced an ideal solution exists. So we go with what we know works. As people experiment, what works changes. The problem is that retrofitting is a bear. (When's the last time you were able to get a $350k PO approved to update cooling to the datacenter? If you can't show a direct ROI, the money people don't like you. And on a more practical line, how many datacenters have you seen where it is physically impossible to remove the CRAC equipment for replacement without first tearing out entire rows of racks or even building walls?) Anyway, my thoughts on the matter. -Wayne --- Wayne Bouchard [EMAIL PROTECTED] Network Dude http://www.typo.org/~web/
