[ http://issues.apache.org/jira/browse/HADOOP-692?page=comments#action_12448252 ] Doug Cutting commented on HADOOP-692: -------------------------------------
Should we think about super-rack locality at this point? For example, Amazon EC2 can allocate nodes in a cluster across multiple datacenters, or at least a few network hops away. This is obviously not an optimal configuration (although, in my 20-node benchmarks, it didn't slow things noticably). Should we try to support things like this now? http://developer.amazonwebservices.com/connect/thread.jspa?threadID=11615 Network hops would be one way to measure things. Localhost is zero hops, same rack is one, different rack in the same datacenter is two, and so on. This could be repressented by a list of ids per node (host, rack, datacenter, ...). > Rack-aware Replica Placement > ---------------------------- > > Key: HADOOP-692 > URL: http://issues.apache.org/jira/browse/HADOOP-692 > Project: Hadoop > Issue Type: Improvement > Components: dfs > Affects Versions: 0.8.0 > Reporter: Hairong Kuang > Assigned To: Hairong Kuang > Fix For: 0.9.0 > > > This issue assumes that HDFS runs on a cluster of computers that spread > across many racks. Communication between two nodes on different racks needs > to go through switches. Bandwidth in/out of a rack may be less than the total > bandwidth of machines in the rack. The purpose of rack-aware replica > placement is to improve data reliability, availability, and network bandwidth > utilization. The basic idea is that each data node determines to which rack > it belongs at the startup time and notifies the name node of the rack id upon > registration. The name node maintains a rackid-to-datanode map and tries to > place replicas across racks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
