[
https://issues.apache.org/jira/browse/HADOOP-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289335#comment-13289335
]
Junping Du commented on HADOOP-8468:
------------------------------------
I would update proposal a bit with listing the first approach. This is a
workaround without hadoop code change. However, this "1-1 mapping" of data node
to physical host will take following restrictions:
1. If nodes' number is larger than the number of physical host.
2. If the number of nodes is smaller than physical hosts, but some hosts are
fully occupied by other logical hadoop clusters or other applications.
3. The clouds/datacenters are formed of heterogeneous hosts that some hosts are
not suitable to deploy hadoop nodes. i.e. attached to shared storage only.
In general, VM placement in cloud is a complex BIN-packing problem which is
NP-hard and should be optimised for a balance of resource utilization and
reliability. Applying an absolute rule like the first approach is not the best
way. In addition, the principle of hadoop network topology should reflect the
physical(or virtual) topology in the bottom layer but should not take strict
requirements/restriction to deploying topology.
Thoughts?
> Umbrella of enhancements to support different failure and locality topologies
> -----------------------------------------------------------------------------
>
> Key: HADOOP-8468
> URL: https://issues.apache.org/jira/browse/HADOOP-8468
> Project: Hadoop Common
> Issue Type: Bug
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Priority: Critical
> Attachments: HADOOP-8468-total-v3.patch, HADOOP-8468-total.patch,
> Proposal for enchanced failure and locality topologies.pdf
>
>
> The current hadoop network topology (described in some previous issues like:
> Hadoop-692) works well in classic three-tiers network when it comes out.
> However, it does not take into account other failure models or changes in the
> infrastructure that can affect network bandwidth efficiency like:
> virtualization.
> Virtualized platform has following genes that shouldn't been ignored by
> hadoop topology in scheduling tasks, placing replica, do balancing or
> fetching block for reading:
> 1. VMs on the same physical host are affected by the same hardware failure.
> In order to match the reliability of a physical deployment, replication of
> data across two virtual machines on the same host should be avoided.
> 2. The network between VMs on the same physical host has higher throughput
> and lower latency and does not consume any physical switch bandwidth.
> Thus, we propose to make hadoop network topology extend-able and introduce a
> new level in the hierarchical topology, a node group level, which maps well
> onto an infrastructure that is based on a virtualized environment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
