[
https://issues.apache.org/jira/browse/HADOOP-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399536#comment-13399536
]
Konstantin Shvachko commented on HADOOP-8468:
---------------------------------------------
It's good that you formulated the policies. Now I can see the differences. In
way-2 you actually don't need to say "virtual node". It is the implementation
detail. You only care that the first replica is on the local physical node. So
way-2 is the same as the original.
In way-1 I agree only one change is needed. Rather surprising.
I briefly checked the patch, and see now that your abstractions are driven by
the implementation. Whether you define it way-1 or way-2 implementation-wise
you still introduce a new inner level in the topology.
I do not think you need the new class InnerNodeWithNodeGroup. It doesn't have
new members or constructors. It overrides isRack(), but only because the old
implementation assumed racks are on the second level. I'd rather add nodeType
member than checking children of children.
So, I think I understand your motivation with the design. Thanks for clarifying
your thoughts to me. I still think that the terminology is better when talking
about extending the topology with new leaves, but your way is also valid and
does not change the policy much. You choose. Either way, please add the full
policy definition in the document.
> Umbrella of enhancements to support different failure and locality topologies
> -----------------------------------------------------------------------------
>
> Key: HADOOP-8468
> URL: https://issues.apache.org/jira/browse/HADOOP-8468
> Project: Hadoop Common
> Issue Type: Bug
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Priority: Critical
> Attachments: HADOOP-8468-total-v3.patch, HADOOP-8468-total.patch,
> Proposal for enchanced failure and locality topologies (revised-1.0).pdf,
> Proposal for enchanced failure and locality topologies.pdf
>
>
> The current hadoop network topology (described in some previous issues like:
> Hadoop-692) works well in classic three-tiers network when it comes out.
> However, it does not take into account other failure models or changes in the
> infrastructure that can affect network bandwidth efficiency like:
> virtualization.
> Virtualized platform has following genes that shouldn't been ignored by
> hadoop topology in scheduling tasks, placing replica, do balancing or
> fetching block for reading:
> 1. VMs on the same physical host are affected by the same hardware failure.
> In order to match the reliability of a physical deployment, replication of
> data across two virtual machines on the same host should be avoided.
> 2. The network between VMs on the same physical host has higher throughput
> and lower latency and does not consume any physical switch bandwidth.
> Thus, we propose to make hadoop network topology extend-able and introduce a
> new level in the hierarchical topology, a node group level, which maps well
> onto an infrastructure that is based on a virtualized environment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
