[
https://issues.apache.org/jira/browse/HADOOP-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726160#action_12726160
]
Scott Carey commented on HADOOP-6121:
-------------------------------------
If you really want to go very robust with the JobTracker, you could implement
it by embedding ZooKeeper and using it for all the RPC coordination and state
tracking. This is a more natural fit than ZK for the NN, IMO.
A job could be a node in ZK, TaskTrackers would be able to see what work they
should do by looking in specific nodes -- and they can register their status
and health as well. Something like a Reduce job getting the map outputs it
needs would simply be listing children from a reduce node for a job.
Fault tolerance would come for free, and this form of RPC would likely be more
efficient than the current one.
Furthermore, with watches firing off, there would be no ping-response sleep
delays which would improve performance, and significantly improve latency for
latency sensitive jobs.
> High Availability support for Hadoop
> ------------------------------------
>
> Key: HADOOP-6121
> URL: https://issues.apache.org/jira/browse/HADOOP-6121
> Project: Hadoop Common
> Issue Type: New Feature
> Components: dfs, mapred
> Reporter: Jie Qiu
>
> Currently, We look at the HA of Hadoop cluster. We need to consider the
> NameNode HA as well as Jobtracker HA. For NameNode, we want to build
> primary/standy or master-slaves pattern to provide NameNode HA. Therefore, we
> need to consider how to ship log between primary/standby/slaves and how
> commit "write" operation to NameNode after the agreement among
> primary/standby/slaves on log. Whether will we use Linux HA package or
> NameNode-built-in HA package without the help of outter Linux HA package.
> After NameNode become high availability, is it necessary to provide HA for
> Jobtracker? Can Jobtracker persist the states of Jobs and tasks into HA
> NameNode? Or Jobtracker also needs the same approach from NameNode for HA
> support.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.