[
https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997600#comment-12997600
]
Arun C Murthy commented on MAPREDUCE-279:
-----------------------------------------
bq. With ZooKeeper in there, I wonder how much of the communicaton now uses
ZooKeeper watches for efficiency and low latency.
Scott - We seriously considered this, but had to continue to use Hadoop RPC for
a couple of reasons:
a) Mahadev, our resident ZK (and the new ResourceManager) expert, was very vary
of using ZK watches for scalability reasons. Consider a 10k node cluster with
25-30 containers per node and 10k running jobs - we'd need at least 10k * 10k
watches which is a *lot* for ZK
b) Security on ZK is still largely unknown, eventually ZK will get there but
we'd need a lot of work to do for delegation tokens etc. since we can't do
kerberos everywhere.
Having said that...
bq. In any case, the new architecture is decoupled and it should be much easier
to make enhancements with this separation.
Exactly. This is something we should definitely re-visit in a subsequent
release. Hopefully that makes sense, thanks!
> Map-Reduce 2.0
> --------------
>
> Key: MAPREDUCE-279
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-279
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker, tasktracker
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
>
> Re-factor MapReduce into a generic resource scheduler and a per-job,
> user-defined component that manages the application execution.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
