[
https://issues.apache.org/jira/browse/MAPREDUCE-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537755#comment-13537755
]
Qinghe Jin commented on MAPREDUCE-2636:
---------------------------------------
Hi Steve, although the number of disks may several times than the number of
nodes, but I think there is only several bits different to identify it.Does it
really matter that much?
It's good idea to consider output and itermediate data, but do we need to think
about it for each task? I think the best configuration is to ensure the
locality of each task, which means it reads, writes to the same disk. In this
way, it makes more sense to the sheduler or user.
Conflict detection is necessary.If we rush to assign task to the busy nodes,
it's not only more harmful to the running tasks, but also will cause load
unblance problem. For conflict detection, there are two ways:1, find out how
many task running on the node; 2,monitor the actual usage of different
resources(for disk, we can use disk waiting time). I prefer the second method
for there maybe more than one hadoop deployment.
> Scheduling over disks horizontally
> ----------------------------------
>
> Key: MAPREDUCE-2636
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2636
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: job submission
> Reporter: Evert Lammerts
> Priority: Minor
>
> Based on this message:
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201106.mbox/browser
> The JT schedules tasks on nodes based on metadata it gets from the NN. The
> namenode does not know on which disk a block resides. It might happen that on
> a node running 4 tasks, all read from the same disk. This can affect
> performance.
> An optimization might be to schedule horizontally over disks instead of
> nodes. Any ideas?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
