[ https://issues.apache.org/jira/browse/MAPREDUCE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578062#comment-13578062 ]
Mayank Bansal commented on MAPREDUCE-4305: ------------------------------------------ Hi, Thanks Arun for your comments. I had a offline discussion with Arun. The reason behind the timeouts which I have added was due to save jobs from starving if we have priority however we need more work in that case. So I am refactoring my patch in to two patches. This patch is mostly Yarn-80 for Hadoop-1 with test framework for testing those scenarios like node local , rack local etc. I will file the JIRA with priority and timeouts and update the patch there. Thanks, Mayank > Implement delay scheduling in capacity scheduler for improving data locality > ---------------------------------------------------------------------------- > > Key: MAPREDUCE-4305 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4305 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Reporter: Mayank Bansal > Assignee: Mayank Bansal > Attachments: MAPREDUCE-4305, MAPREDUCE-4305-1.patch, > PATCH-MAPREDUCE-4305-MR1-1.patch, PATCH-MAPREDUCE-4305-MR1.patch > > > Capacity Scheduler data local tasks are about 40%-50% which is not good. > While my test with 70 node cluster i consistently get data locality around > 40-50% on a free cluster. > I think we need to implement something like delay scheduling in the capacity > scheduler for improving the data locality. > http://radlab.cs.berkeley.edu/publication/308 > After implementing the delay scheduling on Hadoop 22 I am getting 100 % data > locality in free cluster and around 90% data locality in busy cluster. > Thanks, > Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira