[
https://issues.apache.org/jira/browse/HADOOP-5271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy resolved HADOOP-5271.
-----------------------------------
Resolution: Fixed
Fixed via HADOOP-3136.
> JobConf option for minimum progress threshold before reducers are assigned
> --------------------------------------------------------------------------
>
> Key: HADOOP-5271
> URL: https://issues.apache.org/jira/browse/HADOOP-5271
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Tim Williamson
> Attachments: HADOOP-5271.patch
>
>
> A specific sub-case of the general priority inversion problem noted in
> HADOOP-4557 is when many lower priority jobs are submitted and are waiting
> for mappers to free up. Even though they haven't actually done any work,
> they will be assigned any free reducers. If a higher priority job is
> submitted, priority inversion results not just due to the lower priority
> tasks that are in the midst of completing, but also due to the ones that
> haven't yet started but have claimed all the free reducers. A simple
> workaround is to require a job to complete some useful work before assigning
> it a reducer. This can be done in a tunable and backwards compatible manner
> by adding a "minimum map progress percentage before assigning a reducer"
> option to the JobConf. Setting this to 0 would eliminate the common case
> above, and setting it to 100 would technically eliminate the inversion of
> HADOOP-4557, though likely at an unacceptably high cost.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
