[
https://issues.apache.org/jira/browse/MAPREDUCE-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288615#comment-13288615
]
Jason Lowe commented on MAPREDUCE-3927:
---------------------------------------
I'm not sure we should change the denominator of the progress calculation when
map tasks fail. As I see it, the map percentage is a representation of how
many map tasks have been processed regardless of successful or failure outcome.
50% map progress to me means half of the map tasks have been processed and
half have yet to finish processing.
For example, let's say we have 4 map tasks and are configured to allow at least
one to fail. When 1 out of 4 have completed, we're sitting at 25% map
completion. If another map task fails, I'd expect the completion to be 50% as
we've processed two tasks and have yet to finish the other two. However the
proposed change will report 33% completion since it will remove the failed task
from the total task count instead of adding it to the processed task count.
Adding it to the processed task count would also be in line with how a job
reports completed maps, see JobImpl.getCompletedMaps().
> Shuffle hang when set map.failures.percent
> ------------------------------------------
>
> Key: MAPREDUCE-3927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3927
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.0-alpha, 2.0.1-alpha, trunk
> Reporter: MengWang
> Assignee: Bhallamudi Venkata Siva Kamesh
> Priority: Critical
> Labels: patch
> Attachments: MAPREDUCE-3927-1.patch, MAPREDUCE-3927-2.patch,
> MAPREDUCE-3927.patch, MAPREDUCE-3927.patch
>
>
> When set mapred.max.map.failures.percent and there does have some failed
> maps, then shuffle will hang
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
