[
https://issues.apache.org/jira/browse/MAPREDUCE-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sultan Alamro updated MAPREDUCE-7245:
-------------------------------------
Description:
When we set *mapreduce.map.maxattempts* to 1, the reduce tasks should ignore
the output of failed tasks as state it in EventFetch class. However, it turns
out that this only happens when a map task transitions from RUNNING to FAILED,
not from SCHEDULED to FAILED.
I think this problem can be solved in TaskImpl.java file by adding an else
statement if there is not container assigned.
if (attempt.getNodeHttpAddress() != null){
TaskAttemptCompletionEvent tce =
recordFactory.newRecordInstance(TaskAttemptCompletionEvent.class);
tce.setEventId(-1); String scheme = (encryptedShuffle) ? "https://"; :
"http://";
tce.setMapOutputServerAddress(StringInterner.weakIntern(scheme +
attempt.getNodeHttpAddress().split(":")[0] + ":" +
attempt.getShufflePort()));
tce.setStatus(status); tce.setAttemptId(attempt.getID());
int runTime = 0;
if (attempt.getFinishTime() != 0 && attempt.getLaunchTime() !=0)
runTime = (int)(attempt.getFinishTime() - attempt.getLaunchTime());
tce.setAttemptRunTime(runTime);
//raise the event to job so that it adds the completion event to its
//data structures eventHandler.handle(new
JobTaskAttemptCompletedEvent(tce)); }
{color:#de350b}else { {color}
{color:#de350b} TaskAttemptCompletionEvent tce = {color}
{color:#de350b}
recordFactory.newRecordInstance(TaskAttemptCompletionEvent.class); {color}
{color:#de350b} tce.setEventId(-1); tce.setStatus(status);
tce.setAttemptId(attempt.getID()); {color}
{color:#de350b} eventHandler.handle(new
JobTaskAttemptCompletedEvent(tce)); {color}
{color:#de350b} }{color}
was:
When we set *mapreduce.map.maxattempts* to 1, the reduce tasks should ignore
the output of failed tasks as state it in EventFetch class. However, it turns
out that this only happens when a map task transitions from RUNNING to FAILED,
not from SCHEDULED to FAILED.
I think this problem can be solved in TaskImpl.java file by adding an else
statement if there is not container assigned.
if (attempt.getNodeHttpAddress() != null) {
TaskAttemptCompletionEvent tce = recordFactory
.newRecordInstance(TaskAttemptCompletionEvent.class);
tce.setEventId(-1);
String scheme = (encryptedShuffle) ? "https://"; : "http://";;
tce.setMapOutputServerAddress(StringInterner.weakIntern(scheme
+ attempt.getNodeHttpAddress().split(":")[0] + ":"
+ attempt.getShufflePort()));
tce.setStatus(status);
tce.setAttemptId(attempt.getID());
int runTime = 0;
if (attempt.getFinishTime() != 0 && attempt.getLaunchTime() !=0)
runTime = (int)(attempt.getFinishTime() - attempt.getLaunchTime());
tce.setAttemptRunTime(runTime);
//raise the event to job so that it adds the completion event to its
//data structures
eventHandler.handle(new JobTaskAttemptCompletedEvent(tce));
} else {
TaskAttemptCompletionEvent tce = recordFactory
.newRecordInstance(TaskAttemptCompletionEvent.class);
tce.setEventId(-1);
tce.setStatus(status);
tce.setAttemptId(attempt.getID());
eventHandler.handle(new JobTaskAttemptCompletedEvent(tce));
}
> Reduce phase does not continue processing with failed SCHEDULED Map tasks
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-7245
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7245
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.7.2, 3.2.1
> Reporter: Sultan Alamro
> Priority: Major
>
> When we set *mapreduce.map.maxattempts* to 1, the reduce tasks should ignore
> the output of failed tasks as state it in EventFetch class. However, it turns
> out that this only happens when a map task transitions from RUNNING to
> FAILED, not from SCHEDULED to FAILED.
>
> I think this problem can be solved in TaskImpl.java file by adding an else
> statement if there is not container assigned.
>
> if (attempt.getNodeHttpAddress() != null){
> TaskAttemptCompletionEvent tce =
> recordFactory.newRecordInstance(TaskAttemptCompletionEvent.class);
> tce.setEventId(-1); String scheme = (encryptedShuffle) ? "https://"; :
> "http://";
> tce.setMapOutputServerAddress(StringInterner.weakIntern(scheme +
> attempt.getNodeHttpAddress().split(":")[0] + ":" +
> attempt.getShufflePort()));
> tce.setStatus(status); tce.setAttemptId(attempt.getID());
> int runTime = 0;
> if (attempt.getFinishTime() != 0 && attempt.getLaunchTime() !=0)
> runTime = (int)(attempt.getFinishTime() - attempt.getLaunchTime());
>
> tce.setAttemptRunTime(runTime);
> //raise the event to job so that it adds the completion event to its
> //data structures eventHandler.handle(new
> JobTaskAttemptCompletedEvent(tce)); }
> {color:#de350b}else { {color}
> {color:#de350b} TaskAttemptCompletionEvent tce = {color}
> {color:#de350b}
> recordFactory.newRecordInstance(TaskAttemptCompletionEvent.class); {color}
> {color:#de350b} tce.setEventId(-1); tce.setStatus(status);
> tce.setAttemptId(attempt.getID()); {color}
> {color:#de350b} eventHandler.handle(new
> JobTaskAttemptCompletedEvent(tce)); {color}
> {color:#de350b} }{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
