[ http://issues.apache.org/jira/browse/HADOOP-76?page=all ]
Sanjay Dahiya updated HADOOP-76:
--------------------------------
Attachment: Hadoop-76_1.patch
Here is update patch for review
changes in this patch
- Moved temp files to (mapred.system.dir)/<jobid>
- 2 new variables in TIP -
- runningTasks ( to track currently running instances of attempts
for same task, earlier this was done using a boolean)
- hasCompletedTask ( for reduces if a attempts succeeds then all
subsequent failures for same TIP should be ignored, existing setup tries to
schedule another task for that.)
- Added a phaseFileSystem, which takes a jobid, tipid, taskid and creates all
files in (mapred.system.dir)/<jobid>/<tipid>/<taskid>, these are moved to their
final location on commit or deleted.
- Task Constructor needs tip id now, its passed in RPC as well.
- Task.localizeConfiguration adds tip id to conf.
- Minor change in FSDirectory to add exception message in log if rename fails.
+Planning to add an example for using PhasedFileSystem - changing RandomWriter
to use PhasedFileSystem, as Devaraj suggested in a separate patch on the same
issue.
thanks owen for review.
> Implement speculative re-execution of reduces
> ---------------------------------------------
>
> Key: HADOOP-76
> URL: http://issues.apache.org/jira/browse/HADOOP-76
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.1.0
> Reporter: Doug Cutting
> Assigned To: Sanjay Dahiya
> Priority: Minor
> Attachments: Hadoop-76.patch, Hadoop-76_1.patch, spec_reducev.patch
>
>
> As a first step, reduce task outputs should go to temporary files which are
> renamed when the task completes.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
