[
https://issues.apache.org/jira/browse/HADOOP-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541227
]
Alejandro Abdelnur commented on HADOOP-1558:
--------------------------------------------
A couple follow up comments on the solution proposed by this patch:
* The OutputHandler interface is not meant to be a public interface, it is not
expected M/R developers write alternate implementations of it. Just allows
Hadoop to easily plugin an alternate implementation for other type of storage
if required.
* The logic done in the OutpuHandler interface is not much different for what
today is being done in the JobTracker, it is just a refactoring/encapsulation
of the FS operations when handling the output dir/files
While I understand the motivations of Doug and Owen on doing some mayor changes
(either via a new RPC call or a new task), the proposed changes are minor hooks
in the current implementation and it is mostly moving FS operations from one
place to another.
Until a mayor refactoring is done these changes would allow implementing
functionality to restart jobs automatically after a JT failure.
> changes to OutputFormat to work on temporary directory to enable re-running
> crashed jobs (Issue: 1121)
> ------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1558
> URL: https://issues.apache.org/jira/browse/HADOOP-1558
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: all
> Reporter: Alejandro Abdelnur
> Attachments: hadoop-1558-JUL2607-1600.txt
>
>
> Add OutputFormat methods like:
> /** Called to initialize output for this job. */
> void initialize(JobConf job) throws IOException;
> /** Called to finalize output for this job. */
> void commit(JobConf job) throws IOException;
> In the base implemenation for FileSystem output, initialize() might then
> create a temporary directory for the job, removing any that already exists,
> and commit could rename the temporary output directory to the final name.
> The existing checkOutputSpecs() would continue to throw an exception if the
> final output already exists.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
