[
https://issues.apache.org/jira/browse/MAPREDUCE-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832391#action_12832391
]
Amareshwari Sriramadasu commented on MAPREDUCE-1471:
----------------------------------------------------
bq. If the file name output is guaranteed by the client application to be
unique, the temporary files/directories should also be guaranteed to be unique
to avoid this problem
I like this idea. But, this should be done in your own custom
OutputFormat/OutputCommitter.
We cannot change FileOutputFormat/Committer for the following reason:
Currently the output file names are part-m-00000 or part-r-00000 for tasks
m_00000 and r_00000 irrespective of the jobid. Unless user changes these file
names to be different for different jobs, user can not run concurrent jobs with
same output path. If he runs without considering final output file names,
output is unpredictable; One job will replace the output files from the other
job. i.e. part-m-00000 is the output of either job1 or job2 and the other
output is lost!
So, FileOutputFormat makes sure that output directory does not exist already.
If user has made sure that the output file names are unique for each job by
overriding FileOutputFormat, then i would say he can write his own
OutputCommitter for running concurrent jobs.
bq. Currently I have overidden the getOutputCommiter(context) method in my out
FileOutputFormat to work around this.
This is the correct way to plug your custom OutputCommitter.
> FileOutputCommitter does not safely clean up it's temporary files
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1471
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1471
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: Jim Finnessy
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When the FileOutputCommitter cleans up during it's cleanupJob method, it
> potentially deletes the temporary files of other concurrent jobs.
> Since all the temporary files for all concurrent jobs are written to
> working_path/_temporary/ any concurrent tasks that have the same working_path
> will remove all currently executing jobs when it removes
> working_path/_temporary during job cleanup.
> If the file name output is guaranteed by the client application to be unique,
> the temporary files/directories should also be guaranteed to be unique to
> avoid this problem. Suggest modifying cleanupJob to only remove files that it
> created itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
