[
https://issues.apache.org/jira/browse/HADOOP-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9184:
-------------------------------------
Labels: BB2015-05-TBR (was: )
> Some reducers failing to write final output file to s3.
> -------------------------------------------------------
>
> Key: HADOOP-9184
> URL: https://issues.apache.org/jira/browse/HADOOP-9184
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.2
> Reporter: Jeremy Karn
> Labels: BB2015-05-TBR
> Attachments: HADOOP-9184-branch-0.20.patch, example.pig,
> hadoop-9184.patch, task_log.txt
>
>
> We had a Hadoop job that was running 100 reducers with most of the reducers
> expected to write out an empty file. When the final output was to an S3
> bucket we were finding that sometimes we were missing a final part file.
> This was happening approximately 1 job in 3 (so approximately 1 reducer out
> of 300 was failing to output the data properly). I've attached the pig script
> we were using to reproduce the bug.
> After an in depth look and instrumenting the code we traced the problem to
> moveTaskOutputs in FileOutputCommitter.
> The code there looked like:
> {code}
> if (fs.isFile(taskOutput)) {
> … do stuff …
> } else if(fs.getFileStatus(taskOutput).isDir()) {
> … do stuff …
> }
> {code}
> And what we saw happening is that for the problem jobs neither path was being
> exercised. I've attached the task log of our instrumented code. In this
> version we added an else statement and printed out the line "THIS SEEMS LIKE
> WE SHOULD NEVER GET HERE …".
> The root cause of this seems to be an eventual consistency issue with S3.
> You can see in the log that the first time moveTaskOutputs is called it finds
> that the taskOutput is a directory. It goes into the isDir() branch and
> successfully retrieves the list of files in that directory from S3 (in this
> case just one file). This triggers a recursive call to moveTaskOutputs for
> the file found in the directory. But in this pass through moveTaskOutput the
> temporary output file can't be found resulting in both branches of the above
> if statement being skipped and the temporary file never being moved to the
> final output location.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
