[
https://issues.apache.org/jira/browse/HADOOP-9577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13661590#comment-13661590
]
Joshua Caplan commented on HADOOP-9577:
---------------------------------------
To answer your questions:
1. This was the Amazon variant of 1.0.3 with Multipart Uploading disabled,
although looking at the Apache code even as far forward as the 2.0.x branches I
don't see how the logic would be any different
2. Speculative execution was certainly enabled. I suppose if I could live
without speculative execution for the jobs that write directly to S3, I could
use the DirectFileOutputCommitter or some derivative to avoid this issue.
Your comments about blobstores in general and Hadoop's assumptions about
FileSystems are enlightening. I definitely prefer the commit to HDFS and copy
later strategy.
My reason for calling out this particular FileSystem side-effect assumption is
that it seems like a gratuitous use for accomplishing the goal of deciding
whether or not there's temporary output to commit. My workaround has been for
the writers to tell me (in an admittedly clunky way) that they produced output.
I can trust them more than I can trust the FileSystem.
> Actual data loss using s3n (against US Standard region)
> -------------------------------------------------------
>
> Key: HADOOP-9577
> URL: https://issues.apache.org/jira/browse/HADOOP-9577
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1.0.3
> Reporter: Joshua Caplan
> Priority: Critical
>
> The implementation of needsTaskCommit() assumes that the FileSystem used for
> writing temporary outputs is consistent. That happens not to be the case
> when using the S3 native filesystem in the US Standard region. It is
> actually quite common in larger jobs for the exists() call to return false
> even if the task attempt wrote output minutes earlier, which essentially
> cancels the commit operation with no error. That's real life data loss right
> there, folks.
> The saddest part is that the Hadoop APIs do not seem to provide any
> legitimate means for the various RecordWriters to communicate with the
> OutputCommitter. In my projects I have created a static map of semaphores
> keyed by TaskAttemptID, which all my custom RecordWriters have to be aware
> of. That's pretty lame.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
