[
https://issues.apache.org/jira/browse/HADOOP-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132644#comment-18132644
]
Steve Loughran commented on HADOOP-20036:
-----------------------------------------
Christoffer, how were you committing work? file rename committers don't work on
s3. They expect directory rename to be an atomic O(1) operation, and on s3 it
is non-atomic O(data). The O(data) makes it slow, but it is the non-atomicity
which makes it deadly it's impossible to cope with failure during task commit
or partitioned network during task commit.
Use the s3a committers (intermediate manifest) or full manifest-format tables
like iceberg, which give you atomic job commit too, as well as lots of other
features.
> S3A: getFileStatus reports an existing directory as missing when
> ListObjectsV2 returns an empty truncated page
> --------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-20036
> URL: https://issues.apache.org/jira/browse/HADOOP-20036
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.3.6, 3.5.0, 3.4.3
> Reporter: Christoffer Wadum Larsen
> Priority: Major
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> h3. Problem
> S3 ListObjectsV2 may return a page with no keys, {{IsTruncated=true}} and a
> {{{}NextContinuationToken{}}}. The [API
> docs|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html]
> allow this: "NextContinuationToken is sent when isTruncated is true, which
> means there are more keys in the bucket that can be listed" and "The response
> might contain fewer keys but will never contain more".
> {{S3AFileSystem.s3GetFileStatus()}} makes a single {{listObjects()}} call,
> ignores {{{}IsTruncated{}}}, and throws {{FileNotFoundException}} for a
> non-empty directory when that page is empty.
> h3. Impact
> Observed with {{{}FileOutputCommitter{}}}: task output silently skipped.
> ||impact||caller||today||
> |silently
> wrong|[{{create()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2175]|writes
> a file over the directory|
> |silently wrong|[{{rename()}}
> dst|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2482]|moves
> into {{{}dst{}}}, not {{dst/<name>}}|
> |silently
> wrong|[{{delete()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3590]|returns
> false, nothing deleted|
> |silently
> wrong|[{{exists()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4892]
> /
> [{{getFileStatus()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3953]|absent,
> so {{FileOutputCommitter}} skips task output|
> |visible /
> low|[{{access()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3908],
> [{{rename()}}
> src|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2443]|{{FileNotFoundException}}|
> |visible /
> low|[{{isDirectory()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4913]|false|
> |visible /
> low|[{{mkdirs()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MkdirOperation.java#L135]|redundant
> marker PUT|
> |mitigated|[{{listStatus()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3697],
>
> [{{listFiles()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L5170]|{{Listing}}
> follows truncation; only the empty-dir fallback is exposed|
> |unaffected|[{{isFile()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4932]|HEAD
> only|
> h3. Affected versions
> The probe is unchanged from 3.3.x through 3.5.0, on both AWS SDK v1 and v2.
> h3. Proposed fix
> While the page is empty and truncated, {{s3GetFileStatus()}} continues with
> {{{}continueListObjects(){}}}, as {{Listing}} does. It continues only on an
> explicit {{{}IsTruncated=true{}}}; on a missing token/V1 marker it stops with
> a one-time WARN and keeps today's result. No API or config change, and no
> extra request on the normal path. PR to follow.
> h3. Possible follow-ups
> * {{S3ListResult.isTruncated()}} unboxes the flag, so {{Listing}} NPEs if a
> store omits {{IsTruncated}} (pre-existing). The probe reads the flag
> null-safely instead, since a null-safe {{isTruncated()}} would let
> {{Listing}} silently stop paging.
> * Proposal: make {{isTruncated()}} raise a translated {{IOException}} on a
> missing flag, and use that one method in both {{Listing}} and the probe.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
