[
https://issues.apache.org/jira/browse/HADOOP-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132646#comment-18132646
]
Christoffer Wadum Larsen commented on HADOOP-20036:
---------------------------------------------------
I am aware of the magic committer as well, but the non-staging magic committer
suffers this same issue - the failing call happens during the magic committer
as well, just against the __magic attempt directory (needsTaskCommit call).
We will have the bytes as an incomplete multipart upload but the pending file
can be missed and thus won't end up in the pending set and will never see its
CompleteMultipartUpload call.
> S3A: getFileStatus reports an existing directory as missing when
> ListObjectsV2 returns an empty truncated page
> --------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-20036
> URL: https://issues.apache.org/jira/browse/HADOOP-20036
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.3.6, 3.5.0, 3.4.3
> Reporter: Christoffer Wadum Larsen
> Priority: Major
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> h3. Problem
> S3 ListObjectsV2 may return a page with no keys, {{IsTruncated=true}} and a
> {{{}NextContinuationToken{}}}. The [API
> docs|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html]
> allow this: "NextContinuationToken is sent when isTruncated is true, which
> means there are more keys in the bucket that can be listed" and "The response
> might contain fewer keys but will never contain more".
> {{S3AFileSystem.s3GetFileStatus()}} makes a single {{listObjects()}} call,
> ignores {{{}IsTruncated{}}}, and throws {{FileNotFoundException}} for a
> non-empty directory when that page is empty.
> h3. Impact
> Observed with {{{}FileOutputCommitter{}}}: task output silently skipped.
> ||impact||caller||today||
> |silently
> wrong|[{{create()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2175]|writes
> a file over the directory|
> |silently wrong|[{{rename()}}
> dst|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2482]|moves
> into {{{}dst{}}}, not {{dst/<name>}}|
> |silently
> wrong|[{{delete()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3590]|returns
> false, nothing deleted|
> |silently
> wrong|[{{exists()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4892]
> /
> [{{getFileStatus()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3953]|absent,
> so {{FileOutputCommitter}} skips task output|
> |visible /
> low|[{{access()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3908],
> [{{rename()}}
> src|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2443]|{{FileNotFoundException}}|
> |visible /
> low|[{{isDirectory()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4913]|false|
> |visible /
> low|[{{mkdirs()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MkdirOperation.java#L135]|redundant
> marker PUT|
> |mitigated|[{{listStatus()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3697],
>
> [{{listFiles()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L5170]|{{Listing}}
> follows truncation; only the empty-dir fallback is exposed|
> |unaffected|[{{isFile()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4932]|HEAD
> only|
> h3. Affected versions
> The probe is unchanged from 3.3.x through 3.5.0, on both AWS SDK v1 and v2.
> h3. Proposed fix
> While the page is empty and truncated, {{s3GetFileStatus()}} continues with
> {{{}continueListObjects(){}}}, as {{Listing}} does. It continues only on an
> explicit {{{}IsTruncated=true{}}}; on a missing token/V1 marker it stops with
> a one-time WARN and keeps today's result. No API or config change, and no
> extra request on the normal path. PR to follow.
> h3. Possible follow-ups
> * {{S3ListResult.isTruncated()}} unboxes the flag, so {{Listing}} NPEs if a
> store omits {{IsTruncated}} (pre-existing). The probe reads the flag
> null-safely instead, since a null-safe {{isTruncated()}} would let
> {{Listing}} silently stop paging.
> * Proposal: make {{isTruncated()}} raise a translated {{IOException}} on a
> missing flag, and use that one method in both {{Listing}} and the probe.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
