[
https://issues.apache.org/jira/browse/HADOOP-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132319#comment-18132319
]
Daniel Carl Jones commented on HADOOP-20036:
--------------------------------------------
I think the proposal here sounds good. Assuming this is an edge case, I think
the slow down on enumerating pages is acceptable in order to get correctness.
This is only bad if there was a directory with a LOT of objects since deleted
and delete markers not yet expired. As Steve mentions though, if this is the
first we're hearing about it its unlikely that this is not an edge case.
I'd be inclined to lean into correctness here and if we do get problems
downstream, then we can consider options that don't follow the full enumeration.
> S3A: getFileStatus reports an existing directory as missing when
> ListObjectsV2 returns an empty truncated page
> --------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-20036
> URL: https://issues.apache.org/jira/browse/HADOOP-20036
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.3.6, 3.5.0, 3.4.3
> Reporter: Christoffer Wadum Larsen
> Priority: Major
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> h3. Problem
> S3 ListObjectsV2 may return a page with no keys, {{IsTruncated=true}} and a
> {{{}NextContinuationToken{}}}. The [API
> docs|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html]
> allow this: "NextContinuationToken is sent when isTruncated is true, which
> means there are more keys in the bucket that can be listed" and "The response
> might contain fewer keys but will never contain more".
> {{S3AFileSystem.s3GetFileStatus()}} makes a single {{listObjects()}} call,
> ignores {{{}IsTruncated{}}}, and throws {{FileNotFoundException}} for a
> non-empty directory when that page is empty.
> h3. Impact
> Observed with {{{}FileOutputCommitter{}}}: task output silently skipped.
> ||impact||caller||today||
> |silently
> wrong|[{{create()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2175]|writes
> a file over the directory|
> |silently wrong|[{{rename()}}
> dst|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2482]|moves
> into {{{}dst{}}}, not {{dst/<name>}}|
> |silently
> wrong|[{{delete()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3590]|returns
> false, nothing deleted|
> |silently
> wrong|[{{exists()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4892]
> /
> [{{getFileStatus()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3953]|absent,
> so {{FileOutputCommitter}} skips task output|
> |visible /
> low|[{{access()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3908],
> [{{rename()}}
> src|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2443]|{{FileNotFoundException}}|
> |visible /
> low|[{{isDirectory()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4913]|false|
> |visible /
> low|[{{mkdirs()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MkdirOperation.java#L135]|redundant
> marker PUT|
> |mitigated|[{{listStatus()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3697],
>
> [{{listFiles()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L5170]|{{Listing}}
> follows truncation; only the empty-dir fallback is exposed|
> |unaffected|[{{isFile()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4932]|HEAD
> only|
> h3. Affected versions
> The probe is unchanged from 3.3.x through 3.5.0, on both AWS SDK v1 and v2.
> h3. Proposed fix
> While the page is empty and truncated, {{s3GetFileStatus()}} continues with
> {{{}continueListObjects(){}}}, as {{Listing}} does. It continues only on an
> explicit {{{}IsTruncated=true{}}}; on a missing token/V1 marker it stops with
> a one-time WARN and keeps today's result. No API or config change, and no
> extra request on the normal path. PR to follow.
> h3. Possible follow-ups
> * {{S3ListResult.isTruncated()}} unboxes the flag, so {{Listing}} NPEs if a
> store omits {{IsTruncated}} (pre-existing). The probe reads the flag
> null-safely instead, since a null-safe {{isTruncated()}} would let
> {{Listing}} silently stop paging.
> * Proposal: make {{isTruncated()}} raise a translated {{IOException}} on a
> missing flag, and use that one method in both {{Listing}} and the probe.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
