[
https://issues.apache.org/jira/browse/HADOOP-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132480#comment-18132480
]
Christoffer Wadum Larsen commented on HADOOP-20036:
---------------------------------------------------
>From AWS support team themselves, they came back with these pieces of
>information:
{quote}=== Summary ===
* The 13 files were not deleted. Each of the 13 staging prefixes still holds
one current version of its part file, and there is no delete marker under any
of them (point 1).
* For each of the 13 prefixes, S3 returned a ListObjectsV2 response with HTTP
200, zero keys and a NextContinuationToken, completing between 12:43:00.647 and
12:43:01.195 UTC. Each of these requests started after the file under that
prefix had been created (point 2).
* In the Apache Hadoop 3.3.4 source, the existence checks on the task attempt
folder that decide whether its output is committed run before the code that
branches on the algorithm version, so v1 and v2 behave the same way here (point
3).
* have raised these 13 LIST requests with the S3 service team and will update
this case with their findings.
The reason files were not moved to output location is the commit check. Before
Hadoop moves a task's file from _temporary to the output location, the
committer checks whether the task attempt folder exists. It does this with a
single ListObjectsV2 call on that folder. For these 13 tasks, that call came
back with HTTP 200, zero keys and a continuation token, at about
12:43:00-12:43:01 UTC, even though each file had already been written 217 ms to
2,769ms earlier.
{quote}
> S3A: getFileStatus reports an existing directory as missing when
> ListObjectsV2 returns an empty truncated page
> --------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-20036
> URL: https://issues.apache.org/jira/browse/HADOOP-20036
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.3.6, 3.5.0, 3.4.3
> Reporter: Christoffer Wadum Larsen
> Priority: Major
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> h3. Problem
> S3 ListObjectsV2 may return a page with no keys, {{IsTruncated=true}} and a
> {{{}NextContinuationToken{}}}. The [API
> docs|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html]
> allow this: "NextContinuationToken is sent when isTruncated is true, which
> means there are more keys in the bucket that can be listed" and "The response
> might contain fewer keys but will never contain more".
> {{S3AFileSystem.s3GetFileStatus()}} makes a single {{listObjects()}} call,
> ignores {{{}IsTruncated{}}}, and throws {{FileNotFoundException}} for a
> non-empty directory when that page is empty.
> h3. Impact
> Observed with {{{}FileOutputCommitter{}}}: task output silently skipped.
> ||impact||caller||today||
> |silently
> wrong|[{{create()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2175]|writes
> a file over the directory|
> |silently wrong|[{{rename()}}
> dst|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2482]|moves
> into {{{}dst{}}}, not {{dst/<name>}}|
> |silently
> wrong|[{{delete()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3590]|returns
> false, nothing deleted|
> |silently
> wrong|[{{exists()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4892]
> /
> [{{getFileStatus()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3953]|absent,
> so {{FileOutputCommitter}} skips task output|
> |visible /
> low|[{{access()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3908],
> [{{rename()}}
> src|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2443]|{{FileNotFoundException}}|
> |visible /
> low|[{{isDirectory()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4913]|false|
> |visible /
> low|[{{mkdirs()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MkdirOperation.java#L135]|redundant
> marker PUT|
> |mitigated|[{{listStatus()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3697],
>
> [{{listFiles()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L5170]|{{Listing}}
> follows truncation; only the empty-dir fallback is exposed|
> |unaffected|[{{isFile()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4932]|HEAD
> only|
> h3. Affected versions
> The probe is unchanged from 3.3.x through 3.5.0, on both AWS SDK v1 and v2.
> h3. Proposed fix
> While the page is empty and truncated, {{s3GetFileStatus()}} continues with
> {{{}continueListObjects(){}}}, as {{Listing}} does. It continues only on an
> explicit {{{}IsTruncated=true{}}}; on a missing token/V1 marker it stops with
> a one-time WARN and keeps today's result. No API or config change, and no
> extra request on the normal path. PR to follow.
> h3. Possible follow-ups
> * {{S3ListResult.isTruncated()}} unboxes the flag, so {{Listing}} NPEs if a
> store omits {{IsTruncated}} (pre-existing). The probe reads the flag
> null-safely instead, since a null-safe {{isTruncated()}} would let
> {{Listing}} silently stop paging.
> * Proposal: make {{isTruncated()}} raise a translated {{IOException}} on a
> missing flag, and use that one method in both {{Listing}} and the probe.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
