Christoffer Wadum Larsen created HADOOP-20036:
-------------------------------------------------
Summary: S3A: getFileStatus reports an existing directory as
missing when ListObjectsV2 returns an empty truncated page
Key: HADOOP-20036
URL: https://issues.apache.org/jira/browse/HADOOP-20036
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.4.3, 3.5.0, 3.3.6
Reporter: Christoffer Wadum Larsen
h3. Problem
S3 ListObjectsV2 may return a page with no keys, {{IsTruncated=true}} and a
{{{}NextContinuationToken{}}}. The [API
docs|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html]
allow this: "NextContinuationToken is sent when isTruncated is true, which
means there are more keys in the bucket that can be listed" and "The response
might contain fewer keys but will never contain more".
{{S3AFileSystem.s3GetFileStatus()}} makes a single {{listObjects()}} call,
ignores {{{}IsTruncated{}}}, and throws {{FileNotFoundException}} for a
non-empty directory when that page is empty.
h3. Impact
Observed with {{{}FileOutputCommitter{}}}: task output silently skipped.
||impact||caller||today||
|silently
wrong|[{{create()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2175]|writes
a file over the directory|
|silently wrong|[{{rename()}}
dst|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2482]|moves
into {{{}dst{}}}, not {{dst/<name>}}|
|silently
wrong|[{{delete()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3590]|returns
false, nothing deleted|
|silently
wrong|[{{exists()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4892]
/
[{{getFileStatus()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3953]|absent,
so {{FileOutputCommitter}} skips task output|
|visible /
low|[{{access()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3908],
[{{rename()}}
src|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2443]|{{FileNotFoundException}}|
|visible /
low|[{{isDirectory()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4913]|false|
|visible /
low|[{{mkdirs()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MkdirOperation.java#L135]|redundant
marker PUT|
|mitigated|[{{listStatus()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3697],
[{{listFiles()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L5170]|{{Listing}}
follows truncation; only the empty-dir fallback is exposed|
|unaffected|[{{isFile()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4932]|HEAD
only|
h3. Affected versions
The probe is unchanged from 3.3.x through 3.5.0, on both AWS SDK v1 and v2.
h3. Proposed fix
While the page is empty and truncated, {{s3GetFileStatus()}} continues with
{{{}continueListObjects(){}}}, as {{Listing}} does. It continues only on an
explicit {{{}IsTruncated=true{}}}; on a missing token/V1 marker it stops with a
one-time WARN and keeps today's result. No API or config change, and no extra
request on the normal path. PR to follow.
h3. Possible follow-ups
* {{S3ListResult.isTruncated()}} unboxes the flag, so {{Listing}} NPEs if a
store omits {{IsTruncated}} (pre-existing). The probe reads the flag
null-safely instead, since a null-safe {{isTruncated()}} would let {{Listing}}
silently stop paging.
* Proposal: make {{isTruncated()}} raise a translated {{IOException}} on a
missing flag, and use that one method in both {{Listing}} and the probe.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
