[
https://issues.apache.org/jira/browse/HADOOP-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132644#comment-18132644
]
Steve Loughran edited comment on HADOOP-20036 at 10/26/26, 10:38 PM:
---------------------------------------------------------------------
Christoffer, how were you committing work? file rename committers don't work on
s3. They expect directory rename to be an atomic O(1) operation, and on s3 it
is non-atomic O(data). The O(data) makes it slow, but it is the non-atomicity
which makes it deadly it's impossible to cope with failure during task commit
or partitioned network during task commit.
Use the s3a committers (intermediate manifest) or full manifest-format tables
like iceberg, which give you atomic job commit too, as well as lots of other
features.
update: sorry, read the report more clearly. yes, you were using file output
committer.
please don't do that, it's pathologically bad. Embrace the S3a (magic)
committer and your life will get better.
was (Author: [email protected]):
Christoffer, how were you committing work? file rename committers don't work on
s3. They expect directory rename to be an atomic O(1) operation, and on s3 it
is non-atomic O(data). The O(data) makes it slow, but it is the non-atomicity
which makes it deadly it's impossible to cope with failure during task commit
or partitioned network during task commit.
Use the s3a committers (intermediate manifest) or full manifest-format tables
like iceberg, which give you atomic job commit too, as well as lots of other
features.
> S3A: getFileStatus reports an existing directory as missing when
> ListObjectsV2 returns an empty truncated page
> --------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-20036
> URL: https://issues.apache.org/jira/browse/HADOOP-20036
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.3.6, 3.5.0, 3.4.3
> Reporter: Christoffer Wadum Larsen
> Priority: Major
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> h3. Problem
> S3 ListObjectsV2 may return a page with no keys, {{IsTruncated=true}} and a
> {{{}NextContinuationToken{}}}. The [API
> docs|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html]
> allow this: "NextContinuationToken is sent when isTruncated is true, which
> means there are more keys in the bucket that can be listed" and "The response
> might contain fewer keys but will never contain more".
> {{S3AFileSystem.s3GetFileStatus()}} makes a single {{listObjects()}} call,
> ignores {{{}IsTruncated{}}}, and throws {{FileNotFoundException}} for a
> non-empty directory when that page is empty.
> h3. Impact
> Observed with {{{}FileOutputCommitter{}}}: task output silently skipped.
> ||impact||caller||today||
> |silently
> wrong|[{{create()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2175]|writes
> a file over the directory|
> |silently wrong|[{{rename()}}
> dst|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2482]|moves
> into {{{}dst{}}}, not {{dst/<name>}}|
> |silently
> wrong|[{{delete()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3590]|returns
> false, nothing deleted|
> |silently
> wrong|[{{exists()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4892]
> /
> [{{getFileStatus()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3953]|absent,
> so {{FileOutputCommitter}} skips task output|
> |visible /
> low|[{{access()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3908],
> [{{rename()}}
> src|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2443]|{{FileNotFoundException}}|
> |visible /
> low|[{{isDirectory()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4913]|false|
> |visible /
> low|[{{mkdirs()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MkdirOperation.java#L135]|redundant
> marker PUT|
> |mitigated|[{{listStatus()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3697],
>
> [{{listFiles()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L5170]|{{Listing}}
> follows truncation; only the empty-dir fallback is exposed|
> |unaffected|[{{isFile()}}|https://github.com/apache/hadoop/blob/2c01c36fe650a843e1c00c000570f51cfd118c9d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4932]|HEAD
> only|
> h3. Affected versions
> The probe is unchanged from 3.3.x through 3.5.0, on both AWS SDK v1 and v2.
> h3. Proposed fix
> While the page is empty and truncated, {{s3GetFileStatus()}} continues with
> {{{}continueListObjects(){}}}, as {{Listing}} does. It continues only on an
> explicit {{{}IsTruncated=true{}}}; on a missing token/V1 marker it stops with
> a one-time WARN and keeps today's result. No API or config change, and no
> extra request on the normal path. PR to follow.
> h3. Possible follow-ups
> * {{S3ListResult.isTruncated()}} unboxes the flag, so {{Listing}} NPEs if a
> store omits {{IsTruncated}} (pre-existing). The probe reads the flag
> null-safely instead, since a null-safe {{isTruncated()}} would let
> {{Listing}} silently stop paging.
> * Proposal: make {{isTruncated()}} raise a translated {{IOException}} on a
> missing flag, and use that one method in both {{Listing}} and the probe.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
