[
https://issues.apache.org/jira/browse/HADOOP-16840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052706#comment-17052706
]
wujinhu commented on HADOOP-16840:
----------------------------------
Hi [~cheersyang] thank you for your time. Let me explain in more details.
When we list files in OSS versioning bucket with many delete markers in it(in
versioning bucket, latest version of deleted file is a delete marker), OSS may
return empty *Contents* and *CommonPrefixes,* for this case, the bucket is
empty or this listing is full of delete markers. So whether continue to list
files depends on *IsTruncated,* if it is true, it means we still need to do the
listing with *NextMarker* OSS returned.
([https://help.aliyun.com/document_detail/31965.html?spm=5176.8465980.0.0.59221450RgSciY])
OSS lists files in order, if many delete makers come first, this case occurs.
So, the code will not run to endless loop(the worst case will be versioning
bucket full of delete markers, and the whole bucket will be listed).
{code:java}
// code placeholder
---
a/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java
+++
b/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java
@@ -273,12 +273,18 @@ public FileStatus getFileStatus(Path path) throws
IOException {
}
if (meta == null) {
ObjectListing listing = store.listObjects(key, 1, null, false);
- if (CollectionUtils.isNotEmpty(listing.getObjectSummaries()) ||
- CollectionUtils.isNotEmpty(listing.getCommonPrefixes())) {
- return new OSSFileStatus(0, true, 1, 0, 0, qualifiedPath, username);
- } else {
- throw new FileNotFoundException(path + ": No such file or directory!");
- }
+ do {
+ if (CollectionUtils.isNotEmpty(listing.getObjectSummaries()) ||
+ CollectionUtils.isNotEmpty(listing.getCommonPrefixes())) {
+ return new OSSFileStatus(0, true, 1, 0, 0, qualifiedPath, username);
+ } else if (listing.isTruncated()) {
+ listing = store.listObjects(key, 1000, listing.getNextMarker(),
+ false);
+ } else {
+ throw new FileNotFoundException(
+ path + ": No such file or directory!");
+ }
+ } while (true);
{code}
> AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket
> --------------------------------------------------------------------------
>
> Key: HADOOP-16840
> URL: https://issues.apache.org/jira/browse/HADOOP-16840
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/oss
> Affects Versions: 2.10.0, 2.9.2, 3.0.3, 3.2.1, 3.1.3
> Reporter: wujinhu
> Assignee: wujinhu
> Priority: Major
> Attachments: HADOOP-16840.001.patch, HADOOP-16840.002.patch
>
>
> When hadoop lists object in versioning bucket with many delete marker in it,
> OSS will return
> {code:java}
> <?xml version="1.0" encoding="UTF-8"?>
> <ListBucketResult>
> <Name>select-us-east-1</Name>
> <Prefix>test/hadoop/file/</Prefix>
> <Marker></Marker>
> <MaxKeys>100</MaxKeys>
> <Delimiter>/</Delimiter>
> <IsTruncated>true</IsTruncated>
> <NextMarker>test/hadoop/file/sub2</NextMarker>
> </ListBucketResult>
> {code}
> It sets *IsTruncated* to true and without *ObjectSummaries* or
> *CommonPrefixes*, and will throw FileNotFoundException
> {code:java}
> // code placeholder
> java.io.FileNotFoundException: oss://select-us-east-1/test/hadoop/file: No
> such file or directory!java.io.FileNotFoundException:
> oss://select-us-east-1/test/hadoop/file: No such file or directory!
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.getFileStatus(AliyunOSSFileSystem.java:281)
> at
> org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract.testGetFileStatusInVersioningBucket{code}
>
> {code:java}
> // code placeholder
> ObjectListing listing = store.listObjects(key, 1, null, false);
> if (CollectionUtils.isNotEmpty(listing.getObjectSummaries()) ||
> CollectionUtils.isNotEmpty(listing.getCommonPrefixes())) {
> return new OSSFileStatus(0, true, 1, 0, 0, qualifiedPath, username);
> } else {
> throw new FileNotFoundException(path + ": No such file or directory!");
> }
> {code}
> In this case, we should call listObjects until *IsTruncated* is false or
> *ObjectSummaries* is not empty or *CommonPrefixes* is not empty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
