Gargi-jais11 opened a new pull request, #11715: URL: https://github.com/apache/ozone/pull/11715
## What changes were proposed in this pull request? ListObjectsV2 (and OM listKeys) returned no results when **StartAfter** was lexicographically before Prefix, if the bucket also contained keys between them (e.g. dir1/dir2/file1 with prefix=dir1bh and start-after=dir1/dir2/file1). OM used StartAfter as the RocksDB seek key even when it was below Prefix, so the iterator never reached keys under the prefix. When **startKey < keyPrefix**, seek at keyPrefix instead (OBS/LEGACY in OmMetadataManagerImpl; FSO shallow listing in OzoneBucket). Behavior now matches Amazon S3. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17308 ## How was this patch tested? Added unit and integration tests. Tested manually. **Before Results:** ``` B=startafter-obs $E create-bucket --bucket $B for k in dir1/dir2/file1 dir1bh/file1 dir1bha/file1; do $E put-object --bucket $B --key $k --body /tmp/f > /dev/null done { "Location": "http://s3g:9878/startafter-obs"; } // 1. Baseline: expect CommonPrefixes dir1bh/ and dir1bha/ bash-5.1$ $E list-objects-v2 --bucket $B --prefix dir1bh --delimiter / { "CommonPrefixes": [ { "Prefix": "dir1bh/" }, { "Prefix": "dir1bha/" } ], "RequestCharged": null, "Prefix": "dir1bh" } # 2. Bug: expected the same two prefixes, actual is only {"Prefix": "dir1bh"} bash-5.1$ $E list-objects-v2 --bucket $B --prefix dir1bh --delimiter / --start-after dir1/dir2/file1 { "RequestCharged": null, "Prefix": "dir1bh" } # 3. Without delimiter: expected both keys, actual is empty bash-5.1$ $E list-objects-v2 --bucket $B --prefix dir1bh --start-after dir1/dir2/file1 { "RequestCharged": null, "Prefix": "dir1bh" } # 4. Same OM call through the Ozone shell: expected both keys, actual is empty bash-5.1$ ozone sh key list --prefix dir1bh --start dir1/dir2/file1 /s3v/$B [ ] // Control case: in a bucket without dir1/dir2/file1, the same StartAfter request returns both prefixes correctly. This shows the key between StartAfter and the prefix is what triggers the OBS bug. bash-5.1$ B2=startafter-obs-nogap $E create-bucket --bucket $B2 for k in dir1bh/file1 dir1bha/file1; do $E put-object --bucket $B2 --key $k --body /tmp/f > /dev/null; done $E list-objects-v2 --bucket $B2 --prefix dir1bh --delimiter / --start-after dir1/dir2/file1 { "Location": "http://s3g:9878/startafter-obs-nogap"; } { "CommonPrefixes": [ { "Prefix": "dir1bh/" }, { "Prefix": "dir1bha/" } ], "RequestCharged": null, "Prefix": "dir1bh" } // same for FSO bash-5.1$ ozone sh bucket create --layout FILE_SYSTEM_OPTIMIZED /s3v/startafter-fso B=startafter-fso for k in dir1bh/file1 dir1bha/file1; do $E put-object --bucket $B --key $k --body /tmp/f > /dev/null; done $E list-objects-v2 --bucket $B --prefix dir1bh --delimiter / --start-after dir1/dir2/file1 { "RequestCharged": null, "Prefix": "dir1bh" } ``` **After Fix:** ``` bash-5.1$ E="aws s3api --endpoint-url http://s3g:9878/"; echo x > /tmp/f B=startafter-obs $E create-bucket --bucket $B for k in dir1/dir2/file1 dir1bh/file1 dir1bha/file1; do $E put-object --bucket $B --key $k --body /tmp/f > /dev/null done { "Location": "http://s3g:9878/startafter-obs"; } bash-5.1$ $E list-objects-v2 --bucket $B --prefix dir1bh --delimiter / { "CommonPrefixes": [ { "Prefix": "dir1bh/" }, { "Prefix": "dir1bha/" } ], "RequestCharged": null, "Prefix": "dir1bh" } bash-5.1$ $E list-objects-v2 --bucket $B --prefix dir1bh --delimiter / --start-after dir1/dir2/file1 { "CommonPrefixes": [ { "Prefix": "dir1bh/" }, { "Prefix": "dir1bha/" } ], "RequestCharged": null, "Prefix": "dir1bh" } bash-5.1$ $E list-objects-v2 --bucket $B --prefix dir1bh --start-after dir1/dir2/file1 { "Contents": [ { "Key": "dir1bh/file1", "LastModified": "2026-10-23T10:14:44.625Z", "ETag": "\"401b30e3b8b5d629635a5c613cdb7919\"", "Size": 2, "StorageClass": "STANDARD" }, { "Key": "dir1bha/file1", "LastModified": "2026-10-23T10:14:44.840Z", "ETag": "\"401b30e3b8b5d629635a5c613cdb7919\"", "Size": 2, "StorageClass": "STANDARD" } ], "RequestCharged": null, "Prefix": "dir1bh" } bash-5.1$ ozone sh key list --prefix dir1bh --start dir1/dir2/file1 /s3v/$B [ { "volumeName" : "s3v", "bucketName" : "startafter-obs", "name" : "dir1bh/file1", "owner" : "testuser", "dataSize" : 2, "creationTime" : "2026-10-23T10:14:44.598Z", "modificationTime" : "2026-10-23T10:14:44.625Z", "replicationConfig" : { "replicationFactor" : "THREE", "requiredNodes" : 3, "minimumNodes" : 1, "replicationType" : "RATIS" }, "metadata" : { "ETag" : "401b30e3b8b5d629635a5c613cdb7919" }, "tags" : { }, "file" : true }, { "volumeName" : "s3v", "bucketName" : "startafter-obs", "name" : "dir1bha/file1", "owner" : "testuser", "dataSize" : 2, "creationTime" : "2026-10-23T10:14:44.820Z", "modificationTime" : "2026-10-23T10:14:44.840Z", "replicationConfig" : { "replicationFactor" : "THREE", "requiredNodes" : 3, "minimumNodes" : 1, "replicationType" : "RATIS" }, "metadata" : { "ETag" : "401b30e3b8b5d629635a5c613cdb7919" }, "tags" : { }, "file" : true } ] bash-5.1$ B2=startafter-obs-nogap $E create-bucket --bucket $B2 for k in dir1bh/file1 dir1bha/file1; do $E put-object --bucket $B2 --key $k --body /tmp/f > /dev/null; done $E list-objects-v2 --bucket $B2 --prefix dir1bh --delimiter / --start-after dir1/dir2/file1 { "Location": "http://s3g:9878/startafter-obs-nogap"; } { "CommonPrefixes": [ { "Prefix": "dir1bh/" }, { "Prefix": "dir1bha/" } ], "RequestCharged": null, "Prefix": "dir1bh" } bash-5.1$ ozone sh bucket create --layout FILE_SYSTEM_OPTIMIZED /s3v/startafter-fso B=startafter-fso for k in dir1bh/file1 dir1bha/file1; do $E put-object --bucket $B --key $k --body /tmp/f > /dev/null; done $E list-objects-v2 --bucket $B --prefix dir1bh --delimiter / --start-after dir1/dir2/file1 { "CommonPrefixes": [ { "Prefix": "dir1bh/" }, { "Prefix": "dir1bha/" } ], "RequestCharged": null, "Prefix": "dir1bh" } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
