github-actions[bot] commented on code in PR #65318:
URL: https://github.com/apache/doris/pull/65318#discussion_r3535166437
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/S3CompatibleFileSystem.java:
##########
@@ -581,7 +580,12 @@ private void fetchNextPage() throws IOException {
buffer.add(new FileEntry(loc, obj.getSize(), false,
obj.getModificationTime(), List.of()));
}
if (page.isTruncated()) {
- continuationToken = page.getContinuationToken();
+ String next = page.getContinuationToken();
Review Comment:
This guard fixes the iterator path, but the same malformed-page case is
still present in the other pagination loops in this class. For example,
`deleteRecursive()` (used by `InsertIntoTVFCommand.deleteExistingFilesInFE()`
for `delete_existing_files=true`) still does `continuationToken =
batch.isTruncated() ? batch.getContinuationToken() : null`; if a
COS/OBS/OSS-compatible backend returns `isTruncated=true` with a null marker,
Doris stops after page 1 and leaves later objects under the overwrite
directory, and an unchanged marker can keep re-reading the same page.
`renameDirectory()`, `listFiles()`, and `listFilesSingleLevelGlob()` have the
same raw assignment. Please factor the non-null/non-stale token check into a
small helper and use it for all paginated S3-compatible listings, not only
`ObjectStorageFileIterator`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
