github-actions[bot] commented on code in PR #68007:
URL: https://github.com/apache/doris/pull/68007#discussion_r4225995591
##########
fe/fe-core/src/main/java/org/apache/doris/job/offset/s3/S3SourceOffsetProvider.java:
##########
@@ -176,6 +191,9 @@ public void fetchRemoteMeta(Map<String, String> properties)
throws Exception {
@Override
public boolean hasMoreDataToConsume() {
+ if (hasReachedEnd()) {
+ return false;
Review Comment:
[P2] Handle a deleted successor before retrying an ONCE task. With
`s3.max_batch_files=1`, listing A,B marks A nonterminal and caches B as
`maxEndFile`. If A commits and B is removed before the next task lists it, this
readiness check still returns true, but `getNextOffset()` finds no files and
fails four times; the job pauses, and PENDING auto-resume never refreshes
metadata, so it can exhaust its resume budget instead of finishing. This is
distinct from the recovered-tail case, which reaches an empty metadata probe;
the live task fails before one. Treat a successful empty ONCE listing after a
committed cursor as exhaustion (or refresh before dispatch), and test commit A
-> remove B -> next task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
