hudi-agent commented on code in PR #19658:
URL: https://github.com/apache/hudi/pull/19658#discussion_r3823442244
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java:
##########
@@ -471,15 +471,15 @@ private Option<String>
getOffsetsByTimestamp(KafkaConsumer consumer, List<Partit
.map(x -> new
TopicPartition(x.topic(), x.partition()))
.collect(Collectors.toMap(Function.identity(), x -> timestamp));
- Map<TopicPartition, Long> earliestOffsets =
consumer.beginningOffsets(topicPartitions);
+ Map<TopicPartition, Long> endOffsets =
consumer.endOffsets(topicPartitions);
Review Comment:
🤖 One thing worth weighing on the switch approach: routing the fallback
through `auto.offset.reset` means `earliest` users land on `beginningOffsets`,
which is exactly the replay-from-start behavior this PR is trying to eliminate.
Since a timestamp checkpoint explicitly asks to start at records with ts >= T,
resuming at the tip when none qualify seems semantically correct regardless of
`auto.offset.reset` — so I'd lean toward the unconditional end-offset behavior
being the intended contract, just called out explicitly in the description/docs
rather than made configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
