danny0405 commented on code in PR #20564:
URL: https://github.com/apache/hudi/pull/20564#discussion_r4520556348
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/reader/HoodieSourceSplitReader.java:
##########
@@ -181,6 +186,7 @@ public void close() throws Exception {
private RecordsWithSplitIds<HoodieRecordWithPosition<T>> finishSplit() {
RecordsWithSplitIds<HoodieRecordWithPosition<T>> records =
BatchRecords.lastBatchRecords(currentSplit.splitId());
currentSplit = null;
+ readerMetrics.clearCurrentSplit();
Review Comment:
[P2] Could we keep the delay active until the source reader has consumed the
split's buffered records? This runs on the fetcher thread, before Flink's
FetchTask enqueues the finished-split marker. Earlier batches can still be
queued or blocked in downstream emission, so clearing here makes
splitLatestCommitDelay report 0 while the source is still backpressured,
potentially indefinitely. Completion should be tracked at the
source-reader/emission boundary, taking care that completion of an older split
cannot clear the metric for a newer active split. A test that blocks emission
after fetching the final batch would cover this case.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/reader/HoodieSourceSplitReader.java:
##########
@@ -109,6 +113,7 @@ public RecordsWithSplitIds<HoodieRecordWithPosition<T>>
fetch() throws IOExcepti
}
currentSplit = nextSplit;
readerFunction.open(currentSplit);
+ readerMetrics.setSplitLatestCommit(currentSplit.getLatestCommit());
Review Comment:
[P2] Could we set the split metric before calling
readerFunction.open(currentSplit)? AbstractSplitReaderFunction.open() creates
the underlying iterator and synchronously replays/skips the consumed records
when restoring a split. Slow initialization or a large recovery replay
therefore leaves readingSplit false and splitLatestCommitDelay at 0 for the
whole operation, despite the reader actively working on an old split. Setting
it before open() would make those stalls visible too; a test with a blocked
open() would verify the gauge keeps growing during initialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]