cshuo commented on code in PR #19202:
URL: https://github.com/apache/hudi/pull/19202#discussion_r3585515984
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/cdc/CdcIterators.java:
##########
@@ -135,14 +153,25 @@ public boolean hasNext() {
}
@Override
- public RowData next() {
+ public synchronized RowData next() {
Review Comment:
I see some points here:
1. **The task-thread drain path cannot guarantee complete cleanup on forced
cancellation.**
Under normal execution, `SourceReaderBase` drains a `BatchRecords` and
invokes `recycle()`. During forced cancellation, however, the mailbox may stop
before the active batch is fully consumed. A batch already held as
`SourceReaderBase.currentFetch` is no longer in the fetcher queue and may never
be recycled.
2. **Closing only `HoodieCdcSplitReaderFunction.currentIterator` is not a
complete cleanup mechanism either.**
`currentIterator` represents only the most recently created iterator.
Previously returned iterators may still be referenced by undrained
`BatchRecords`, so closing `currentIterator` does not guarantee that all
outstanding reader resources are released.
The complete solution is to remove lazy reading from `BatchRecords`.
`BatchRecords` should contain an actually materialized, bounded minibatch of
records rather than a live iterator. The reader function should retain
ownership of the iterator, `CdcImageManager`, file-group readers, and other I/O
resources. It should read each minibatch on the split-fetcher thread and close
those resources on EOF, failure, or cancellation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
