ravjot28 opened a new issue, #20609:
URL: https://github.com/apache/hudi/issues/20609
### Bug Description
**What happened:**
`LazyConcatenatingIterator` closes an inner iterator before it drops its
reference to it. When that `close()` throws,
or when opening the next inner iterator throws, the closed iterator stays
referenced. The next `hasNext()` or
`close()` calls its `close()` again, and a failed `close()` of the outer
iterator leaves it marked as open.
**What you expected:**
Each inner iterator is closed at most once, and after `close()` the iterator
is closed even if an inner close failed.
**Steps to reproduce:**
1. Concatenate two iterators whose first throws from `close()`.
2. Read the first iterator to its end: `hasNext()` closes it and throws.
3. Call `hasNext()` again: the first iterator's `close()` runs a second time
and throws again; the second iterator
is never reached.
4. Separately: with the first iterator open, call `close()` twice: the first
iterator's `close()` runs twice.
**Root cause (master 9a7728c5862):**
`hudi-common/src/main/java/org/apache/hudi/common/util/collection/LazyConcatenatingIterator.java`:
- `close()` (`:49-58`) calls `itr.close()` (`:52`) before clearing `itr` and
before setting `closed` (`:56`).
- `hasNext()` (`:61-76`) calls `itr.close()` (`:68`) and then opens the next
iterator (`:71`) while `itr` still points
at the closed one.
The class is used by Spark clustering (`MultipleSparkJobExecutionStrategy`,
`SingleSparkJobConsistentHashingExecutionStrategy`), Java clustering
(`JavaExecutionStrategy`) and metadata table
reads (`HoodieBackedTableMetadata`), where the inner iterators hold file
readers.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT at 9a7728c5862), 1.2.1
**Query engine:** any (hudi-common)
**Relevant configs:** none
### Logs and Stack Trace
The second close surfaces as whatever the inner reader throws when closed
twice, after the first failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]