david-mollitor-db commented on PR #58844:
URL: https://github.com/apache/spark/pull/58844#issuecomment-5707122650
Thanks for the careful review, @dongjoon-hyun -- you're right about the
retained capacity.
Addressed in `69bf503`: `hasNext()` now records each batch's high-water mark
(the buffer is always
drained to empty before it refills, so `currentRows.size()` right after
`processNext()` returns is
the batch high-water mark) and replaces the deque with a fresh one once a
batch exceeds 1024 rows.
The check is once per batch, so the steady-state path keeps the
per-row-allocation-free behavior
and only a large fan-out sheds its array.
```java
private static final int SHRINK_BUFFER_THRESHOLD = 1024;
...
public boolean hasNext() throws IOException {
if (currentRows.isEmpty()) {
if (shrinkBuffer) {
currentRows = new ArrayDeque<>();
}
processNext();
shrinkBuffer = currentRows.size() > SHRINK_BUFFER_THRESHOLD;
}
return !currentRows.isEmpty();
}
```
`WholeStageCodegenSuite` and `GeneratorFunctionSuite` pass. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
