cloud-fan commented on code in PR #58844:
URL: https://github.com/apache/spark/pull/58844#discussion_r4039333305
##########
sql/core/src/main/java/org/apache/spark/sql/execution/BufferedRowIterator.java:
##########
@@ -41,7 +49,11 @@ public abstract class BufferedRowIterator {
public boolean hasNext() throws IOException {
if (currentRows.isEmpty()) {
+ if (shrinkBuffer) {
+ currentRows = new ArrayDeque<>();
+ }
processNext();
+ shrinkBuffer = currentRows.size() > SHRINK_BUFFER_THRESHOLD;
Review Comment:
**Non-blocking (P2):** This branch is the only protection against
reintroducing the retained-capacity regression, but the existing suites assert
only emitted rows. If either the flag assignment or the deque replacement
disappears, they still pass while a wide fan-out can again pin the grown
backing array for the rest of the task. Could we add a focused
BufferedRowIterator test that crosses 1024 rows, drains the batch, triggers the
next refill, and verifies a fresh queue, along with the at-threshold
non-replacement case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
