david-mollitor-db commented on PR #58844:
URL: https://github.com/apache/spark/pull/58844#issuecomment-5734188593
Thanks, @dongjoon-hyun. Both items are addressed in `17d8bad6c38`:
1. Updated the PR description to match the commit message -- it now
documents the
`SHRINK_BUFFER_THRESHOLD` (1024) buffer reset, and I folded in your note
that a sustained
fan-out above the threshold re-grows the deque once per batch (still not
a regression versus the
per-row `LinkedList` nodes).
2. Added `BufferedRowIteratorSuite` with the focused threshold coverage
@cloud-fan requested: a
batch above the threshold (1025 rows) drains and is replaced with a fresh
deque on the next
refill, and a batch at the threshold (1024 rows) reuses the same deque.
It drives a minimal
`BufferedRowIterator` subclass and asserts on the buffer's identity
across batches.
`BufferedRowIteratorSuite`, `WholeStageCodegenSuite`, and
`GeneratorFunctionSuite` pass. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
