peterxcli commented on code in PR #5493:
URL: https://github.com/apache/datafusion-comet/pull/5493#discussion_r3869570679
##########
spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometBypassMergeSortShuffleWriter.java:
##########
@@ -176,6 +177,10 @@ public void write(Iterator<Product2<K, V>> records) throws
IOException {
CometShuffleExternalSorter.MAXIMUM_PAGE_SIZE_BYTES,
memoryManager.pageSizeBytes()));
+ // This task's disk writers. Under memory pressure a writer spills its
sibling writers in
+ // this list, never those of other tasks.
+ final LinkedList<CometDiskBlockWriter> taskWriters = new LinkedList<>();
Review Comment:
Good catch, thanks — fixed in 3cee056cd. The spill registries stay per-task,
and allocation progress on the shared on-heap pool is preserved by making the
post-spill retry (`SpillWriter.initialCurrentPage`) call a new
`allocateBlocking` on `CometBoundedShuffleMemoryAllocator`, which waits (up to
60s) for other tasks to free pool memory; `free()` notifies waiters. That retry
point is only reached after the requesting task has already spilled everything
it owns, so a waiting task holds no pool memory itself and the tasks still
holding it can always progress and free it — no cross-task waiting cycle is
possible. The off-heap allocator keeps the non-waiting default since Spark's
unified memory manager already arbitrates between tasks there. Added an on-heap
regression test mirroring your reproducer (1 MiB pool, 256 KiB pages, task B
idle with 4 pages, task A's first row): it fails with the same
`UNABLE_TO_ACQUIRE_MEMORY` error without the change and passes with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
