peterxcli commented on code in PR #5493:
URL: https://github.com/apache/datafusion-comet/pull/5493#discussion_r3870225127
##########
spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometBypassMergeSortShuffleWriter.java:
##########
@@ -176,6 +177,10 @@ public void write(Iterator<Product2<K, V>> records) throws
IOException {
CometShuffleExternalSorter.MAXIMUM_PAGE_SIZE_BYTES,
memoryManager.pageSizeBytes()));
+ // This task's disk writers. Under memory pressure a writer spills its
sibling writers in
+ // this list, never those of other tasks.
+ final LinkedList<CometDiskBlockWriter> taskWriters = new LinkedList<>();
Review Comment:
Thanks for re-testing — addressed in 4c12797e0. You're right that any finite
deadline fails some workload, so I removed it: `allocateBlocking` now waits
until memory is freed, mirroring how Spark's unified memory manager blocks a
task until memory becomes available. Liveness still holds for the same reason
as before (a waiter has already spilled everything it owns, so pool memory is
only held by tasks that can progress and free it), interruption (task kill)
still aborts the wait, and the allocator logs a warning when a task starts
waiting so long waits are diagnosable. I validated your exact scenario locally:
1-MiB pool, 256-KiB pages, task B holding four pages (751 rows) pausing 65 s —
task A's first insert waited ~65.1 s, then both tasks completed with all 752
rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
