sunchao commented on code in PR #58763:
URL: https://github.com/apache/spark/pull/58763#discussion_r4010940844
##########
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala:
##########
@@ -211,9 +211,22 @@ private[spark] class MemoryStore(
// Keep track of unroll memory used by this particular block /
putIterator() operation
var unrollMemoryUsedByThisBlock = 0L
+ def reserveUnrollMemory(memory: Long): Boolean = {
+ try {
+ reserveUnrollMemoryForThisTask(blockId, memory, memoryMode)
+ } catch {
+ case error: Throwable =>
+ // No entry or partial iterator can own these values when
reclamation throws.
+ // A normal denial must retain them for the returned partial
iterator.
+ Utils.tryWithSafeFinally { throw error } {
+ releaseUnrollMemoryForThisTask(memoryMode,
unrollMemoryUsedByThisBlock)
+ freeUnrolledValues(valuesHolder)
Review Comment:
Updated in `79b3464d7889fcc66282e47804706178f6a2ec42`.
Fixed both unroll and transfer failure cleanup. Deserialized values remain
caller-owned until the put succeeds; only this operation's unroll credits and
Spark-created serialized buffers are released on failure. The regression
injects a real eviction IOException with no reclaimer and checks that the
consumed AutoCloseable remains open, the old block remains present, and unroll
accounting is restored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
