JoshRosen commented on a change in pull request #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again URL: https://github.com/apache/spark/pull/25953#discussion_r329325054
########## File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java ########## @@ -787,8 +783,16 @@ private void allocate(int capacity) { assert (capacity >= 0); capacity = Math.max((int) Math.min(MAX_CAPACITY, ByteArrayMethods.nextPowerOf2(capacity)), 64); assert (capacity <= MAX_CAPACITY); - longArray = allocateArray(capacity * 2L); - longArray.zeroOut(); + try { + longArray = allocateArray(capacity * 2L); + longArray.zeroOut(); + } catch (SparkOutOfMemoryError e) { + // When OOM, allocated page was already freed by `TaskMemoryManager`. + // We should not keep it in `longArray`. Otherwise it might be freed again in task + // complete listeners and cause unnecessary error. + longArray = null; Review comment: I realize that I attached this comment to the wrong line in the GitHub diff, so I've re-worded it in an attempt to be clearer. I meant to post this at ```java try { longArray = allocateArray(capacity * 2L); // <-- here ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org