This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git
View the commit online: https://github.com/apache/spark/commit/12ec3387af1ab5d0e5687f9b8277b3c96b7cc11f The following commit(s) were added to refs/heads/branch-2.4 by this push: new 12ec338 [SPARK-27558][CORE] Gracefully cleanup task when it fails with OOM exception 12ec338 is described below commit 12ec3387af1ab5d0e5687f9b8277b3c96b7cc11f Author: yudovin <artsiom.yudo...@profitero.com> AuthorDate: Wed Nov 20 18:18:29 2019 -0600 [SPARK-27558][CORE] Gracefully cleanup task when it fails with OOM exception ### What changes were proposed in this pull request? When a task fails with OOM exception, the UnsafeInMemorySorter.array could be null. In the meanwhile, the cleanupResources() on task completion would call UnsafeInMemorySorter.getMemoryUsage in turn, and that lead to another NPE thrown. ### Why are the changes needed? Check if array is null in UnsafeInMemorySorter.getMemoryUsage and it should help to avoid NPE. ### Does this PR introduce any user-facing change? No ### How was this patch tested? It was tested manually. Closes #26606 from ayudovin/fix-npe-in-listener-2.4. Authored-by: yudovin <artsiom.yudo...@profitero.com> Signed-off-by: Sean Owen <sean.o...@databricks.com> --- .../spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java index 75690ae..aedc7ec 100644 --- a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java +++ b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java @@ -205,6 +205,10 @@ public final class UnsafeInMemorySorter { } public long getMemoryUsage() { + if (array == null) { + return 0L; + } + return array.size() * 8; } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org