Repository: spark
Updated Branches:
refs/heads/master cbbcd8e42 -> eabc7b8ee
[SPARK-12690][CORE] Fix NPE in UnsafeInMemorySorter.free()
I hit the exception below. The `UnsafeKVExternalSorter` does pass `null` as the
consumer when creating an `UnsafeInMemorySorter`. Normally the NPE doesn't
occur because the `inMemSorter` is set to null later and the `free()` method is
not called. It happens when there is another exception like OOM thrown before
setting `inMemSorter` to null. Anyway, we can add the null check to avoid it.
```
ERROR spark.TaskContextImpl: Error in TaskCompletionListener
java.lang.NullPointerException
at
org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.free(UnsafeInMemorySorter.java:110)
at
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.cleanupResources(UnsafeExternalSorter.java:288)
at
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter$1.onTaskCompletion(UnsafeExternalSorter.java:141)
at
org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:79)
at
org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:77)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:77)
at org.apache.spark.scheduler.Task.run(Task.scala:91)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
```
Author: Carson Wang <carson.w...@intel.com>
Closes #10637 from carsonwang/FixNPE.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/eabc7b8e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/eabc7b8e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/eabc7b8e
Branch: refs/heads/master
Commit: eabc7b8ee7e809bab05361ed154f87bff467bd88
Parents: cbbcd8e
Author: Carson Wang <carson.w...@intel.com>
Authored: Wed Jan 13 13:28:39 2016 -0800
Committer: Josh Rosen <joshro...@databricks.com>
Committed: Wed Jan 13 13:28:39 2016 -0800
----------------------------------------------------------------------
.../util/collection/unsafe/sort/UnsafeInMemorySorter.java | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/eabc7b8e/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
----------------------------------------------------------------------
diff --git
a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
index f71b8d1..d1b0bc5 100644
---
a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
+++
b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
@@ -116,8 +116,10 @@ public final class UnsafeInMemorySorter {
* Free the memory used by pointer array.
*/
public void free() {
- consumer.freeArray(array);
- array = null;
+ if (consumer != null) {
+ consumer.freeArray(array);
+ array = null;
+ }
}
public void reset() {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
