[jira] [Commented] (SPARK-22622) OutOfMemory thrown by Closure Serializer without proper failure propagation
[
https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269974#comment-16269974
]
Raghavendra commented on SPARK-22622:
-
[~srowen] I am broadcasting all my large data strucures and im assuming that
the actual RDD is not serialized via the closure serializer. Then what is
causing this error?
Also, how about my second issue. Even though the issue could be large data, the
Job must end gracefully. The Log4j Logs are receiving this stack trace but the
Spark UI is not showing anything and the Job is not ending on this failure,
instead the "dag-event-loop" thread is killed and the jobs continues
indefinitely. I think the latter is a bug that needs to be handled at least.
Kindly clarify.
> OutOfMemory thrown by Closure Serializer without proper failure propagation
> ---
>
> Key: SPARK-22622
> URL: https://issues.apache.org/jira/browse/SPARK-22622
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.2.0
> Environment: Spark 2.2.0
> Hadoop 2.9.0
>Reporter: Raghavendra
>Priority: Critical
>
> While moving from a Stage to another, the Closure serializer is trying to
> Serialize the Closures and throwing OOMs.
> This is happening when the RDD size crosses 70 GB.
> I set the Driver Memory to 225 GB and yet the error persist.
> There are two issues here
> * OOM thrown when there is almost 3 times of Driver memory provided than the
> last Stage RDD size.(Even tried caching this into the disk before moving it
> into the current stage)
> * After the Error is thrown, the Spark Job does not exit. it just continues
> in the same state without propagating the error into the Spark UI.
> *Scenario 1*
> {color:red}Exception in thread "dag-scheduler-event-loop"
> java.lang.OutOfMemoryError: Requested array size exceeds VM limit
> at java.util.Arrays.copyOf(Arrays.java:3236)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> at
> org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41)
> at
> java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
> at
> java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> at
> org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:43)
> at
> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
> at
> org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1003)
> at
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:930)
> at
> org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:874)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1677)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {color}
> *Scenario 2*
> {color:red}
>Exception in thread "dag-scheduler-event-loop"
> java.lang.OutOfMemoryError
> at
> java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> at
> org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41)
> at
> java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
> at
> java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> at
> org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:43)
> at
> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
> at
> org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1003)
> at
>
[jira] [Commented] (SPARK-22622) OutOfMemory thrown by Closure Serializer without proper failure propagation
[
https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269935#comment-16269935
]
Sean Owen commented on SPARK-22622:
---
The error isn't out of memory. It's that an array is too large, more than 2GB.
This is either an issue with your code, or is simply another instance of the
"2GB limit" in Spark and the JVM. That's a real issue, but already in JIRA.
either way please don't reopen this.
> OutOfMemory thrown by Closure Serializer without proper failure propagation
> ---
>
> Key: SPARK-22622
> URL: https://issues.apache.org/jira/browse/SPARK-22622
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.2.0
> Environment: Spark 2.2.0
> Hadoop 2.9.0
>Reporter: Raghavendra
>Priority: Critical
>
> While moving from a Stage to another, the Closure serializer is trying to
> Serialize the Closures and throwing OOMs.
> This is happening when the RDD size crosses 70 GB.
> I set the Driver Memory to 225 GB and yet the error persist.
> There are two issues here
> * OOM thrown when there is almost 3 times of Driver memory provided than the
> last Stage RDD size.(Even tried caching this into the disk before moving it
> into the current stage)
> * After the Error is thrown, the Spark Job does not exit. it just continues
> in the same state without propagating the error into the Spark UI.
> *Scenario 1*
> {color:red}Exception in thread "dag-scheduler-event-loop"
> java.lang.OutOfMemoryError: Requested array size exceeds VM limit
> at java.util.Arrays.copyOf(Arrays.java:3236)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> at
> org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41)
> at
> java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
> at
> java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> at
> org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:43)
> at
> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
> at
> org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1003)
> at
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:930)
> at
> org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:874)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1677)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {color}
> *Scenario 2*
> {color:red}
>Exception in thread "dag-scheduler-event-loop"
> java.lang.OutOfMemoryError
> at
> java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> at
> org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41)
> at
> java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
> at
> java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> at
> org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:43)
> at
> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
> at
> org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1003)
> at
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:930)
> at
> org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:874)
> at
>
[jira] [Commented] (SPARK-22622) OutOfMemory thrown by Closure Serializer without proper failure propagation
[
https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269917#comment-16269917
]
Raghavendra commented on SPARK-22622:
-
The output of previous Stage is around 70GB and my driver memory is 225GB. Why
am i getting an OOM when there is sufficient memory.
Also, since this is closure serializer, isnt it only serializing the Physical
Plan and not the RDD?
17/11/28 07:22:53 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD
[25]
at parquet at LogProcessor.java:473), which has no missing parents
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 25 - 0 - 0 -
StorageLevel(1 replicas) - parquet at ***.java:473
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 20 - 0 - 0 -
StorageLevel(1 replicas) - persist at ***.java:283
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 19 - 0 - 0 -
StorageLevel(1 replicas) - persist at ***.java:283
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 12 - 0 - 0 -
StorageLevel(1 replicas) - persist at ***.java:88
17/11/28 07:22:53 ERROR LogListener: RDD Info : FileScanRDD - 11 - 0 - 0 -
StorageLevel(1 replicas) - persist at ***.java:88
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 22 - 0 - 0 -
StorageLevel(1 replicas) - persist at ***.java:283
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 18 - 0 - 0 -
StorageLevel(1 replicas) - persist at ***.java:283
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 21 - 0 - 0 -
StorageLevel(1 replicas) - persist at ***.java:283
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 24 - 0 - 0 -
StorageLevel(1 replicas) - parquet at ***.java:473
Exception in thread "dag-scheduler-event-loop" java.lang.OutOfMemoryError
at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at
org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41)
at
java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
at
java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:43)
at
org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
at
org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1003)
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:930)
at
org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:874)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1677)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658)
> OutOfMemory thrown by Closure Serializer without proper failure propagation
> ---
>
> Key: SPARK-22622
> URL: https://issues.apache.org/jira/browse/SPARK-22622
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.2.0
> Environment: Spark 2.2.0
> Hadoop 2.9.0
>Reporter: Raghavendra
>Priority: Critical
>
> While moving from a Stage to another, the Closure serializer is trying to
> Serialize the Closures and throwing OOMs.
> This is happening when the RDD size crosses 70 GB.
> I set the Driver Memory to 225 GB and yet the error persist.
> There are two issues here
> * OOM thrown when there is almost 3 times of Driver memory provided than the
> last Stage RDD size.(Even tried caching this into the disk before moving it
> into the current stage)
> * After the Error is thrown, the Spark Job does not exit. it just continues
> in the same state without propagating the error into the Spark UI.
> *Scenario 1*
> {color:red}Exception in thread "dag-scheduler-event-loop"
> java.lang.OutOfMemoryError: Requested array size exceeds VM limit
> at java.util.Arrays.copyOf(Arrays.java:3236)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> at
>
[jira] [Commented] (SPARK-22622) OutOfMemory thrown by Closure Serializer
[
https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269894#comment-16269894
]
Raghavendra commented on SPARK-22622:
-
The output of previous Stage is around 70GB and my driver memory is 225GB. Why
am i getting an OOM when there is sufficient memory.
Also, since this is closure serializer, isnt it only serializing the Physical
Plan and not the RDD?
17/11/28 07:22:53 INFO DAGScheduler: Submitting ResultStage 3
(MapPartitionsRDD[25] at parquet at LogProcessor.java:473), which has no
missing parents
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 25 - 0 - 0 -
StorageLevel(1 replicas) - parquet at LogProcessor.java:473
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 20 - 0 - 0 -
StorageLevel(1 replicas) - persist at LogProcessor.java:283
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 19 - 0 - 0 -
StorageLevel(1 replicas) - persist at LogProcessor.java:283
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 12 - 0 - 0 -
StorageLevel(1 replicas) - persist at LogProcessor.java:88
17/11/28 07:22:53 ERROR LogListener: RDD Info : FileScanRDD - 11 - 0 - 0 -
StorageLevel(1 replicas) - persist at LogProcessor.java:88
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 22 - 0 - 0 -
StorageLevel(1 replicas) - persist at LogProcessor.java:283
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 18 - 0 - 0 -
StorageLevel(1 replicas) - persist at LogProcessor.java:283
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 21 - 0 - 0 -
StorageLevel(1 replicas) - persist at LogProcessor.java:283
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 24 - 0 - 0 -
StorageLevel(1 replicas) - parquet at LogProcessor.java:473
> OutOfMemory thrown by Closure Serializer
>
>
> Key: SPARK-22622
> URL: https://issues.apache.org/jira/browse/SPARK-22622
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.2.0
> Environment: Spark 2.2.0
> Hadoop 2.9.0
>Reporter: Raghavendra
>Priority: Critical
>
> While moving from a Stage to another, the Closure serializer is trying to
> Serialize the Closures and throwing OOMs.
> This is happening when the RDD size crosses 70 GB.
> I set the Driver Memory to 225 GB and yet the error persists.
> {color:red}Exception in thread "dag-scheduler-event-loop"
> java.lang.OutOfMemoryError: Requested array size exceeds VM limit
> at java.util.Arrays.copyOf(Arrays.java:3236)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> at
> org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41)
> at
> java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
> at
> java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> at
> org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:43)
> at
> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
> at
> org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1003)
> at
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:930)
> at
> org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:874)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1677)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {color}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
