Looks like SparkListenerSuite doesn't OOM on QA runs compared to Jenkins builds.
I wonder if this is due to difference between machines running QA tests vs machines running Jenkins builds. On Fri, Oct 30, 2015 at 1:19 PM, Ted Yu <yuzhih...@gmail.com> wrote: > I noticed that the SparkContext created in each sub-test is not stopped > upon finishing sub-test. > > Would stopping each SparkContext make a difference in terms of heap memory > consumption ? > > Cheers > > On Fri, Oct 30, 2015 at 12:04 PM, Mridul Muralidharan <mri...@gmail.com> > wrote: > >> It is giving OOM at 32GB ? Something looks wrong with that ... that is >> already on the higher side. >> >> Regards, >> Mridul >> >> >> On Fri, Oct 30, 2015 at 11:28 AM, shane knapp <skn...@berkeley.edu> >> wrote: >> > here's the current heap settings on our workers: >> > InitialHeapSize == 2.1G >> > MaxHeapSize == 32G >> > >> > system ram: 128G >> > >> > we can bump it pretty easily... it's just a matter of deciding if we >> > want to do this globally (super easy, but will affect ALL maven builds >> > on our system -- not just spark) or on a per-job basis (this doesn't >> > scale that well). >> > >> > thoughts? >> > >> > On Fri, Oct 30, 2015 at 9:47 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> >> This happened recently on Jenkins: >> >> >> >> >> https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.3,label=spark-test/3964/console >> >> >> >> On Sun, Oct 18, 2015 at 7:54 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> >> >>> From >> >>> >> https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=spark-test/3846/console >> >>> : >> >>> >> >>> SparkListenerSuite: >> >>> - basic creation and shutdown of LiveListenerBus >> >>> - bus.stop() waits for the event queue to completely drain >> >>> - basic creation of StageInfo >> >>> - basic creation of StageInfo with shuffle >> >>> - StageInfo with fewer tasks than partitions >> >>> - local metrics >> >>> - onTaskGettingResult() called when result fetched remotely *** >> FAILED *** >> >>> org.apache.spark.SparkException: Job aborted due to stage failure: >> Task >> >>> 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in >> stage >> >>> 0.0 (TID 0, localhost): java.lang.OutOfMemoryError: Java heap space >> >>> at java.util.Arrays.copyOf(Arrays.java:2271) >> >>> at >> java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) >> >>> at >> >>> >> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) >> >>> at >> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) >> >>> at >> >>> >> java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1852) >> >>> at java.io.ObjectOutputStream.write(ObjectOutputStream.java:708) >> >>> at org.apache.spark.util.Utils$.writeByteBuffer(Utils.scala:182) >> >>> at >> >>> >> org.apache.spark.scheduler.DirectTaskResult$$anonfun$writeExternal$1.apply$mcV$sp(TaskResult.scala:52) >> >>> at >> org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1160) >> >>> at >> >>> >> org.apache.spark.scheduler.DirectTaskResult.writeExternal(TaskResult.scala:49) >> >>> at >> >>> >> java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1458) >> >>> at >> >>> >> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1429) >> >>> at >> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) >> >>> at >> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) >> >>> at >> >>> >> org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44) >> >>> at >> >>> >> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101) >> >>> at >> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:256) >> >>> at >> >>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> >>> at >> >>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> >>> at java.lang.Thread.run(Thread.java:745) >> >>> >> >>> >> >>> Should more heap be given to test suite ? >> >>> >> >>> >> >>> Cheers >> >> >> >> >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> > For additional commands, e-mail: dev-h...@spark.apache.org >> > >> > >
