I have a PR which tries to address this issue:
https://github.com/apache/spark/pull/9384
Comment is welcome.
On Mon, Nov 2, 2015 at 9:53 AM, Patrick Wendell wrote:
> I believe this is some bug in our tests. For some reason we are using way
> more memory than necessary. We'll probably need to lo
I believe this is some bug in our tests. For some reason we are using way
more memory than necessary. We'll probably need to log into Jenkins and
heap dump some running tests and figure out what is going on.
On Mon, Nov 2, 2015 at 7:42 AM, Ted Yu wrote:
> Looks like SparkListenerSuite doesn't OO
Looks like SparkListenerSuite doesn't OOM on QA runs compared to Jenkins
builds.
I wonder if this is due to difference between machines running QA tests vs
machines running Jenkins builds.
On Fri, Oct 30, 2015 at 1:19 PM, Ted Yu wrote:
> I noticed that the SparkContext created in each sub-test
I noticed that the SparkContext created in each sub-test is not stopped
upon finishing sub-test.
Would stopping each SparkContext make a difference in terms of heap memory
consumption ?
Cheers
On Fri, Oct 30, 2015 at 12:04 PM, Mridul Muralidharan
wrote:
> It is giving OOM at 32GB ? Something l
It is giving OOM at 32GB ? Something looks wrong with that ... that is
already on the higher side.
Regards,
Mridul
On Fri, Oct 30, 2015 at 11:28 AM, shane knapp wrote:
> here's the current heap settings on our workers:
> InitialHeapSize == 2.1G
> MaxHeapSize == 32G
>
> system ram: 128G
>
> we c
here's the current heap settings on our workers:
InitialHeapSize == 2.1G
MaxHeapSize == 32G
system ram: 128G
we can bump it pretty easily... it's just a matter of deciding if we
want to do this globally (super easy, but will affect ALL maven builds
on our system -- not just spark) or on a per-j
This happened recently on Jenkins:
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.3,label=spark-test/3964/console
On Sun, Oct 18, 2015 at 7:54 AM, Ted Yu wrote:
> From
> https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADO
From
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=spark-test/3846/console
:
SparkListenerSuite:- basic creation and shutdown of LiveListenerBus-
bus.stop() waits for the event queue to completely drain- basic
creation of StageInfo- basic c
