Re: spark-submit memory too larger

2014-10-25 Thread marylucy
Version: spark 1.1.0 42 workers,40g memory per worker Running graphx componentgraph ,use five hours > 在 Oct 25, 2014,1:27,"Sameer Farooqui" 写道: > > That does seem a bit odd. How many Executors are running under this Driver? > > Does the spark-submit process start out using ~60GB of memory righ

Re: spark-submit memory too larger

2014-10-24 Thread Sameer Farooqui
That does seem a bit odd. How many Executors are running under this Driver? Does the spark-submit process start out using ~60GB of memory right away or does it start out smaller and slowly build up to that high? If so, how long does it take to get that high? Also, which version of Spark are you u

spark-submit memory too larger

2014-10-24 Thread marylucy
i used standalone spark,set spark.driver.memory=5g,but spark-submit process use 57g memory, is this normal?how to decrease it?