In addition to setting the Standalone memory, you'll also need to tell your
SparkContext to claim the extra resources. Set spark.executor.memory to
1600m as well. This should be a system property set in SPARK_JAVA_OPTS in
conf/spark-env.sh (in 0.9.1, which you appear to be using) -- e.g.,
export SPARK_JAVA_OPTS=-Dspark.executor.memory=1600mb
On Sun, Jun 1, 2014 at 7:36 PM, Yunmeng Ban banyunm...@gmail.com wrote:
Hi,
I'm running the example of JavaKafkaWordCount in a standalone cluster. I
want to set 1600MB memory for each slave node. I wrote in the
spark/conf/spark-env.sh
SPARK_WORKER_MEMORY=1600m
But the logs on slave nodes looks this:
Spark Executor Command: /usr/java/latest/bin/java -cp
:/~path/spark/conf:/~path/spark/assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar
-Xms512M -Xmx512M
org.apache.spark.executor.CoarseGrainedExecutorBackend
The memory seems to be the default number, not 1600M.
I don't how to make SPARK_WORKER_MEMORY work.
Can anyone help me?
Many thanks in advance.
Yunmeng
