I am trying to set the available memory on a stand alone spark streaming job
and had no success.
I have tried these options on spark-env.sh
export SPARK_DAEMON_MEMORY=15g
export SPARK_WORKER_MEMORY=15g
export SPARK_DAEMON_JAVA_OPTS="-Xms20g -Xmx20g"
export SPARK_JAVA_OPTS="-Xms20g -Xmx20g"
Also tried this inside the scala file:
"System.setProperty("spark.executor.memory","5g")". This option worked on a
normal spark job running on a cluster. However, it will not work with spark
streaming.
Has anyone managed to run a spark streaming job with a custom value of java
memory ?
If so, how did you do it ?
Thanks
