Thanks Andrew, I tried and it works.
On Fri, Jul 18, 2014 at 12:53 AM, Andrew Or and...@databricks.com wrote:
You will need to include that in the SPARK_JAVA_OPTS environment variable,
so add the following line to spark-env.sh:
export SPARK_JAVA_OPTS= -XX:+UseConcMarkSweepGC
This should
I am using spark 0.9.0 and I am able to submit job to YARN,
https://spark.apache.org/docs/0.9.0/running-on-yarn.html.
I am trying to turn on gc logging on executors but could not find a way to
set extra Java opts for workers.
I tried to set spark.executor.extraJavaOptions but that did not work.
Can you check in the environment tab of Spark web ui to see whether this
configuration parameter is in effect?
TD
On Thu, Jul 17, 2014 at 2:05 PM, Chen Song chen.song...@gmail.com wrote:
I am using spark 0.9.0 and I am able to submit job to YARN,
Hi Chen,
spark.executor.extraJavaOptions is introduced in Spark 1.0, not in Spark
0.9. You need to
export SPARK_JAVA_OPTS= -Dspark.config1=value1 -Dspark.config2=value2
in conf/spark-env.sh.
Let me know if that works.
Andrew
2014-07-17 18:15 GMT-07:00 Tathagata Das
Thanks Andrew.
Say that I want to turn on CMS gc for each worker.
All I need to do is add the following line to conf/spark-env.sh on node
where I submit the application.
-XX:+UseConcMarkSweepGC
Is that correct?
Will this option be populated to each worker in yarn?
On Thu, Jul 17, 2014 at
You will need to include that in the SPARK_JAVA_OPTS environment variable,
so add the following line to spark-env.sh:
export SPARK_JAVA_OPTS= -XX:+UseConcMarkSweepGC
This should propagate to the executors. (Though you should double check,
since 0.9 is a little old and I could be forgetting
