Thanks Andrew. Say that I want to turn on CMS gc for each worker.
All I need to do is add the following line to conf/spark-env.sh on node where I submit the application. -XX:+UseConcMarkSweepGC Is that correct? Will this option be populated to each worker in yarn? On Thu, Jul 17, 2014 at 9:26 PM, Andrew Or <and...@databricks.com> wrote: > Hi Chen, > > spark.executor.extraJavaOptions is introduced in Spark 1.0, not in Spark > 0.9. You need to > > export SPARK_JAVA_OPTS=" -Dspark.config1=value1 -Dspark.config2=value2" > > in conf/spark-env.sh. > > Let me know if that works. > Andrew > > > 2014-07-17 18:15 GMT-07:00 Tathagata Das <tathagata.das1...@gmail.com>: > > Can you check in the environment tab of Spark web ui to see whether this >> configuration parameter is in effect? >> >> TD >> >> >> On Thu, Jul 17, 2014 at 2:05 PM, Chen Song <chen.song...@gmail.com> >> wrote: >> >>> I am using spark 0.9.0 and I am able to submit job to YARN, >>> https://spark.apache.org/docs/0.9.0/running-on-yarn.html. >>> >>> I am trying to turn on gc logging on executors but could not find a way >>> to set extra Java opts for workers. >>> >>> I tried to set spark.executor.extraJavaOptions but that did not work. >>> >>> Any idea on how I should do this? >>> >>> -- >>> Chen Song >>> >>> >> > -- Chen Song
