You will need to include that in the SPARK_JAVA_OPTS environment variable, so add the following line to spark-env.sh:
export SPARK_JAVA_OPTS=" -XX:+UseConcMarkSweepGC" This should propagate to the executors. (Though you should double check, since 0.9 is a little old and I could be forgetting something) If you wish to add spark options in addition to this, simply append them to the environment variable: export SPARK_JAVA_OPTS=" -XX:+UseConcMarkSweepGC -Dspark.config.one=value -Dspark.config.two=value" (Please note that this is only for Spark 0.9. The part where we set Spark options within SPARK_JAVA_OPTS is deprecated as of 1.0) 2014-07-17 21:08 GMT-07:00 Chen Song <chen.song...@gmail.com>: > Thanks Andrew. > > Say that I want to turn on CMS gc for each worker. > > All I need to do is add the following line to conf/spark-env.sh on node > where I submit the application. > > -XX:+UseConcMarkSweepGC > > Is that correct? > > Will this option be populated to each worker in yarn? > > > > On Thu, Jul 17, 2014 at 9:26 PM, Andrew Or <and...@databricks.com> wrote: > >> Hi Chen, >> >> spark.executor.extraJavaOptions is introduced in Spark 1.0, not in Spark >> 0.9. You need to >> >> export SPARK_JAVA_OPTS=" -Dspark.config1=value1 -Dspark.config2=value2" >> >> in conf/spark-env.sh. >> >> Let me know if that works. >> Andrew >> >> >> 2014-07-17 18:15 GMT-07:00 Tathagata Das <tathagata.das1...@gmail.com>: >> >> Can you check in the environment tab of Spark web ui to see whether this >>> configuration parameter is in effect? >>> >>> TD >>> >>> >>> On Thu, Jul 17, 2014 at 2:05 PM, Chen Song <chen.song...@gmail.com> >>> wrote: >>> >>>> I am using spark 0.9.0 and I am able to submit job to YARN, >>>> https://spark.apache.org/docs/0.9.0/running-on-yarn.html. >>>> >>>> I am trying to turn on gc logging on executors but could not find a way >>>> to set extra Java opts for workers. >>>> >>>> I tried to set spark.executor.extraJavaOptions but that did not work. >>>> >>>> Any idea on how I should do this? >>>> >>>> -- >>>> Chen Song >>>> >>>> >>> >> > > > -- > Chen Song > >
