Re: how to pass extra Java opts to workers for spark streaming jobs
Thanks Andrew, I tried and it works. On Fri, Jul 18, 2014 at 12:53 AM, Andrew Or wrote: > You will need to include that in the SPARK_JAVA_OPTS environment variable, > so add the following line to spark-env.sh: > > export SPARK_JAVA_OPTS=" -XX:+UseConcMarkSweepGC" > > This should propagate to the executors. (Though you should double check, > since 0.9 is a little old and I could be forgetting something) If you wish > to add spark options in addition to this, simply append them to the > environment variable: > > export SPARK_JAVA_OPTS=" -XX:+UseConcMarkSweepGC -Dspark.config.one=value > -Dspark.config.two=value" > > (Please note that this is only for Spark 0.9. The part where we set Spark > options within SPARK_JAVA_OPTS is deprecated as of 1.0) > > > 2014-07-17 21:08 GMT-07:00 Chen Song : > > Thanks Andrew. >> >> Say that I want to turn on CMS gc for each worker. >> >> All I need to do is add the following line to conf/spark-env.sh on node >> where I submit the application. >> >> -XX:+UseConcMarkSweepGC >> >> Is that correct? >> >> Will this option be populated to each worker in yarn? >> >> >> >> On Thu, Jul 17, 2014 at 9:26 PM, Andrew Or wrote: >> >>> Hi Chen, >>> >>> spark.executor.extraJavaOptions is introduced in Spark 1.0, not in Spark >>> 0.9. You need to >>> >>> export SPARK_JAVA_OPTS=" -Dspark.config1=value1 -Dspark.config2=value2" >>> >>> in conf/spark-env.sh. >>> >>> Let me know if that works. >>> Andrew >>> >>> >>> 2014-07-17 18:15 GMT-07:00 Tathagata Das : >>> >>> Can you check in the environment tab of Spark web ui to see whether this configuration parameter is in effect? TD On Thu, Jul 17, 2014 at 2:05 PM, Chen Song wrote: > I am using spark 0.9.0 and I am able to submit job to YARN, > https://spark.apache.org/docs/0.9.0/running-on-yarn.html. > > I am trying to turn on gc logging on executors but could not find a > way to set extra Java opts for workers. > > I tried to set spark.executor.extraJavaOptions but that did not work. > > Any idea on how I should do this? > > -- > Chen Song > > >>> >> >> >> -- >> Chen Song >> >> > -- Chen Song
Re: how to pass extra Java opts to workers for spark streaming jobs
You will need to include that in the SPARK_JAVA_OPTS environment variable, so add the following line to spark-env.sh: export SPARK_JAVA_OPTS=" -XX:+UseConcMarkSweepGC" This should propagate to the executors. (Though you should double check, since 0.9 is a little old and I could be forgetting something) If you wish to add spark options in addition to this, simply append them to the environment variable: export SPARK_JAVA_OPTS=" -XX:+UseConcMarkSweepGC -Dspark.config.one=value -Dspark.config.two=value" (Please note that this is only for Spark 0.9. The part where we set Spark options within SPARK_JAVA_OPTS is deprecated as of 1.0) 2014-07-17 21:08 GMT-07:00 Chen Song : > Thanks Andrew. > > Say that I want to turn on CMS gc for each worker. > > All I need to do is add the following line to conf/spark-env.sh on node > where I submit the application. > > -XX:+UseConcMarkSweepGC > > Is that correct? > > Will this option be populated to each worker in yarn? > > > > On Thu, Jul 17, 2014 at 9:26 PM, Andrew Or wrote: > >> Hi Chen, >> >> spark.executor.extraJavaOptions is introduced in Spark 1.0, not in Spark >> 0.9. You need to >> >> export SPARK_JAVA_OPTS=" -Dspark.config1=value1 -Dspark.config2=value2" >> >> in conf/spark-env.sh. >> >> Let me know if that works. >> Andrew >> >> >> 2014-07-17 18:15 GMT-07:00 Tathagata Das : >> >> Can you check in the environment tab of Spark web ui to see whether this >>> configuration parameter is in effect? >>> >>> TD >>> >>> >>> On Thu, Jul 17, 2014 at 2:05 PM, Chen Song >>> wrote: >>> I am using spark 0.9.0 and I am able to submit job to YARN, https://spark.apache.org/docs/0.9.0/running-on-yarn.html. I am trying to turn on gc logging on executors but could not find a way to set extra Java opts for workers. I tried to set spark.executor.extraJavaOptions but that did not work. Any idea on how I should do this? -- Chen Song >>> >> > > > -- > Chen Song > >
Re: how to pass extra Java opts to workers for spark streaming jobs
Thanks Andrew. Say that I want to turn on CMS gc for each worker. All I need to do is add the following line to conf/spark-env.sh on node where I submit the application. -XX:+UseConcMarkSweepGC Is that correct? Will this option be populated to each worker in yarn? On Thu, Jul 17, 2014 at 9:26 PM, Andrew Or wrote: > Hi Chen, > > spark.executor.extraJavaOptions is introduced in Spark 1.0, not in Spark > 0.9. You need to > > export SPARK_JAVA_OPTS=" -Dspark.config1=value1 -Dspark.config2=value2" > > in conf/spark-env.sh. > > Let me know if that works. > Andrew > > > 2014-07-17 18:15 GMT-07:00 Tathagata Das : > > Can you check in the environment tab of Spark web ui to see whether this >> configuration parameter is in effect? >> >> TD >> >> >> On Thu, Jul 17, 2014 at 2:05 PM, Chen Song >> wrote: >> >>> I am using spark 0.9.0 and I am able to submit job to YARN, >>> https://spark.apache.org/docs/0.9.0/running-on-yarn.html. >>> >>> I am trying to turn on gc logging on executors but could not find a way >>> to set extra Java opts for workers. >>> >>> I tried to set spark.executor.extraJavaOptions but that did not work. >>> >>> Any idea on how I should do this? >>> >>> -- >>> Chen Song >>> >>> >> > -- Chen Song
Re: how to pass extra Java opts to workers for spark streaming jobs
Hi Chen, spark.executor.extraJavaOptions is introduced in Spark 1.0, not in Spark 0.9. You need to export SPARK_JAVA_OPTS=" -Dspark.config1=value1 -Dspark.config2=value2" in conf/spark-env.sh. Let me know if that works. Andrew 2014-07-17 18:15 GMT-07:00 Tathagata Das : > Can you check in the environment tab of Spark web ui to see whether this > configuration parameter is in effect? > > TD > > > On Thu, Jul 17, 2014 at 2:05 PM, Chen Song wrote: > >> I am using spark 0.9.0 and I am able to submit job to YARN, >> https://spark.apache.org/docs/0.9.0/running-on-yarn.html. >> >> I am trying to turn on gc logging on executors but could not find a way >> to set extra Java opts for workers. >> >> I tried to set spark.executor.extraJavaOptions but that did not work. >> >> Any idea on how I should do this? >> >> -- >> Chen Song >> >> >
Re: how to pass extra Java opts to workers for spark streaming jobs
Can you check in the environment tab of Spark web ui to see whether this configuration parameter is in effect? TD On Thu, Jul 17, 2014 at 2:05 PM, Chen Song wrote: > I am using spark 0.9.0 and I am able to submit job to YARN, > https://spark.apache.org/docs/0.9.0/running-on-yarn.html. > > I am trying to turn on gc logging on executors but could not find a way to > set extra Java opts for workers. > > I tried to set spark.executor.extraJavaOptions but that did not work. > > Any idea on how I should do this? > > -- > Chen Song > >
