Hi Koert and Lukasz, The recommended way of not hard-coding configurations in your application is through conf/spark-defaults.conf as documented here: http://spark.apache.org/docs/latest/configuration.html#dynamically-loading-spark-properties. However, this is only applicable to spark-submit, so this may not be useful to you.
Depending on how you launch your Spark applications, you can workaround this by manually specifying these configs as -Dspark.x=y in your java command to launch Spark. This is actually how SPARK_JAVA_OPTS used to work before 1.0. Note that spark-submit does essentially the same thing, but sets these properties programmatically by reading from the conf/spark-defaults.conf file and calling System.setProperty("spark.x", "y"). Note that spark.executor.extraJavaOpts not intended for spark configuration (see http://spark.apache.org/docs/latest/configuration.html). SPARK_DAEMON_JAVA_OPTS, as you pointed out, is for Spark daemons like the standalone master, worker, and the history server; it is also not intended for spark configurations to be picked up by Spark executors and drivers. In general, any reference to "java opts" in any variable or config refers to java options, as the name implies, not Spark configuration. Unfortunately, it just so happened that we used to mix the two in the same environment variable before 1.0. Is there a reason you're not using spark-submit? Is it for legacy reasons? As of 1.0, most changes to launching Spark applications will be done through spark-submit, so you may miss out on relevant new features or bug fixes. Andrew 2014-06-19 7:41 GMT-07:00 Koert Kuipers <ko...@tresata.com>: > still struggling with SPARK_JAVA_OPTS being deprecated. i am using spark > standalone. > > for example if i have a akka timeout setting that i would like to be > applied to every piece of the spark framework (so spark master, spark > workers, spark executor sub-processes, spark-shell, etc.). i used to do > that with SPARK_JAVA_OPTS. now i am unsure. > > SPARK_DAEMON_JAVA_OPTS works for the master and workers, but not for the > spark-shell i think? i tried using SPARK_DAEMON_JAVA_OPTS, and it does not > seem that useful. for example for a worker it does not apply the settings > to the executor sub-processes, while for SPARK_JAVA_OPTS it does do that. > so seems like SPARK_JAVA_OPTS is my only way to change settings for the > executors, yet its deprecated? > > > On Wed, Jun 11, 2014 at 10:59 PM, elyast <lukasz.jastrzeb...@gmail.com> > wrote: > >> Hi, >> >> I tried to use SPARK_JAVA_OPTS in spark-env.sh as well as conf/java-opts >> file to set additional java system properties. In this case I could >> connect >> to tachyon without any problem. >> >> However when I tried setting executor and driver extraJavaOptions in >> spark-defaults.conf it doesn't. >> >> I suspect the root cause may be following: >> >> SparkSubmit doesn't fork additional JVM to actually run either driver or >> executor process and additional system properties are set after JVM is >> created and other classes are loaded. It may happen that Tachyon >> CommonConf >> class is already being loaded and since its Singleton it won't pick up and >> changes to system properties. >> >> Please let me know what do u think. >> >> Can I use conf/java-opts ? since it's not really documented anywhere? >> >> Best regards >> Lukasz >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/little-confused-about-SPARK-JAVA-OPTS-alternatives-tp5798p7448.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > >