Hi Koert and Lukasz,
The recommended way of not hard-coding configurations in your application
is through conf/spark-defaults.conf as documented here:
http://spark.apache.org/docs/latest/configuration.html#dynamically-loading-spark-properties.
However, this is only applicable to
spark-submit, so this may not be useful to you.
Depending on how you launch your Spark applications, you can workaround
this by manually specifying these configs as -Dspark.x=y
in your java command to launch Spark. This is actually how SPARK_JAVA_OPTS
used to work before 1.0. Note that spark-submit does
essentially the same thing, but sets these properties programmatically by
reading from the conf/spark-defaults.conf file and calling
System.setProperty("spark.x", "y").
Note that spark.executor.extraJavaOpts not intended for spark configuration
(see http://spark.apache.org/docs/latest/configuration.html).
SPARK_DAEMON_JAVA_OPTS, as you pointed out, is for Spark daemons like the
standalone master, worker, and the history server;
it is also not intended for spark configurations to be picked up by Spark
executors and drivers. In general, any reference to "java opts"
in any variable or config refers to java options, as the name implies, not
Spark configuration. Unfortunately, it just so happened that we
used to mix the two in the same environment variable before 1.0.
Is there a reason you're not using spark-submit? Is it for legacy reasons?
As of 1.0, most changes to launching Spark applications
will be done through spark-submit, so you may miss out on relevant new
features or bug fixes.
Andrew
2014-06-19 7:41 GMT-07:00 Koert Kuipers <ko...@tresata.com>:
> still struggling with SPARK_JAVA_OPTS being deprecated. i am using spark
> standalone.
>
> for example if i have a akka timeout setting that i would like to be
> applied to every piece of the spark framework (so spark master, spark
> workers, spark executor sub-processes, spark-shell, etc.). i used to do
> that with SPARK_JAVA_OPTS. now i am unsure.
>
> SPARK_DAEMON_JAVA_OPTS works for the master and workers, but not for the
> spark-shell i think? i tried using SPARK_DAEMON_JAVA_OPTS, and it does not
> seem that useful. for example for a worker it does not apply the settings
> to the executor sub-processes, while for SPARK_JAVA_OPTS it does do that.
> so seems like SPARK_JAVA_OPTS is my only way to change settings for the
> executors, yet its deprecated?
>
>
> On Wed, Jun 11, 2014 at 10:59 PM, elyast <lukasz.jastrzeb...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I tried to use SPARK_JAVA_OPTS in spark-env.sh as well as conf/java-opts
>> file to set additional java system properties. In this case I could
>> connect
>> to tachyon without any problem.
>>
>> However when I tried setting executor and driver extraJavaOptions in
>> spark-defaults.conf it doesn't.
>>
>> I suspect the root cause may be following:
>>
>> SparkSubmit doesn't fork additional JVM to actually run either driver or
>> executor process and additional system properties are set after JVM is
>> created and other classes are loaded. It may happen that Tachyon
>> CommonConf
>> class is already being loaded and since its Singleton it won't pick up and
>> changes to system properties.
>>
>> Please let me know what do u think.
>>
>> Can I use conf/java-opts ? since it's not really documented anywhere?
>>
>> Best regards
>> Lukasz
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/little-confused-about-SPARK-JAVA-OPTS-alternatives-tp5798p7448.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>
