for a jvm application its not very appealing to me to use spark submit.... my application uses hadoop, so i should use "hadoop jar", and my application uses spark, so it should use "spark-submit". if i add a piece of code that uses some other system there will be yet another suggested way to launch it. thats not very scalable, since i can only launch it one way in the end...
On Thu, Jun 19, 2014 at 4:58 PM, Andrew Or <and...@databricks.com> wrote: > Hi Koert and Lukasz, > > The recommended way of not hard-coding configurations in your application > is through conf/spark-defaults.conf as documented here: > http://spark.apache.org/docs/latest/configuration.html#dynamically-loading-spark-properties. > However, this is only applicable to > spark-submit, so this may not be useful to you. > > Depending on how you launch your Spark applications, you can workaround > this by manually specifying these configs as -Dspark.x=y > in your java command to launch Spark. This is actually how SPARK_JAVA_OPTS > used to work before 1.0. Note that spark-submit does > essentially the same thing, but sets these properties programmatically by > reading from the conf/spark-defaults.conf file and calling > System.setProperty("spark.x", "y"). > > Note that spark.executor.extraJavaOpts not intended for spark > configuration (see http://spark.apache.org/docs/latest/configuration.html > ). > SPARK_DAEMON_JAVA_OPTS, as you pointed out, is for Spark daemons like the > standalone master, worker, and the history server; > it is also not intended for spark configurations to be picked up by Spark > executors and drivers. In general, any reference to "java opts" > in any variable or config refers to java options, as the name implies, not > Spark configuration. Unfortunately, it just so happened that we > used to mix the two in the same environment variable before 1.0. > > Is there a reason you're not using spark-submit? Is it for legacy reasons? > As of 1.0, most changes to launching Spark applications > will be done through spark-submit, so you may miss out on relevant new > features or bug fixes. > > Andrew > > > > 2014-06-19 7:41 GMT-07:00 Koert Kuipers <ko...@tresata.com>: > > still struggling with SPARK_JAVA_OPTS being deprecated. i am using spark >> standalone. >> >> for example if i have a akka timeout setting that i would like to be >> applied to every piece of the spark framework (so spark master, spark >> workers, spark executor sub-processes, spark-shell, etc.). i used to do >> that with SPARK_JAVA_OPTS. now i am unsure. >> >> SPARK_DAEMON_JAVA_OPTS works for the master and workers, but not for the >> spark-shell i think? i tried using SPARK_DAEMON_JAVA_OPTS, and it does not >> seem that useful. for example for a worker it does not apply the settings >> to the executor sub-processes, while for SPARK_JAVA_OPTS it does do that. >> so seems like SPARK_JAVA_OPTS is my only way to change settings for the >> executors, yet its deprecated? >> >> >> On Wed, Jun 11, 2014 at 10:59 PM, elyast <lukasz.jastrzeb...@gmail.com> >> wrote: >> >>> Hi, >>> >>> I tried to use SPARK_JAVA_OPTS in spark-env.sh as well as conf/java-opts >>> file to set additional java system properties. In this case I could >>> connect >>> to tachyon without any problem. >>> >>> However when I tried setting executor and driver extraJavaOptions in >>> spark-defaults.conf it doesn't. >>> >>> I suspect the root cause may be following: >>> >>> SparkSubmit doesn't fork additional JVM to actually run either driver or >>> executor process and additional system properties are set after JVM is >>> created and other classes are loaded. It may happen that Tachyon >>> CommonConf >>> class is already being loaded and since its Singleton it won't pick up >>> and >>> changes to system properties. >>> >>> Please let me know what do u think. >>> >>> Can I use conf/java-opts ? since it's not really documented anywhere? >>> >>> Best regards >>> Lukasz >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/little-confused-about-SPARK-JAVA-OPTS-alternatives-tp5798p7448.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >> >> >
