Ah, I guess I missed that by only looking in the YARN config docs, but this is a more general parameter and not documented there. Thanks!
On Thu, Aug 18, 2016 at 2:51 PM dhruve ashar <dhruveas...@gmail.com> wrote: > Hi Ryan, > > You can get more info on this here: Spark documentation > <http://spark.apache.org/docs/latest/configuration.html>. > > The page addresses what you need. You can look for > spark.executorEnv.[EnvironmentVariableName] > and set your java home as > spark.executorEnv.JAVA_HOME=.... > > Regards, > Dhruve > > On Thu, Aug 18, 2016 at 12:49 PM, Ryan Williams < > ryan.blake.willi...@gmail.com> wrote: > >> I need to tell YARN a JAVA_HOME to use when spawning containers (to run a >> Java 8 app on Java 7 YARN). >> >> The only way I've found that works is >> setting SPARK_YARN_USER_ENV="JAVA_HOME=/path/to/java8". >> >> The code >> <https://github.com/apache/spark/blob/b72bb62d421840f82d663c6b8e3922bd14383fbb/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L762> >> implies that this is deprecated and users should use "the config", but I >> can't figure out what config is being referenced. >> >> Passing "--conf spark.yarn.appMasterEnv.JAVA_HOME=/path/to/java8" seems >> to set it for the AM but not for executors. >> >> Likewise, spark.executor.extraLibraryPath and >> spark.driver.extraLibraryPath don't appear to set JAVA_HOME (and maybe >> aren't even supposed to?). >> >> The 1.0.1 docs >> <https://spark.apache.org/docs/1.0.1/running-on-yarn.html#environment-variables> >> are the last ones to reference the SPARK_YARN_USER_ENV var, afaict. >> >> What's the preferred way of passing YARN a custom JAVA_HOME that will be >> applied to executors' containers? >> >> Thanks! >> > > > > -- > -Dhruve Ashar > >
