Hi Ryan, You can get more info on this here: Spark documentation <http://spark.apache.org/docs/latest/configuration.html>.
The page addresses what you need. You can look for spark.executorEnv.[EnvironmentVariableName] and set your java home as spark.executorEnv.JAVA_HOME=.... Regards, Dhruve On Thu, Aug 18, 2016 at 12:49 PM, Ryan Williams < ryan.blake.willi...@gmail.com> wrote: > I need to tell YARN a JAVA_HOME to use when spawning containers (to run a > Java 8 app on Java 7 YARN). > > The only way I've found that works is setting SPARK_YARN_USER_ENV=" > JAVA_HOME=/path/to/java8". > > The code > <https://github.com/apache/spark/blob/b72bb62d421840f82d663c6b8e3922bd14383fbb/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L762> > implies that this is deprecated and users should use "the config", but I > can't figure out what config is being referenced. > > Passing "--conf spark.yarn.appMasterEnv.JAVA_HOME=/path/to/java8" seems > to set it for the AM but not for executors. > > Likewise, spark.executor.extraLibraryPath and > spark.driver.extraLibraryPath don't appear to set JAVA_HOME (and maybe > aren't even supposed to?). > > The 1.0.1 docs > <https://spark.apache.org/docs/1.0.1/running-on-yarn.html#environment-variables> > are the last ones to reference the SPARK_YARN_USER_ENV var, afaict. > > What's the preferred way of passing YARN a custom JAVA_HOME that will be > applied to executors' containers? > > Thanks! > -- -Dhruve Ashar
