Re: Setting YARN executors' JAVA_HOME

[Ryan Williams]

Ah, I guess I missed that by only looking in the YARN config docs, but this
is a more general parameter and not documented there. Thanks!

On Thu, Aug 18, 2016 at 2:51 PM dhruve ashar  wrote:

> Hi Ryan,
>
> You can get more info on this here:  Spark documentation
> .
>
> The page addresses what you need. You can look for 
> spark.executorEnv.[EnvironmentVariableName]
> and set your java home as
> spark.executorEnv.JAVA_HOME=
>
> Regards,
> Dhruve
>
> On Thu, Aug 18, 2016 at 12:49 PM, Ryan Williams <
> ryan.blake.willi...@gmail.com> wrote:
>
>> I need to tell YARN a JAVA_HOME to use when spawning containers (to run a
>> Java 8 app on Java 7 YARN).
>>
>> The only way I've found that works is
>> setting SPARK_YARN_USER_ENV="JAVA_HOME=/path/to/java8".
>>
>> The code
>> 
>> implies that this is deprecated and users should use "the config", but I
>> can't figure out what config is being referenced.
>>
>> Passing "--conf spark.yarn.appMasterEnv.JAVA_HOME=/path/to/java8" seems
>> to set it for the AM but not for executors.
>>
>> Likewise, spark.executor.extraLibraryPath and
>> spark.driver.extraLibraryPath don't appear to set JAVA_HOME (and maybe
>> aren't even supposed to?).
>>
>> The 1.0.1 docs
>> 
>>  are the last ones to reference the SPARK_YARN_USER_ENV var, afaict.
>>
>> What's the preferred way of passing YARN a custom JAVA_HOME that will be
>> applied to executors' containers?
>>
>> Thanks!
>>
>
>
>
> --
> -Dhruve Ashar
>
>

Re: Setting YARN executors' JAVA_HOME

[dhruve ashar]

Hi Ryan,

You can get more info on this here:  Spark documentation
.

The page addresses what you need. You can look for
spark.executorEnv.[EnvironmentVariableName]
and set your java home as
spark.executorEnv.JAVA_HOME=

Regards,
Dhruve

On Thu, Aug 18, 2016 at 12:49 PM, Ryan Williams <
ryan.blake.willi...@gmail.com> wrote:

> I need to tell YARN a JAVA_HOME to use when spawning containers (to run a
> Java 8 app on Java 7 YARN).
>
> The only way I've found that works is setting SPARK_YARN_USER_ENV="
> JAVA_HOME=/path/to/java8".
>
> The code
> 
> implies that this is deprecated and users should use "the config", but I
> can't figure out what config is being referenced.
>
> Passing "--conf spark.yarn.appMasterEnv.JAVA_HOME=/path/to/java8" seems
> to set it for the AM but not for executors.
>
> Likewise, spark.executor.extraLibraryPath and
> spark.driver.extraLibraryPath don't appear to set JAVA_HOME (and maybe
> aren't even supposed to?).
>
> The 1.0.1 docs
> 
>  are the last ones to reference the SPARK_YARN_USER_ENV var, afaict.
>
> What's the preferred way of passing YARN a custom JAVA_HOME that will be
> applied to executors' containers?
>
> Thanks!
>



-- 
-Dhruve Ashar

Setting YARN executors' JAVA_HOME

[Ryan Williams]

I need to tell YARN a JAVA_HOME to use when spawning containers (to run a
Java 8 app on Java 7 YARN).

The only way I've found that works is
setting SPARK_YARN_USER_ENV="JAVA_HOME=/path/to/java8".

The code

implies that this is deprecated and users should use "the config", but I
can't figure out what config is being referenced.

Passing "--conf spark.yarn.appMasterEnv.JAVA_HOME=/path/to/java8" seems to
set it for the AM but not for executors.

Likewise, spark.executor.extraLibraryPath and spark.driver.extraLibraryPath
don't appear to set JAVA_HOME (and maybe aren't even supposed to?).

The 1.0.1 docs

 are the last ones to reference the SPARK_YARN_USER_ENV var, afaict.

What's the preferred way of passing YARN a custom JAVA_HOME that will be
applied to executors' containers?

Thanks!