[jira] [Commented] (SPARK-16265) Add option to SparkSubmit to ship driver JRE to YARN
[
https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366251#comment-15366251
]
Thomas Graves commented on SPARK-16265:
---
Have you tried shipping your jre as gzip file in the distributed cache and
pointing to it with: spark.yarn.appMasterEnv.JAVA_HOME= and
spark.executorEnv.JAVA_HOME=?
Not positive that works but you should try it first.
I'd rather not get into the business of installing software with Spark. It
then starts to extend to things like I want to install python version x, I want
to install R version Y, etc.
Also as stated above generally in the yarn environment java would be one of
those things installed on the clusters. yes we try to keep things isolated and
you should ship as much stuff as possible with your application but at this
point we also rely on the hadoop libraries and other things that you could have
incompatibilities with. obviously docker or something like that could give you
an entire environment.
> Add option to SparkSubmit to ship driver JRE to YARN
>
>
> Key: SPARK-16265
> URL: https://issues.apache.org/jira/browse/SPARK-16265
> Project: Spark
> Issue Type: Improvement
>Affects Versions: 1.6.2
>Reporter: Andrew Duffy
>
> Add an option to {{SparkSubmit}} to allow the driver to package up it's
> version of the JRE to be shipped to a YARN cluster. This allows deploying
> Spark applications to a YARN cluster in which its required Java version need
> not match one of the versions already installed on the YARN cluster, useful
> in situations in which the Spark Application developer does not have
> administrative access over the YARN cluster (ex. school or corporate
> environment) but still wants to use certain language features in their code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16265) Add option to SparkSubmit to ship driver JRE to YARN
[
https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365998#comment-15365998
]
Andrew Duffy commented on SPARK-16265:
--
Hi Sean, yeah I can see where you're coming from, but I feel like this change
is simple and targeted enough (meant to be used with the {{SparkLauncher}} API)
that it can actually be useful without adding much (if any) maintenance load.
If anything I would argue it at least deserves consideration as an experimental
feature, as users who write programs that use SparkLauncher are going to have
to split Java versions for the code that launches and interacts with the Spark
app and the Spark app itself if the application is eg. written for one
environment and then deployed in another uncontrolled customer environment
where the cluster does not have Java 8 installed.
> Add option to SparkSubmit to ship driver JRE to YARN
>
>
> Key: SPARK-16265
> URL: https://issues.apache.org/jira/browse/SPARK-16265
> Project: Spark
> Issue Type: Improvement
>Affects Versions: 1.6.2
>Reporter: Andrew Duffy
>
> Add an option to {{SparkSubmit}} to allow the driver to package up it's
> version of the JRE to be shipped to a YARN cluster. This allows deploying
> Spark applications to a YARN cluster in which its required Java version need
> not match one of the versions already installed on the YARN cluster, useful
> in situations in which the Spark Application developer does not have
> administrative access over the YARN cluster (ex. school or corporate
> environment) but still wants to use certain language features in their code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16265) Add option to SparkSubmit to ship driver JRE to YARN
[
https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15354207#comment-15354207
]
Saisai Shao commented on SPARK-16265:
-
If you want to run Spark on different JVM other than YARN, you could set
{{spark.yarn.appMasterEnv.JAVA_HOME=}} and
{{spark.executorEnv.JAVA_HOME=}}, the prerequisite is that JDK is
already installed on the cluster. This could partially address your problem.
> Add option to SparkSubmit to ship driver JRE to YARN
>
>
> Key: SPARK-16265
> URL: https://issues.apache.org/jira/browse/SPARK-16265
> Project: Spark
> Issue Type: Improvement
>Affects Versions: 1.6.2
>Reporter: Andrew Duffy
> Fix For: 2.1.0
>
>
> Add an option to {{SparkSubmit}} to allow the driver to package up it's
> version of the JRE to be shipped to a YARN cluster. This allows deploying
> Spark applications to a YARN cluster in which its required Java version need
> not match one of the versions already installed on the YARN cluster, useful
> in situations in which the Spark Application developer does not have
> administrative access over the YARN cluster (ex. school or corporate
> environment) but still wants to use certain language features in their code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16265) Add option to SparkSubmit to ship driver JRE to YARN
[
https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353709#comment-15353709
]
Apache Spark commented on SPARK-16265:
--
User 'andreweduffy' has created a pull request for this issue:
https://github.com/apache/spark/pull/13953
> Add option to SparkSubmit to ship driver JRE to YARN
>
>
> Key: SPARK-16265
> URL: https://issues.apache.org/jira/browse/SPARK-16265
> Project: Spark
> Issue Type: Improvement
>Affects Versions: 1.6.2
>Reporter: Andrew Duffy
> Fix For: 2.1.0
>
>
> Add an option to {{SparkSubmit}} to allow the driver to package up it's
> version of the JRE to be shipped to a YARN cluster. This allows deploying
> Spark applications to a YARN cluster in which its required Java version need
> not match one of the versions already installed on the YARN cluster, useful
> in situations in which the Spark Application developer does not have
> administrative access over the YARN cluster (ex. school or corporate
> environment) but still wants to use certain language features in their code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16265) Add option to SparkSubmit to ship driver JRE to YARN
[
https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353682#comment-15353682
]
Sean Owen commented on SPARK-16265:
---
I don't think this is in scope for Spark, since I assume you mean that your
cluster has Java 6, and Spark requires Java 7. Spark simply doesn't support
that and the answer really is: upgrade. You have other problems besides this
one in this scenario anyway. If you just want to run Java 8 on a Java 7 cluster
-- does not seem compelling compared to the complexity.
> Add option to SparkSubmit to ship driver JRE to YARN
>
>
> Key: SPARK-16265
> URL: https://issues.apache.org/jira/browse/SPARK-16265
> Project: Spark
> Issue Type: Improvement
>Affects Versions: 1.6.2
>Reporter: Andrew Duffy
> Fix For: 2.1.0
>
>
> Add an option to {{SparkSubmit}} to allow the driver to package up it's
> version of the JRE to be shipped to a YARN cluster. This allows deploying
> Spark applications to a YARN cluster in which its required Java version need
> not match one of the versions already installed on the YARN cluster, useful
> in situations in which the Spark Application developer does not have
> administrative access over the YARN cluster (ex. school or corporate
> environment) but still wants to use certain language features in their code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
