[jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on command line
[ https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081490#comment-14081490 ] Marcelo Vanzin commented on SPARK-1576: --- With Sandy's recent patch (https://github.com/apache/spark/pull/1253) this should be easy to do (spark-submit --conf spark.executor.extraJavaOptions=blah). > Passing of JAVA_OPTS to YARN on command line > > > Key: SPARK-1576 > URL: https://issues.apache.org/jira/browse/SPARK-1576 > Project: Spark > Issue Type: Improvement >Affects Versions: 0.9.0, 0.9.1, 0.9.2 >Reporter: Nishkam Ravi > Attachments: SPARK-1576.patch > > > JAVA_OPTS can be passed by using either env variables (i.e., SPARK_JAVA_OPTS) > or as config vars (after Patrick's recent change). It would be good to allow > the user to pass them on command line as well to restrict scope to single > application invocation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on command line
[ https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981473#comment-13981473 ] Nishkam Ravi commented on SPARK-1576: - We have been using SPARK_JAVA_OPTS and JAVA_OPTS interchangeably. SPARK_JAVA_OPTS are JAVA_OPTS :) > Passing of JAVA_OPTS to YARN on command line > > > Key: SPARK-1576 > URL: https://issues.apache.org/jira/browse/SPARK-1576 > Project: Spark > Issue Type: Improvement >Affects Versions: 0.9.0, 1.0.0, 0.9.1 >Reporter: Nishkam Ravi > Fix For: 0.9.0, 1.0.0, 0.9.1 > > Attachments: SPARK-1576.patch > > > JAVA_OPTS can be passed by using either env variables (i.e., SPARK_JAVA_OPTS) > or as config vars (after Patrick's recent change). It would be good to allow > the user to pass them on command line as well to restrict scope to single > application invocation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on command line
[ https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981313#comment-13981313 ] Mridul Muralidharan commented on SPARK-1576: There is a misunderstanding here - it is to pass SPARK_JAVA_OPTS : not JAVA_OPTS. Directly passing JAVA_OPTS has beem removed > Passing of JAVA_OPTS to YARN on command line > > > Key: SPARK-1576 > URL: https://issues.apache.org/jira/browse/SPARK-1576 > Project: Spark > Issue Type: Improvement >Affects Versions: 0.9.0, 1.0.0, 0.9.1 >Reporter: Nishkam Ravi > Fix For: 0.9.0, 1.0.0, 0.9.1 > > Attachments: SPARK-1576.patch > > > JAVA_OPTS can be passed by using either env variables (i.e., SPARK_JAVA_OPTS) > or as config vars (after Patrick's recent change). It would be good to allow > the user to pass them on command line as well to restrict scope to single > application invocation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on command line
[ https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13979184#comment-13979184 ] Nishkam Ravi commented on SPARK-1576: - Thomas, --driver-java-options would not work with yarn client though (only spark-submit). > Passing of JAVA_OPTS to YARN on command line > > > Key: SPARK-1576 > URL: https://issues.apache.org/jira/browse/SPARK-1576 > Project: Spark > Issue Type: Improvement >Affects Versions: 0.9.0, 1.0.0, 0.9.1 >Reporter: Nishkam Ravi > Fix For: 0.9.0, 1.0.0, 0.9.1 > > Attachments: SPARK-1576.patch > > > JAVA_OPTS can be passed by using either env variables (i.e., SPARK_JAVA_OPTS) > or as config vars (after Patrick's recent change). It would be good to allow > the user to pass them on command line as well to restrict scope to single > application invocation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on command line
[
https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13979173#comment-13979173
]
Nishkam Ravi commented on SPARK-1576:
-
Bit of a race condition here it seems. Patrick made a few changes yesterday
around the same time as I did (in ClientBase.scala):
for ((k, v) <- sys.props.filterKeys(_.startsWith("spark"))) {
JAVA_OPTS += "-D" + k + "=" + "\\\"" + v + "\\\""
}
This would allow JAVA_OPTS to be passed on the command line to the
ApplicationMaster, and accomplishes the same things as creation of a new
command line flag --spark-java-opts.
> Passing of JAVA_OPTS to YARN on command line
>
>
> Key: SPARK-1576
> URL: https://issues.apache.org/jira/browse/SPARK-1576
> Project: Spark
> Issue Type: Improvement
>Affects Versions: 0.9.0, 1.0.0, 0.9.1
>Reporter: Nishkam Ravi
> Fix For: 0.9.0, 1.0.0, 0.9.1
>
> Attachments: SPARK-1576.patch
>
>
> JAVA_OPTS can be passed by using either env variables (i.e., SPARK_JAVA_OPTS)
> or as config vars (after Patrick's recent change). It would be good to allow
> the user to pass them on command line as well to restrict scope to single
> application invocation.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on command line
[ https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978164#comment-13978164 ] Thomas Graves commented on SPARK-1576: -- Is this meant for the driver or the executors? The spark-submit script has a command line option for the driver: --driver-java-options. I believe the intent of https://github.com/apache/spark/pull/299 was to not expose SPARK_JAVA_OPTS to the user anymore. > Passing of JAVA_OPTS to YARN on command line > > > Key: SPARK-1576 > URL: https://issues.apache.org/jira/browse/SPARK-1576 > Project: Spark > Issue Type: Improvement >Affects Versions: 0.9.0, 1.0.0, 0.9.1 >Reporter: Nishkam Ravi > Fix For: 0.9.0, 1.0.0, 0.9.1 > > Attachments: SPARK-1576.patch > > > JAVA_OPTS can be passed by using either env variables (i.e., SPARK_JAVA_OPTS) > or as config vars (after Patrick's recent change). It would be good to allow > the user to pass them on command line as well to restrict scope to single > application invocation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on command line
[ https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977922#comment-13977922 ] Nishkam Ravi commented on SPARK-1576: - https://github.com/apache/spark/pull/492 > Passing of JAVA_OPTS to YARN on command line > > > Key: SPARK-1576 > URL: https://issues.apache.org/jira/browse/SPARK-1576 > Project: Spark > Issue Type: Improvement >Affects Versions: 0.9.0, 1.0.0, 0.9.1 >Reporter: Nishkam Ravi > Fix For: 0.9.0, 1.0.0, 0.9.1 > > Attachments: SPARK-1576.patch > > > JAVA_OPTS can be passed by using either env variables (i.e., SPARK_JAVA_OPTS) > or as config vars (after Patrick's recent change). It would be good to allow > the user to pass them on command line as well to restrict scope to single > application invocation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on command line
[ https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977918#comment-13977918 ] Sandy Ryza commented on SPARK-1576: --- Thanks, mind posting a link to it on this JIRA? > Passing of JAVA_OPTS to YARN on command line > > > Key: SPARK-1576 > URL: https://issues.apache.org/jira/browse/SPARK-1576 > Project: Spark > Issue Type: Improvement >Affects Versions: 0.9.0, 1.0.0, 0.9.1 >Reporter: Nishkam Ravi > Fix For: 0.9.0, 1.0.0, 0.9.1 > > Attachments: SPARK-1576.patch > > > JAVA_OPTS can be passed by using either env variables (i.e., SPARK_JAVA_OPTS) > or as config vars (after Patrick's recent change). It would be good to allow > the user to pass them on command line as well to restrict scope to single > application invocation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on command line
[ https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977906#comment-13977906 ] Nishkam Ravi commented on SPARK-1576: - Hi Sandy, I've initiated a pull request for this fix. > Passing of JAVA_OPTS to YARN on command line > > > Key: SPARK-1576 > URL: https://issues.apache.org/jira/browse/SPARK-1576 > Project: Spark > Issue Type: Improvement >Affects Versions: 0.9.0, 1.0.0, 0.9.1 >Reporter: Nishkam Ravi > Fix For: 0.9.0, 1.0.0, 0.9.1 > > Attachments: SPARK-1576.patch > > > JAVA_OPTS can be passed by using either env variables (i.e., SPARK_JAVA_OPTS) > or as config vars (after Patrick's recent change). It would be good to allow > the user to pass them on command line as well to restrict scope to single > application invocation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on command line
[ https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977754#comment-13977754 ] Sandy Ryza commented on SPARK-1576: --- Hi [~nravi], Spark patches should be submitted as pull requests against the Spark github - https://github.com/apache/spark. > Passing of JAVA_OPTS to YARN on command line > > > Key: SPARK-1576 > URL: https://issues.apache.org/jira/browse/SPARK-1576 > Project: Spark > Issue Type: Improvement >Affects Versions: 0.9.0, 1.0.0, 0.9.1 >Reporter: Nishkam Ravi > Fix For: 0.9.0, 1.0.0, 0.9.1 > > Attachments: SPARK-1576.patch > > > JAVA_OPTS can be passed by using either env variables (i.e., SPARK_JAVA_OPTS) > or as config vars (after Patrick's recent change). It would be good to allow > the user to pass them on command line as well to restrict scope to single > application invocation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on command line
[ https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977472#comment-13977472 ] Nishkam Ravi commented on SPARK-1576: - Attached is a patch for this. > Passing of JAVA_OPTS to YARN on command line > > > Key: SPARK-1576 > URL: https://issues.apache.org/jira/browse/SPARK-1576 > Project: Spark > Issue Type: Improvement >Affects Versions: 0.9.0, 1.0.0, 0.9.1 >Reporter: Nishkam Ravi > Fix For: 0.9.0, 1.0.0, 0.9.1 > > > JAVA_OPTS can be passed by using either env variables (i.e., SPARK_JAVA_OPTS) > or as config vars (after Patrick's recent change). It would be good to allow > the user to pass them on command line as well to restrict scope to single > application invocation. -- This message was sent by Atlassian JIRA (v6.2#6252)
