[jira] [Commented] (SPARK-3107) Don't pass null jar to executor in yarn-client mode
[
https://issues.apache.org/jira/browse/SPARK-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132164#comment-14132164
]
Andrew Or commented on SPARK-3107:
--
I see. Yes, setting it to an empty value is semantically different from not
setting it, and that is exactly the reason why we shouldn't set these to empty
strings when what we really mean to do is to not set it. I think we still do
that in a few places here and there.
Don't pass null jar to executor in yarn-client mode
---
Key: SPARK-3107
URL: https://issues.apache.org/jira/browse/SPARK-3107
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 1.1.0
Reporter: Andrew Or
In the following line, ExecutorLauncher's `--jar` takes in null.
{code}
14/08/18 20:52:43 INFO yarn.Client: command: $JAVA_HOME/bin/java -server
-Xmx512m ... org.apache.spark.deploy.yarn.ExecutorLauncher --class 'notused'
--jar null --arg 'ip-172-31-0-12.us-west-2.compute.internal:56838'
--executor-memory 1024 --executor-cores 1 --num-executors 2
{code}
Also it appears that we set a bunch of system properties to empty strings
(not shown). We should avoid setting these if they don't actually contain
values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3107) Don't pass null jar to executor in yarn-client mode
[
https://issues.apache.org/jira/browse/SPARK-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107476#comment-14107476
]
Marcelo Vanzin commented on SPARK-3107:
---
The {{--jar}} issue is fixed in SPARK-2933.
Where do you see the sys properties issue? Setting a system property to an
empty value is semantically different from not setting it, although I'm
sceptical it would make a difference here.
Don't pass null jar to executor in yarn-client mode
---
Key: SPARK-3107
URL: https://issues.apache.org/jira/browse/SPARK-3107
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 1.1.0
Reporter: Andrew Or
In the following line, ExecutorLauncher's `--jar` takes in null.
{code}
14/08/18 20:52:43 INFO yarn.Client: command: $JAVA_HOME/bin/java -server
-Xmx512m ... org.apache.spark.deploy.yarn.ExecutorLauncher --class 'notused'
--jar null --arg 'ip-172-31-0-12.us-west-2.compute.internal:56838'
--executor-memory 1024 --executor-cores 1 --num-executors 2
{code}
Also it appears that we set a bunch of system properties to empty strings
(not shown). We should avoid setting these if they don't actually contain
values.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
