[jira] [Commented] (SPARK-25995) sparkR should ensure user args are after the argument used for the port
[ https://issues.apache.org/jira/browse/SPARK-25995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683876#comment-16683876 ] Thomas Graves commented on SPARK-25995: --- I haven't looked at the details but I would say whatever is easier. I'm not sure what users do with the args in this case anyway. Can they be used? Another option would be to make the port file an actual spark internal arg or configuration. It looks like it has a SparkConf object at the time its read https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/r/RBackend.scala#L135 > sparkR should ensure user args are after the argument used for the port > --- > > Key: SPARK-25995 > URL: https://issues.apache.org/jira/browse/SPARK-25995 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 2.3.2 >Reporter: Thomas Graves >Priority: Minor > > Currently if you run sparkR and accidentally specify an argument, it fails > with a useless error message. For example: > $SPARK_HOME/bin/sparkR --master yarn --deploy-mode client fooarg > This gets turned into: > Launching java with spark-submit command spark-submit "--master" "yarn" > "--deploy-mode" "client" "sparkr-shell" "fooarg" > /tmp/Rtmp6XBGz2/backend_port162806ea36bca > Notice that "fooarg" got put before /tmp file which is how R and jvm know > which port to connect to. SparkR eventually fails with timeout exception > after 10 seconds. > > SparkR should either not allow args or make sure the order is correct so the > backend_port is always first. see > https://github.com/apache/spark/blob/master/R/pkg/R/sparkR.R#L129 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25995) sparkR should ensure user args are after the argument used for the port
[
https://issues.apache.org/jira/browse/SPARK-25995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682762#comment-16682762
]
Felix Cheung commented on SPARK-25995:
--
sparkR is just taking the whole string as-is
[https://github.com/apache/spark/blob/141953f4c44dbad1c2a7059e92bec5fe770af932/R/pkg/R/client.R#L59]
you see sparkSubmitOpts is before args (args is the file with port number)
I think we should avoid duplicating the submit arg parsing in R, which we would
need to break before
{code:java}
fooarg
{code}
?
Is it easier/better to always set the temp file with port as the last arg
instead?
> sparkR should ensure user args are after the argument used for the port
> ---
>
> Key: SPARK-25995
> URL: https://issues.apache.org/jira/browse/SPARK-25995
> Project: Spark
> Issue Type: Bug
> Components: SparkR
>Affects Versions: 2.3.2
>Reporter: Thomas Graves
>Priority: Minor
>
> Currently if you run sparkR and accidentally specify an argument, it fails
> with a useless error message. For example:
> $SPARK_HOME/bin/sparkR --master yarn --deploy-mode client fooarg
> This gets turned into:
> Launching java with spark-submit command spark-submit "--master" "yarn"
> "--deploy-mode" "client" "sparkr-shell" "fooarg"
> /tmp/Rtmp6XBGz2/backend_port162806ea36bca
> Notice that "fooarg" got put before /tmp file which is how R and jvm know
> which port to connect to. SparkR eventually fails with timeout exception
> after 10 seconds.
>
> SparkR should either not allow args or make sure the order is correct so the
> backend_port is always first. see
> https://github.com/apache/spark/blob/master/R/pkg/R/sparkR.R#L129
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25995) sparkR should ensure user args are after the argument used for the port
[ https://issues.apache.org/jira/browse/SPARK-25995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682322#comment-16682322 ] Hyukjin Kwon commented on SPARK-25995: -- Sounds related with SPARK-18340 as well. > sparkR should ensure user args are after the argument used for the port > --- > > Key: SPARK-25995 > URL: https://issues.apache.org/jira/browse/SPARK-25995 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 2.3.2 >Reporter: Thomas Graves >Priority: Minor > > Currently if you run sparkR and accidentally specify an argument, it fails > with a useless error message. For example: > $SPARK_HOME/bin/sparkR --master yarn --deploy-mode client fooarg > This gets turned into: > Launching java with spark-submit command spark-submit "--master" "yarn" > "--deploy-mode" "client" "sparkr-shell" "fooarg" > /tmp/Rtmp6XBGz2/backend_port162806ea36bca > Notice that "fooarg" got put before /tmp file which is how R and jvm know > which port to connect to. SparkR eventually fails with timeout exception > after 10 seconds. > > SparkR should either not allow args or make sure the order is correct so the > backend_port is always first. see > https://github.com/apache/spark/blob/master/R/pkg/R/sparkR.R#L129 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
