[jira] [Commented] (SPARK-1606) spark-submit needs `--arg` for every application parameter

[Mridul Muralidharan (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13988756#comment-13988756
 ] 

Mridul Muralidharan commented on SPARK-1606:


Crap, got to this too late.
We really should not have added this - and I would have -1'ed it.

We have had too many issues with trying to parse the user command line and 
getting into all sorts of issues which are entirely avoidable by simply being 
explicit about what the user wants to pass.
Trying to pass strings which contain escape characters and/or whitespace, etc 
is just going to be a nightmare about this change.

 spark-submit needs `--arg` for every application parameter
 --

 Key: SPARK-1606
 URL: https://issues.apache.org/jira/browse/SPARK-1606
 Project: Spark
  Issue Type: Improvement
  Components: Spark Core
Affects Versions: 1.0.0
Reporter: Xiangrui Meng
Assignee: Patrick Wendell
Priority: Blocker
 Fix For: 1.0.0


 If the application has a few parameters, the spark-submit command looks like 
 the following:
 {code}
 spark-submit --master yarn-cluster --class main.Class --arg --numPartitions 
 --arg 8 --arg --kryo --arg true
 {code}
 It is a little bit hard to read and modify. Maybe it is okay to treat all 
 arguments after `main.Class` as application parameters.
 {code}
 spark-submit --master yarn-cluster --class main.Class --numPartitions 8 
 --kryo true
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (SPARK-1606) spark-submit needs `--arg` for every application parameter

[Patrick Wendell (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981851#comment-13981851
 ] 

Patrick Wendell commented on SPARK-1606:


I submitted a PR here that adds the following syntax.

{code}
./bin/spark-submit [options] user.jar [user options]
{code}

 spark-submit needs `--arg` for every application parameter
 --

 Key: SPARK-1606
 URL: https://issues.apache.org/jira/browse/SPARK-1606
 Project: Spark
  Issue Type: Improvement
  Components: Spark Core
Affects Versions: 1.0.0
Reporter: Xiangrui Meng
Assignee: Patrick Wendell
Priority: Blocker

 If the application has a few parameters, the spark-submit command looks like 
 the following:
 {code}
 spark-submit --master yarn-cluster --class main.Class --arg --numPartitions 
 --arg 8 --arg --kryo --arg true
 {code}
 It is a little bit hard to read and modify. Maybe it is okay to treat all 
 arguments after `main.Class` as application parameters.
 {code}
 spark-submit --master yarn-cluster --class main.Class --numPartitions 8 
 --kryo true
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)