[ https://issues.apache.org/jira/browse/SPARK-17814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated SPARK-17814: ------------------------------ Priority: Minor (was: Major) OK, yeah with quoting fixed, I think there's still an issue here. I think the issue is that with cluster mode, these args become an argument again to a new command line, and although it's escaped again, it seems something goes funny with curly braces. In cluster mode, {code} spark2-submit --verbose --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi /opt/cloudera/parcels/SPARK2/lib/spark2/examples/jars/spark-examples_2.11-2.0.0.cloudera.beta1.jar "{\"mode\":\"wf\", \"arrays\":{\"array\":[1]}}" ... childArgs [{"mode":"wf", "arrays":{"array":[1]}}] ... --arg {"mode":"wf", "arrays":{"array":[1]}} {code} I end up with the error ... {code} java.lang.NumberFormatException: For input string: "{"mode":"wf", "arrays":{"array":[1]" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:580) at java.lang.Integer.parseInt(Integer.java:615) at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272) at scala.collection.immutable.StringOps.toInt(StringOps.scala:29) at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:32) ... {code} Obviously a NFE is expected here, but, the input string it fails on is not expected. It seems specific to braces, but not sure why that is. > spark submit arguments are truncated in yarn-cluster mode > --------------------------------------------------------- > > Key: SPARK-17814 > URL: https://issues.apache.org/jira/browse/SPARK-17814 > Project: Spark > Issue Type: Bug > Components: Spark Submit, YARN > Affects Versions: 1.6.1 > Reporter: shreyas subramanya > Priority: Minor > > {noformat} > One of our customers is trying to pass in json through spark-submit as > follows: > spark-submit --verbose --class SimpleClass --master yarn-cluster ./simple.jar > "{\"mode\":\"wf\", \"arrays\":{\"array\":[1]}}" > The application reports the passed arguments as: {"mode":"wf", > "arrays":{"array":[1] > If the same application is submitted in yarn-client mode, as follows: > spark-submit --verbose --class SimpleClass --master yarn-client ./simple.jar > "{\"mode\":\"wf\", \"arrays\":{\"array\":[1]}}" > The application reports the passed args as: {"mode":"wf", > "arrays":{"array":[1]}} > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org