Emma Dickson created SPARK-28095: ------------------------------------ Summary: Pyspark on kubernetes doesn't parse arguments with spaces as expected. Key: SPARK-28095 URL: https://issues.apache.org/jira/browse/SPARK-28095 Project: Spark Issue Type: Bug Components: Kubernetes, PySpark Affects Versions: 2.4.3 Environment: Python 2.7.13
Spark 2.4.3 Kubernetes Reporter: Emma Dickson When passing in arguments to a bash script that calls a python file which sets up a pyspark context strings with spaces are processed as individual strings even when encased in double quotes, using backslashes or unicode escape characters. Example Command entered {code:java} ./scripts/spark-k8s.sh v0.0.32 --job-args "cos://waas-logentries.mycos/Logentries/IBM-b634032e/Github/Load Balancer" --job pages{code} Error Message {code:java} + exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=172.30.83.253 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.deploy.PythonRunner /opt/spark/work-dir/main.py --job-args cos://waas-logentries.mycos/Logentries/IBM-b634032e/Github/Load Balancer --job pages 19/06/18 19:28:35 WARN Utils: Kubernetes master URL uses HTTP instead of HTTPS. 19/06/18 19:28:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable usage: main.py [-h] --job JOB --job-args JOB_ARGS main.py: error: unrecognized arguments: Balancer {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org