Hi Yuqi, Just curious can you share the spark-submit script and what are you passing as --master argument?
Thanks & Regards Biplob Biswas On Wed, Oct 31, 2018 at 10:34 AM Gourav Sengupta <gourav.sengu...@gmail.com> wrote: > Just out of curiosity why would you not use Glue (which is Spark on > kubernetes) or EMR? > > Regards, > Gourav Sengupta > > On Mon, Oct 29, 2018 at 1:29 AM Zhang, Yuqi <yuqi.zh...@teradata.com> > wrote: > >> Hello guys, >> >> >> >> I am Yuqi from Teradata Tokyo. Sorry to disturb but I have some problem >> regarding using spark 2.4 client mode function on kubernetes cluster, so I >> would like to ask if there is some solution to my problem. >> >> >> >> The problem is when I am trying to run spark-shell on kubernetes v1.11.3 >> cluster on AWS environment, I couldn’t successfully run stateful set using >> the docker image built from spark 2.4. The error message is showing below. >> The version I am using is spark v2.4.0-rc3. >> >> >> >> Also, I wonder if there is more documentation on how to use client-mode >> or integrate spark-shell on kubernetes cluster. From the documentation on >> https://github.com/apache/spark/blob/v2.4.0-rc3/docs/running-on-kubernetes.md >> there is only a brief description. I understand it’s not the official >> released version yet, but If there is some more documentation, could you >> please share with me? >> >> >> >> Thank you very much for your help! >> >> >> >> >> >> Error msg: >> >> + env >> >> + sed 's/[^=]*=\(.*\)/\1/g' >> >> + sort -t_ -k4 -n >> >> + grep SPARK_JAVA_OPT_ >> >> + readarray -t SPARK_EXECUTOR_JAVA_OPTS >> >> + '[' -n '' ']' >> >> + '[' -n '' ']' >> >> + PYSPARK_ARGS= >> >> + '[' -n '' ']' >> >> + R_ARGS= >> >> + '[' -n '' ']' >> >> + '[' '' == 2 ']' >> >> + '[' '' == 3 ']' >> >> + case "$SPARK_K8S_CMD" in >> >> + CMD=("$SPARK_HOME/bin/spark-submit" --conf >> "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client >> "$@") >> >> + exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf >> spark.driver.bindAddress= --deploy-mode client >> >> Error: Missing application resource. >> >> Usage: spark-submit [options] <app jar | python file | R file> [app >> arguments] >> >> Usage: spark-submit --kill [submission ID] --master [spark://...] >> >> Usage: spark-submit --status [submission ID] --master [spark://...] >> >> Usage: spark-submit run-example [options] example-class [example args] >> >> >> >> >> >> -- >> >> Yuqi Zhang >> >> Software Engineer >> >> m: 090-6725-6573 >> >> >> [image: signature_147554612] <http://www.teradata.com/> >> >> 2 Chome-2-23-1 Akasaka >> >> Minato, Tokyo 107-0052 >> teradata.com <http://www.teradata.com> >> >> >> This e-mail is from Teradata Corporation and may contain information that >> is confidential or proprietary. If you are not the intended recipient, do >> not read, copy or distribute the e-mail or any attachments. Instead, please >> notify the sender and delete the e-mail and any attachments. Thank you. >> >> Please consider the environment before printing. >> >> >> >> >> >
