Which version of Spark are you using to run spark-submit, and which version of Spark your container image is based off? This looks to be caused my mismatched versions of Spark used for spark-submit and for the driver/executor at runtime.
On Mon, Apr 30, 2018 at 12:00 PM, Holden Karau <hol...@pigscanfly.ca> wrote: > So, while its not perfect, I have a guide focused on running custom Spark > on GKE https://cloud.google.com/blog/big-data/2018/03/testing- > future-apache-spark-releases-and-changes-on-google- > kubernetes-engine-and-cloud-dataproc and if you want to run pre-built > Spark on GKE there is a solutions article - https://cloud.google.com/ > solutions/spark-on-kubernetes-engine which could be relevant. > > On Mon, Apr 30, 2018 at 7:51 PM, Eric Wang <ericswang1...@gmail.com> > wrote: > >> Hello all, >> >> I've been trying to spark-submit a job to the Google Kubernetes Engine >> but I keep encountering a "Exception in thread "main" >> java.lang.IllegalArgumentException: Server properties file given at >> /opt/spark/work-dir/driver does not exist or is not a file." >> error. I'm unsure of how to even begin debugging this so any help would >> be greatly appreciated. I've attached the logs and the full spark-submit >> command I'm running here: https://gist.github.com/ >> erkkel/c04a0b5ca60ad755cf62e9ad18e5b7ed >> >> For reference, I've been following this guide: https://apache-spark-on >> -k8s.github.io/userdocs/running-on-kubernetes.html >> >> Thanks, >> Eric >> >> > > > -- > Twitter: https://twitter.com/holdenkarau >
