Re: Running Spark on Kubernetes (GKE) - failing on spark-submit

You can try this gsutil cp src/StructuredStream-on-gke.py gs://codes/ where you create a bucket on gcs called codes Then in you spark-submit do spark-submit --verbose \ --master k8s://https://$KUBERNETES_MASTER_IP:443 \ --deploy-mode cluster \ --name

Re: Running Spark on Kubernetes (GKE) - failing on spark-submit

thnks, Mich .. let me check this On Wed, Feb 15, 2023 at 1:42 AM Mich Talebzadeh wrote: > > It may help to check this article of mine > > > Spark on Kubernetes, A Practitioner’s Guide >

Re: Running Spark on Kubernetes (GKE) - failing on spark-submit

It may help to check this article of mine Spark on Kubernetes, A Practitioner’s Guide HTH view my Linkedin profile

Re: Running Spark on Kubernetes (GKE) - failing on spark-submit

Your submit command spark-submit --master k8s://https://34.74.22.140:7077 --deploy-mode cluster --name pyspark-example --conf spark.kubernetes.container.image=pyspark-example:0.1 --conf spark.kubernetes.file.upload.path=/myexample src/StructuredStream-on-gke.py pay attention to what it says

Re: Running Spark on Kubernetes (GKE) - failing on spark-submit

Hi Ye, This is the error i get when i don't set the spark.kubernetes.file.upload.path Any ideas on how to fix this ? ``` Exception in thread "main" org.apache.spark.SparkException: Please specify spark.kubernetes.file.upload.path property. at

Re: Running Spark on Kubernetes (GKE) - failing on spark-submit

The configuration of ‘…file.upload.path’ is wrong. it means a distributed fs path to store your archives/resource/jars temporarily, then distributed by spark to drivers/executors. For your cases, you don’t need to set this configuration.Sent from my iPhoneOn Feb 14, 2023, at 5:43 AM, karan alang

Re: Running Spark on Kubernetes (GKE) - failing on spark-submit

I am not k8s expert but I think you got permission issue. Try 777 as an example to see if it works. On Mon, 13 Feb 2023, 21:42 karan alang, wrote: > Hello All, > > I'm trying to run a simple application on GKE (Kubernetes), and it is > failing: > Note : I have spark(bitnami spark chart)

Running Spark on Kubernetes (GKE) - failing on spark-submit

Hello All, I'm trying to run a simple application on GKE (Kubernetes), and it is failing: Note : I have spark(bitnami spark chart) installed on GKE using helm install Here is what is done : 1. created a docker image using Dockerfile Dockerfile : ``` FROM python:3.7-slim RUN apt-get update &&