Re: Running Spark on Kubernetes (GKE) - failing on spark-submit

Hi Ye, This is the error i get when i don't set the spark.kubernetes.file.upload.path Any ideas on how to fix this ? ``` Exception in thread "main" org.apache.spark.SparkException: Please specify spark.kubernetes.file.upload.path property. at

ADLS Gen2 adfs sample yaml configuration

Hello, I need help/sample adfs(Active Directory Federation Services) (ADLS GEN2) on how to configure ADLS GEN2(adfs) configurations in yaml file with spark history server ?? I would like to see running jobs from JupiterLab notebook with SparkOnK8sV3.0.2 Kernel shell. Any help is much

How to explode array columns of a dataframe having the same length

Hello guys, I have the following dataframe: *col1* *col2* *col3* ["A","B","null"] ["C","D","null"] ["E","null","null"] I want to explode it to the following dataframe: *col1* *col2* *col3* "A" "C" "E" "B" "D" "null" "null" "null" "null" How to do that (preferably in Java) using

Re: Running Spark on Kubernetes (GKE) - failing on spark-submit

The configuration of ‘…file.upload.path’ is wrong. it means a distributed fs path to store your archives/resource/jars temporarily, then distributed by spark to drivers/executors. For your cases, you don’t need to set this configuration.Sent from my iPhoneOn Feb 14, 2023, at 5:43 AM, karan alang

Re: Running Spark on Kubernetes (GKE) - failing on spark-submit

I am not k8s expert but I think you got permission issue. Try 777 as an example to see if it works. On Mon, 13 Feb 2023, 21:42 karan alang, wrote: > Hello All, > > I'm trying to run a simple application on GKE (Kubernetes), and it is > failing: > Note : I have spark(bitnami spark chart)