Re: Spark DF does not rename the column

Hi, I think you have a typo : root |-- ceated: string (nullable = true) and then: withColumnRenamed("created","Date Calculated"). On Mon, 4 Jan 2021 at 19:12, Lalwani, Jayesh wrote: > You don’t have a column named “created”. The column name is “ceated”, > without the “r” > > > >

Suggestion on Spark 2.4.7 vs Spark 3 for Kubernetes

Hi Users, Could you please tell which Spark version have you used in Production for Kubernetes. Which is a recommended version for Production provided that both Streaming and core apis have to be used using Pyspark. Thanks ! Kind Regards, Sachit Murarka

Re: Spark DF does not rename the column

Thanks Jayesh for spotting that typo. Cheers Mich LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * *Disclaimer:* Use it at your own risk. Any and all

Re: Spark DF does not rename the column

You don’t have a column named “created”. The column name is “ceated”, without the “r” From: Mich Talebzadeh Date: Monday, January 4, 2021 at 1:06 PM To: "user @spark" Subject: [EXTERNAL] Spark DF does not rename the column CAUTION: This email originated from outside of the organization. Do

Spark DF does not rename the column

Hi, version 2.4.3 I don't know the cause of this. This renaming of DF columns used to work fine. I did couple of changes to spark/Scala code not relevant to this table and it refuses to rename the columns for a table!. val summaryACC = HiveContext.table("summaryACC") summaryACC.printSchema()

Re: Error while running Spark on K8s

Can you please paste the full exception trace, and mention spark and k8s version? On Mon, Jan 4, 2021 at 6:19 PM Sachit Murarka wrote: > Hi Prashant > > Thanks for the response! > > I created the service account with the permissions and following is the > command: > > spark-submit

Re: Error while running Spark on K8s

Hi Prashant Thanks for the response! I created the service account with the permissions and following is the command: spark-submit --deploy-mode cluster --master k8s://http://ip:port --name "sachit" --conf spark.kubernetes.pyspark.pythonVersion=3 --conf spark.kubernetes.namespace=spark-test

Re: Error while running Spark on K8s

Hi Sachit, Can you give more details on how did you run? i.e. spark submit command. My guess is, a service account with sufficient privilege is not provided. Please see: http://spark.apache.org/docs/latest/running-on-kubernetes.html#rbac Thanks, On Mon, Jan 4, 2021 at 5:27 PM Sachit Murarka

Error while running Spark on K8s

Hi All, I am getting the below error when I am trying to run the spark job on Kubernetes, I am running it in cluster mode. Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create] for kind: [Pod] with name: [null] in namespace: [spark-test] failed. at