You don’t have a column named “created”. The column name is “ceated”, without the “r”
From: Mich Talebzadeh <mich.talebza...@gmail.com> Date: Monday, January 4, 2021 at 1:06 PM To: "user @spark" <user@spark.apache.org> Subject: [EXTERNAL] Spark DF does not rename the column CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hi, version 2.4.3 I don't know the cause of this. This renaming of DF columns used to work fine. I did couple of changes to spark/Scala code not relevant to this table and it refuses to rename the columns for a table!. val summaryACC = HiveContext.table("summaryACC") summaryACC.printSchema() root |-- ceated: string (nullable = true) |-- hashtag: string (nullable = true) |-- paid: float (nullable = true) |-- received: float (nullable = true) summaryACC. orderBy(desc("paid"),desc("received")). withColumnRenamed("created","Date Calculated"). withColumnRenamed("hashtag","Who"). withColumn(("received"),format_number(col("received"),2)). withColumn(("paid"),format_number(col("paid"),2)). withColumnRenamed("paid","paid out/GBP"). withColumnRenamed("received","paid in/GBP"). withColumn("paid in/GBP",when(col("paid in/GBP") === "0.00","--").otherwise(col("paid in/GBP"))). withColumn("paid out/GBP",when(col("paid out/GBP") === "0.00","--").otherwise(col("paid out/GBP"))). select("Date Calculated","Who","paid in/GBP","paid out/GBP").show(1000,false) and this is the error org.apache.spark.sql.AnalysisException: cannot resolve '`Date Calculated`' given input columns: [alayer.summaryacc.ceated, Who, paid out/GBP, paid in/GBP];; This used to work before! +----------------------------+------------------+-----------+------------+ |Date Calculated |Who |paid in/GBP|paid out/GBP| +----------------------------+------------------+-----------+------------+ |Mon Jan 04 14:22:17 GMT 2021|paypal |579.98 |1,526.86 | Appreciate any ideas. Thanks, Mich