Re: Kyro Serializer not getting set : Spark3

Hi rajat I’m guessing you are setting the configuration at runtime, and correct me if I’m wrong. Only certain subset of Spark SQL properties (prefixed with spark.sql) can be set on runtime, please refer to SparkConf.scala

Kyro Serializer not getting set : Spark3

Hello Users, While using below setting getting exception spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") User class threw exception: org.apache.spark.sql.AnalysisException: Cannot modify the value of a Spark config: spark.serializer at org.apache.spark.sql.error

HELP, Populating an empty pyspark dataframe with auto-generated dates

I need help populating an empty dataframe in pyspark with auto-generated dates in a column in the format -mm-dd from 1900-01-01 to 2030-12-31. Kindly help.