Thanks Qian and Vinod for response. Yes , legacy way works same as for other properties, was thinking if there is any other way.
Thanks Rajat On Fri, Sep 23, 2022 at 7:43 AM Qian SUN <qian.sun2...@gmail.com> wrote: > Hi rajat > > I’m guessing you are setting the configuration at runtime, and correct me > if I’m wrong. > Only certain subset of Spark SQL properties (prefixed with spark.sql) can > be set on runtime, please refer to SparkConf.scala > <https://github.com/apache/spark/blob/500f3097111a6bf024acf41400660c199a150350/core/src/main/scala/org/apache/spark/SparkConf.scala#L51-L52> > > Once a SparkConf object is passed to Spark, it is cloned and can no longer > be modified by the user. Spark does not support modifying the configuration > at runtime. > > So, remaining options have to be set before SparkContext is initalized. > > val spark = SparkSession.builder.config("spark.serializer", > "org.apache.spark.serializer.KryoSerializer"").getOrCreate > > > rajat kumar <kumar.rajat20...@gmail.com> 于2022年9月23日周五 05:58写道： > >> Hello Users, >> >> While using below setting getting exception >> spark.conf.set("spark.serializer", >> "org.apache.spark.serializer.KryoSerializer") >> >> User class threw exception: org.apache.spark.sql.AnalysisException: >> Cannot modify the value of a Spark config: spark.serializer at >> org.apache.spark.sql.errors.QueryCompilationErrors$.cannotModifyValueOfSparkConfigError(QueryCompilationErrors.scala:2322) >> >> Can we safely skip setting it or is there any changed way? >> >> Thanks >> Rajat >> >> >> > > -- > Best! > Qian SUN >
