Hi, I am trying to migrate one spark application from Spark 2.3 to 3.0.1.
The issue can be reproduced using below sample code: object TestMain { val session = SparkSession.builder().appName("test").enableHiveSupport().getOrCreate() def main(args: Array[String]): Unit = { import session.implicits._ val a = *session.*sparkContext.parallelize(*Array*(("A",1),("B",2))).toDF("_ c1","_c2").*rdd*.map(x=> x(0).toString).collect() *println*(a.mkString("|")) } } It runs successfully in Spark 2.3 but fails with Spark 3.0.1 with below exception: Caused by: org.apache.spark.SparkException: A master URL must be set in your configuration at org.apache.spark.SparkContext.<init>(SparkContext.scala:394) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690) at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943) at TestMain$.<init>(TestMain.scala:7) at TestMain$.<clinit>(TestMain.scala) >From the exception it appears that it tries to create spark session on executor also in Spark 3 whereas its not created again on executor in Spark 2.3. Can anyone help in identfying why there is this change in behavior? Thanks and Regards, Shrikant -- Regards, Shrikant Prasad