Could someone please help me.
On Thu, Oct 17, 2019 at 7:29 PM Chetan Khatri <[email protected]>
wrote:
> Hi Users,
>
> I am setting spark configuration in below way;
>
> val spark = SparkSession.builder().appName(APP_NAME).getOrCreate()
>
> spark.conf.set("spark.speculation", "false")
> spark.conf.set("spark.broadcast.compress", "true")
> spark.conf.set("spark.sql.broadcastTimeout", "36000")
> spark.conf.set("spark.network.timeout", "2500s")
> spark.conf.set("spark.serializer",
> "org.apache.spark.serializer.KryoSerializer")
> spark.conf.set("spark.driver.memory", "10g")
> spark.conf.set("spark.executor.memory", "10g")
>
> import spark.implicits._
>
>
> and submitting spark job with spark - submit. but none of the above
> configuration is
>
> getting reflected to the job, I have checked at Spark-UI.
>
> I know setting up like this while creation of spark object, it's working well.
>
>
> val spark = SparkSession.builder().appName(APP_NAME)
> .config("spark.network.timeout", "1500s")
> .config("spark.broadcast.compress", "true")
> .config("spark.sql.broadcastTimeout", "36000")
> .getOrCreate()
>
> import spark.implicits._
>
>
> Can someone please throw light?
>
>
