How to pass sparkSession from driver to executor

[Chackravarthy Esakkimuthu]

Hi,

I want to know how to pass sparkSession from driver to executor.

I have a spark program (batch job) which does following,

#################

val spark = SparkSession.builder().appName("SampleJob").config(
"spark.master", "local") .getOrCreate()

val df = this is dataframe which has list of file names (hdfs)

df.foreach { fileName =>

      *spark.read.json(fileName)*

      ...... some logic here....
}

#################


*spark.read.json(fileName) --- this fails as it runs in executor. When I
put it outside foreach, i.e. in driver, it works.*

As I am trying to use spark (sparkSession) in executor which is not visible
outside driver. But I want to read hdfs files inside foreach, how do I do
it.

Can someone help how to do this.

Thanks,
Chackra