Re: Need config params while doing rdd.foreach or map

Spark context runs in driver whereas the func inside foreach runs in executor. You can pass on the param in the func so it is available in executor On Thu, 22 Mar 2018 at 8:18 pm, Kamalanathan Venkatesan < kamalanatha...@in.ey.com> wrote: > Hello All, > > > > I have custom parameter say for

Need config params while doing rdd.foreach or map

Hello All, I have custom parameter say for example file name added to the conf of spark context example SparkConf.set(INPUT_FILE_NAME, fileName). I need this value inside foreach performed on an RDD, but the when access spark context inside foreach, I receive spark context is null exception!