Yes, but i need to read from cassandra db within a spark transformation..something like..
dstream.forachRDD{ rdd=> rdd.foreach { message => sc.cassandraTable() . . . } } Since rdd.foreach gets executed on workers, how can i make sparkContext available on workers ??? Regards, Padma Ch On Mon, Sep 21, 2015 at 5:10 PM, Ted Yu <yuzhih...@gmail.com> wrote: > You can use broadcast variable for passing connection information. > > Cheers > > On Sep 21, 2015, at 4:27 AM, Priya Ch <learnings.chitt...@gmail.com> > wrote: > > can i use this sparkContext on executors ?? > In my application, i have scenario of reading from db for certain records > in rdd. Hence I need sparkContext to read from DB (cassandra in our case), > > If sparkContext couldn't be sent to executors , what is the workaround for > this ?????? > > On Mon, Sep 21, 2015 at 3:06 PM, Petr Novak <oss.mli...@gmail.com> wrote: > >> add @transient? >> >> On Mon, Sep 21, 2015 at 11:27 AM, Priya Ch <learnings.chitt...@gmail.com> >> wrote: >> >>> Hello All, >>> >>> How can i pass sparkContext as a parameter to a method in an object. >>> Because passing sparkContext is giving me TaskNotSerializable Exception. >>> >>> How can i achieve this ? >>> >>> Thanks, >>> Padma Ch >>> >> >> >