foreach is something that runs on the driver, not the workers. if you want to perform some function on each record from cassandra, you need to do cassandraRdd.map(func), which will run distributed on the spark workers
*Romi Kuntsman*, *Big Data Engineer* http://www.totango.com On Mon, Sep 21, 2015 at 3:29 PM, Priya Ch <learnings.chitt...@gmail.com> wrote: > Yes, but i need to read from cassandra db within a spark > transformation..something like.. > > dstream.forachRDD{ > > rdd=> rdd.foreach { > message => > sc.cassandraTable() > . > . > . > } > } > > Since rdd.foreach gets executed on workers, how can i make sparkContext > available on workers ??? > > Regards, > Padma Ch > > On Mon, Sep 21, 2015 at 5:10 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> You can use broadcast variable for passing connection information. >> >> Cheers >> >> On Sep 21, 2015, at 4:27 AM, Priya Ch <learnings.chitt...@gmail.com> >> wrote: >> >> can i use this sparkContext on executors ?? >> In my application, i have scenario of reading from db for certain records >> in rdd. Hence I need sparkContext to read from DB (cassandra in our case), >> >> If sparkContext couldn't be sent to executors , what is the workaround >> for this ?????? >> >> On Mon, Sep 21, 2015 at 3:06 PM, Petr Novak <oss.mli...@gmail.com> wrote: >> >>> add @transient? >>> >>> On Mon, Sep 21, 2015 at 11:27 AM, Priya Ch <learnings.chitt...@gmail.com >>> > wrote: >>> >>>> Hello All, >>>> >>>> How can i pass sparkContext as a parameter to a method in an >>>> object. Because passing sparkContext is giving me TaskNotSerializable >>>> Exception. >>>> >>>> How can i achieve this ? >>>> >>>> Thanks, >>>> Padma Ch >>>> >>> >>> >> >
