I have scenario like this - I read dstream of messages from kafka. Now if my rdd contains 10 messages, for each message I need to query the cassandraDB, do some modification and update the records in DB. If there is no option of passing sparkContext to workers to read.write into DB, the only option is to use CassandraConnextor.withSession ???? If yes, for writing to table, should i construct the entire INSERT statement for thousands of fields in the DB ? Is this way of writing code is an optimized way ???
On Tue, Sep 22, 2015 at 1:32 AM, Romi Kuntsman <r...@totango.com> wrote: > Cody, that's a great reference! > As shown there - the best way to connect to an external database from the > workers is to create a connection pool on (each) worker. > The driver mass pass, via broadcast, the connection string, but not the > connect object itself and not the spark context. > > On Mon, Sep 21, 2015 at 5:31 PM Cody Koeninger <c...@koeninger.org> wrote: > >> That isn't accurate, I think you're confused about foreach. >> >> Look at >> >> >> http://spark.apache.org/docs/latest/streaming-programming-guide.html#design-patterns-for-using-foreachrdd >> >> >> On Mon, Sep 21, 2015 at 7:36 AM, Romi Kuntsman <r...@totango.com> wrote: >> >>> foreach is something that runs on the driver, not the workers. >>> >>> if you want to perform some function on each record from cassandra, you >>> need to do cassandraRdd.map(func), which will run distributed on the spark >>> workers >>> >>> *Romi Kuntsman*, *Big Data Engineer* >>> http://www.totango.com >>> >>> On Mon, Sep 21, 2015 at 3:29 PM, Priya Ch <learnings.chitt...@gmail.com> >>> wrote: >>> >>>> Yes, but i need to read from cassandra db within a spark >>>> transformation..something like.. >>>> >>>> dstream.forachRDD{ >>>> >>>> rdd=> rdd.foreach { >>>> message => >>>> sc.cassandraTable() >>>> . >>>> . >>>> . >>>> } >>>> } >>>> >>>> Since rdd.foreach gets executed on workers, how can i make sparkContext >>>> available on workers ??? >>>> >>>> Regards, >>>> Padma Ch >>>> >>>> On Mon, Sep 21, 2015 at 5:10 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>> >>>>> You can use broadcast variable for passing connection information. >>>>> >>>>> Cheers >>>>> >>>>> On Sep 21, 2015, at 4:27 AM, Priya Ch <learnings.chitt...@gmail.com> >>>>> wrote: >>>>> >>>>> can i use this sparkContext on executors ?? >>>>> In my application, i have scenario of reading from db for certain >>>>> records in rdd. Hence I need sparkContext to read from DB (cassandra in >>>>> our >>>>> case), >>>>> >>>>> If sparkContext couldn't be sent to executors , what is the workaround >>>>> for this ?????? >>>>> >>>>> On Mon, Sep 21, 2015 at 3:06 PM, Petr Novak <oss.mli...@gmail.com> >>>>> wrote: >>>>> >>>>>> add @transient? >>>>>> >>>>>> On Mon, Sep 21, 2015 at 11:27 AM, Priya Ch < >>>>>> learnings.chitt...@gmail.com> wrote: >>>>>> >>>>>>> Hello All, >>>>>>> >>>>>>> How can i pass sparkContext as a parameter to a method in an >>>>>>> object. Because passing sparkContext is giving me TaskNotSerializable >>>>>>> Exception. >>>>>>> >>>>>>> How can i achieve this ? >>>>>>> >>>>>>> Thanks, >>>>>>> Padma Ch >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>
