rdd count is throwing null pointer exception
Hi All, Thank you very much for the detailed explanation. I have scenario like this- I have rdd of ticket records and another rdd of booking records. for each ticket record, i need to check whether any link exists in booking table. val ticketCachedRdd = ticketRdd.cache ticketRdd.foreach{ ticket => val bookingRecords = queryOnBookingTable (date, flightNumber, flightCarrier) // this function queries the booking table and retrieves the booking rows println(ticketCachedRdd.count) // this is throwing Null pointer exception } Is there somthing wrong in the count, i am trying to use the count of cached rdd when looping through the actual rdd. whats wrong in this ? Thanks, Padma Ch
Re: rdd count is throwing null pointer exception
The error could be because of the missing brackets after the word cache - .ticketRdd.cache() > On Aug 17, 2015, at 7:26 AM, Priya Ch wrote: > > Hi All, > > Thank you very much for the detailed explanation. > > I have scenario like this- > I have rdd of ticket records and another rdd of booking records. for each > ticket record, i need to check whether any link exists in booking table. > > val ticketCachedRdd = ticketRdd.cache > > ticketRdd.foreach{ > ticket => > val bookingRecords = queryOnBookingTable (date, flightNumber, flightCarrier) > // this function queries the booking table and retrieves the booking rows > println(ticketCachedRdd.count) // this is throwing Null pointer exception > > } > > Is there somthing wrong in the count, i am trying to use the count of cached > rdd when looping through the actual rdd. whats wrong in this ? > > Thanks, > Padma Ch - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: rdd count is throwing null pointer exception
Looks like because of Spark-5063 RDD transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(x => rdd2.values.count() * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more information, see SPARK-5063. On Mon, Aug 17, 2015 at 8:13 PM, Preetam wrote: > The error could be because of the missing brackets after the word cache - > .ticketRdd.cache() > > > On Aug 17, 2015, at 7:26 AM, Priya Ch > wrote: > > > > Hi All, > > > > Thank you very much for the detailed explanation. > > > > I have scenario like this- > > I have rdd of ticket records and another rdd of booking records. for > each ticket record, i need to check whether any link exists in booking > table. > > > > val ticketCachedRdd = ticketRdd.cache > > > > ticketRdd.foreach{ > > ticket => > > val bookingRecords = queryOnBookingTable (date, flightNumber, > flightCarrier) // this function queries the booking table and retrieves > the booking rows > > println(ticketCachedRdd.count) // this is throwing Null pointer exception > > > > } > > > > Is there somthing wrong in the count, i am trying to use the count of > cached rdd when looping through the actual rdd. whats wrong in this ? > > > > Thanks, > > Padma Ch >
Re: rdd count is throwing null pointer exception
Move your count operation outside the foreach and use a broadcast to access it inside the foreach. On Aug 17, 2015 10:34 AM, "Priya Ch" wrote: > Looks like because of Spark-5063 > RDD transformations and actions can only be invoked by the driver, not > inside of other transformations; for example, rdd1.map(x => > rdd2.values.count() * x) is invalid because the values transformation and > count action cannot be performed inside of the rdd1.map transformation. For > more information, see SPARK-5063. > > On Mon, Aug 17, 2015 at 8:13 PM, Preetam wrote: > >> The error could be because of the missing brackets after the word cache - >> .ticketRdd.cache() >> >> > On Aug 17, 2015, at 7:26 AM, Priya Ch >> wrote: >> > >> > Hi All, >> > >> > Thank you very much for the detailed explanation. >> > >> > I have scenario like this- >> > I have rdd of ticket records and another rdd of booking records. for >> each ticket record, i need to check whether any link exists in booking >> table. >> > >> > val ticketCachedRdd = ticketRdd.cache >> > >> > ticketRdd.foreach{ >> > ticket => >> > val bookingRecords = queryOnBookingTable (date, flightNumber, >> flightCarrier) // this function queries the booking table and retrieves >> the booking rows >> > println(ticketCachedRdd.count) // this is throwing Null pointer >> exception >> > >> > } >> > >> > Is there somthing wrong in the count, i am trying to use the count of >> cached rdd when looping through the actual rdd. whats wrong in this ? >> > >> > Thanks, >> > Padma Ch >> > >