Thanks , So if i load some static data from database and then i need to use than in my map function to filter records what will be the best way to do it,
Ashish On Wed, Jul 1, 2015 at 10:45 PM, Raghavendra Pandey < raghavendra.pan...@gmail.com> wrote: > You cannot refer to one rdd inside another rdd.map function... > Rdd object is not serialiable. Whatever objects you use inside map > function should be serializable as they get transferred to executor nodes. > On Jul 2, 2015 6:13 AM, "Ashish Soni" <asoni.le...@gmail.com> wrote: > >> Hi All , >> >> I am not sure what is the wrong with below code as it give below error >> when i access inside the map but it works outside >> >> JavaRDD<Charge> rdd2 = rdd.map(new Function<Charge, Charge>() { >> >> @Override >> public Charge call(Charge ch) throws Exception { >> >> >> * DataFrame df = accountRdd.filter("login=test");* >> >> return ch; >> } >> >> }); >> >> 5/07/01 20:38:08 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID >> 0) >> java.lang.NullPointerException >> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:129) >> at org.apache.spark.sql.DataFrame.org >> $apache$spark$sql$DataFrame$$logicalPlanToDataFrame(DataFrame.scala:154) >> >