You cannot refer to one rdd inside another rdd.map function...
Rdd object is not serialiable. Whatever objects you use inside map
function should be serializable as they get transferred to executor nodes.
On Jul 2, 2015 6:13 AM, "Ashish Soni" <asoni.le...@gmail.com> wrote:
> Hi All ,
>
> I am not sure what is the wrong with below code as it give below error
> when i access inside the map but it works outside
>
> JavaRDD<Charge> rdd2 = rdd.map(new Function<Charge, Charge>() {
>
> @Override
> public Charge call(Charge ch) throws Exception {
>
>
> * DataFrame df = accountRdd.filter("login=test");*
>
> return ch;
> }
>
> });
>
> 5/07/01 20:38:08 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
> java.lang.NullPointerException
> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:129)
> at org.apache.spark.sql.DataFrame.org
> $apache$spark$sql$DataFrame$$logicalPlanToDataFrame(DataFrame.scala:154)
>
