I actually only ran into this issue recently after we upgraded to Spark
1.1. Within the REPL for Spark 1.0 everything works fine but within the
REPL for 1.1, it is not. FYI I am also only doing simple regex matching
functions within an RDD... Now when I am running the same code as App
everything is working fine... it leads me to believe that it is a bug
within the REPL for 1.1....
Can anyone else confirm this?
ᐧ
*JIMMY MCERLAIN*
DATA SCIENTIST (NERD)
*. . . . . . . . . . . . . . . . . .*
*IF WE CAN’T DOUBLE YOUR SALES,*
*ONE OF US IS IN THE WRONG BUSINESS.*
*E*: ji...@sellpoints.com
*M*: *510.303.7751*
On Thu, Oct 16, 2014 at 7:56 AM, Akshat Aranya <aara...@gmail.com> wrote:
> Hi,
>
> Can anyone explain how things get captured in a closure when runing
> through the REPL. For example:
>
> def foo(..) = { .. }
>
> rdd.map(foo)
>
> sometimes complains about classes not being serializable that are
> completely unrelated to foo. This happens even when I write it such:
>
> object Foo {
> def foo(..) = { .. }
> }
>
> rdd.map(Foo.foo)
>
> It also doesn't happen all the time.
>
