`func1` and `func2` never get serialized. They must exist on the other
end in the form of a class loaded by the JVM.
What gets serialized is an instance of a particular closure (the
argument to your "map" function). That's a separate class. The
instance of that class that is serialized contains references to all
other instances it needs to execute its "apply" method (or "run" or
whatever is the correct method name). In this case, nothing is needed,
since all it does is pass its argument in a call to a static method
(Util.func1).
Hope that helps, these things can be really confusing. You can play
with "javap -c" to disassemble the class files to understand better
how it all happens under the hood.
On Mon, Feb 9, 2015 at 1:56 PM, Yitong Zhou <timyit...@gmail.com> wrote:
> If we define an Utils object:
>
> object Utils {
> def func1 = {..}
> def func2 = {..}
> }
>
> And then in a RDD we refer to one of the function:
>
> rdd.map{r => Utils.func1(r)}
>
> Will Utils.func2 also get serialized or not?
>
> Thanks,
> Yitong
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Will-Spark-serialize-an-entire-Object-or-just-the-method-referred-in-an-object-tp21566.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
