On Fri, Jun 5, 2015 at 12:55 PM, Lee McFadden <splee...@gmail.com> wrote:
> Regarding serialization, I'm still confused as to why I was getting a > serialization error in the first place as I'm executing these Runnable > classes from a java thread pool. I'm fairly new to Scala/JVM world and > there doesn't seem to be any Spark documentation to explain *why* I need to > declare the sc variable as @transient (or even that I should). > > I was under the impression that objects only need to be serializable when > they are sent over the network, and that doesn't seem to be occurring as > far as I can tell. > You didn't show the error so the only thing we can do is speculate. You're probably sending the object that's holding the SparkContext reference over the network at some point (e.g. it's used by a task run in an executor), and that's why you were getting that exception. -- Marcelo
