This is more about mechanism of Scala compiler and Java serialization. By default, Java will serialize an object deeply and recursively. Secondly, how Scala compiler generates the byte codes does matter. I'm not a Scala expert. Here is just some observation:
1. If the function does not use any outer variable, it should be able to serialized. 2. If the function uses some outer variables in a Scala `object`, it does not require the outer `object` and variables be Serializable. 3. If the function uses some outer variables in a Scala `class` instance, this class should be Serializable because the function will have a field which refer to this outer class instance. 4. If the function uses some outer variables in a method, these variables should be Serializable because the function will have a field which refer to them. At last, "javap" is a friend to diagnose such serialization error. Best Regards, Shixiong Zhu 2014-11-06 7:56 GMT+08:00 ankits <[email protected]>: > In my spark job, I have a loop something like this: > > bla.forEachRdd(rdd => { > //init some vars > rdd.forEachPartition(partiton => { > //init some vars > partition.foreach(kv => { > ... > > I am seeing serialization errors (unread block data), because I think spark > is trying to serialize the whole containing class. But I have been careful > not to reference instance vars in the block. > > Is there a way to see exactly what class is failing serialization, and > maybe > how spark decided it needs to be serialized? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-trace-debug-serialization-tp18230.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
