In my spark job, I have a loop something like this:
bla.forEachRdd(rdd => {
//init some vars
rdd.forEachPartition(partiton => {
//init some vars
partition.foreach(kv => {
...
I am seeing serialization errors (unread block data), because I think spark
is trying to serialize the whole containing class. But I have been careful
not to reference instance vars in the block.
Is there a way to see exactly what class is failing serialization, and maybe
how spark decided it needs to be serialized?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-trace-debug-serialization-tp18230.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
