I'm trying to run a Spark 1.4.1 job on my CDH5.4 cluster, through Yarn. Serialization is set to use Kryo.
I have a large object which I send to the executors as a Broadcast. The object seems to serialize just fine. When it attempts to deserialize, though, Kryo throws a ClassNotFoundException... for a class that I include in the fat jar that I spark-submit. What could be causing this classpath issue with Kryo on the executors? Where should I even start looking to try to diagnose the problem? I appreciate any help you can provide. Thank you! -- Nick