Thans, Igor; I've got it running again right now, and can attach the stack trace when it finishes.
In the mean time, I've noticed something interesting: in the Spark UI, the application jar that I submit is not being included on the classpath. It has been successfully uploaded to the nodes -- in the nodemanager directory for the application, I see __app__.jar and __spark__.jar. The directory itself is on the classpath, and __spark__.jar and __hadoop_conf__ are as well. When I do everything the same but switch the master to local[*], the jar I submit IS added to the classpath. This seems like a likely culprit. What could cause this, and how can I fix it? Best, Nick On Tue, Sep 8, 2015 at 1:14 AM Igor Berman <igor.ber...@gmail.com> wrote: > as a starting point, attach your stacktrace... > ps: look for duplicates in your classpath, maybe you include another jar > with same class > > On 8 September 2015 at 06:38, Nicholas R. Peterson <nrpeter...@gmail.com> > wrote: > >> I'm trying to run a Spark 1.4.1 job on my CDH5.4 cluster, through Yarn. >> Serialization is set to use Kryo. >> >> I have a large object which I send to the executors as a Broadcast. The >> object seems to serialize just fine. When it attempts to deserialize, >> though, Kryo throws a ClassNotFoundException... for a class that I include >> in the fat jar that I spark-submit. >> >> What could be causing this classpath issue with Kryo on the executors? >> Where should I even start looking to try to diagnose the problem? I >> appreciate any help you can provide. >> >> Thank you! >> >> -- Nick >> > >
