Maybe this is me misunderstanding the Spark system property behavior, but
I'm not clear why the class being loaded ends up having '/' rather than '.'
in it's fully qualified name. When I tested this out locally, the '/' were
preventing the class from being loaded.
On Fri, Jul 25, 2014 at 2:27 PM, Gary Malouf wrote:
> After upgrading to Spark 1.0.1 from 0.9.1 everything seemed to be going
> well. Looking at the Mesos slave logs, I noticed:
>
> ERROR KryoSerializer: Failed to run spark.kryo.registrator
> java.lang.ClassNotFoundException:
> com/mediacrossing/verrazano/kryo/MxDataRegistrator
>
> My spark-env.sh has the following when I run the Spark Shell:
>
> export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so
>
> export MASTER=mesos://zk://n-01:2181,n-02:2181,n-03:2181/masters
>
> export ADD_JARS=/opt/spark/mx-lib/verrazano-assembly.jar
>
>
> # -XX:+UseCompressedOops must be disabled to use more than 32GB RAM
>
> SPARK_JAVA_OPTS="-Xss2m -XX:+UseCompressedOops
> -Dspark.local.dir=/opt/mesos-tmp -Dspark.executor.memory=4g
> -Dspark.serializer=org.apache.spark.serializer.KryoSerializer
> -Dspark.kryo.registrator=com.mediacrossing.verrazano.kryo.MxDataRegistrator
> -Dspark.kryoserializer.buffer.mb=16 -Dspark.akka.askTimeout=30"
>
>
> I was able to verify that our custom jar was being copied to each worker,
> but for some reason it is not finding my registrator class. Is anyone else
> struggling with Kryo on 1.0.x branch?
>
