Hi Matei, Yes, I'm 100% positive the jar on the executors is the same version. I am building everything and deploying myself. Additionally, while debugging the issue, I forked spark's git repo and added additional logging, which I could see in the driver and executors. These debugging jars exhibited the same behaviour.
I agree that the code being called in the executors and the driver *should* be the same, but I found that not to be the case. I forgot to mention that this issue only exhibits it's self under mesos; running in local mode or a standalone cluster (with a single worker, all processes running on my laptop) I wasn't able to reproduce the issue. The classes in question should get registered by chill-scala's AllScalaRegistrar here: https://github.com/twitter/chill/blob/0.3.6/chill-scala/src/main/scala/com/twitter/chill/ScalaKryoInstantiator.scala#L166 but, for a reason I haven't tracked down, they don't in my mesos executor. I don't really have a way to test if this is an issue specific only to my mesos cluster, or if it exhibits in all mesos clusters. fwiw, I am running spark-0.9.1 with hadoop 2.0.0-mr1-cdh4.6.0 under mesos 0.18.1 On Mon, May 12, 2014 at 12:02 PM, Matei Zaharia <matei.zaha...@gmail.com>wrote: > Hey Soren, are you sure that the JAR you used on the executors is for the > right version of Spark? Maybe they’re running an older version. The Kryo > serializer should be initialized the same way on both. > > Matei > > On May 12, 2014, at 10:39 AM, Soren Macbeth <so...@yieldbot.com> wrote: > > > I finally managed to track down the source of the kryo issues that I was > > having under mesos. > > > > What happens is the for a reason that I haven't tracked down yet, a > handful > > of the scala collection classes from chill-scala down get registered by > the > > mesos executors, but they do all get registered in the driver process. > > > > This led to scala.Some classes which were serialized by the executors > being > > incorrectly deserialized as scala.collections.Wrappers$SeqWrapper in > driver > > during task deserialization, causing a KryoException. > > > > I resolved this issue in my spark job by explicitly registering the > classes > > in my Registrator like so: > > > > > > kryo.register(scala.collection.convert.Wrappers.IteratorWrapper.class); > > kryo.register(scala.collection.convert.Wrappers.SeqWrapper.class); > > kryo.register(scala.collection.convert.Wrappers.MapWrapper.class); > > kryo.register(scala.collection.convert.Wrappers.JListWrapper.class); > > kryo.register(scala.collection.convert.Wrappers.JMapWrapper.class); > > > > Again, I'm not sure why they don't get registered in the mesos executors, > > but I wanted to report wht I found as well as a workaround in case anyone > > else hit this (extraordinarily frustrating) issue again. > > > > Some interactive debugging note are available in this gist: > > > > https://gist.github.com/sorenmacbeth/28707a7a973f7a1982dc > > > > Cheers, > > Soren > >
