that would violate my personal oath of never writing a single line of scala, but I might be able to do that if I can get past the issue this issue I'm struggling with in this thread.
On Sunday, May 4, 2014, Reynold Xin <r...@databricks.com> wrote: > Thanks. Do you mind playing with chill-scala a little bit and see if it > actually works well for closures? One way to try is to hard code the > serializer to use Kryo with chill-scala, and then run through all the unit > tests. > > If it works well, we can incorporate that in the next release (probably not > 1.0, but after that). > > > On Sun, May 4, 2014 at 9:08 PM, Soren Macbeth <so...@yieldbot.com> wrote: > > > fwiw, it seems like it wouldn't be very difficult to integrate > chill-scala, > > since you're already chill-java and probably get kryo serialization of > > closures and all sorts of other scala stuff for free. All that would be > > needed would be to include the dependency and then update KryoSerializer > to > > register the stuff in chill-scala. > > > > In that case, you could probably safely make kryo the default serializer, > > which I think would be desirable in general. > > > > > > On Sun, May 4, 2014 at 8:48 PM, Reynold Xin <r...@databricks.com> wrote: > > > > > Good idea. I submitted a pull request for the doc update here: > > > https://github.com/apache/spark/pull/642 > > > > > > > > > On Sun, May 4, 2014 at 3:54 PM, Soren Macbeth <so...@yieldbot.com> > > wrote: > > > > > > > Thanks for the reply! > > > > > > > > Ok, if that's the case, I'd recommend a note to that affect in the > docs > > > at > > > > least. > > > > > > > > Just to give some more context here, I'm working on a Clojure DSL for > > > Spark > > > > called Flambo, which I plan to open source shortly. If I could I'd > like > > > to > > > > focus on the initial bug that I hit. > > > > > > > > Exception in thread "main" org.apache.spark.SparkException: Job > > aborted: > > > > Exception while deserializing and fetching task: > > > > com.esotericsoftware.kryo.KryoException: > > > > java.lang.IllegalArgumentException: Can not set final > > > > scala.collection.convert.Wrappers field > > > > scala.collection.convert.Wrappers$SeqWrapper.$outer to > > > > clojure.lang.PersistentVector > > > > Serialization trace: > > > > $outer (scala.collection.convert.Wrappers$SeqWrapper) > > > > at > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020) > > > > at > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018) > > > > at > > > > > > > > > > > > > > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > > > > at > > > > scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > > > > at org.apache.spark.scheduler.DAGScheduler.org > > > > > > $apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018) > > > > at > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604) > > > > at > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604) > > > > at scala.Option.foreach(Option.scala:236) > > > > at > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:604) > > > > at > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190) > > > > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) > > > > at akka.actor.ActorCell.invoke(ActorCell.scala:456) > > > >
