I agree with Sean's answer, you can check out the relevant serializer here https://github.com/twitter/chill/blob/develop/chill-scala/src/main/scala/com/twitter/chill/Traversable.scala
On Wed, Sep 28, 2016 at 3:11 AM, Sean Owen <so...@cloudera.com> wrote: > My guess is that Kryo specially handles Maps generically or relies on > some mechanism that does, and it happens to iterate over all > key/values as part of that and of course there aren't actually any > key/values in the map. The Java serialization is a much more literal > (expensive) field-by-field serialization which works here because > there's no special treatment. I think you could register a custom > serializer that handles this case. Or work around it in your client > code. I know there have been other issues with Kryo and Map because, > for example, sometimes a Map in an application is actually some > non-serializable wrapper view. > > On Wed, Sep 28, 2016 at 3:18 AM, Maciej Szymkiewicz > <mszymkiew...@gmail.com> wrote: >> Hi everyone, >> >> I suspect there is no point in submitting a JIRA to fix this (not a Spark >> issue?) but I would like to know if this problem is documented anywhere. >> Somehow Kryo is loosing default value during serialization: >> >> scala> import org.apache.spark.{SparkContext, SparkConf} >> import org.apache.spark.{SparkContext, SparkConf} >> >> scala> val aMap = Map[String, Long]().withDefaultValue(0L) >> aMap: scala.collection.immutable.Map[String,Long] = Map() >> >> scala> aMap("a") >> res6: Long = 0 >> >> scala> val sc = new SparkContext(new >> SparkConf().setAppName("bar").set("spark.serializer", >> "org.apache.spark.serializer.KryoSerializer")) >> >> scala> sc.parallelize(Seq(aMap)).map(_("a")).first >> 16/09/28 09:13:47 ERROR Executor: Exception in task 2.0 in stage 2.0 (TID 7) >> java.util.NoSuchElementException: key not found: a >> >> while Java serializer works just fine: >> >> scala> val sc = new SparkContext(new >> SparkConf().setAppName("bar").set("spark.serializer", >> "org.apache.spark.serializer.JavaSerializer")) >> >> scala> sc.parallelize(Seq(aMap)).map(_("a")).first >> res9: Long = 0 >> >> -- >> Best regards, >> Maciej > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org