bq. at akka.serialization.JavaSerializer.toBinary(Serializer.scala:129) It was Akka which uses JavaSerializer
Cheers On Sat, May 7, 2016 at 11:13 AM, Nirav Patel <npa...@xactlycorp.com> wrote: > Hi, > > I thought I was using kryo serializer for shuffle. I could verify it from > spark UI - Environment tab that > spark.serializer org.apache.spark.serializer.KryoSerializer > spark.kryo.registrator > com.myapp.spark.jobs.conf.SparkSerializerRegistrator > > > But when I see following error in Driver logs it looks like spark is using > JavaSerializer > > 2016-05-06 09:49:26,490 ERROR > [sparkDriver-akka.actor.default-dispatcher-17] akka.actor.ActorSystemImpl: > Uncaught fatal error from thread > [sparkDriver-akka.remote.default-remote-dispatcher-6] shutting down > ActorSystem [sparkDriver] > > java.lang.OutOfMemoryError: Java heap space > > at java.util.Arrays.copyOf(Arrays.java:2271) > > at > java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) > > at > java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) > > at > java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) > > at > java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1876) > > at > java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1785) > > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1188) > > at > java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) > > at > akka.serialization.JavaSerializer$$anonfun$toBinary$1.apply$mcV$sp(Serializer.scala:129) > > at > akka.serialization.JavaSerializer$$anonfun$toBinary$1.apply(Serializer.scala:129) > > at > akka.serialization.JavaSerializer$$anonfun$toBinary$1.apply(Serializer.scala:129) > > at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) > > at akka.serialization.JavaSerializer.toBinary(Serializer.scala:129) > > at > akka.remote.MessageSerializer$.serialize(MessageSerializer.scala:36) > > at > akka.remote.EndpointWriter$$anonfun$serializeMessage$1.apply(Endpoint.scala:843) > > at > akka.remote.EndpointWriter$$anonfun$serializeMessage$1.apply(Endpoint.scala:843) > > at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) > > at akka.remote.EndpointWriter.serializeMessage(Endpoint.scala:842) > > at akka.remote.EndpointWriter.writeSend(Endpoint.scala:743) > > at > akka.remote.EndpointWriter$$anonfun$2.applyOrElse(Endpoint.scala:718) > > at akka.actor.Actor$class.aroundReceive(Actor.scala:467) > > at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:411) > > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) > > at akka.actor.ActorCell.invoke(ActorCell.scala:487) > > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) > > at akka.dispatch.Mailbox.run(Mailbox.scala:220) > > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) > > at > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > > > What I am missing here? > > Thanks > > > > [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/> > > <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] > <https://www.linkedin.com/company/xactly-corporation> [image: Twitter] > <https://twitter.com/Xactly> [image: Facebook] > <https://www.facebook.com/XactlyCorp> [image: YouTube] > <http://www.youtube.com/xactlycorporation>
