Here is the definition of EsDoc
case class EsDoc(id: Long, isExample: Boolean, docSetIds: Array[String],
randomId: Double, vector: String) extends Serializable
Note that it is not EsDoc having problem with registration. It is the EsDoc[]
(the array class of EsDoc) that has problem with registration.
I have tried to replace the class EsDoc by the Map class, I also got the
following error ask me to register the Map[] (array of Map) class
java.lang.IllegalArgumentException: Class is not registered:
scala.collection.immutable.Map[]
Note: To register this class use:
kryo.register(scala.collection.immutable.Map[].class);
So the question is how to register Array class? Adding the following in
spark-defauls.conf does not work
spark.kryo.classesToRegister
scala.collection.immutable.Map,scala.collection.immutable.Map[]
Ningjun
From: Ted Yu [mailto:yuzhih...@gmail.com]
Sent: Friday, July 31, 2015 11:49 AM
To: Wang, Ningjun (LNG-NPV)
Cc: user@spark.apache.org
Subject: Re: How to register array class with Kyro in spark-defaults.conf
For the second exception, was there anything following SparkException which
would give us more clue ?
Can you tell us how EsDoc is structured ?
Thanks
On Fri, Jul 31, 2015 at 8:42 AM, Wang, Ningjun (LNG-NPV)
ningjun.w...@lexisnexis.commailto:ningjun.w...@lexisnexis.com wrote:
Does anybody have any idea how to solve this problem?
Ningjun
From: Wang, Ningjun (LNG-NPV)
Sent: Thursday, July 30, 2015 11:06 AM
To: user@spark.apache.orgmailto:user@spark.apache.org
Subject: How to register array class with Kyro in spark-defaults.conf
I register my class with Kyro in spark-defaults.conf as follow
spark.serializer
org.apache.spark.serializer.KryoSerializer
spark.kryo.registrationRequired true
spark.kryo.classesToRegister ltn.analytics.es.EsDoc
But I got the following exception
java.lang.IllegalArgumentException: Class is not registered:
ltn.analytics.es.EsDoc[]
Note: To register this class use: kryo.register(ltn.analytics.es.EsDoc[].class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:565)
at
org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:162)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
The error message seems to suggest that I should also register the array class
EsDoc[]. So I add it to spark-defaults.conf as follow
spark.kryo.classesToRegister ltn.analytics.es.EsDoc,ltn.analytics.es.EsDoc[]
Then I got the following error
org.apache.spark.SparkException: Failed to register classes with Kryo
at
org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:101)
at
org.apache.spark.serializer.KryoSerializerInstance.init(KryoSerializer.scala:153)
at
org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:115)
at
org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:200)
at
org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:101)
at
org.apache.spark.broadcast.TorrentBroadcast.init(TorrentBroadcast.scala:84)
at
org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
at
org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29)
at
org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1051)
at ltn.analytics.index.Index.addDocuments(Index.scala:82)
Please advise.
Thanks.
Ningjun
