Re: Kryo serializer Exception during serialization: java.io.IOException: java.lang.IllegalArgumentException:
Could you disable `spark.kryo.registrationRequired`? Some classes may not be registered but they work well with Kryo's default serializer. On Fri, Jan 8, 2016 at 8:58 AM, Ted Yu wrote: > bq. try adding scala.collection.mutable.WrappedArray > > But the hint said registering > scala.collection.mutable.WrappedArray$ofRef.class > , right ? > > On Fri, Jan 8, 2016 at 8:52 AM, jiml wrote: > >> (point of post is to see if anyone has ideas about errors at end of post) >> >> In addition, the real way to test if it's working is to force >> serialization: >> >> In Java: >> >> Create array of all your classes: >> // for kyro serializer it wants to register all classes that need to be >> serialized >> Class[] kryoClassArray = new Class[]{DropResult.class, >> DropEvaluation.class, >> PrintHetSharing.class}; >> >> in the builder for your SparkConf (or in conf/spark-defaults.sh) >> .set("spark.serializer", >> "org.apache.spark.serializer.KryoSerializer") >> //require registration of all classes with Kyro >> .set("spark.kryo.registrationRequired", "true") >> // don't forget to register ALL classes or will get error >> .registerKryoClasses(kryoClassArray); >> >> Then you will start to get neat errors like the one I am working on: >> >> Exception in thread "main" org.apache.spark.SparkException: Job aborted >> due >> to stage failure: Failed to serialize task 0, not attempting to retry it. >> Exception during serialization: java.io.IOException: >> java.lang.IllegalArgumentException: Class is not registered: >> scala.collection.mutable.WrappedArray$ofRef >> Note: To register this class use: >> kryo.register(scala.collection.mutable.WrappedArray$ofRef.class); >> >> I did try adding scala.collection.mutable.WrappedArray to the Class array >> up >> top but no luck. Thanks >> >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/kryos-serializer-tp16454p25921.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
Re: Kryo serializer Exception during serialization: java.io.IOException: java.lang.IllegalArgumentException:
bq. try adding scala.collection.mutable.WrappedArray But the hint said registering scala.collection.mutable.WrappedArray$ofRef.class , right ? On Fri, Jan 8, 2016 at 8:52 AM, jiml wrote: > (point of post is to see if anyone has ideas about errors at end of post) > > In addition, the real way to test if it's working is to force > serialization: > > In Java: > > Create array of all your classes: > // for kyro serializer it wants to register all classes that need to be > serialized > Class[] kryoClassArray = new Class[]{DropResult.class, > DropEvaluation.class, > PrintHetSharing.class}; > > in the builder for your SparkConf (or in conf/spark-defaults.sh) > .set("spark.serializer", > "org.apache.spark.serializer.KryoSerializer") > //require registration of all classes with Kyro > .set("spark.kryo.registrationRequired", "true") > // don't forget to register ALL classes or will get error > .registerKryoClasses(kryoClassArray); > > Then you will start to get neat errors like the one I am working on: > > Exception in thread "main" org.apache.spark.SparkException: Job aborted due > to stage failure: Failed to serialize task 0, not attempting to retry it. > Exception during serialization: java.io.IOException: > java.lang.IllegalArgumentException: Class is not registered: > scala.collection.mutable.WrappedArray$ofRef > Note: To register this class use: > kryo.register(scala.collection.mutable.WrappedArray$ofRef.class); > > I did try adding scala.collection.mutable.WrappedArray to the Class array > up > top but no luck. Thanks > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/kryos-serializer-tp16454p25921.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: Kryo serializer Exception during serialization: java.io.IOException: java.lang.IllegalArgumentException:
(point of post is to see if anyone has ideas about errors at end of post) In addition, the real way to test if it's working is to force serialization: In Java: Create array of all your classes: // for kyro serializer it wants to register all classes that need to be serialized Class[] kryoClassArray = new Class[]{DropResult.class, DropEvaluation.class, PrintHetSharing.class}; in the builder for your SparkConf (or in conf/spark-defaults.sh) .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") //require registration of all classes with Kyro .set("spark.kryo.registrationRequired", "true") // don't forget to register ALL classes or will get error .registerKryoClasses(kryoClassArray); Then you will start to get neat errors like the one I am working on: Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Failed to serialize task 0, not attempting to retry it. Exception during serialization: java.io.IOException: java.lang.IllegalArgumentException: Class is not registered: scala.collection.mutable.WrappedArray$ofRef Note: To register this class use: kryo.register(scala.collection.mutable.WrappedArray$ofRef.class); I did try adding scala.collection.mutable.WrappedArray to the Class array up top but no luck. Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/kryos-serializer-tp16454p25921.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org