[ https://issues.apache.org/jira/browse/SPARK-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14714161#comment-14714161 ]
Ram Sriharsha edited comment on SPARK-10251 at 8/26/15 3:36 PM: ---------------------------------------------------------------- as far as I can see, this happens from spark 1.2 onward. haven't gone back yet to see if this was present before spark 1.2 a temporary workaround is to register the necessary classes manually by setting the following conf property: --conf "spark.kryo.classesToRegister = [Lscala.Tuple2;" I'm looking into a better solution now: one option is to automatically register such classes (i.e. arrays of tuples, lists of tuples, None, etc) was (Author: rams): as far as I can see, this happens from spark 1.2 onward. haven't gone back yet to see if this was present before spark 1.2 a temporary workaround is to register the necessary classes manually by setting the following conf property: --conf "spark.kryo.classesToRegister = [Lscala.Tuple2;" I'm looking into a better solution now > Some internal spark classes are not registered with kryo > -------------------------------------------------------- > > Key: SPARK-10251 > URL: https://issues.apache.org/jira/browse/SPARK-10251 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.4.1 > Reporter: Soren Macbeth > Assignee: Ram Sriharsha > > When running a job using kryo serialization and setting > `spark.kryo.registrationRequired=true` some internal classes are not > registered, causing the job to die. This is still a problem when this setting > is false (which is the default) because it makes the space required to store > serialized objects in memory or disk much much more expensive in terms of > runtime and storage space. > {code} > 15/08/25 20:28:21 WARN spark.scheduler.TaskSetManager: Lost task 0.0 in stage > 0.0 (TID 0, a.b.c.d): java.lang.IllegalArgumentException: Class is not > registered: scala.Tuple2[] > Note: To register this class use: kryo.register(scala.Tuple2[].class); > at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442) > at > com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79) > at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:565) > at > org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:250) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:236) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org