[jira] [Commented] (SPARK-19938) java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field
[ https://issues.apache.org/jira/browse/SPARK-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200272#comment-17200272 ] Igor Kamyshnikov commented on SPARK-19938: -- [~rdblue], my analysis shows the different root cause of the problem: https://bugs.openjdk.java.net/browse/JDK-8024931 https://github.com/scala/bug/issues/9777 It's about circular references among the objects being serialized: RDD1.dependencies_ = Seq1[RDD2] RDD2.dependences_ = Seq2[RDD3] RDD3 with some Dataset/catalyst magic can refer back to the Seq1[RDD2] Seq are instances of scala.collection.immutable.List which uses writeReplace, giving an instance of 'SerializationProxy'. The serialization of RDD3 puts a reference to the Seq1's SerializationProxy. When the deserialization works, it reads that reference to SerializationProxy earlier than the 'readResolve' method is called (see the JDK bug reported). > java.lang.ClassCastException: cannot assign instance of > scala.collection.immutable.List$SerializationProxy to field > --- > > Key: SPARK-19938 > URL: https://issues.apache.org/jira/browse/SPARK-19938 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 2.0.2 >Reporter: srinivas thallam >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19938) java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field
[ https://issues.apache.org/jira/browse/SPARK-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16530741#comment-16530741 ] ant_nebula commented on SPARK-19938: I solve the problem.Look SPARK-24728 > java.lang.ClassCastException: cannot assign instance of > scala.collection.immutable.List$SerializationProxy to field > --- > > Key: SPARK-19938 > URL: https://issues.apache.org/jira/browse/SPARK-19938 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 2.0.2 >Reporter: srinivas thallam >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19938) java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field
[
https://issues.apache.org/jira/browse/SPARK-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105449#comment-16105449
]
Ryan Blue commented on SPARK-19938:
---
[~snavatski], I just hit this problem also and found out what causes it.
There's a [similar issue with Java serialization on a SO
question|https://stackoverflow.com/questions/9110677/readresolve-not-working-an-instance-of-guavas-serializedform-appears/18647941]
that I found helpful.
The cause of this is one of two problems during deserialization:
# The classloader can't find the class of objects in the list
# The classloader used by Java deserialization differs from the one that loaded
the class of objects in the list
These cases end up causing the deserialization code to take a path where
{{readResolve}} isn't called on the list's {{SerializationProxy}}. When the
list is set on the object that contains it, the type doesn't match and you get
this exception.
To fix this problem, check the following things:
* Make sure Jars loaded on the driver are in the executor's classpath
* Make sure Jars provided by Spark aren't included in your application (to
avoid loading with different classloaders).
> java.lang.ClassCastException: cannot assign instance of
> scala.collection.immutable.List$SerializationProxy to field
> ---
>
> Key: SPARK-19938
> URL: https://issues.apache.org/jira/browse/SPARK-19938
> Project: Spark
> Issue Type: Bug
> Components: Tests
>Affects Versions: 2.0.2
>Reporter: srinivas thallam
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19938) java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field
[ https://issues.apache.org/jira/browse/SPARK-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079985#comment-16079985 ] Siarhei Navatski commented on SPARK-19938: -- [~sthallam] Have you resolve your issue? I have the same with Idea and maven behaviour I can't mark spark dependencies as provided because I need them to run Spark in local mode > java.lang.ClassCastException: cannot assign instance of > scala.collection.immutable.List$SerializationProxy to field > --- > > Key: SPARK-19938 > URL: https://issues.apache.org/jira/browse/SPARK-19938 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 2.0.2 >Reporter: srinivas thallam > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19938) java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field
[ https://issues.apache.org/jira/browse/SPARK-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907608#comment-15907608 ] srinivas thallam commented on SPARK-19938: -- The tests are working fine in IDE(intellij) but when I run through sbt i am getting the following error.Any help would be appreciated. [info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, lo calhost): java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rd d$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD [info] at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2133) [info] at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1305) [info] at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2237) [info] at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) [info] at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) [info] at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [info] at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) [info] at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) [info] at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) [info] at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [info] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) [info] at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75) [info] at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114) [info] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) [info] at org.apache.spark.scheduler.Task.run(Task.scala:86) [info] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) [info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [info] at java.lang.Thread.run(Thread.java:745) [info] [info] Driver stacktrace: [info] at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1454) [info] at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1442) [info] at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1
