[ https://issues.apache.org/jira/browse/SPARK-19675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15877262#comment-15877262 ]
Shixiong Zhu edited comment on SPARK-19675 at 2/22/17 1:59 AM: --------------------------------------------------------------- [~taroplus] ExecutorClassLoader does try to load from its parent classloader: https://github.com/apache/spark/blob/master/repl/src/main/scala/org/apache/spark/repl/ExecutorClassLoader.scala#L77 The issue you reported is that it should load classes from the remote driver instead of the current parent classloader. Right? was (Author: zsxwing): [~taroplus] ExecutorClassLoader does try to load from its parent classloader: https://github.com/apache/spark/blob/master/repl/src/main/scala/org/apache/spark/repl/ExecutorClassLoader.scala#L77 The issue you reported is that it should load from the remote driver instead of the current parent classloader. Right? > ExecutorClassLoader loads classes from SystemClassLoader > -------------------------------------------------------- > > Key: SPARK-19675 > URL: https://issues.apache.org/jira/browse/SPARK-19675 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.1.0, 2.2.0 > Environment: sbt / Play Framework > Reporter: Kohki Nishio > Priority: Minor > > Spark Executor loads classes from SystemClassLoader which contains > sbt-launch.jar and it contains Scala2.10 binary, however Spark itself is > built on Scala2.11, thus it's throwing InvalidClassException > java.io.InvalidClassException: scala.Option; local class incompatible: stream > classdesc serialVersionUID = -114498752079829388, local class > serialVersionUID = 5081326844987135632 > at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) > at > java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630) > at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521) > at > java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630) > at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) > ExecutorClassLoader's desired class loder (parentLoader) actually contains > the correct path (scala-library-2.11.8.jar) but it is not being used. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org