Re: Linkage error - duplicate class definition
Have you solved this problem? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Linkage-error-duplicate-class-definition-tp9482p21260.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Linkage error - duplicate class definition
Facing a funny issue with the Spark class loader. Testing out a basic functionality on a vagrant VM with spark running - looks like it's attempting to ship the jar to a remote instance (in this case local) and somehow is encountering the jar twice? 14/07/11 23:27:59 INFO DAGScheduler: Got job 0 (count at GenerateSEOContent.java:75) with 1 output partitions (allowLocal=false) 14/07/11 23:27:59 INFO DAGScheduler: Final stage: Stage 0(count at GenerateSEOContent.java:75) 14/07/11 23:27:59 INFO DAGScheduler: Parents of final stage: List() 14/07/11 23:27:59 INFO DAGScheduler: Missing parents: List() 14/07/11 23:27:59 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[1] at map at GenerateSEOContent.java:67), which has no missing parents 14/07/11 23:27:59 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (MappedRDD[1] at map at GenerateSEOContent.java:67) 14/07/11 23:27:59 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks 14/07/11 23:27:59 INFO TaskSetManager: Starting task 0.0:0 as TID 0 on executor localhost: localhost (PROCESS_LOCAL) 14/07/11 23:27:59 INFO TaskSetManager: Serialized task 0.0:0 as 3287 bytes in 4 ms 14/07/11 23:27:59 INFO Executor: Running task ID 0 14/07/11 23:27:59 INFO Executor: Fetching http://10.141.141.10:36365/jars/rickshaw-spark-0.0.1-SNAPSHOT.jar with timestamp 1405121278732 14/07/11 23:27:59 INFO Utils: Fetching http://10.141.141.10:36365/jars/rickshaw-spark-0.0.1-SNAPSHOT.jar to /tmp/fetchFileTemp2298196547032055523.tmp 14/07/11 23:27:59 INFO Executor: Adding file:/tmp/spark-defa5d35-1853-492f-b8e0-e7ac30a370b1/rickshaw-spark-0.0.1-SNAPSHOT.jar to class loader 14/07/11 23:27:59 ERROR Executor: Exception in task ID 0 java.lang.LinkageError: loader (instance of org/apache/spark/executor/ChildExecutorURLClassLoader$userClassLoader$): attempted duplicate class definition for name: com/evocalize/rickshaw/spark/util/HdfsUtil at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at org.apache.spark.executor.ChildExecutorURLClassLoader$userClassLoader$.findClass(ExecutorURLClassLoader.scala:42) at org.apache.spark.executor.ChildExecutorURLClassLoader.findClass(ExecutorURLClassLoader.scala:50) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:270) at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:60) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Linkage-error-duplicate-class-definition-tp9482.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
