(Marcelo you might have some insight on this one)
Warning: this may just be because I'm doing something non-standard --
trying embed Spark in a Java app and feed it all the classpath it
needs manually. But this was surprising enough I wanted to ask.
I have an app that includes among other things SLF4J. I have set
spark.{driver,executor}.userClassPathFirst to true. If I run it and
let it start a Spark job, it quickly fails with:
2015-05-20 04:35:01,747 WARN TaskSetManager:71 Lost task 0.0 in stage
0.0 (TID 0, x.cloudera.com): java.lang.LinkageError: loader constraint
violation: loader (instance of
org/apache/spark/util/ChildFirstURLClassLoader) previously initiated
loading for a different type with name "org/slf4j/Logger"
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at
org.apache.spark.util.ChildFirstURLClassLoader.liftedTree1$1(MutableURLClassLoader.scala:74)
at
org.apache.spark.util.ChildFirstURLClassLoader.loadClass(MutableURLClassLoader.scala:73)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.spark.streaming.kafka.KafkaRDD.compute(KafkaRDD.scala:89)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
...
I can see that this class was loaded from my app JAR:
[Loaded org.slf4j.Logger from file:/home/sowen/oryx-batch-2.0.0-SNAPSHOT.jar]
I'm assuming it's also loaded in some Spark classloader.
Tracing the code, I don't see that it ever gets to consulting any
other classloader; this happens during its own child-first attempt to
load the class.
This didn't happen in 1.2, FWIW, when the implementation was
different, but that's only to say it was different, not correct.
Anyone have thoughts on what this indicates? something to be expected
or surprising?
I think that disabling userClassPathFirst gets rid of this of course,
although that may cause other issues later.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
