I get the same error even when I set the SPARK_CLASSPATH: export SPARK_CLASSPATH=/.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.1.jar:/.m2/repository/ch/qos/logback/logback-core/1.1.2/logback-core-1.1.2.jar And I run the job like this: /spark-1.4.1-bin-hadoop2.4/bin/spark-submit --class runner.SparkRunner target/simspark-0.1-SNAPSHOT-jar-with-dependencies.jar
I am not able to find the code in spark which adds these jars before the spark classes in classpath. Or maybe its a bug. Any suggestions on workarounds? Thanks, -Utkarsh On Mon, Aug 24, 2015 at 4:32 PM, Utkarsh Sengar <utkarsh2...@gmail.com> wrote: > I assumed that's the case beacause of the error I got and the > documentation which says: "Extra classpath entries to append to the > classpath of the driver." > > This is where I stand now: > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-core_2.10</artifactId> > <version>1.4.1</version> > <exclusions> > <exclusion> > <groupId>org.slf4j</groupId> > <artifactId>slf4j-log4j12</artifactId> > </exclusion> > </exclusions> > </dependency> > > And no exclusions from my logging lib. > > And I submit this task: spark-1.4.1-bin-hadoop2.4/bin/spark-submit --class > runner.SparkRunner --conf > "spark.driver.extraClassPath=/.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar" > --conf > "spark.executor.extraClassPath=/.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar" > --conf > "spark.driver.extraClassPath=/.m2/repository/ch/qos/logback/logback-core/1.1.2/logback-core-1.1.2.jar" > --conf > "spark.executor.extraClassPath=/.m2/repository/ch/qos/logback/logback-core/1.1.2/logback-core-1.1.2.jar" > target/simspark-0.1-SNAPSHOT-jar-with-dependencies.jar > > And I get the same error: > Caused by: java.lang.ClassCastException: org.slf4j.impl.Log4jLoggerFactory > cannot be cast to ch.qos.logback.classic.LoggerContext > at > com.opentable.logging.AssimilateForeignLogging.assimilate(AssimilateForeignLogging.java:68) > at > com.opentable.logging.AssimilateForeignLoggingHook.automaticAssimilationHook(AssimilateForeignLoggingHook.java:28) > at com.opentable.logging.Log.<clinit>(Log.java:31) > ... 16 more > > > Thanks, > -Utkarsh > > On Mon, Aug 24, 2015 at 4:11 PM, Marcelo Vanzin <van...@cloudera.com> > wrote: > >> On Mon, Aug 24, 2015 at 3:58 PM, Utkarsh Sengar <utkarsh2...@gmail.com> >> wrote: >> > That didn't work since "extraClassPath" flag was still appending the >> jars at >> > the end, so its still picking the slf4j jar provided by spark. >> >> Out of curiosity, how did you verify this? The "extraClassPath" >> options are supposed to prepend entries to the classpath, and the code >> seems to be doing that. If it's not really doing that in some case, >> it's a bug that needs to be fixed. >> >> Another option is those is setting the "SPARK_CLASSPATH" env variable, >> which is deprecated, but might come in handy in case there is actually >> a bug in handling those options. >> >> >> -- >> Marcelo >> > > > > -- > Thanks, > -Utkarsh > -- Thanks, -Utkarsh
