Hi Utkarsh, A quick look at slf4j's source shows it loads the first "StaticLoggerBinder" in your classpath. How are you adding the logback jar file to spark-submit?
If you use "spark.driver.extraClassPath" and "spark.executor.extraClassPath" to add the jar, it should take precedence over the log4j binding embedded in the Spark assembly. On Mon, Aug 24, 2015 at 3:15 PM, Utkarsh Sengar <utkarsh2...@gmail.com> wrote: > Hi Marcelo, > > When I add this exclusion rule to my pom: > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-core_2.10</artifactId> > <version>1.4.1</version> > <exclusions> > <exclusion> > <groupId>org.slf4j</groupId> > <artifactId>slf4j-log4j12</artifactId> > </exclusion> > </exclusions> > </dependency> > > The SparkRunner class works fine (from IntelliJ) but when I build a jar and > submit it to spark-submit: > > I get this error: > Caused by: java.lang.ClassCastException: org.slf4j.impl.Log4jLoggerFactory > cannot be cast to ch.qos.logback.classic.LoggerContext > at > com.opentable.logging.AssimilateForeignLogging.assimilate(AssimilateForeignLogging.java:68) > at > com.opentable.logging.AssimilateForeignLoggingHook.automaticAssimilationHook(AssimilateForeignLoggingHook.java:28) > at com.opentable.logging.Log.<clinit>(Log.java:31) > > Which is this here (our logging lib is open sourced): > https://github.com/opentable/otj-logging/blob/master/logging/src/main/java/com/opentable/logging/AssimilateForeignLogging.java#L68 > > Thanks, > -Utkarsh > > > > > On Mon, Aug 24, 2015 at 3:04 PM, Marcelo Vanzin <van...@cloudera.com> wrote: >> >> Hi Utkarsh, >> >> Unfortunately that's not going to be easy. Since Spark bundles all >> dependent classes into a single fat jar file, to remove that >> dependency you'd need to modify Spark's assembly jar (potentially in >> all your nodes). Doing that per-job is even trickier, because you'd >> probably need some kind of script to inject the correct binding into >> Spark's classpath. >> >> That being said, that message is not an error, it's more of a noisy >> warning. I'd expect slf4j to use the first binding available - in your >> case, logback-classic. Is that not the case? >> >> >> On Mon, Aug 24, 2015 at 2:50 PM, Utkarsh Sengar <utkarsh2...@gmail.com> >> wrote: >> > Continuing this discussion: >> > >> > http://apache-spark-user-list.1001560.n3.nabble.com/same-log4j-slf4j-error-in-spark-9-1-td5592.html >> > >> > I am getting this error when I use logback-classic. >> > >> > SLF4J: Class path contains multiple SLF4J bindings. >> > SLF4J: Found binding in >> > >> > [jar:file:.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> > SLF4J: Found binding in >> > >> > [jar:file:.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> > >> > I need to use logback-classic for my current project, so I am trying to >> > ignore "slf4j-log4j12" from spark: >> > <dependency> >> > <groupId>org.apache.spark</groupId> >> > <artifactId>spark-core_2.10</artifactId> >> > <version>1.4.1</version> >> > <exclusions> >> > <exclusion> >> > <groupId>org.slf4j</groupId> >> > <artifactId>slf4j-log4j12</artifactId> >> > </exclusion> >> > </exclusions> >> > </dependency> >> > >> > Now, when I run my job from Intellij (which sets the classpath), things >> > work >> > perfectly. >> > >> > But when I run my job via spark-submit: >> > ~/spark-1.4.1-bin-hadoop2.4/bin/spark-submit --class runner.SparkRunner >> > spark-0.1-SNAPSHOT-jar-with-dependencies.jar >> > My job fails because spark-submit sets up the classpath and it re-adds >> > the >> > slf4j-log4j12. >> > >> > I am not adding spark jar to the uber-jar via the maven assembly plugin: >> > <dependencySets> >> > <dependencySet> >> > ...... >> > <useTransitiveDependencies>false</useTransitiveDependencies> >> > <excludes> >> > <exclude>org.apache.spark:spark-core_2.10</exclude> >> > </excludes> >> > </dependencySet> >> > </dependencySets> >> > >> > So how can I exclude "slf4j-log4j12.jar" when I submit a job via >> > spark-submit (on a per job basis)? >> > >> > -- >> > Thanks, >> > -Utkarsh >> >> >> >> -- >> Marcelo > > > > > -- > Thanks, > -Utkarsh -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
