Hi Marcelo,
When I add this exclusion rule to my pom:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.4.1</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
The SparkRunner class works fine (from IntelliJ) but when I build a jar and
submit it to spark-submit:
I get this error:
Caused by: java.lang.ClassCastException: org.slf4j.impl.Log4jLoggerFactory
cannot be cast to ch.qos.logback.classic.LoggerContext
at
com.opentable.logging.AssimilateForeignLogging.assimilate(AssimilateForeignLogging.java:68)
at
com.opentable.logging.AssimilateForeignLoggingHook.automaticAssimilationHook(AssimilateForeignLoggingHook.java:28)
at com.opentable.logging.Log.<clinit>(Log.java:31)
Which is this here (our logging lib is open sourced):
https://github.com/opentable/otj-logging/blob/master/logging/src/main/java/com/opentable/logging/AssimilateForeignLogging.java#L68
Thanks,
-Utkarsh
On Mon, Aug 24, 2015 at 3:04 PM, Marcelo Vanzin <[email protected]> wrote:
> Hi Utkarsh,
>
> Unfortunately that's not going to be easy. Since Spark bundles all
> dependent classes into a single fat jar file, to remove that
> dependency you'd need to modify Spark's assembly jar (potentially in
> all your nodes). Doing that per-job is even trickier, because you'd
> probably need some kind of script to inject the correct binding into
> Spark's classpath.
>
> That being said, that message is not an error, it's more of a noisy
> warning. I'd expect slf4j to use the first binding available - in your
> case, logback-classic. Is that not the case?
>
>
> On Mon, Aug 24, 2015 at 2:50 PM, Utkarsh Sengar <[email protected]>
> wrote:
> > Continuing this discussion:
> >
> http://apache-spark-user-list.1001560.n3.nabble.com/same-log4j-slf4j-error-in-spark-9-1-td5592.html
> >
> > I am getting this error when I use logback-classic.
> >
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> >
> [jar:file:.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> >
> [jar:file:.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> >
> > I need to use logback-classic for my current project, so I am trying to
> > ignore "slf4j-log4j12" from spark:
> > <dependency>
> > <groupId>org.apache.spark</groupId>
> > <artifactId>spark-core_2.10</artifactId>
> > <version>1.4.1</version>
> > <exclusions>
> > <exclusion>
> > <groupId>org.slf4j</groupId>
> > <artifactId>slf4j-log4j12</artifactId>
> > </exclusion>
> > </exclusions>
> > </dependency>
> >
> > Now, when I run my job from Intellij (which sets the classpath), things
> work
> > perfectly.
> >
> > But when I run my job via spark-submit:
> > ~/spark-1.4.1-bin-hadoop2.4/bin/spark-submit --class runner.SparkRunner
> > spark-0.1-SNAPSHOT-jar-with-dependencies.jar
> > My job fails because spark-submit sets up the classpath and it re-adds
> the
> > slf4j-log4j12.
> >
> > I am not adding spark jar to the uber-jar via the maven assembly plugin:
> > <dependencySets>
> > <dependencySet>
> > ......
> > <useTransitiveDependencies>false</useTransitiveDependencies>
> > <excludes>
> > <exclude>org.apache.spark:spark-core_2.10</exclude>
> > </excludes>
> > </dependencySet>
> > </dependencySets>
> >
> > So how can I exclude "slf4j-log4j12.jar" when I submit a job via
> > spark-submit (on a per job basis)?
> >
> > --
> > Thanks,
> > -Utkarsh
>
>
>
> --
> Marcelo
>
--
Thanks,
-Utkarsh
