Re: Exclude slf4j-log4j12 from the classpath via spark-submit

[Utkarsh Sengar]

I assumed that's the case beacause of the error I got and the documentation
which says: "Extra classpath entries to append to the classpath of the
driver."

This is where I stand now:
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>1.4.1</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

And no exclusions from my logging lib.

And I submit this task: spark-1.4.1-bin-hadoop2.4/bin/spark-submit --class
runner.SparkRunner --conf
"spark.driver.extraClassPath=/.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar"
--conf
"spark.executor.extraClassPath=/.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar"
--conf
"spark.driver.extraClassPath=/.m2/repository/ch/qos/logback/logback-core/1.1.2/logback-core-1.1.2.jar"
--conf
"spark.executor.extraClassPath=/.m2/repository/ch/qos/logback/logback-core/1.1.2/logback-core-1.1.2.jar"
target/simspark-0.1-SNAPSHOT-jar-with-dependencies.jar

And I get the same error:
Caused by: java.lang.ClassCastException: org.slf4j.impl.Log4jLoggerFactory
cannot be cast to ch.qos.logback.classic.LoggerContext
    at
com.opentable.logging.AssimilateForeignLogging.assimilate(AssimilateForeignLogging.java:68)
    at
com.opentable.logging.AssimilateForeignLoggingHook.automaticAssimilationHook(AssimilateForeignLoggingHook.java:28)
    at com.opentable.logging.Log.<clinit>(Log.java:31)
    ... 16 more


Thanks,
-Utkarsh

On Mon, Aug 24, 2015 at 4:11 PM, Marcelo Vanzin <van...@cloudera.com> wrote:

> On Mon, Aug 24, 2015 at 3:58 PM, Utkarsh Sengar <utkarsh2...@gmail.com>
> wrote:
> > That didn't work since "extraClassPath" flag was still appending the
> jars at
> > the end, so its still picking the slf4j jar provided by spark.
>
> Out of curiosity, how did you verify this? The "extraClassPath"
> options are supposed to prepend entries to the classpath, and the code
> seems to be doing that. If it's not really doing that in some case,
> it's a bug that needs to be fixed.
>
> Another option is those is setting the "SPARK_CLASSPATH" env variable,
> which is deprecated, but might come in handy in case there is actually
> a bug in handling those options.
>
>
> --
> Marcelo
>



-- 
Thanks,
-Utkarsh