I assumed that's the case beacause of the error I got and the documentation
which says: "Extra classpath entries to append to the classpath of the
driver."
This is where I stand now:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.4.1</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
And no exclusions from my logging lib.
And I submit this task: spark-1.4.1-bin-hadoop2.4/bin/spark-submit --class
runner.SparkRunner --conf
"spark.driver.extraClassPath=/.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar"
--conf
"spark.executor.extraClassPath=/.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar"
--conf
"spark.driver.extraClassPath=/.m2/repository/ch/qos/logback/logback-core/1.1.2/logback-core-1.1.2.jar"
--conf
"spark.executor.extraClassPath=/.m2/repository/ch/qos/logback/logback-core/1.1.2/logback-core-1.1.2.jar"
target/simspark-0.1-SNAPSHOT-jar-with-dependencies.jar
And I get the same error:
Caused by: java.lang.ClassCastException: org.slf4j.impl.Log4jLoggerFactory
cannot be cast to ch.qos.logback.classic.LoggerContext
at
com.opentable.logging.AssimilateForeignLogging.assimilate(AssimilateForeignLogging.java:68)
at
com.opentable.logging.AssimilateForeignLoggingHook.automaticAssimilationHook(AssimilateForeignLoggingHook.java:28)
at com.opentable.logging.Log.<clinit>(Log.java:31)
... 16 more
Thanks,
-Utkarsh
On Mon, Aug 24, 2015 at 4:11 PM, Marcelo Vanzin <[email protected]> wrote:
> On Mon, Aug 24, 2015 at 3:58 PM, Utkarsh Sengar <[email protected]>
> wrote:
> > That didn't work since "extraClassPath" flag was still appending the
> jars at
> > the end, so its still picking the slf4j jar provided by spark.
>
> Out of curiosity, how did you verify this? The "extraClassPath"
> options are supposed to prepend entries to the classpath, and the code
> seems to be doing that. If it's not really doing that in some case,
> it's a bug that needs to be fixed.
>
> Another option is those is setting the "SPARK_CLASSPATH" env variable,
> which is deprecated, but might come in handy in case there is actually
> a bug in handling those options.
>
>
> --
> Marcelo
>
--
Thanks,
-Utkarsh
