GitHub user srowen opened a pull request:
https://github.com/apache/incubator-spark/pull/570
SPARK-1071: Tidy logging strategy and use of log4j
Prompted by a recent thread on the mailing list, I tried and failed to see
if Spark can be made independent of log4j. There are a few cases where control
of the underlying logging is pretty useful, and to do that, you have to bind to
a specific logger.
Instead I propose some tidying that leaves Spark's use of log4j, but gets
rid of warnings and should still enable downstream users to switch. The idea is
to pipe everything (except log4j) through SLF4J, and have Spark use SLF4J
directly when logging, and where Spark needs to output info (REPL and tests),
bind from SLF4J to log4j.
This leaves the same behavior in Spark. It means that downstream users who
want to use something except log4j should:
- Exclude dependencies on log4j, slf4j-log4j12 from Spark
- Include dependency on log4j-over-slf4j
- Include dependency on another logger X, and another slf4j-X
- Recreate any log config that Spark does, that is needed, in the other
logger's config
That sounds about right.
Here are the key changes:
- Include the jcl-over-slf4j shim everywhere by depending on it in core.
- Exclude dependencies on commons-logging from third-party libraries.
- Include the jul-to-slf4j shim everywhere by depending on it in core.
- Exclude slf4j-* dependencies from third-party libraries to prevent
collision or warnings
- Added missing slf4j-log4j12 binding to GraphX, Bagel module tests
And minor/incidental changes:
- Update to SLF4J 1.7.5, which happily matches Hadoop 2âs version and is
a recommended update over 1.7.2
- (Remove a duplicate HBase dependency declaration in SparkBuild.scala)
- (Remove a duplicate mockito dependency declaration that was causing
warnings and bugging me)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-spark SPARK-1071
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-spark/pull/570.patch
----
commit 4f27b339cb12a61e70d61d99594cca00340f31c8
Author: Sean Owen <so...@cloudera.com>
Date: 2014-02-09T22:51:17Z
SPARK-1071: Tidy logging strategy and use of log4j
----
