Hi Patrick, Thanks for the patch. I tried building a patched version of spark-core_2.10-0.9.0-incubating.jar but the Maven build fails: *[ERROR] /home/das/Work/thx/incubator-spark/core/src/main/scala/org/apache/spark/Logging.scala:22: object impl is not a member of package org.slf4j* *[ERROR] import org.slf4j.impl.StaticLoggerBinder* *[ERROR] ^* *[ERROR] /home/das/Work/thx/incubator-spark/core/src/main/scala/org/apache/spark/Logging.scala:106: not found: value StaticLoggerBinder* *[ERROR] val binder = StaticLoggerBinder.getSingleton* *[ERROR] ^* *[ERROR] two errors found*
The module only has compile dependency on slf4j-api, and not on slf4j-log4j12 or any other slf4j logging modules which provide org.slf4j.impl.StaticLoggerBinder*. *Adding slf4j-log4j12 with compile scope helps, and I confirm the logging is redirected to slf4j/Logback correctly now with the patched module. I'm not sure however if using compile scope for slf4j-log4j12 is a good idea. -- Best regards, Sergey Parhomenko On 5 March 2014 20:11, Patrick Wendell <[email protected]> wrote: > Hey All, > > We have a fix for this but it didn't get merged yet. I'll put it as a > blocker for Spark 0.9.1. > > > https://github.com/pwendell/incubator-spark/commit/66594e88e5be50fca073a7ef38fa62db4082b3c8 > > https://spark-project.atlassian.net/browse/SPARK-1190 > > Sergey if you could try compiling Spark with this batch and seeing if > it works that would be great. > > Thanks, > Patrick > > > On Wed, Mar 5, 2014 at 10:26 AM, Paul Brown <[email protected]> wrote: > > > > Hi, Sergey -- > > > > Here's my recipe, implemented via Maven; YMMV if you need to do it via > sbt, > > etc., but it should be equivalent: > > > > 1) Replace org.apache.spark.Logging trait with this: > > https://gist.github.com/prb/bc239b1616f5ac40b4e5 (supplied by Patrick > during > > the discussion on the dev list) > > 2) Amend your POM using the fragment that's in the same gist. > > > > We build two shaded JARs from the same build, one for the driver and one > for > > the worker; to ensure that our Logging trait is the one in use in the > driver > > (where it matters), we exclude that same class from the Spark JAR in the > > shade plugin configuration. > > > > Best. > > -- Paul > > > > > > -- > > [email protected] | Multifarious, Inc. | http://mult.ifario.us/ > > > > > > On Wed, Mar 5, 2014 at 10:02 AM, Sergey Parhomenko < > [email protected]> > > wrote: > >> > >> Hi Sean, > >> > >> We're not using log4j actually, we're trying to redirect all logging to > >> slf4j which then uses logback as the logging implementation. > >> > >> The fix you mentioned - am I right to assume it is not part of the > latest > >> released Spark version (0.9.0)? If so, are there any workarounds or > advices > >> on how to avoid this issue in 0.9.0? > >> > >> -- > >> Best regards, > >> Sergey Parhomenko > >> > >> > >> On 5 March 2014 14:40, Sean Owen <[email protected]> wrote: > >>> > >>> Yes I think that issue is fixed (Patrick you had the last eyes on it > >>> IIRC?) > >>> > >>> If you are using log4j, in general, do not redirect log4j to slf4j. > >>> Stuff using log4j is already using log4j, done. > >>> -- > >>> Sean Owen | Director, Data Science | London > >>> > >>> > >>> On Wed, Mar 5, 2014 at 1:12 PM, Sergey Parhomenko < > [email protected]> > >>> wrote: > >>> > Hi, > >>> > > >>> > I'm trying to redirect Spark logs to slf4j. Spark seem to be using > >>> > Log4J, so > >>> > I did the typical steps of forcing a Log4J-based framework to use > slf4j > >>> > - > >>> > manually excluded slf4j-log4j12 and log4j, and included > >>> > log4j-over-slf4j. > >>> > When doing that however Spark starts failing on initialization with: > >>> > java.lang.StackOverflowError > >>> > at java.lang.ThreadLocal.access$400(ThreadLocal.java:72) > >>> > at > java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:376) > >>> > at > >>> > java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:261) > >>> > at java.lang.ThreadLocal.get(ThreadLocal.java:146) > >>> > at java.lang.StringCoding.deref(StringCoding.java:63) > >>> > at java.lang.StringCoding.encode(StringCoding.java:330) > >>> > at java.lang.String.getBytes(String.java:916) > >>> > at java.io.UnixFileSystem.getBooleanAttributes0(Native Method) > >>> > at > java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242) > >>> > at java.io.File.exists(File.java:813) > >>> > at > sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080) > >>> > at > >>> > sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java:1047) > >>> > at sun.misc.URLClassPath.findResource(URLClassPath.java:176) > >>> > at java.net.URLClassLoader$2.run(URLClassLoader.java:551) > >>> > at java.net.URLClassLoader$2.run(URLClassLoader.java:549) > >>> > at java.security.AccessController.doPrivileged(Native Method) > >>> > at java.net.URLClassLoader.findResource(URLClassLoader.java:548) > >>> > at java.lang.ClassLoader.getResource(ClassLoader.java:1147) > >>> > at > org.apache.spark.Logging$class.initializeLogging(Logging.scala:109) > >>> > at > >>> > > org.apache.spark.Logging$class.initializeIfNecessary(Logging.scala:97) > >>> > at org.apache.spark.Logging$class.log(Logging.scala:36) > >>> > at org.apache.spark.util.Utils$.log(Utils.scala:47) > >>> > <last 4 lines repeated many, many times> > >>> > > >>> > There's some related work done in SPARK-1071, but it was resolved > after > >>> > 0.9.0 was released. In the last comment Sean refers to a > >>> > StackOverflowError > >>> > which was discussed in the mailing list, I assume it might be a > problem > >>> > similar to mine but I was not able to find that discussion. > >>> > Is anyone aware of a way to redirect Spark 0.9.0 logs to slf4j? > >>> > > >>> > -- > >>> > Best regards, > >>> > Sergey Parhomenko > >> > >> > > >
