Hey All, We have a fix for this but it didn't get merged yet. I'll put it as a blocker for Spark 0.9.1.
https://github.com/pwendell/incubator-spark/commit/66594e88e5be50fca073a7ef38fa62db4082b3c8 https://spark-project.atlassian.net/browse/SPARK-1190 Sergey if you could try compiling Spark with this batch and seeing if it works that would be great. Thanks, Patrick On Wed, Mar 5, 2014 at 10:26 AM, Paul Brown <[email protected]> wrote: > > Hi, Sergey -- > > Here's my recipe, implemented via Maven; YMMV if you need to do it via sbt, > etc., but it should be equivalent: > > 1) Replace org.apache.spark.Logging trait with this: > https://gist.github.com/prb/bc239b1616f5ac40b4e5 (supplied by Patrick during > the discussion on the dev list) > 2) Amend your POM using the fragment that's in the same gist. > > We build two shaded JARs from the same build, one for the driver and one for > the worker; to ensure that our Logging trait is the one in use in the driver > (where it matters), we exclude that same class from the Spark JAR in the > shade plugin configuration. > > Best. > -- Paul > > > -- > [email protected] | Multifarious, Inc. | http://mult.ifario.us/ > > > On Wed, Mar 5, 2014 at 10:02 AM, Sergey Parhomenko <[email protected]> > wrote: >> >> Hi Sean, >> >> We're not using log4j actually, we're trying to redirect all logging to >> slf4j which then uses logback as the logging implementation. >> >> The fix you mentioned - am I right to assume it is not part of the latest >> released Spark version (0.9.0)? If so, are there any workarounds or advices >> on how to avoid this issue in 0.9.0? >> >> -- >> Best regards, >> Sergey Parhomenko >> >> >> On 5 March 2014 14:40, Sean Owen <[email protected]> wrote: >>> >>> Yes I think that issue is fixed (Patrick you had the last eyes on it >>> IIRC?) >>> >>> If you are using log4j, in general, do not redirect log4j to slf4j. >>> Stuff using log4j is already using log4j, done. >>> -- >>> Sean Owen | Director, Data Science | London >>> >>> >>> On Wed, Mar 5, 2014 at 1:12 PM, Sergey Parhomenko <[email protected]> >>> wrote: >>> > Hi, >>> > >>> > I'm trying to redirect Spark logs to slf4j. Spark seem to be using >>> > Log4J, so >>> > I did the typical steps of forcing a Log4J-based framework to use slf4j >>> > - >>> > manually excluded slf4j-log4j12 and log4j, and included >>> > log4j-over-slf4j. >>> > When doing that however Spark starts failing on initialization with: >>> > java.lang.StackOverflowError >>> > at java.lang.ThreadLocal.access$400(ThreadLocal.java:72) >>> > at java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:376) >>> > at >>> > java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:261) >>> > at java.lang.ThreadLocal.get(ThreadLocal.java:146) >>> > at java.lang.StringCoding.deref(StringCoding.java:63) >>> > at java.lang.StringCoding.encode(StringCoding.java:330) >>> > at java.lang.String.getBytes(String.java:916) >>> > at java.io.UnixFileSystem.getBooleanAttributes0(Native Method) >>> > at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242) >>> > at java.io.File.exists(File.java:813) >>> > at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080) >>> > at >>> > sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java:1047) >>> > at sun.misc.URLClassPath.findResource(URLClassPath.java:176) >>> > at java.net.URLClassLoader$2.run(URLClassLoader.java:551) >>> > at java.net.URLClassLoader$2.run(URLClassLoader.java:549) >>> > at java.security.AccessController.doPrivileged(Native Method) >>> > at java.net.URLClassLoader.findResource(URLClassLoader.java:548) >>> > at java.lang.ClassLoader.getResource(ClassLoader.java:1147) >>> > at org.apache.spark.Logging$class.initializeLogging(Logging.scala:109) >>> > at >>> > org.apache.spark.Logging$class.initializeIfNecessary(Logging.scala:97) >>> > at org.apache.spark.Logging$class.log(Logging.scala:36) >>> > at org.apache.spark.util.Utils$.log(Utils.scala:47) >>> > <last 4 lines repeated many, many times> >>> > >>> > There's some related work done in SPARK-1071, but it was resolved after >>> > 0.9.0 was released. In the last comment Sean refers to a >>> > StackOverflowError >>> > which was discussed in the mailing list, I assume it might be a problem >>> > similar to mine but I was not able to find that discussion. >>> > Is anyone aware of a way to redirect Spark 0.9.0 logs to slf4j? >>> > >>> > -- >>> > Best regards, >>> > Sergey Parhomenko >> >> >
