A config option e.g. could just be to add: spark.logging.loadDefaultLogger (default true) If set to true, Spark will try to initialize a log4j logger if none is detected. Otherwise Spark will not modify logging behavior.
Then users could just set this to false if they have a logging set-up that conflicts with this. Maybe there is a nicer fix... On Fri, Feb 7, 2014 at 12:03 AM, Patrick Wendell <pwend...@gmail.com> wrote: > Hey Paul, > > Thanks for digging this up. I worked on this feature and the intent > was to give users good default behavior if they didn't include any > logging configuration on the classpath. > > The problem with assuming that CL tooling is going to fix the job is > that many people link against spark as a library and run their > application using their own scripts. In this case the first thing > people see when they run an application that links against Spark was a > big ugly logging warning. > > I'm not super familiar with log4j-over-slf4j, but this behavior of > returning null for the appenders seems a little weird. What is the use > case for using this and not just directly use slf4j-log4j12 like Spark > itself does? > > Did you have a more general fix for this in mind? Or was your plan to > just revert the existing behavior... We might be able to add a > configuration option to disable this logging default stuff. Or we > could just rip it out - but I'd like to avoid that if possible. > > - Patrick > > On Thu, Feb 6, 2014 at 11:41 PM, Paul Brown <p...@mult.ifario.us> wrote: >> We have a few applications that embed Spark, and in 0.8.0 and 0.8.1, we >> were able to use slf4j, but 0.9.0 broke that and unintentionally forces >> direct use of log4j as the logging backend. >> >> The issue is here in the org.apache.spark.Logging trait: >> >> https://github.com/apache/incubator-spark/blame/master/core/src/main/scala/org/apache/spark/Logging.scala#L107 >> >> log4j-over-slf4j *always* returns an empty enumeration for appenders to the >> ROOT logger: >> >> https://github.com/qos-ch/slf4j/blob/master/log4j-over-slf4j/src/main/java/org/apache/log4j/Category.java?source=c#L81 >> >> And this causes an infinite loop and an eventual stack overflow. >> >> I'm happy to submit a Jira and a patch, but it would be significant enough >> reversal of recent changes that it's probably worth discussing before I >> sink a half hour into it. My suggestion would be that initialization (or >> not) should be left to the user with reasonable default behavior supplied >> by the spark commandline tooling and not forced on applications that >> incorporate Spark. >> >> Thoughts/opinions? >> >> -- Paul >> -- >> p...@mult.ifario.us | Multifarious, Inc. | http://mult.ifario.us/
