[ https://issues.apache.org/jira/browse/SPARK-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell resolved SPARK-2646. ------------------------------------ Resolution: Fixed Fix Version/s: 1.1.0 Issue resolved by pull request 1547 [https://github.com/apache/spark/pull/1547] > log4j initialization not quite compatible with log4j 2.x > -------------------------------------------------------- > > Key: SPARK-2646 > URL: https://issues.apache.org/jira/browse/SPARK-2646 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.0.0, 1.0.1 > Reporter: Sean Owen > Priority: Minor > Fix For: 1.1.0 > > > The logging code that handles log4j initialization leads to an stack overflow > error when used with log4j 2.x, which has just been released. This occurs > even a downstream project has correctly adjusted SLF4J bindings, and that is > the right thing to do for log4j 2.x, since it is effectively a separate > project from 1.x. > Here is the relevant bit of Logging.scala: > {code} > private def initializeLogging() { > // If Log4j is being used, but is not initialized, load a default > properties file > val binder = StaticLoggerBinder.getSingleton > val usingLog4j = > binder.getLoggerFactoryClassStr.endsWith("Log4jLoggerFactory") > val log4jInitialized = > LogManager.getRootLogger.getAllAppenders.hasMoreElements > if (!log4jInitialized && usingLog4j) { > val defaultLogProps = "org/apache/spark/log4j-defaults.properties" > Option(Utils.getSparkClassLoader.getResource(defaultLogProps)) match { > case Some(url) => > PropertyConfigurator.configure(url) > log.info(s"Using Spark's default log4j profile: $defaultLogProps") > case None => > System.err.println(s"Spark was unable to load $defaultLogProps") > } > } > Logging.initialized = true > // Force a call into slf4j to initialize it. Avoids this happening from > mutliple threads > // and triggering this: > http://mailman.qos.ch/pipermail/slf4j-dev/2010-April/002956.html > log > } > {code} > The first minor issue is that there is a call to a logger inside this method, > which is initializing logging. In this situation, it ends up causing the > initialization to be called recursively until the stack overflow. It would be > slightly tidier to log this only after Logging.initialized = true. Or not at > all. But it's not the root problem, or else, it would not work at all now. > The calls to log4j classes here always reference log4j 1.2 no matter what. > For example, there is not getAllAppenders in log4j 2.x. That's fine. Really, > "usingLog4j" means "using log4j 1.2" and "log4jInitialized" means "log4j 1.2 > is initialized". > usingLog4j should be false for log4j 2.x, because the initialization only > matters for log4j 1.2. But, it's true, and that's the real issue. And > log4jInitialized is always false, since calls to the log4j 1.2 API are stubs > and no-ops in this setup, where the caller has swapped in log4j 2.x. Hence > the loop. > This is fixed, I believe, if "usingLog4j" can be false for log4j 2.x. The > SLF4J static binding class has the same name for both versions, > unfortunately, which causes the issue. However they're in different packages. > For example, if the test included "... and begins with org.slf4j", it should > work, as the SLF4J binding for log4j 2.x is provided by log4j 2.x at the > moment, and is in package org.apache.logging.slf4j. > Of course, I assume that SLF4J will eventually offer its own binding. I hope > to goodness they at least name the binding class differently, or else this > will again not work. But then some other check can probably be made. > (Credit to Agust Egilsson for finding this; at his request I'm opening a JIRA > for him. I'll propose a PR too.) -- This message was sent by Atlassian JIRA (v6.2#6252)