[ https://issues.apache.org/jira/browse/SPARK-47194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821543#comment-17821543 ]
Yang Jie edited comment on SPARK-47194 at 2/28/24 7:19 AM: ----------------------------------------------------------- It seems that the `-Dlog4j2.debug` option may not be working in 2.23.0, perhaps we should skip this upgrade. I have tested the following scenarios: 1. run `dev/make-distribution.sh --tgz` to build a Spark Client 2. add `log4j2.properties` and `spark-defaults.conf` with the same content as test case `Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties` ``` log4j2.properties # This log4j config file is for integration test SparkConfPropagateSuite. rootLogger.level = debug rootLogger.appenderRef.stdout.ref = console appender.console.type = Console appender.console.name = console appender.console.target = SYSTEM_ERR appender.console.layout.type = PatternLayout appender.console.layout.pattern = %d\{HH:mm:ss.SSS} %p %c: %maxLen\{%m} {512} %n%ex\{8}%n ``` ``` spark-defaults.conf spark.driver.extraJavaOptions -Dlog4j2.debug spark.executor.extraJavaOptions -Dlog4j2.debug spark.kubernetes.executor.deleteOnTermination false ``` 3. run `bin/run-example SparkPi` When using log4j 2.22.1, we can have the following log: ``` ... TRACE StatusLogger DefaultConfiguration cleaning Appenders from 1 LoggerConfigs. DEBUG StatusLogger Stopped org.apache.logging.log4j.core.config.DefaultConfiguration@384ad17b OK TRACE StatusLogger Reregistering MBeans after reconfigure. Selector=org.apache.logging.log4j.core.selector.ClassLoaderContextSelector@5852c06f TRACE StatusLogger Reregistering context (1/1): '5ffd2b27' org.apache.logging.log4j.core.LoggerContext@31190526 TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27' TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27,component=StatusLogger' TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27,component=ContextSelector' TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27,component=Loggers,name=*' TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27,component=Appenders,name=*' TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27,component=AsyncAppenders,name=*' TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27,component=AsyncLoggerRingBuffer' TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27,component=Loggers,name=*,subtype=RingBuffer' DEBUG StatusLogger Registering MBean org.apache.logging.log4j2:type=5ffd2b27 DEBUG StatusLogger Registering MBean org.apache.logging.log4j2:type=5ffd2b27,component=StatusLogger DEBUG StatusLogger Registering MBean org.apache.logging.log4j2:type=5ffd2b27,component=ContextSelector DEBUG StatusLogger Registering MBean org.apache.logging.log4j2:type=5ffd2b27,component=Loggers,name= DEBUG StatusLogger Registering MBean org.apache.logging.log4j2:type=5ffd2b27,component=Appenders,name=console TRACE StatusLogger Using default SystemClock for timestamps. DEBUG StatusLogger org.apache.logging.log4j.core.util.SystemClock supports precise timestamps. TRACE StatusLogger Using DummyNanoClock for nanosecond timestamps. DEBUG StatusLogger Reconfiguration complete for context[name=5ffd2b27] at URI /Users/yangjie01/Tools/4.0/spark-4.0.0-SNAPSHOT-bin-3.3.6/conf/log4j2.properties (org.apache.logging.log4j.core.LoggerContext@31190526) with optional ClassLoader: null DEBUG StatusLogger Shutdown hook enabled. Registering a new one. ... ``` But when using log4j 2.23.0, no logs related to `StatusLogger` are printed. So let's skip this upgrade was (Author: luciferyang): It seems that the `-Dlog4j2.debug` option may not be working in 2.23.0, perhaps we should skip this upgrade. I have tested the following scenarios: 1. run `dev/make-distribution.sh --tgz` to build a Spark Client 2. add `log4j2.properties` and `spark-defaults.conf` with the same content as test case `Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties` ``` log4j2.properties # This log4j config file is for integration test SparkConfPropagateSuite. rootLogger.level = debug rootLogger.appenderRef.stdout.ref = console appender.console.type = Console appender.console.name = console appender.console.target = SYSTEM_ERR appender.console.layout.type = PatternLayout appender.console.layout.pattern = %d\{HH:mm:ss.SSS} %p %c: %maxLen\{%m}{512}%n%ex\{8}%n ``` ``` spark-defaults.conf spark.driver.extraJavaOptions -Dlog4j2.debug spark.executor.extraJavaOptions -Dlog4j2.debug spark.kubernetes.executor.deleteOnTermination false ``` 3. run `bin/run-example SparkPi` When using log4j 2.22.1, we can have the following log: ``` ... TRACE StatusLogger DefaultConfiguration cleaning Appenders from 1 LoggerConfigs. DEBUG StatusLogger Stopped org.apache.logging.log4j.core.config.DefaultConfiguration@384ad17b OK TRACE StatusLogger Reregistering MBeans after reconfigure. Selector=org.apache.logging.log4j.core.selector.ClassLoaderContextSelector@5852c06f TRACE StatusLogger Reregistering context (1/1): '5ffd2b27' org.apache.logging.log4j.core.LoggerContext@31190526 TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27' TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27,component=StatusLogger' TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27,component=ContextSelector' TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27,component=Loggers,name=*' TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27,component=Appenders,name=*' TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27,component=AsyncAppenders,name=*' TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27,component=AsyncLoggerRingBuffer' TRACE StatusLogger Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=5ffd2b27,component=Loggers,name=*,subtype=RingBuffer' DEBUG StatusLogger Registering MBean org.apache.logging.log4j2:type=5ffd2b27 DEBUG StatusLogger Registering MBean org.apache.logging.log4j2:type=5ffd2b27,component=StatusLogger DEBUG StatusLogger Registering MBean org.apache.logging.log4j2:type=5ffd2b27,component=ContextSelector DEBUG StatusLogger Registering MBean org.apache.logging.log4j2:type=5ffd2b27,component=Loggers,name= DEBUG StatusLogger Registering MBean org.apache.logging.log4j2:type=5ffd2b27,component=Appenders,name=console TRACE StatusLogger Using default SystemClock for timestamps. DEBUG StatusLogger org.apache.logging.log4j.core.util.SystemClock supports precise timestamps. TRACE StatusLogger Using DummyNanoClock for nanosecond timestamps. DEBUG StatusLogger Reconfiguration complete for context[name=5ffd2b27] at URI /Users/yangjie01/Tools/4.0/spark-4.0.0-SNAPSHOT-bin-3.3.6/conf/log4j2.properties (org.apache.logging.log4j.core.LoggerContext@31190526) with optional ClassLoader: null DEBUG StatusLogger Shutdown hook enabled. Registering a new one. ... ``` But when using log4j 2.23.0, no logs related to `StatusLogger` are printed. cc @dongjoon-hyun > Upgrade log4j2 to 2.23.0 > ------------------------ > > Key: SPARK-47194 > URL: https://issues.apache.org/jira/browse/SPARK-47194 > Project: Spark > Issue Type: Sub-task > Components: Build > Affects Versions: 4.0.0 > Reporter: Yang Jie > Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org