[
https://issues.apache.org/jira/browse/SPARK-49145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amanda Liu updated SPARK-49145:
-------------------------------
Description:
Prior to this update, the OSS Spark logs were difficult to interpret. The logs
followed a JSON output format which is not optimal for human consumption:
{code:java}
{"ts":"2024-07-26T18:45:17.712Z","level":"INFO","msg":"Running Spark version
4.0.0-SNAPSHOT","context":{"spark_version":"4.0.0-SNAPSHOT"},"logger":"SparkContext"}{"ts":"2024-07-26T18:45:17.715Z","level":"INFO","msg":"OS
info Mac OS X, 14.4.1, aarch64","context":{"os_arch":"aarch64","os_name":"Mac
OS
X","os_version":"14.4.1"},"logger":"SparkContext"}{"ts":"2024-07-26T18:45:17.716Z","level":"INFO","msg":"Java
version
17.0.11","context":{"java_version":"17.0.11"},"logger":"SparkContext"}{"ts":"2024-07-26T18:45:17.761Z","level":"WARN","msg":"Unable
to load native-hadoop library for your platform... using builtin-java classes
where
applicable","logger":"NativeCodeLoader"}{"ts":"2024-07-26T18:45:17.783Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}{"ts":"2024-07-26T18:45:17.783Z","level":"INFO","msg":"No
custom resources configured for
spark.driver.","logger":"ResourceUtils"}{"ts":"2024-07-26T18:45:17.783Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}{"ts":"2024-07-26T18:45:17.784Z","level":"INFO","msg":"Submitted
application: Spark Pi","context":{"app_name":"Spark
Pi"},"logger":"SparkContext"}...{"ts":"2024-07-26T18:45:18.036Z","level":"INFO","msg":"Start
Jetty 0.0.0.0:4040 for
SparkUI","context":{"host":"0.0.0.0","port":"4040","server_name":"SparkUI"},"logger":"JettyUtils"}{"ts":"2024-07-26T18:45:18.044Z","level":"INFO","msg":"jetty-11.0.20;
built: 2024-01-29T21:04:22.394Z; git:
922f8dc188f7011e60d0361de585fd4ac4d63064; jvm
17.0.11+9-LTS","logger":"Server"}{"ts":"2024-07-26T18:45:18.054Z","level":"INFO","msg":"Started
Server@22c75c01{STARTING}[11.0.20,sto=30000] @1114ms","logger":"Server"} {code}
This issue updates the default `log4j.properties.template` with the following
improvements for console logging format:
* Use PatternLayout for improved human readability
* Color-code log levels to simplify logging output
* Visually partition the threadName and contextInfo for easy interpretation
was:
Prior to this update, the OSS Spark logs were difficult to interpret. The logs
followed a JSON output format which is not optimal for human consumption:
{code:java}
{"ts":"2024-07-26T18:45:17.712Z","level":"INFO","msg":"Running Spark version
4.0.0-SNAPSHOT","context":{"spark_version":"4.0.0-SNAPSHOT"},"logger":"SparkContext"}{"ts":"2024-07-26T18:45:17.715Z","level":"INFO","msg":"OS
info Mac OS X, 14.4.1, aarch64","context":{"os_arch":"aarch64","os_name":"Mac
OS
X","os_version":"14.4.1"},"logger":"SparkContext"}{"ts":"2024-07-26T18:45:17.716Z","level":"INFO","msg":"Java
version
17.0.11","context":{"java_version":"17.0.11"},"logger":"SparkContext"}{"ts":"2024-07-26T18:45:17.761Z","level":"WARN","msg":"Unable
to load native-hadoop library for your platform... using builtin-java classes
where
applicable","logger":"NativeCodeLoader"}{"ts":"2024-07-26T18:45:17.783Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}{"ts":"2024-07-26T18:45:17.783Z","level":"INFO","msg":"No
custom resources configured for
spark.driver.","logger":"ResourceUtils"}{"ts":"2024-07-26T18:45:17.783Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}{"ts":"2024-07-26T18:45:17.784Z","level":"INFO","msg":"Submitted
application: Spark Pi","context":{"app_name":"Spark
Pi"},"logger":"SparkContext"}...{"ts":"2024-07-26T18:45:18.036Z","level":"INFO","msg":"Start
Jetty 0.0.0.0:4040 for
SparkUI","context":{"host":"0.0.0.0","port":"4040","server_name":"SparkUI"},"logger":"JettyUtils"}{"ts":"2024-07-26T18:45:18.044Z","level":"INFO","msg":"jetty-11.0.20;
built: 2024-01-29T21:04:22.394Z; git:
922f8dc188f7011e60d0361de585fd4ac4d63064; jvm
17.0.11+9-LTS","logger":"Server"}{"ts":"2024-07-26T18:45:18.054Z","level":"INFO","msg":"Started
Server@22c75c01{STARTING}[11.0.20,sto=30000] @1114ms","logger":"Server"} {code}
This effort updates the default `log4j.properties.template` with the following
improvements for console logging format:
* Use PatternLayout for improved human readability
* Color-code log levels to simplify logging output
* Visually partition the threadName and contextInfo for easy interpretation
> Improve readability of log4j console log output
> -----------------------------------------------
>
> Key: SPARK-49145
> URL: https://issues.apache.org/jira/browse/SPARK-49145
> Project: Spark
> Issue Type: Task
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: Amanda Liu
> Priority: Major
>
> Prior to this update, the OSS Spark logs were difficult to interpret. The
> logs followed a JSON output format which is not optimal for human consumption:
>
> {code:java}
> {"ts":"2024-07-26T18:45:17.712Z","level":"INFO","msg":"Running Spark version
> 4.0.0-SNAPSHOT","context":{"spark_version":"4.0.0-SNAPSHOT"},"logger":"SparkContext"}{"ts":"2024-07-26T18:45:17.715Z","level":"INFO","msg":"OS
> info Mac OS X, 14.4.1,
> aarch64","context":{"os_arch":"aarch64","os_name":"Mac OS
> X","os_version":"14.4.1"},"logger":"SparkContext"}{"ts":"2024-07-26T18:45:17.716Z","level":"INFO","msg":"Java
> version
> 17.0.11","context":{"java_version":"17.0.11"},"logger":"SparkContext"}{"ts":"2024-07-26T18:45:17.761Z","level":"WARN","msg":"Unable
> to load native-hadoop library for your platform... using builtin-java
> classes where
> applicable","logger":"NativeCodeLoader"}{"ts":"2024-07-26T18:45:17.783Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}{"ts":"2024-07-26T18:45:17.783Z","level":"INFO","msg":"No
> custom resources configured for
> spark.driver.","logger":"ResourceUtils"}{"ts":"2024-07-26T18:45:17.783Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}{"ts":"2024-07-26T18:45:17.784Z","level":"INFO","msg":"Submitted
> application: Spark Pi","context":{"app_name":"Spark
> Pi"},"logger":"SparkContext"}...{"ts":"2024-07-26T18:45:18.036Z","level":"INFO","msg":"Start
> Jetty 0.0.0.0:4040 for
> SparkUI","context":{"host":"0.0.0.0","port":"4040","server_name":"SparkUI"},"logger":"JettyUtils"}{"ts":"2024-07-26T18:45:18.044Z","level":"INFO","msg":"jetty-11.0.20;
> built: 2024-01-29T21:04:22.394Z; git:
> 922f8dc188f7011e60d0361de585fd4ac4d63064; jvm
> 17.0.11+9-LTS","logger":"Server"}{"ts":"2024-07-26T18:45:18.054Z","level":"INFO","msg":"Started
> Server@22c75c01{STARTING}[11.0.20,sto=30000] @1114ms","logger":"Server"}
> {code}
>
>
> This issue updates the default `log4j.properties.template` with the following
> improvements for console logging format:
> * Use PatternLayout for improved human readability
> * Color-code log levels to simplify logging output
> * Visually partition the threadName and contextInfo for easy interpretation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
