[ https://issues.apache.org/jira/browse/SPARK-47240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gengliang Wang updated SPARK-47240: ----------------------------------- Summary: SPIP: Structured Logging Framework for Apache Spark (was: SPIP: Structured Spark Logging) > SPIP: Structured Logging Framework for Apache Spark > --------------------------------------------------- > > Key: SPARK-47240 > URL: https://issues.apache.org/jira/browse/SPARK-47240 > Project: Spark > Issue Type: New Feature > Components: Project Infra > Affects Versions: 4.0.0 > Reporter: Gengliang Wang > Priority: Major > > This proposal aims to enhance Apache Spark's logging system by implementing > structured logging. This transition will change the format of the default log > files from plain text to JSON, making them more accessible and analyzable. > The new logs will include crucial identifiers such as worker, executor, > query, job, stage, and task IDs, thereby making the logs more informative and > facilitating easier search and analysis. > h2. Current Logging Format > The current format of Spark logs is plain text, which can be challenging to > parse and analyze efficiently. An example of the current log format is as > follows: > {code:java} > 23/11/29 17:53:44 ERROR TaskSchedulerImpl: Lost executor 289 on 100.116.29.4: > Executor heartbeat timed out after 150300 ms{code} > h2. Proposed Structured Logging Format > The proposed change involves structuring the logs in JSON format, which > organizes the log information into easily identifiable fields. Here is how > the new structured log format would look: > {code:java} > {"ts": "23/11/29 17:53:44","level": "ERROR", "message": "Lost executor 289 on > 100.116.29.4: Executor heartbeat timed out after 150300 ms", "logger": > "TaskSchedulerImpl","executor_id": 289, "host": "100.116.29.4"} > {code} > This format will enable users to upload and directly query > driver/executor/master/worker log files using Spark SQL for more effective > problem-solving and analysis, such as tracking executor losses or identifying > faulty tasks: > {code:java} > spark.read.json("hdfs://hdfs_host/logs").createOrReplaceTempView("logs") > /* To get all the executor lost logs */ > SELECT * FROM logs WHERE contains(message, 'Lost executor'); > /* To get all the distributed logs about executor 289 */ > SELECT * FROM logs WHERE executor_id = 289; > /* To get all the errors on host 100.116.29.4 */ > SELECT * FROM logs WHERE host = "100.116.29.4" and log_level="ERROR"; > {code} > > SPIP doc: > [https://docs.google.com/document/d/1rATVGmFLNVLmtxSpWrEceYm7d-ocgu8ofhryVs4g3XU/edit?usp=sharing] -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org