Gengliang Wang created SPARK-50036:
--------------------------------------
Summary: Include SPARK_LOG_SCHEMA in the context of repl shell
Key: SPARK-50036
URL: https://issues.apache.org/jira/browse/SPARK-50036
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 4.0.0
Reporter: Gengliang Wang
Assignee: Gengliang Wang
*Before the Change:*
Users needed to import {{LOG_SCHEMA}} to read structured logs as a JSON data
source:
import org.apache.spark.util.LogUtils.LOG_SCHEMA
val logDf = spark.read.schema(LOG_SCHEMA).json("path/to/logs")
*After the Change:*
* *No Import Needed in REPL Shells:* You can now use {{SPARK_LOG_SCHEMA}}
directly in REPL environments like {{spark-shell}} and {{pyspark}} without
importing it.
* *Renamed for Clarity:* {{LOG_SCHEMA}} has been renamed to
{{SPARK_LOG_SCHEMA}} to make its purpose more clear.
Now, you can read structured logs without the import:
val logDf = spark.read.schema(SPARK_LOG_SCHEMA).json("path/to/logs")
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
