[
https://issues.apache.org/jira/browse/SPARK-48689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17856910#comment-17856910
]
Wei Guo edited comment on SPARK-48689 at 6/22/24 2:33 PM:
----------------------------------------------------------
This is controlled by option `maxStringLen`, the default value of it is
20,000,000. If you set the option big enough when reading, you can get the
right result.
{code:java}
spark.read.option("maxStringLen", 100000000).json(path){code}
I did a test with a long string of length 20,000,010 and proved that:
!image-2024-06-22-15-33-38-833.png!
was (Author: wayne guo):
This is controlled by option `maxStringLen`, the default value of it is
20,000,000. If you set the option when reading, you can get the right result.
{code:java}
spark.read.option("maxStringLen", 100000000).json(path){code}
I did a test with a long string of length 20,000,010 and proved that:
!image-2024-06-22-15-33-38-833.png!
> Reading lengthy JSON results in a corrupted record.
> ---------------------------------------------------
>
> Key: SPARK-48689
> URL: https://issues.apache.org/jira/browse/SPARK-48689
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.5.1
> Environment: Ubuntu 22.04, Python 3.11, and OpenJDK 22
> Reporter: Yuxiang Wei
> Priority: Major
> Labels: Reader
> Attachments: image-2024-06-22-15-33-38-833.png
>
>
> When reading a data frame from a JSON file including a very long string,
> spark will incorrectly make it a corrupted record even though the format is
> correct. Here is a minimal example with PySpark:
> {{import json}}
> {{import tempfile}}
> {{{}from pyspark.sql import SparkSession{}}}{{{}# Create a Spark session{}}}
> {{spark = (SparkSession.builde}}
> {{ .appName("PySpark JSON Example")}}
> {{ .getOrCreate()}}
> {{{}){}}}{{{}# Define the JSON content{}}}
> {{data = {}}
> {{ "text": "a" * 100000000}}
> {{{}}{}}}{{{}# Create a temporary file{}}}
> {{with tempfile.NamedTemporaryFile(delete=False, suffix=".json", mode="w") as
> tmp_file:}}
> {{ # Write the JSON content to the temporary file}}
> {{ tmp_file.write(json.dumps(data) + "\n")}}
> {{ tmp_file_path = tmp_file.name}}{{ # Load the JSON file into a
> PySpark DataFrame}}
> {{ df = spark.read.json(tmp_file_path)}}{{ # Print the schema}}
> {{ print(df)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
