jahnavi created SPARK-56654:
-------------------------------
Summary: Enforce strict Unicode validation in JSON parsing
Key: SPARK-56654
URL: https://issues.apache.org/jira/browse/SPARK-56654
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.0.0
Reporter: jahnavi
The JVM execution path uses a Jackson-based parser (e.g.,
ReaderBasedJsonParser), which is permissive. It does not strictly validate
Unicode surrogate pairs and will accept invalid sequences such as a lone high
surrogate (\uD835). When this happens, it silently replaces the invalid
character with a placeholder (?) and returns a parsed result.
The alternative execution like simdjson uses a stricter JSON parser this is
used in photon in Databricks. It explicitly checks for correct surrogate
pairing and rejects malformed sequences. As a result, it returns NULL for
try_parse_json or throws an error for parse_json.
Because of this difference:
The same input JSON can be rejected in one execution path and accepted (with
data corruption) in another.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]