Jubin Soni created SPARK-61726:
----------------------------------

             Summary: Honor the parse mode in schema_of_json instead of 
surfacing raw Jackson exceptions on malformed input
                 Key: SPARK-61726
                 URL: https://issues.apache.org/jira/browse/SPARK-61726
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.3.0, 4.4.0
            Reporter: Jubin Soni


h2. Bug / gap
schema_of_json does not handle malformed or over-limit JSON input. 
SchemaOfJsonEvaluator.evaluate 
(sql/catalyst/.../expressions/json/JsonExpressionEvalUtils.scala) calls 
JsonInferSchema.inferField with no error handling, so Jackson's 
JsonEOFException, JsonParseException and StreamConstraintsException surface to 
the user as raw internal exceptions (a bare Py4JJavaError in PySpark). The 
'mode' option is ignored: FAILFAST and PERMISSIVE both throw the same raw 
exception. The JSON data source (JsonInferSchema.infer) and schema_of_xml 
(XmlInferSchema.infer) both route the same failures through the parse mode.
h2. Reproduction
{code}
SELECT schema_of_json('{"a":');
-- com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input 
within/between Object entries
SELECT schema_of_json('{"a":', map('mode', 'FAILFAST'));
-- com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input 
within/between Object entries
-- PySpark, default options, nesting deeper than Jackson's default limit of 
1000:
spark.range(1).select(F.schema_of_json(F.lit("[" * 1001 + "1" + "]" * 
1001))).collect()
-- com.fasterxml.jackson.core.exc.StreamConstraintsException: Document nesting 
depth (1001) exceeds the maximum allowed (1000, ...)
-- For comparison:
SELECT schema_of_xml('<a><b>');                          -- 
STRUCT<_corrupt_record: STRING>
SELECT schema_of_xml('<a><b>', map('mode','FAILFAST'));  -- 
[MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION]
spark.read.json(sc.parallelize(['{"a":'])).schema        -- 
struct<_corrupt_record:string>
{code}
Expected:
{code}
SELECT schema_of_json('{"a":');                          -- 
STRUCT<_corrupt_record: STRING>
SELECT schema_of_json('{"a":', map('mode','FAILFAST'));  -- 
[MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION] ...
{code}
h2. Why it matters
schema_of_json is typically used on a sample record to build a from_json 
schema. A truncated or malformed sample, or a deeply nested one, fails with a 
raw Jackson exception instead of a named error condition, and the documented 
parse-mode option has no effect. This is inconsistent with schema_of_xml and 
with JSON data source schema inference.
h2. Related work
SPARK-57517 fixed schema_of_json for non-string literal input (type check) but 
did not touch the evaluation path. No related ticket found: searched JIRA for 
schema_of_json, SchemaOfJson, JsonEOFException, JsonParseException, 
StreamConstraintsException, maxNestingDepth, and GitHub PRs for schema_of_json 
/ SchemaOfJsonEvaluator / JsonExpressionEvalUtils on 2026-11-14.
h2. Proposed fix
In SchemaOfJsonEvaluator.evaluate, catch JsonProcessingException (which 
includes StreamConstraintsException) and MalformedInputException, and apply the 
same parse-mode handling as JsonInferSchema.infer: PERMISSIVE returns 
STRUCT<_corrupt_record: STRING> (honoring columnNameOfCorruptRecord), 
DROPMALFORMED returns STRUCT<>, FAILFAST raises MALFORMED_RECORD_IN_PARSING. 
Reuse JsonInferSchema's handler rather than duplicating it. Add a migration 
guide note, since the default (PERMISSIVE) behaviour changes from an exception 
to a returned schema.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to