Jubin Soni created SPARK-61726:
----------------------------------
Summary: Honor the parse mode in schema_of_json instead of
surfacing raw Jackson exceptions on malformed input
Key: SPARK-61726
URL: https://issues.apache.org/jira/browse/SPARK-61726
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.3.0, 4.4.0
Reporter: Jubin Soni
h2. Bug / gap
schema_of_json does not handle malformed or over-limit JSON input.
SchemaOfJsonEvaluator.evaluate
(sql/catalyst/.../expressions/json/JsonExpressionEvalUtils.scala) calls
JsonInferSchema.inferField with no error handling, so Jackson's
JsonEOFException, JsonParseException and StreamConstraintsException surface to
the user as raw internal exceptions (a bare Py4JJavaError in PySpark). The
'mode' option is ignored: FAILFAST and PERMISSIVE both throw the same raw
exception. The JSON data source (JsonInferSchema.infer) and schema_of_xml
(XmlInferSchema.infer) both route the same failures through the parse mode.
h2. Reproduction
{code}
SELECT schema_of_json('{"a":');
-- com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input
within/between Object entries
SELECT schema_of_json('{"a":', map('mode', 'FAILFAST'));
-- com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input
within/between Object entries
-- PySpark, default options, nesting deeper than Jackson's default limit of
1000:
spark.range(1).select(F.schema_of_json(F.lit("[" * 1001 + "1" + "]" *
1001))).collect()
-- com.fasterxml.jackson.core.exc.StreamConstraintsException: Document nesting
depth (1001) exceeds the maximum allowed (1000, ...)
-- For comparison:
SELECT schema_of_xml('<a><b>'); --
STRUCT<_corrupt_record: STRING>
SELECT schema_of_xml('<a><b>', map('mode','FAILFAST')); --
[MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION]
spark.read.json(sc.parallelize(['{"a":'])).schema --
struct<_corrupt_record:string>
{code}
Expected:
{code}
SELECT schema_of_json('{"a":'); --
STRUCT<_corrupt_record: STRING>
SELECT schema_of_json('{"a":', map('mode','FAILFAST')); --
[MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION] ...
{code}
h2. Why it matters
schema_of_json is typically used on a sample record to build a from_json
schema. A truncated or malformed sample, or a deeply nested one, fails with a
raw Jackson exception instead of a named error condition, and the documented
parse-mode option has no effect. This is inconsistent with schema_of_xml and
with JSON data source schema inference.
h2. Related work
SPARK-57517 fixed schema_of_json for non-string literal input (type check) but
did not touch the evaluation path. No related ticket found: searched JIRA for
schema_of_json, SchemaOfJson, JsonEOFException, JsonParseException,
StreamConstraintsException, maxNestingDepth, and GitHub PRs for schema_of_json
/ SchemaOfJsonEvaluator / JsonExpressionEvalUtils on 2026-11-14.
h2. Proposed fix
In SchemaOfJsonEvaluator.evaluate, catch JsonProcessingException (which
includes StreamConstraintsException) and MalformedInputException, and apply the
same parse-mode handling as JsonInferSchema.infer: PERMISSIVE returns
STRUCT<_corrupt_record: STRING> (honoring columnNameOfCorruptRecord),
DROPMALFORMED returns STRUCT<>, FAILFAST raises MALFORMED_RECORD_IN_PARSING.
Reuse JsonInferSchema's handler rather than duplicating it. Add a migration
guide note, since the default (PERMISSIVE) behaviour changes from an exception
to a returned schema.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]