Yang Jie created SPARK-58707:
--------------------------------
Summary: JSON schema pruning returns a null corrupt record column
Key: SPARK-58707
URL: https://issues.apache.org/jira/browse/SPARK-58707
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Yang Jie
{{OptimizeCsvJsonExprs}} prunes the schema of a {{JsonToStructs}} down to the
single field being
selected. When that field is the corrupt record column, the pruned schema
leaves the parser nothing
to convert, so a malformed value in a dropped-away field is never detected and
the corrupt record
column comes back NULL instead of the record text.
This is reachable with default configs: the rule requires {{options.isEmpty}},
which is the
PERMISSIVE default, and
{{spark.sql.optimizer.enableJsonExpressionOptimization}} defaults to true.
Reproduction, schema {{a int, b int, _corrupt_record string}}, input {{ {"a":
1, "b": "bad"} }}:
{code:scala}
val df = Seq("""{"a": 1, "b": "bad"}""").toDS()
.selectExpr("from_json(value, 'a int, b int, _corrupt_record string') as p")
.selectExpr("p._corrupt_record")
{code}
||spark.sql.optimizer.enableJsonExpressionOptimization||result||
|true|null|
|false|{{ {"a": 1, "b": "bad"} }}|
The optimized plan shows the schema pruned to the corrupt column alone:
{noformat}
Project [from_json(StructField(_corrupt_record,StringType,true), value#1,
...)._corrupt_record ...]
{noformat}
Two branches are affected, {{GetStructField}} and {{CreateNamedStruct}}. The
CSV branch added by
SPARK-32968 already guards with {{schema(ordinal).name !=
nameOfCorruptRecord}}; the JSON branches
do not.
{{GetArrayStructFields}} is not affected: an {{ArrayType}} schema takes
{{JsonToStructsEvaluator}}'s
{{case other =>}}, so {{corruptFieldIndex}} is {{None}} and corrupt-record
semantics do not apply
on that path.
The existing SPARK-33907 test misses this because its bad record is
structurally malformed
({{ {"a" 1, ...} }}), which fails at tokenization regardless of the requested
schema. Exposing the
defect needs a type mismatch on a field that pruning drops.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
