[
https://issues.apache.org/jira/browse/SPARK-61831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61831:
-----------------------------------
Labels: pull-request-available (was: )
> Cast from VARIANT to a type with a non-nullable nested field reads a JSON
> null or missing key as 0
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-61831
> URL: https://issues.apache.org/jira/browse/SPARK-61831
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.3.1
> Reporter: Linhong Liu
> Priority: Major
> Labels: pull-request-available
>
> A cast from VARIANT to a type with a non-nullable nested slot, for example
> {{CAST(v AS STRUCT<user_id: BIGINT NOT NULL, country: STRING>)}} (also
> {{v::STRUCT<...>}}, {{try_cast}}, and the same
> struct nested in an ARRAY or MAP), is accepted, and its result type keeps
> {{user_id NOT NULL}}. When the JSON value is
> {{null}} or the key is missing, the cast still puts NULL into that field.
> Everything downstream trusts the NOT NULL type,
> so the NULL is read as 0: {{IS NULL}} is false, {{coalesce}} is skipped, and
> an INSERT into a NOT NULL column stores 0.
> {code:sql}
> SELECT event_id, p.user_id, p.user_id IS NULL FROM (
> SELECT DISTINCT event_id, payload::STRUCT<user_id: BIGINT NOT NULL> AS p
> FROM (
> SELECT 1 AS event_id, parse_json('{"user_id": 42}') AS payload UNION ALL
> SELECT 2, parse_json('{"user_id": null}') UNION ALL
> SELECT 3, parse_json('{}')));
> -- returns (2, 0, false) and (3, 0, false); the user_id of rows 2 and 3 is
> NULL
> {code}
> {{Cast.canAnsiCast}} and {{Cast.canCast}} accept any VARIANT target that
> {{VariantGet.checkDataType}} accepts, which ignores
> nullability, while {{VariantGet.cast}} leaves a struct field NULL when the
> key is missing or its value is null.
> {{variant_get}} avoids this by typing itself {{targetType.asNullable}}.
> Elsewhere Cast already rejects a nullable source
> slot for a non-nullable target ({{resolvableNullability}}), including a
> nested VARIANT.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]