Linhong Liu created SPARK-61831:
-----------------------------------
Summary: Cast from VARIANT to a type with a non-nullable nested
field reads a JSON null or missing key as 0
Key: SPARK-61831
URL: https://issues.apache.org/jira/browse/SPARK-61831
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.3.1
Reporter: Linhong Liu
A cast from VARIANT to a type with a non-nullable nested slot, for example
{{CAST(v AS STRUCT<user_id: BIGINT NOT NULL, country: STRING>)}} (also
{{v::STRUCT<...>}}, {{try_cast}}, and the same
struct nested in an ARRAY or MAP), is accepted, and its result type keeps
{{user_id NOT NULL}}. When the JSON value is
{{null}} or the key is missing, the cast still puts NULL into that field.
Everything downstream trusts the NOT NULL type,
so the NULL is read as 0: {{IS NULL}} is false, {{coalesce}} is skipped, and an
INSERT into a NOT NULL column stores 0.
{code:sql}
SELECT event_id, p.user_id, p.user_id IS NULL FROM (
SELECT DISTINCT event_id, payload::STRUCT<user_id: BIGINT NOT NULL> AS p FROM
(
SELECT 1 AS event_id, parse_json('{"user_id": 42}') AS payload UNION ALL
SELECT 2, parse_json('{"user_id": null}') UNION ALL
SELECT 3, parse_json('{}')));
-- returns (2, 0, false) and (3, 0, false); the user_id of rows 2 and 3 is NULL
{code}
{{Cast.canAnsiCast}} and {{Cast.canCast}} accept any VARIANT target that
{{VariantGet.checkDataType}} accepts, which ignores
nullability, while {{VariantGet.cast}} leaves a struct field NULL when the key
is missing or its value is null.
{{variant_get}} avoids this by typing itself {{targetType.asNullable}}.
Elsewhere Cast already rejects a nullable source
slot for a non-nullable target ({{resolvableNullability}}), including a nested
VARIANT.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]