Linhong Liu created SPARK-61831:
-----------------------------------

             Summary: Cast from VARIANT to a type with a non-nullable nested 
field reads a JSON null or missing key as 0
                 Key: SPARK-61831
                 URL: https://issues.apache.org/jira/browse/SPARK-61831
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.3.1
            Reporter: Linhong Liu


A cast from VARIANT to a type with a non-nullable nested slot, for example
{{CAST(v AS STRUCT<user_id: BIGINT NOT NULL, country: STRING>)}} (also 
{{v::STRUCT<...>}}, {{try_cast}}, and the same
struct nested in an ARRAY or MAP), is accepted, and its result type keeps 
{{user_id NOT NULL}}. When the JSON value is
{{null}} or the key is missing, the cast still puts NULL into that field. 
Everything downstream trusts the NOT NULL type,
so the NULL is read as 0: {{IS NULL}} is false, {{coalesce}} is skipped, and an 
INSERT into a NOT NULL column stores 0.

{code:sql}
SELECT event_id, p.user_id, p.user_id IS NULL FROM (
  SELECT DISTINCT event_id, payload::STRUCT<user_id: BIGINT NOT NULL> AS p FROM 
(
    SELECT 1 AS event_id, parse_json('{"user_id": 42}') AS payload UNION ALL
    SELECT 2, parse_json('{"user_id": null}') UNION ALL
    SELECT 3, parse_json('{}')));
-- returns (2, 0, false) and (3, 0, false); the user_id of rows 2 and 3 is NULL
{code}

{{Cast.canAnsiCast}} and {{Cast.canCast}} accept any VARIANT target that 
{{VariantGet.checkDataType}} accepts, which ignores
nullability, while {{VariantGet.cast}} leaves a struct field NULL when the key 
is missing or its value is null.
{{variant_get}} avoids this by typing itself {{targetType.asNullable}}. 
Elsewhere Cast already rejects a nullable source
slot for a non-nullable target ({{resolvableNullability}}), including a nested 
VARIANT.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to