marcuslin123 opened a new pull request, #57318:
URL: https://github.com/apache/spark/pull/57318
### What changes were proposed in this pull request?
Add a new built-in SQL function `variant_strip_nulls(v) -> variant` that
returns a variant value with all variant-null values removed from its objects,
applied recursively. Returns NULL if the input is NULL.
```sql
SELECT variant_strip_nulls(parse_json('{"a":1, "b":null}')); --
{"a":1}
SELECT variant_strip_nulls(parse_json('{"a":{"b":null, "c":2}}')); --
{"a":{"c":2}}
SELECT variant_strip_nulls(parse_json('[1, null, 2]')); --
[1,null,2]
```
The function is exposed across all surfaces: SQL, the Scala DataFrame API
(`variant_strip_nulls(col)`), and PySpark (`F.variant_strip_nulls(col)`),
including Spark Connect.
**Implementation:** Mirrors the existing `variant_delete` (`VariantDelete` /
`VariantBuilder`) pattern. The core logic is `VariantBuilder.stripNulls`, which
recursively rebuilds the variant into a fresh `VariantBuilder`, dropping object
fields whose value is a variant null. The expression is a `RuntimeReplaceable`
that delegates to `VariantExpressionEvalUtils.variantStripNulls`, so
interpreted eval and codegen both come for free.
### Design decisions (open to reviewer feedback)
The JIRA did not specify exact semantics, so I chose what I believe is the
least-surprising behavior:
1. **Recursive.** The purpose of `variant_strip_nulls` is to remove
null-valued fields from the variant, so it is applied at every level of nesting
rather than only the top level. Stripping only the top level would
inconsistently leave nested nulls behind.
2. **Array elements are preserved.** Array positions are meaningful —
removing a null element would shift the indices of every subsequent element and
silently change the data. So the function recurses *into* array elements (to
strip nulls from nested objects) but never removes the elements themselves. A
variant null that is an array element is therefore kept.
3. **Only object-field variant nulls are stripped.** A SQL `NULL` input
returns `NULL` (handled by the expression layer).
Happy to adjust any of these if reviewers prefer different semantics.
### Does this PR introduce _any_ user-facing change?
Yes. A new `variant_strip_nulls` function is available in SQL, the Scala
DataFrame API, and PySpark (classic and Connect).
### How was this patch tested?
- Catalyst unit test in `VariantExpressionSuite` covering: no-op cases,
top-level null stripping, recursive nested-object stripping, array elements
preserved, and nested objects inside arrays stripped.
- End-to-end test in `VariantEndToEndSuite` exercising the SQL string form,
the Scala function, and codegen.
- Spark Connect plan golden files regenerated.
- Expression schema golden file regenerated.
- PySpark doctest.
### Was this patch authored or co-authored using generative AI tooling?
Generative AI tooling (Claude Code) was used as an assistive tool for
implementation guidance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
