yongster opened a new issue, #10992:
URL: https://github.com/apache/arrow-rs/issues/10992
### Is your feature request related to a problem or challenge?
This issue follows up on #10909 and accompanies the fix for `nullif` on
RunEndEncoded and Union arrays.
Several compute kernels can introduce logical nulls, for example through null
take indices or a `nullif` condition.
For most Arrow arrays, a top-level validity bitmap represents these nulls.
RunEndEncoded and Union arrays are different:
- RunEndEncoded derives logical nulls from its `values` child.
- Union has no top-level validity bitmap; a logical null must be represented
by
a nullable child selected by its type id.
As a result, a kernel can currently produce physical nulls even when the
corresponding field metadata is marked as non-nullable.
### Describe the solution you'd like
A kernel must not silently drop or fabricate requested nulls.
- For `RunEndEncoded`, if the `values` field is non-nullable and an operation
needs to introduce a null, return a compute error.
- For `Union`, a null may be represented using a nullable child field.
If no child field is nullable and an operation needs to introduce a null,
return a compute error.
- Kernels that preserve an existing null should continue to do so without
requiring a new nullable representation.
- This policy should be applied consistently to kernels that can introduce
nulls, including `take`, `nullif`, and similar compute kernels.
### Describe alternatives you've considered
Together with #10909 and the `nullif` fix, this policy keeps the physical
result consistent with its field metadata.
In particular, a field marked as non-nullable must not produce output
containing nulls. Returning an error when a requested null cannot be
represented is preferable to producing a result whose data contradicts its
declared nullability.
### Additional context
PR #10909 fixed handling of null take indices for RunEndEncoded and Union
arrays. This issue tracks the broader nullability contract and its consistent
application across compute kernels.
This issue tracks the broader nullability contract for REE and Union arrays
and
the consistent application of that contract across compute kernels.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
