sdf-jkl opened a new issue, #11939:
URL: https://github.com/apache/arrow-rs/issues/11939
## Describe the bug
`cast_to_variant_with_options` with `CastOptions { safe: true, .. }` encodes
conversion failures as `Variant::Null` while leaving the output row valid.
Arrow's cast kernel marks failed conversions null in the output array, as
documented by
[`CastOptions::safe`](https://github.com/apache/arrow-rs/blob/5746b443f70b4e55676be59cea317501a94ec810/arrow-cast/src/cast/mod.rs#L231-L235).
Found while implementing Spark-compatible SQL `TRY_CAST(... AS VARIANT)` in
[datafusion-variant](https://github.com/datafusion-contrib/datafusion-variant).
[Spark documents `TRY_CAST` as returning SQL NULL on runtime conversion errors,
including
overflow](https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html#cast).
Downstream code must currently convert these failed results to SQL NULL
explicitly.
## To Reproduce
Reproduced with `arrow = "60.0.0"` and `parquet-variant-compute = "60.0.0"`:
```rust
use arrow::array::{Array, TimestampSecondArray};
use arrow::compute::{CastOptions, cast_with_options};
use arrow::datatypes::{DataType, TimeUnit};
use parquet_variant_compute::cast_to_variant_with_options;
fn main() {
let input = TimestampSecondArray::from(vec![Some(i64::MAX), Some(0),
None]);
let options = CastOptions { safe: true, ..Default::default() };
let arrow = cast_with_options(
&input,
&DataType::Timestamp(TimeUnit::Microsecond, None),
&options,
).unwrap();
let variant = cast_to_variant_with_options(&input, &options).unwrap();
println!("Arrow overflow is null: {}", arrow.is_null(0));
println!("Variant overflow is null: {}", variant.is_null(0));
println!("Variant overflow value: {:?}", variant.value(0));
assert!(!variant.is_null(1));
assert!(variant.is_null(2));
assert!(arrow.is_null(0));
assert!(variant.is_null(0)); // Fails: overflow is encoded as
Variant::Null.
}
```
Output before the assertion failure:
```text
Arrow overflow is null: true
Variant overflow is null: false
Variant overflow value: Null
```
## Expected behavior
For a failed top-level conversion with `safe: true`, mark the output row
null, matching Arrow's cast kernel. Valid conversions and input nulls should
retain their current behavior; `safe: false` should continue to return an error.
## Additional context
Source inspection of `main` at `5746b443f7` confirms the same behavior: the
[failure
branch](https://github.com/apache/arrow-rs/blob/5746b443f70b4e55676be59cea317501a94ec810/parquet-variant-compute/src/arrow_to_variant.rs#L367-L372)
appends `Variant::Null`, and the [timestamp overflow
test](https://github.com/apache/arrow-rs/blob/5746b443f70b4e55676be59cea317501a94ec810/parquet-variant-compute/src/cast_to_variant.rs#L916-L934)
expects it.
The [original discussion in
#8059](https://github.com/apache/arrow-rs/issues/8059#issuecomment-3167642459)
proposed setting the array element null on overflow, following Arrow's cast
model. A [later discussion in
#8233](https://github.com/apache/arrow-rs/pull/8233#discussion_r2308739717)
suggested `Variant::Null` instead of `None`.
[#8984](https://github.com/apache/arrow-rs/pull/8984) subsequently unified the
options with Arrow's `CastOptions` while preserving behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
