sdf-jkl opened a new issue, #11301:
URL: https://github.com/apache/arrow-rs/issues/11301
## Describe the bug
`variant_get` can turn Variant null and non-object values into present Arrow
structs when the selected input has a `typed_value` column. The same values
produce NULL structs or cast errors when unshredded.
The source does not need to be shredded as a struct. Shredding integer `42`
as `Int64`, then requesting `Struct<a: Int64>`, is enough to suppress the
strict-cast error and return `{a: NULL}`.
| Input | Cast mode | Unshredded result | Shredded result |
| --- | --- | --- | --- |
| Variant null | Safe | NULL struct | Present `{a: NULL}` |
| Integer `42` | Safe | NULL struct | Present `{a: NULL}` |
| Integer `42` | Strict | `CastError` | Present `{a: NULL}` |
This changes parent validity and strict-cast behavior based on physical
representation. Consumers can observe different `IS NULL` and `COUNT(expr)`
results for equivalent values. The inputs below are built entirely with
`VariantArray::from_iter` and `shred_variant`.
Reproduced on main at
[`8208506f8f9ec193c08023ac2477d211d1d86d20`](https://github.com/apache/arrow-rs/commit/8208506f8f9ec193c08023ac2477d211d1d86d20)
(workspace version 60.0.0), checked on 2026-09-29.
## To reproduce
Put this in `parquet-variant-compute/tests/shredded_struct_semantics.rs`:
```rust
use arrow::array::ArrayRef;
use arrow::compute::CastOptions;
use arrow::datatypes::{DataType, Field};
use parquet_variant::Variant;
use parquet_variant_compute::{GetOptions, VariantArray, shred_variant,
variant_get};
use std::sync::Arc;
#[test]
fn variant_null_to_struct_preserves_null() {
let input = VariantArray::from_iter([Variant::Null]);
// A primitive shredding schema is sufficient to trigger the bug.
let shredded = shred_variant(&input, &DataType::Int64).unwrap();
let target = DataType::Struct(vec![Field::new("a", DataType::Int64,
true)].into());
let options = GetOptions::new()
.with_as_type(Some(Arc::new(Field::new("result", target, true))));
let before = variant_get(&ArrayRef::from(input),
options.clone()).unwrap();
let after = variant_get(&ArrayRef::from(shredded), options).unwrap();
println!("unshredded is_null: {}", before.is_null(0));
println!("shredded is_null: {}", after.is_null(0));
assert!(before.is_null(0));
assert!(after.is_null(0)); // fails: present {a: NULL}
}
#[test]
fn scalar_to_struct_preserves_strict_cast_error() {
let input = VariantArray::from_iter([Variant::Int64(42)]);
let shredded = shred_variant(&input, &DataType::Int64).unwrap();
let target = DataType::Struct(vec![Field::new("a", DataType::Int64,
true)].into());
let options = GetOptions::new()
.with_as_type(Some(Arc::new(Field::new("result", target, true))))
.with_cast_options(CastOptions { safe: false, ..Default::default()
});
let before = variant_get(&ArrayRef::from(input), options.clone());
let after = variant_get(&ArrayRef::from(shredded), options);
println!("unshredded: {before:?}");
println!("shredded: {after:?}");
assert!(before.is_err());
assert!(after.is_err()); // fails: Ok(present {a: NULL})
}
```
Run from the repository root:
```sh
cargo test -p parquet-variant-compute --test shredded_struct_semantics --
--nocapture
```
Both tests fail. The first prints `true` before shredding and `false` after
shredding. The second returns
`CastError("Failed to extract struct from variant Int64(42)")` before
shredding
and a present struct with a null `a` child afterward. With `safe: true`, the
scalar case instead returns a NULL struct before shredding and `{a: NULL}`
afterward.
## Expected behavior
For equivalent shredded and unshredded values, struct conversion should
agree:
- Variant null becomes a NULL struct.
- A non-object scalar becomes a NULL struct in safe mode and a cast error in
strict mode.
- An object with a missing field can become a present struct with a null
child.
It must remain distinguishable from a NULL struct.
## Scope and apparent cause
The failure also reproduces for nested struct targets and when a path selects
an element of a shredded list and requests it as a struct. Struct, primitive,
and list shredding schemas all trigger it. Corresponding unshredded Variant
conversions, including nested structs and lists of structs, preserve the
expected semantics in these checks. Ordinary Arrow struct-to-struct casts
also preserve parent validity in the cases checked.
The [struct conversion
branch](https://github.com/apache/arrow-rs/blob/8208506f8f9ec193c08023ac2477d211d1d86d20/parquet-variant-compute/src/variant_get.rs#L338)
uses the row builder when `typed_value` is absent. When it is present, the
branch extracts the requested fields separately and constructs a
`StructArray`
with `target.nulls()` as parent validity, without first checking whether each
parent value is an object.
Variant null and integer `42` are present Variant values, so that outer
validity is true. Their missing `a` field becomes NULL, and the branch
assembles
`{a: NULL}`. It never applies the parent null/type checks that the unshredded
row builder performs. Unshredding the selected value before typed conversion
is a workaround.
## Related work
- [#8153](https://github.com/apache/arrow-rs/issues/8153) tracks broader
shredded
struct access; this report concerns incorrect results in the existing
conversion to an ordinary Arrow struct.
- [#11191](https://github.com/apache/arrow-rs/issues/11191) concerns missing
object-field extraction as Variant. Here the parent Variant is present, and
its conversion to a struct incorrectly succeeds or changes null semantics.
AI assistance: OpenAI Codex assisted with investigation, reproducer
execution,
and drafting this report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
