sdf-jkl opened a new issue, #11948:
URL: https://github.com/apache/arrow-rs/issues/11948
## Describe the bug
`cast_to_variant`, which uses strict mode, accepts repeated struct field
names and repeated map keys and silently discards earlier values. Fields `("a",
1), ("a", 2)` become `{"a":2}` without an error or an explicit request for
overwrite semantics.
Found while implementing Spark-compatible `to_variant_object` in
[datafusion-variant](https://github.com/datafusion-contrib/datafusion-variant).
Silently choosing one value loses input data and hides an ambiguous conversion.
The resulting Variant has unique names and passes full validation; the defect
is the conversion's silent overwrite policy, not malformed output encoding.
## To Reproduce
Executed with `arrow` and `parquet-variant-compute` version `60.0.0`:
```rust
use std::sync::Arc;
use arrow::array::{ArrayRef, Int32Array, StructArray};
use arrow::datatypes::{DataType, Field};
use parquet_variant_compute::{cast_to_variant, variant_to_json};
fn main() {
let field = Arc::new(Field::new("a", DataType::Int32, true));
let input = StructArray::new(
vec![field.clone(), field].into(),
vec![
Arc::new(Int32Array::from(vec![Some(1), Some(1)])),
Arc::new(Int32Array::from(vec![Some(2), None])),
],
None,
);
// cast_to_variant uses strict mode.
let output = ArrayRef::from(cast_to_variant(&input).unwrap());
let json = variant_to_json(&output).unwrap();
println!("{}", json.value(0));
println!("{}", json.value(1));
}
```
Actual output:
```json
{"a":2}
{"a":1}
```
Both rows have repeated field name `a`, but strict conversion succeeds. The
second row also demonstrates the separate null-member omission problem: its
final null is skipped, so even a simple last-value-wins description is
incomplete.
Additional probes on 60.0.0 reproduced silent overwrite for duplicate map
keys, and for duplicate-bearing structs/maps nested inside objects, all five
list layouts, dictionaries, dense/sparse unions, and run-end encoded arrays.
Safe mode produced the same successful outputs. Reusing a field name in
separate objects remains valid, as expected.
## Expected behavior
Strict conversion should return a duplicate-key error for repeated names
within one output object, including nested objects and duplicates whose values
are null. It should neither panic nor silently discard a value. Any overwrite
policy should be an explicit option rather than the behavior of strict casting.
Check uniqueness per object: repeated array elements, identical field names
in separate objects, and case-distinct names are not duplicate object keys.
## Additional context
The [Variant encoding
specification](https://github.com/apache/parquet-format/blob/e365c6f7c05eea69c5aa11f1ead6c3db261ca9f9/VariantEncoding.md#L454-L466)
requires case-sensitive field names to be unique within each encoded object.
It does not itself prescribe how an Arrow cast must resolve conflicting source
entries. This report requests an error instead of implicit data loss in strict
conversion; output validation alone cannot detect the discarded values.
On current `main` at `fd2b5d156d`,
[struct](https://github.com/apache/arrow-rs/blob/fd2b5d156d80c986ff9ba5ace2db84500edf50ec/parquet-variant-compute/src/arrow_to_variant.rs#L593-L604)
and
[map](https://github.com/apache/arrow-rs/blob/fd2b5d156d80c986ff9ba5ace2db84500edf50ec/parquet-variant-compute/src/arrow_to_variant.rs#L652-L662)
conversion use object builders without enabling duplicate rejection. The
builder already has [configurable uniqueness
validation](https://github.com/apache/arrow-rs/blob/fd2b5d156d80c986ff9ba5ace2db84500edf50ec/parquet-variant/src/builder/object.rs#L186-L194),
but non-fallible insertion methods panic when it is enabled. Integration must
propagate an error through the kernel's fallible API.
Related completed work:
[#7777](https://github.com/apache/arrow-rs/issues/7777) introduced configurable
builder validation, and [#8170](https://github.com/apache/arrow-rs/issues/8170)
addressed incomplete enforcement for nested builders.
[#11351](https://github.com/apache/arrow-rs/pull/11351) rejects duplicate
fields during shredding; it does not cover Arrow-to-Variant conversion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
