This is an automated email from the ASF dual-hosted git repository.
Jefffrey pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new a5158c8bf9 arrow-row: Fix decode_fixed_size_list to apply the
corrected_type step for dictionary children (#10414)
a5158c8bf9 is described below
commit a5158c8bf926bd74a01093a05187d25b2dfc1bc3
Author: Qi Zhu <[email protected]>
AuthorDate: Fri Jul 24 09:13:15 2026 +0800
arrow-row: Fix decode_fixed_size_list to apply the corrected_type step for
dictionary children (#10414)
# Which issue does this PR close?
Closes #10413.
# Rationale for this change
`RowConverter::convert_rows` used to fail with a schema-mismatch
`InvalidArgumentError` on any target type that contained a
`FixedSizeList<Dictionary<K, V>>`:
```text
InvalidArgumentError("FixedSizeListArray expected data type
Dictionary(Int32, Utf8) got Utf8 for \"item\"")
```
`decode_fixed_size_list` was building the returned array with the
declared `element_field` (which still carried `Dictionary<...>`) while
the children came from `converter.convert_raw`, which returns the
flattened (non-dictionary) type. `try_new_with_length` validates the
child type against `element_field` and returned `Err`.
The other list-like decoders — `List`, `LargeList`, `ListView`,
`LargeListView`, `Map` — already reconcile declared-vs-actual via a
`corrected_type` step (see `list::decode<L>` at [list.rs:~260 on
main](https://github.com/apache/arrow-rs/blob/main/arrow-row/src/list.rs#L259-L320)),
and `Codec::Struct` in `lib.rs` does the same via `corrected_fields`.
`decode_fixed_size_list` was the only list-like decoder that skipped
this pattern.
# What changes are included in this PR?
- `arrow-row/src/list.rs::decode_fixed_size_list`: adopt the
`corrected_type` step, mirroring the shape of every other list-like
decoder in the file.
-
`arrow-row/src/lib.rs::tests::test_fixed_size_list_of_dictionaries_round_trips`:
regression test that reproduces the original failure on one row `["a",
"b"]` of `FixedSizeList<Dictionary<Int32, Utf8>, 2>`.
Diff on the decoder is ~10 lines:
```rust
let mut children = unsafe { converter.convert_raw(&mut child_rows,
validate_utf8) }?;
assert_eq!(children.len(), 1);
// Since `RowConverter` flattens certain data types (i.e. `Dictionary`),
// we need to use the child's actual data type rather than the declared
// `element_field`'s. Mirrors the `corrected_type` logic in `decode<L>`
// above and `Codec::Struct`'s `corrected_fields` in `lib.rs`.
let corrected_element_field = Arc::new(
element_field
.as_ref()
.clone()
.with_data_type(children[0].data_type().clone()),
);
FixedSizeListArray::try_new_with_length(
corrected_element_field,
*size,
children.pop().unwrap(),
nulls,
num_rows,
)
```
# Are these changes tested?
Yes:
- `test_fixed_size_list_of_dictionaries_round_trips` — regression test
in `arrow-row/src/lib.rs`. Pre-fix it panics at the first
`convert_rows(...).unwrap()` (matching the report in #10413); post-fix
it returns a `FixedSizeList<Utf8>` (same flattened shape produced today
for `List<Dictionary<...>>`) whose values are `"a"`, `"b"`.
- All existing `arrow-row` tests continue to pass (`cargo test -p
arrow-row`).
- `cargo clippy -p arrow-row --all-targets -- -D warnings` clean.
# Are there any user-facing changes?
Behaviour change (bug fix): `RowConverter::convert_rows` now succeeds
where it previously returned an `InvalidArgumentError` for
`FixedSizeList<Dictionary<...>>` (and any type containing one, e.g.
`FSL<Struct<Dict>>`, `List<FSL<Dict>>`). The returned array flattens the
dictionary child to its native representation — same behaviour today's
`List<Dict>` / `LargeList<Dict>` / `Map<Dict>` decoders exhibit. Callers
that need the declared dictionary type back can re-encode after
`convert_rows` (that's the documented contract).
No public API surface changes.
# Downstream context
DataFusion hit this in https://github.com/apache/datafusion/pull/23523
(nested-type support in `GroupValuesColumn`) where a `GROUP BY fsl_col`
on `FixedSizeList<Dictionary<Int32, Utf8>>` would panic on emit. The
DataFusion PR is landing a defensive blacklist in the meantime; this fix
will let that blacklist be lifted in a follow-up.
---------
Co-authored-by: Jeffrey Vo <[email protected]>
---
arrow-row/src/lib.rs | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++
arrow-row/src/list.rs | 11 ++++++++++-
2 files changed, 65 insertions(+), 1 deletion(-)
diff --git a/arrow-row/src/lib.rs b/arrow-row/src/lib.rs
index f1cb42d3d8..b647dab288 100644
--- a/arrow-row/src/lib.rs
+++ b/arrow-row/src/lib.rs
@@ -6169,6 +6169,61 @@ mod tests {
assert_eq!(&list, &back[0]);
}
+ /// Ensure dictionaries nested within FixedSizeLists are not flattened
+ #[test]
+ fn test_fixed_size_list_of_dictionaries_round_trips() {
+ // Build one row = ["a", "b"] as
+ // `FixedSizeList<Dictionary<Int32, Utf8>, 2>`.
+ let dict_dt = DataType::Dictionary(Box::new(DataType::Int32),
Box::new(DataType::Utf8));
+ let element_field = Arc::new(Field::new("item", dict_dt.clone(),
true));
+ let fsl_dt = DataType::FixedSizeList(Arc::clone(&element_field), 2);
+
+ let values = Arc::new(StringArray::from(vec!["a", "b"]));
+ let keys = Int32Array::from(vec![0, 1]);
+ let dict = DictionaryArray::<Int32Type>::try_new(keys,
values).unwrap();
+ let fsl: ArrayRef = Arc::new(FixedSizeListArray::new(
+ Arc::clone(&element_field),
+ 2,
+ Arc::new(dict),
+ None,
+ ));
+
+ assert!(RowConverter::supports_fields(&[SortField::new(
+ fsl_dt.clone()
+ )]));
+
+ let converter =
RowConverter::new(vec![SortField::new(fsl_dt.clone())]).unwrap();
+ let rows = converter.convert_columns(&[Arc::clone(&fsl)]).unwrap();
+
+ // Before the fix this panicked at the `.unwrap()` because
+ // `convert_rows` returned `Err(InvalidArgumentError(...))`.
+ let back = converter.convert_rows(&rows).unwrap();
+ assert_eq!(back.len(), 1);
+
+ // The returned array is a `FixedSizeList` with a decoded
+ // (flattened) child — same self-consistent shape the other
+ // list-like decoders produce for dictionary children.
+ let out = back[0]
+ .as_any()
+ .downcast_ref::<FixedSizeListArray>()
+ .expect("decoded array must be a FixedSizeListArray");
+ assert_eq!(out.len(), 1);
+ assert_eq!(out.value_length(), 2);
+ // Child data type is the flattened `Utf8`, not the declared
+ // `Dictionary`. Callers that want the dictionary back need to
+ // re-encode (see the module docs).
+ assert_eq!(out.values().data_type(), &DataType::Utf8);
+
+ // Sanity: values survived the round trip.
+ let values = out
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .expect("child must be a StringArray after flattening");
+ assert_eq!(values.value(0), "a");
+ assert_eq!(values.value(1), "b");
+ }
+
// Test List<Null> with various combinations of nulls and empty lists
#[test]
fn test_list_null_variations() {
diff --git a/arrow-row/src/list.rs b/arrow-row/src/list.rs
index c35e614a9f..751ecc12ad 100644
--- a/arrow-row/src/list.rs
+++ b/arrow-row/src/list.rs
@@ -421,8 +421,17 @@ pub unsafe fn decode_fixed_size_list(
let mut children = unsafe { converter.convert_raw(&mut child_rows,
validate_utf8) }?;
assert_eq!(children.len(), 1);
+ // Since RowConverter flattens certain data types (i.e. Dictionary),
+ // we need to use updated data type instead of original field
+ let corrected_element_field = Arc::new(
+ element_field
+ .as_ref()
+ .clone()
+ .with_data_type(children[0].data_type().clone()),
+ );
+
FixedSizeListArray::try_new_with_length(
- Arc::clone(element_field),
+ corrected_element_field,
*size,
children.pop().unwrap(),
nulls,
