comphead opened a new issue, #11612:
URL: https://github.com/apache/arrow-rs/issues/11612
**Describe the bug**
`Fields::filter_leaves` and `Fields::try_filter_leaves` treat `ListView` and
`LargeListView` as leaves. The docs say the filter is invoked "for each
[`FieldRef`] containing no child [`FieldRef`], a leaf field", but a list view
has a child, its element. So the filter is called once for the list view field
itself instead of for the leaves under its element, and the leaf index of every
later field is off by that number of leaves minus one. `List`, `LargeList`, and
`FixedSizeList` are descended into.
**To Reproduce**
With `arrow-schema = "=60.0.0"`:
```rust
use std::sync::Arc;
use arrow_schema::{DataType, Field, Fields};
fn main() {
// An element with two leaves, `a` and `b`.
let element = Arc::new(Field::new(
"element",
DataType::Struct(Fields::from(vec![
Field::new("a", DataType::Int32, true),
Field::new("b", DataType::Int32, true),
])),
true,
));
for (name, list) in [
("List", DataType::List(Arc::clone(&element))),
("ListView", DataType::ListView(Arc::clone(&element))),
("LargeListView", DataType::LargeListView(Arc::clone(&element))),
] {
let fields = Fields::from(vec![
Field::new("l", list, true),
Field::new("id", DataType::Int32, true),
]);
let mut leaves = Vec::new();
fields.filter_leaves(|idx, field| {
leaves.push(format!("{idx}: {}", field.name()));
false
});
let kept = fields.filter_leaves(|idx, _| idx == 0);
println!("{name}: leaves {leaves:?}, keeping leaf 0 keeps {}",
kept[0].data_type());
}
}
```
```text
List: leaves ["0: a", "1: b", "2: id"], keeping leaf 0 keeps
List(Struct("a": Int32), field: 'element')
ListView: leaves ["0: l", "1: id"], keeping leaf 0 keeps
ListView(Struct("a": Int32, "b": Int32), field: 'element')
LargeListView: leaves ["0: l", "1: id"], keeping leaf 0 keeps
LargeListView(Struct("a": Int32, "b": Int32), field: 'element')
```
**Expected behavior**
List views are descended into like `List` and `LargeList`. The leaves are
`a`, `b`, and `id`, and keeping leaf 0 keeps `ListView(Struct("a": Int32))`.
The match in `try_filter_leaves` on `main` (`fedfd0dbb`) has no list view arms,
so adding `ListView` and `LargeListView` next to `List` and `LargeList` should
be enough.
**Additional context**
iceberg-rust uses these leaf indices as Parquet leaf column indices for
`ProjectionMask::leaves`. parquet-rs reads a Parquet LIST as `ListView` when
the file's embedded Arrow schema says so, so the two numberings disagree. A
list view column then read as NULL, and a column after a list view of structs
was read from the wrong Parquet leaf (apache/iceberg-rust#3393). DataFusion's
own leaf counting already descends into list views (`count_leaves` in
`datafusion/datasource-parquet/src/nested_schema_pruning.rs`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
