mbutrovich opened a new issue, #11586:
URL: https://github.com/apache/arrow-rs/issues/11586
### Describe the bug
Writing a sliced list whose child is a union to IPC and reading it back
loses data. For a dense union the rows come back with wrong values. For a
sparse union the reader rejects the batch.
### To Reproduce
```rust
use std::sync::Arc;
use arrow_array::{Array, ArrayRef, Int32Array, LargeListArray, ListArray,
RecordBatch, UnionArray};
use arrow_buffer::OffsetBuffer;
use arrow_ipc::{reader::StreamReader, writer::StreamWriter};
use arrow_schema::{DataType, Field, Schema, UnionFields};
fn round_trip(a: &ArrayRef) -> Result<ArrayRef, String> {
let schema = Arc::new(Schema::new(vec![Field::new("v",
a.data_type().clone(), true)]));
let batch = RecordBatch::try_new(schema.clone(),
vec![a.clone()]).unwrap();
let mut bytes = vec![];
let mut w = StreamWriter::try_new(&mut bytes, &schema).unwrap();
w.write(&batch).unwrap();
w.finish().unwrap();
drop(w);
let mut r = StreamReader::try_new(std::io::Cursor::new(bytes),
None).unwrap();
r.next().unwrap().map(|b| b.column(0).clone()).map_err(|e| e.to_string())
}
fn main() {
let fields = UnionFields::try_new([0], [Field::new("a", DataType::Int32,
true)]).unwrap();
// Dense: rows [[1], [null]], sliced to the second row
let child: ArrayRef = Arc::new(Int32Array::from(vec![Some(1), None]));
let dense: ArrayRef = Arc::new(UnionArray::try_new(fields.clone(),
vec![0i8, 0].into(), Some(vec![0, 1].into()), vec![child]).unwrap());
let item = Arc::new(Field::new("item", dense.data_type().clone(), true));
let list: ArrayRef = Arc::new(LargeListArray::new(item,
OffsetBuffer::from_lengths([1, 1]), dense, None));
let back = round_trip(&list.slice(1, 1)).unwrap();
let row =
back.as_any().downcast_ref::<LargeListArray>().unwrap().value(0);
println!("{:?}",
row.as_any().downcast_ref::<UnionArray>().unwrap().value(0)); // [1], expected
[null]
// Sparse: rows [[10], [20, 30]], sliced to the second row
let child: ArrayRef = Arc::new(Int32Array::from(vec![10, 20, 30]));
let sparse: ArrayRef = Arc::new(UnionArray::try_new(fields, vec![0i8, 0,
0].into(), None, vec![child]).unwrap());
let item = Arc::new(Field::new("item", sparse.data_type().clone(),
true));
let list: ArrayRef = Arc::new(ListArray::new(item,
OffsetBuffer::from_lengths([1, 2]), sparse, None));
println!("{:?}", round_trip(&list.slice(1, 1)).map(|a| a.len()));
// Err("Invalid argument error: Sparse union child arrays must be equal
in length to the length of the union")
}
```
### Expected behavior
The round trip returns the rows that were written: `[[null]]` for the dense
case and `[[20, 30]]` for the sparse case.
### Additional context
Without the slice, both round trips are correct. I have not located the
cause in the writer. The sparse case may share its cause with the sparse union
slicing issue, where a sparse union's children are not sliced with its offset.
Found with property-based tests that build arrays in every physical layout
the Arrow format allows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
