mbutrovich opened a new issue, #11598: URL: https://github.com/apache/arrow-rs/issues/11598
### Describe the bug The arrow-row docs say rows "compare the same as when using `lexsort`" ([lib.rs](https://github.com/apache/arrow-rs/blob/72887ed91c80002bb7ec755ce1e964bab07bebea/arrow-row/src/lib.rs#L26-L27)). For union columns they do not, in two cases: 1. With `descending: true`, the type ids sort descending but the values within one type stay ascending. 2. A row whose selected child is null sorts by its type id, among that type's values. `lexsort` treats it as a null row and places it by `nulls_first`, because `UnionArray::logical_nulls` reports it as null. ### To Reproduce ```rust use std::sync::Arc; use arrow_array::{Array, ArrayRef, Int32Array, UnionArray}; use arrow_ord::sort::{lexsort_to_indices, SortColumn}; use arrow_row::{RowConverter, SortField}; use arrow_schema::{DataType, Field, SortOptions, UnionFields}; fn orders(union: &ArrayRef, opts: SortOptions) -> (Vec<u32>, Vec<u32>) { let conv = RowConverter::new(vec![SortField::new_with_options(union.data_type().clone(), opts)]).unwrap(); let rows = conv.convert_columns(&[union.clone()]).unwrap(); let mut by_rows: Vec<u32> = (0..union.len() as u32).collect(); by_rows.sort_by(|&a, &b| rows.row(a as usize).cmp(&rows.row(b as usize))); let lex = lexsort_to_indices(&[SortColumn { values: union.clone(), options: Some(opts) }], None).unwrap(); (by_rows, lex.values().to_vec()) } fn main() { let fields = UnionFields::try_new([0, 1], [Field::new("a", DataType::Int32, true), Field::new("b", DataType::Int32, true)]).unwrap(); // Rows: a=1, a=2, a=null, b=0 let a: ArrayRef = Arc::new(Int32Array::from(vec![Some(1), Some(2), None, Some(0)])); let b: ArrayRef = Arc::new(Int32Array::from(vec![Some(0), Some(0), Some(0), Some(0)])); let union: ArrayRef = Arc::new(UnionArray::try_new(fields, vec![0i8, 0, 0, 1].into(), None, vec![a, b]).unwrap()); let descending = SortOptions { descending: true, nulls_first: false }; println!("{:?}", orders(&union, descending)); // ([3, 2, 0, 1], [3, 1, 0, 2]): rows put a null, 1, 2 inside type a; lexsort puts a=2, a=1, then the null last let nulls_last = SortOptions { descending: false, nulls_first: false }; println!("{:?}", orders(&union, nulls_last)); // ([0, 1, 2, 3], [0, 1, 3, 2]): rows keep the null row with type a; lexsort puts it last } ``` ### Expected behavior Sorting by rows gives the same order as `lexsort_to_indices`: `[3, 1, 0, 2]` and `[0, 1, 3, 2]`. ### Additional context The union codec builds the child converters with `descending: false` and `nulls_first != descending`, with the comment that "the encoded contents will be inverted if descending is set" ([lib.rs](https://github.com/apache/arrow-rs/blob/72887ed91c80002bb7ec755ce1e964bab07bebea/arrow-row/src/lib.rs#L736-L765)). The encoder negates only the type id byte and copies the child bytes as they are ([lib.rs](https://github.com/apache/arrow-rs/blob/72887ed91c80002bb7ec755ce1e964bab07bebea/arrow-row/src/lib.rs#L2206-L2241)), so nothing inverts them. The list codec uses the same child options and inverts the child bytes when descending ([list.rs](https://github.com/apache/arrow-rs/blob/72887ed91c80002bb7ec755ce1e964bab07bebea/arrow-row/src/list.rs#L242-L244)). For null rows, the encoder writes the type id and the child's null encoding, and never checks the union's logical nulls. The [Union Ordering](https://github.com/apache/arrow-rs/blob/72887ed91c80002bb7ec755ce1e964bab07bebea/arrow-row/src/lib.rs#L562-L566) section of the docs does not say where null rows go. If ordering null rows by type id is intended, the docs could say so, and `lexsort` and the row format would still disagree. Found with property-based tests that build arrays in every physical layout the Arrow format allows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
