[ https://issues.apache.org/jira/browse/ARROW-8598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yordan Pavlov updated ARROW-8598: --------------------------------- Description: the simd_compare_op function defined here [https://github.com/apache/arrow/blob/master/rust/arrow/src/compute/kernels/comparison.rs#L229] appears to only work correctly when item count is a multiple of T::lanes(). Otherwise the resulting boolean array is created with a buffer of incorrect length and subsequent binary operations such as compute::and return an error. The no_simd_compare_op function defined in the same module does not appear to have this problem. This bug can be reproduced with the following code: {code:java} fn main() { let lanes = Int8Type::lanes(); println("i8 lanes: {}", lanes); // 64 // let item_count = 128; // this works because item_count divides by 64 (lanes) without remainder let item_count = 130; // this fails because item_count divides by 64 (lanes) with remainder // item_count = 130 should return error: // ComputeError("Buffers must be the same size to apply Bitwise AND.") // create boolean array let mut select_mask: BooleanArray = vec![true; item_count].into(); // create arrays with i8 values let value_array: PrimitiveArray<Int8Type> = vec![1; item_count].into(); let filter_array: PrimitiveArray<Int8Type> = vec![2; item_count].into(); // compare i8 arrays and produce a boolean array let result_mask = compute::gt_eq(&value_array, &filter_array).unwrap(); // compare boolean arrays select_mask = compute::and(&select_mask, &result_mask).unwrap(); // print result, should be all false println!("select mask: {:?}", select_mask); } {code} was: the simd_compare_op function defined here [https://github.com/apache/arrow/blob/master/rust/arrow/src/compute/kernels/comparison.rs#L229] appears to only work correctly when item count is a multiple of T::lanes(). Otherwise the resulting boolean array is created with a buffer of incorrect length and subsequent binary operations such as compute::and return an error. This bug can be reproduced with the following code: {code:java} fn main() { let lanes = Int8Type::lanes(); println("i8 lanes: {}", lanes); // 64 // let item_count = 128; // this works because item_count divides by 64 (lanes) without remainder let item_count = 130; // this fails because item_count divides by 64 (lanes) with remainder // item_count = 130 should return error: // ComputeError("Buffers must be the same size to apply Bitwise AND.") // create boolean array let mut select_mask: BooleanArray = vec![true; item_count].into(); // create arrays with i8 values let value_array: PrimitiveArray<Int8Type> = vec![1; item_count].into(); let filter_array: PrimitiveArray<Int8Type> = vec![2; item_count].into(); // compare i8 arrays and produce a boolean array let result_mask = compute::gt_eq(&value_array, &filter_array).unwrap(); // compare boolean arrays select_mask = compute::and(&select_mask, &result_mask).unwrap(); // print result, should be all false println!("select mask: {:?}", select_mask); } {code} > [Rust] simd_compare_op creates buffer of incorrect length when item count is > not a multiple of T::lanes() > --------------------------------------------------------------------------------------------------------- > > Key: ARROW-8598 > URL: https://issues.apache.org/jira/browse/ARROW-8598 > Project: Apache Arrow > Issue Type: Bug > Components: Rust > Affects Versions: 0.17.0 > Reporter: Yordan Pavlov > Priority: Major > > the simd_compare_op function defined here > [https://github.com/apache/arrow/blob/master/rust/arrow/src/compute/kernels/comparison.rs#L229] > appears to only work correctly when item count is a multiple of T::lanes(). > Otherwise the resulting boolean array is created with a buffer of incorrect > length and subsequent binary operations such as compute::and return an error. > The no_simd_compare_op function defined in the same module does not appear to > have this problem. > This bug can be reproduced with the following code: > > {code:java} > fn main() { > let lanes = Int8Type::lanes(); > println("i8 lanes: {}", lanes); // 64 > // let item_count = 128; // this works because item_count divides by 64 > (lanes) without remainder > let item_count = 130; // this fails because item_count divides by 64 > (lanes) with remainder > // item_count = 130 should return error: > // ComputeError("Buffers must be the same size to apply Bitwise AND.") > > // create boolean array > let mut select_mask: BooleanArray = vec![true; item_count].into(); > // create arrays with i8 values > let value_array: PrimitiveArray<Int8Type> = vec![1; item_count].into(); > let filter_array: PrimitiveArray<Int8Type> = vec![2; item_count].into(); > // compare i8 arrays and produce a boolean array > let result_mask = compute::gt_eq(&value_array, &filter_array).unwrap(); > // compare boolean arrays > select_mask = compute::and(&select_mask, &result_mask).unwrap(); > // print result, should be all false > println!("select mask: {:?}", select_mask); > } > {code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005)