yordan-pavlov commented on a change in pull request #8364:
URL: https://github.com/apache/arrow/pull/8364#discussion_r503415507
##########
File path: rust/arrow/src/compute/kernels/filter.rs
##########
@@ -230,6 +231,86 @@ macro_rules! filter_dictionary_array {
}};
}
+macro_rules! filter_primitive_item_list_array {
+ ($context:expr, $array:expr, $item_type:ident) => {{
+ let input_array = $array.as_any().downcast_ref::<ListArray>().unwrap();
+ let values_builder =
PrimitiveBuilder::<$item_type>::new($context.filtered_count);
+ let mut builder = ListBuilder::new(values_builder);
+ for i in 0..$context.filter_u64.len() {
+ // foreach u64 batch
+ let filter_batch = $context.filter_u64[i];
+ if filter_batch == 0 {
+ // if batch == 0: skip
Review comment:
the meaning of this condition is not that batch 0 is skipped, but rather
that if the current batch (of 64 bits / elements from the filter array)
contains only 0s (i.e. all items are filtered out) then the entire batch should
be skipped as there is no items to copy to the output array (all items are
filtered out); this continues until a batch is encountered where at least one
item is not filtered out (and therefore the numeric representation of the batch
of filter values is != 0); this skipping of entire batches of 64 items is a
major factor to the performance improvements recently implemented in the filter
kernel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
