Rich-T-kid commented on code in PR #11918:
URL: https://github.com/apache/arrow-rs/pull/11918#discussion_r4381827821
##########
arrow-select/src/coalesce/boolean.rs:
##########
@@ -19,10 +19,136 @@ use super::InProgressArray;
use crate::filter::FilterPredicate;
use arrow_array::cast::AsArray;
use arrow_array::{Array, ArrayRef, BooleanArray};
-use arrow_buffer::{BooleanBuffer, BooleanBufferBuilder, NullBufferBuilder};
+use arrow_buffer::bit_chunk_iterator::BitChunks;
+use arrow_buffer::{BooleanBuffer, BooleanBufferBuilder, NullBufferBuilder,
bit_util};
use arrow_schema::ArrowError;
use std::sync::Arc;
+/// Joins compressed bit fragments into complete output words.
+#[derive(Default)]
+struct WordPacker {
+ /// Bits waiting to be appended.
+ pending_bits: u64,
+ /// Number of populated low bits in `pending_bits`.
+ pending_len: u32,
+}
+
+impl WordPacker {
+ fn push(&mut self, compressed_bits: u64, bit_count: u32) -> Option<u64> {
+ self.pending_bits |= compressed_bits << self.pending_len;
+ let total_len = self.pending_len + bit_count;
+ if total_len < 64 {
+ self.pending_len = total_len;
+ return None;
+ }
+
+ let complete_word = self.pending_bits;
+ // Keep bits that crossed the word boundary. Split the shift to avoid
+ // shifting by 64 when `pending_len == 0`.
+ self.pending_bits = (compressed_bits >> 1) >> (63 - self.pending_len);
+ self.pending_len = total_len - 64;
+ Some(complete_word)
+ }
+
+ fn finish(self) -> Option<(u64, usize)> {
+ (self.pending_len != 0).then_some((self.pending_bits, self.pending_len
as usize))
+ }
+}
Review Comment:
I tried this, but the existing Packer isn’t reusable as-is. It is local to
`filter_bits_compress` and owns the raw pointer and index for that function’s
preallocated `Vec<u64>`, while the coalescer writes through
BooleanBufferBuilder.
going to update some things so that we can re-use this and make it
destination agnostic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
