adriangb commented on code in PR #11235:
URL: https://github.com/apache/arrow-rs/pull/11235#discussion_r4222010418
##########
parquet/src/util/push_buffers.rs:
##########
@@ -180,15 +185,156 @@ impl PushBuffers {
new_buffers.push(buffer.clone());
}
}
+ self.buffered_bytes = new_ranges.iter().map(|r| r.end - r.start).sum();
self.ranges = new_ranges;
self.buffers = new_buffers;
}
+ /// Remove all buffered bytes in `ranges`, whatever the shape of the
+ /// pushed buffers.
+ ///
+ /// A buffer that overlaps a range is trimmed or split, and the parts
+ /// outside the range are kept. The kept parts are zero-copy slices. Thus,
+ /// the allocator frees the memory of a pushed [`Bytes`] only after all of
+ /// its parts are removed.
+ ///
+ /// If the buffers are sorted by start, they stay sorted. The order of
+ /// buffers with the same start is not specified. If no buffer overlaps
+ /// `ranges`, the buffers do not change.
+ #[cfg(feature = "arrow")]
+ pub(crate) fn release_ranges(&mut self, ranges: &[Range<u64>]) {
Review Comment:
I don't think much simpler, but we could make it faster using binary search.
I can do that in a followup PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
