ariel-miculas commented on code in PR #22416:
URL: https://github.com/apache/datafusion/pull/22416#discussion_r3283452344
##########
datafusion/common/src/utils/mod.rs:
##########
@@ -395,6 +395,70 @@ pub fn longest_consecutive_prefix<T: Borrow<usize>>(
count
}
+/// Splits `vec` at index `n`, returning the first `n` elements and leaving the
+/// remaining `vec.len() - n` elements in `vec`.
+///
+/// Allocates for whichever side is smaller, so the new allocation is
+/// `min(n, vec.len() - n)` rather than always `n` (as
`vec.drain(0..n).collect()`
+/// would). This matters when the split emits a prefix under memory pressure,
+/// where `n` can be close to `vec.len()`.
+pub fn split_vec_min_alloc<T>(vec: &mut Vec<T>, n: usize) -> Vec<T> {
+ if n * 2 <= vec.len() {
+ vec.drain(0..n).collect()
+ } else {
+ let remaining = vec.split_off(n);
Review Comment:
I was meaning to add this, but since you're in this area, I appreciate it if
you could apply it.
This is important because in datafusion memory tracking is done by looking
at capacity, not length, so we shouldn't hand off `Vec`s with small length but
very large capacity.
```suggestion
let remaining = vec.split_off(n);
vec.shrink_to_fit();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
