rich7420 opened a new issue, #7145: URL: https://github.com/apache/datafusion-comet/issues/7145
### What is the problem the feature request solves? `list_extract` (#6498) and `map_extract` (#6845) compact a nested `take` result when a buffer's capacity exceeds 4× its used size plus 64 bytes. The threshold has only been measured at the kernel level, where compaction looks like pure cost: for 8,192 `map<int,array<int>>` rows with 128-element values and ~75% missing keys, `m[1]` runs 29–32% slower while output memory drops from 4,228,100 to 1,079,300 bytes ([review](https://github.com/apache/datafusion-comet/pull/6845#discussion_r4323356562)). The benefit shows up downstream: operators that buffer batches reserve by capacity, so oversized output spills earlier and holds more memory. ### Describe the potential solution Add a query-level benchmark that feeds nested `m[k]`/`element_at` and `arr[i]` results into a buffering operator (for example, sort or a hash-join build), so that it measures CPU time, peak memory and spills together. Then tune one shared threshold in `nested_compaction.rs` for both kernels, or keep 4× if it holds up. ### Additional context Follow-up from @andygrove's [approval](https://github.com/apache/datafusion-comet/pull/6845#pullrequestreview-5585674483) of #6845. Both kernels should keep one policy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
