kosiew opened a new pull request, #26154: URL: https://github.com/apache/datafusion/pull/26154
## Which issue does this PR close? - Part of #23393 ## Rationale for this change Grouped byte MIN/MAX accumulators can underreport memory usage because their accounting is based on the current data length and number of group slots rather than the memory allocations they retain. For example, replacing a 128-byte string with a one-byte string reuses the existing vector allocation, but the previous accounting reports only the new string length. Similarly, unused capacity in the outer group vector is not counted. This change makes memory accounting reflect retained allocations, providing more accurate memory usage estimates for grouped MIN/MAX operations. ## What changes are included in this PR? - Add `total_data_capacity` to `MinMaxBytesState` to track retained inner `Vec<u8>` capacities independently of `total_data_bytes`, which remains responsible for output preallocation. - Update `set_value()` to maintain capacity accounting when values are inserted, replaced, or cause an existing vector to grow. - Update `size()` to account for the outer `min_max` vector's capacity and the cached inner vector capacities, without double-counting the inline `Option<Vec<u8>>` descriptors. - Keep `size()` O(1) by maintaining the inner capacity total incrementally. - Update `emit_to()` to reset capacity accounting on full emission and subtract emitted inner vector capacities on partial emission. ## Are these changes tested? Three regression tests are added: - `size_counts_outer_capacity`: Verifies that memory accounting includes allocated outer vector capacity, even when fewer group slots are initialized. - `size_retains_inner_capacity`: Tests both MIN and MAX accumulators, verifying that replacing a long string with a shorter one preserves retained-capacity accounting and that partial and full emission update it correctly. - `size_tracks_growth_and_emit_reuse`: Verifies capacity accounting when an inner vector grows, groups are partially emitted, and the state is reused after full emission. ## Are there any user-facing changes? No changes to SQL MIN/MAX results or public APIs are intended. The change corrects internal memory accounting for grouped byte MIN/MAX accumulators, allowing retained allocations to be reported more accurately. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
