mohitgurav20 opened a new pull request, #25750:
URL: https://github.com/apache/datafusion/pull/25750
## Which issue does this PR close?
Closes #25735
Related to #25188
## Rationale for this change
### Background
`GroupValuesRows` is the general-purpose **fallback** `GroupValues`
implementation used for nested types (`Struct`, `List`, `Map`,
`RunEndEncoded`), multi-column schemas that cannot be handled by
`GroupValuesColumn`, and any schema containing unsupported column types.
Its `size()` method is the authoritative byte count fed into DataFusion's
query-execution **memory pool**, which uses it to decide when to apply spill
pressure. Under-reporting here directly weakens memory-spill signals for
exactly the schemas — nested and multi-column fallback — that are the most
expensive at runtime.
### Root Cause
Before this PR, `GroupValuesRows` tracked its hash-table footprint through an
incrementally maintained `map_size: usize` side-channel, updated in two
places:
**During insertion** via `HashTableAllocExt::insert_accounted` in `proxy.rs`:
```rust
if self.len() == self.capacity() {
let bump_elements = self.capacity().max(16);
let bump_size = bump_elements * size_of::<T>(); // ← entry slots only
*accounting = (*accounting).checked_add(bump_size).expect("overflow");
self.reserve(bump_elements, &hasher);
}
self.insert_unique(hash, x, hasher);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
