kosiew commented on code in PR #25188:
URL: https://github.com/apache/datafusion/pull/25188#discussion_r4101104022
##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/partial_table.rs:
##########
@@ -106,6 +107,65 @@ impl AggregateHashTable<PartialMarker> {
})
}
+ /// Starts a bounded-memory drain of partial aggregate states.
+ pub(in crate::aggregates) fn start_early_emit(&mut self) {
+ self.start_outputting();
+ }
+
+ /// Emits at most one output batch while releasing its groups from the
table.
+ ///
+ /// Unlike terminal output, this must not materialize all states: early
+ /// emission can be triggered precisely because the complete state does not
+ /// fit in the memory pool. Once drained, rebuild an empty table so raw
input
+ /// aggregation can resume.
+ pub(in crate::aggregates) fn next_early_emit_batch(
+ &mut self,
+ ) -> Result<Option<RecordBatch>> {
+ let state_schema = Arc::clone(&self.state_schema);
+ let accumulator_metrics =
Arc::clone(&self.aggregate_accumulator_metrics);
+ let group_by_metrics = self.group_by_metrics.clone();
+ let AggregateHashTableState::Outputting(mut state) =
+ std::mem::replace(&mut self.state, AggregateHashTableState::Done)
+ else {
+ return Ok(None);
+ };
+
+ let emit_to =
EmitTo::First(self.batch_size.min(state.group_values.len()));
Review Comment:
[8b69b81](https://github.com/apache/datafusion/pull/25188/commits/8b69b81c03a2c794d6f818b6e8b86093e890d58b)-
restore `take_state_batch` and the materialized terminal-output path, retain
the `GroupValues::size()` fixes, and raise the grouping-sets spill budget to
700 B. Bounded early emission needs separate design and performance coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
