2010YOUY01 commented on code in PR #24061:
URL: https://github.com/apache/datafusion/pull/24061#discussion_r3725842781
##########
datafusion/physical-plan/src/aggregates/hash_stream.rs:
##########
@@ -804,119 +1125,305 @@ impl FinalHashAggregateStream {
fn handle_reading_input(
&mut self,
cx: &mut Context<'_>,
- mut original_state: FinalHashAggregateState,
+ original_state: FinalHashAggregateState,
) -> FinalHashAggregateStateTransition {
- debug_assert!(matches!(
- &original_state,
- FinalHashAggregateState::ReadingInput { .. }
- ));
- debug_assert!(original_state.hash_table().is_building());
+ let FinalHashAggregateState::ReadingInput {
+ mut hash_table,
+ spill_context,
+ } = original_state
+ else {
+ return Self::break_with_internal_err(
+ "Final hash aggregate stream expected ReadingInput state",
+ );
+ };
match self.input.poll_next_unpin(cx) {
- Poll::Pending => ControlFlow::Break((Poll::Pending,
original_state)),
+ Poll::Pending => ControlFlow::Break((
+ Poll::Pending,
+ FinalHashAggregateState::ReadingInput {
+ hash_table,
+ spill_context,
+ },
+ )),
Poll::Ready(Some(Ok(batch))) => {
let elapsed_compute =
self.baseline_metrics.elapsed_compute().clone();
let timer = elapsed_compute.timer();
- let result =
original_state.hash_table_mut().aggregate_batch(&batch);
+ let result = hash_table.aggregate_batch(&batch);
timer.done();
if let Err(e) = result {
- return ControlFlow::Break((
- Poll::Ready(Some(Err(e))),
- original_state,
- ));
+ return Self::break_with_err(e);
}
- if self.hit_soft_group_limit(original_state.hash_table()) {
+ if self.hit_soft_group_limit(&hash_table) {
Review Comment:
Updated in
[084f633](https://github.com/apache/datafusion/pull/24061/commits/084f633eefead6041cd1ee4aa229ce3dc810cde7)
I applied the same logic: if spilled then don't trigger soft limit
optimization. Those limits are usually small constants, so they're unlikely to
be co-exist with spilling, so I think this extra check would be enough.
BTW I found only good AI models are able to find such tricky bugs
effectively, they're really hard to construct tests for. It's quite valuable to
let coding agents scan the codebase to find logic inconsistencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
