comphead opened a new issue, #27591:
URL: https://github.com/apache/datafusion/issues/27591
### Is your feature request related to a problem or challenge?
Once `SkipAggregationProbe` decides to skip partial aggregation, the
decision is final. `update_state` sets `is_locked = should_skip`, and the
stream converts every remaining batch to state rows without aggregating. The
probe only sees the rows just past
`skip_partial_aggregation_probe_rows_threshold`, so two kinds of input lose:
- **Keys whose pattern changes after the probe.** 400K distinct keys
followed by 1.6M rows over 1000 keys:
```sql
SET datafusion.execution.target_partitions = 2;
EXPLAIN ANALYZE
SELECT CASE WHEN value <= 400000 THEN value ELSE value % 1000 END AS k,
SUM(value)
FROM generate_series(1, 2000000)
GROUP BY k;
```
The Partial skips 1.79M rows and sends 2.00M rows downstream. With
`skip_partial_aggregation_probe_ratio_threshold = 1.0` it sends 401K. This
happens with or without a memory limit. Query time is about the same on one
node (0.012 s and 0.013 s), so the cost is the 5x larger Partial output, which
matters most where that output is shuffled over the network.
- **A burst of memory pressure.** With #27351 the probe also counts groups
emitted early under memory pressure. If the pool is tight while the probe is
still deciding, for example because another operator or a concurrent query
holds memory, the bounded table reduces little and the stream starts skipping.
It keeps skipping after the memory is released. The same data shows how much
the decision depends on the memory at that moment: 20M rows over 50K shuffled
keys skip under a 16M fair pool but reduce 50x without a limit.
### Describe the solution you'd like
Re-probe after skipping. Trino re-enables partial aggregation after
processing 200 times the window it used to disable it, and resets its counters
(`PartialAggregationController`). DuckDB resumes hash table lookups after
skipping 128K to 1M rows (`RadixHTAdaptivity::MaybeResumeLookups`). In
DataFusion this could be: after skipping some multiple of
`skip_partial_aggregation_probe_rows_threshold` rows, aggregate the next window
into an empty table, check its ratio, and then keep aggregating or skip again.
This is point 3 of #22405. Its prototype deferred re-probing because
re-entering aggregation after a skip panicked in `multi_group_by/primitive.rs`.
That looks like the stale hash map after `emit(EmitTo::All)` that #27501 fixed.
### Describe alternatives you've considered
- Re-probe only when the memory pool frees up. That covers the burst case
but not a key pattern that changes.
- Estimate repetition in the skipped batches with a sketch instead of
aggregating them, as Polars does with an F2 sketch before growing its bounded
table. That adds hashing to every skipped row.
### Additional context
Raised in the review of #27351:
https://github.com/apache/datafusion/pull/27351#discussion_r4551339139.
Related: #22405, #27350, #26124.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]