2010YOUY01 commented on code in PR #25696:
URL: https://github.com/apache/datafusion/pull/25696#discussion_r4152896619
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -952,25 +1175,42 @@ impl AggregateExec {
let mut new = self.clone();
match &mut new.kind {
AggregateKind::General { aggr_expr: old, .. } => *old = aggr_expr,
- AggregateKind::DistinctLimit { .. } if aggr_expr.is_empty() => {}
- AggregateKind::DistinctLimit { group_by, .. } => {
- // An accumulator rewrite cannot inherit DISTINCT's early stop.
+ AggregateKind::DistinctLimit { .. } | AggregateKind::TopKDistinct
{ .. }
+ if aggr_expr.is_empty() => {}
+ // Ordering optimization can revisit an existing TopK with the
+ // same expressions. Preserve its specialization in that case.
+ AggregateKind::TopKMinMax {
+ aggr_expr: existing,
+ ..
+ } if matches!(aggr_expr.as_ref(), [new] if Arc::ptr_eq(existing,
new)) => {}
+ AggregateKind::DistinctLimit { group_by, .. }
+ | AggregateKind::TopKMinMax { group_by, .. }
+ | AggregateKind::TopKDistinct { group_by, .. } => {
+ // A replacement expression cannot inherit a specialization
+ // validated for the previous aggregate.
new.kind = AggregateKind::General {
group_by: Arc::clone(group_by),
- // The previous `DistinctLimit` type doesn't include filter
filter_expr: vec![None; aggr_expr.len()].into(),
aggr_expr,
- limit_options: None,
};
}
}
new.metrics = ExecutionPlanMetricsSet::new();
new
}
- /// Clone this exec, overriding only the limit hint.
+ /// Clone with a validated legacy limit hint, falling back to ordinary
+ /// aggregation when the request is unsupported.
+ #[deprecated(
+ since = "56.0.0",
+ note = "This API is intended for internal use only and was
inadvertently made public. Do not use this API."
Review Comment:
This API seems impossible to use it correctly. The updated
`try_optimize_topk` API makes the mutation itself atomic and safe, but it still
has preconditions on the surrounding plan shape. (In this case, the parent
`Limit` operator must remain in the plan.)
For that reason, I would prefer to stay conservative and keep the
replacement API internal for now. Since this change is part of a deprecation,
we can revisit the decision if an external use case comes up.
I updated it in
[0880e98](https://github.com/apache/datafusion/pull/25696/commits/0880e988132bc32ce227f627b6c9c4c55340c957)
to leave a note, let me know if you have different ideas
> The replacement API is [`try_optimize_topk`]. It still requires specific
plan-shape invariants to be used correctly, so it remains internal for now. If
you have a use case that requires this API to be public, please open an issue
in DataFusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
