[
https://issues.apache.org/jira/browse/CALCITE-7687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Etienne Pelissier updated CALCITE-7687:
---------------------------------------
Description:
{{[RelMdSelectivity#getSelectivity|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdSelectivity.java#L178](Aggregate
rel, RelMetadataQuery mq, @Nullable RexNode predicate)}} method:
{code:java}
public @Nullable Double getSelectivity(Aggregate rel, RelMetadataQuery mq,
@Nullable RexNode predicate) {
...
RelOptUtil.splitFilters(
rel.getGroupSet(),
predicate,
pushable,
notPushable);
final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
RexNode childPred =
RexUtil.composeConjunction(rexBuilder, pushable, true);
Double selectivity = mq.getSelectivity(rel.getInput(), childPred);
...
}
{code}
{{[RelMdDistinctRowCount#getDistinctRowCount|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdDistinctRowCount.java#L168](Aggregate
rel, RelMetadataQuery mq, ImmutableBitSet groupKey, @Nullable RexNode
predicate)}} method:
{code:java}
public @Nullable Double getDistinctRowCount(Aggregate rel, RelMetadataQuery
mq,
ImmutableBitSet groupKey, @Nullable RexNode predicate) {
...
RelOptUtil.splitFilters(
ImmutableBitSet.range(rel.getGroupCount()),
predicate,
pushable,
notPushable);
final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
RexNode childPreds =
RexUtil.composeConjunction(rexBuilder, pushable, true);
// set the bits as they correspond to the child input
ImmutableBitSet.Builder childKey = ImmutableBitSet.builder();
RelMdUtil.setAggChildKeys(groupKey, rel, childKey);
Double distinctRowCount =
mq.getDistinctRowCount(rel.getInput(), childKey.build(), childPreds);
...
}
{code}
[CALCITE-4414|https://issues.apache.org/jira/browse/CALCITE-4414] fixed the
analogous problem in RelMdSelectivity#getSelectivity for Calc: it pushes a
predicate to its input, and RelOptUtil.pushPastCalc converts it to input-column
references first, which is possible because Calc output fields are themselves
expressions over the input.
Aggregate's group-key output fields are also references to specific input
fields, so predicates that reference only group keys can be translated the same
way, via an analogous RelOptUtil.pushPastAggregate, which currently doesn't
exist. Aggregate call outputs (e.g., COUNT(x)) have no equivalent input
expression, so a predicate referencing one must not be pushed at all.
FilterAggregateTransposeRule already computes the same group-key mapping to
push a Filter past an Aggregate, and can be used as a basis to implement the
missing RelOptUtil.pushPastAggregate.
The two examples below show the problem for
RelMdSelectivity#getSelectivity(Aggregate) and
RelMdDistinctRowCount#getDistinctRowCount(Aggregate), respectively.
In both cases, the predicates reach the input with the wrong field references,
silently returning statistics for the wrong column rather than raising an
exception.
was:
{{[RelMdSelectivity#getSelectivity|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdSelectivity.java#L178](Aggregate
rel, RelMetadataQuery mq, @Nullable RexNode predicate)}} method:
{code:java}
public @Nullable Double getSelectivity(Aggregate rel, RelMetadataQuery mq,
@Nullable RexNode predicate) {
...
RelOptUtil.splitFilters(
rel.getGroupSet(), // [2]
predicate,
pushable,
notPushable);
final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
RexNode childPred =
RexUtil.composeConjunction(rexBuilder, pushable, true);
// [1]
Double selectivity = mq.getSelectivity(rel.getInput(), childPred);
...
}
{code}
{{[RelMdDistinctRowCount#getDistinctRowCount|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdDistinctRowCount.java#L168](Aggregate
rel, RelMetadataQuery mq, ImmutableBitSet groupKey, @Nullable RexNode
predicate)}} method:
{code:java}
public @Nullable Double getDistinctRowCount(Aggregate rel, RelMetadataQuery
mq,
ImmutableBitSet groupKey, @Nullable RexNode predicate) {
...
RelOptUtil.splitFilters(
ImmutableBitSet.range(rel.getGroupCount()),
predicate,
pushable,
notPushable);
final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
RexNode childPreds =
RexUtil.composeConjunction(rexBuilder, pushable, true);
// set the bits as they correspond to the child input
ImmutableBitSet.Builder childKey = ImmutableBitSet.builder();
RelMdUtil.setAggChildKeys(groupKey, rel, childKey);
// [1]
Double distinctRowCount =
mq.getDistinctRowCount(rel.getInput(), childKey.build(), childPreds);
...
}
{code}
both currently pass down the predicate to their input [1] without considering
any possible translation, since an {{Aggregate}} derives its row type as
{{(group keys..., aggregate calls...)}} and so its output field {{i}} is input
field {{groupSet.nth(i\)}}; hence when the Aggregate's input analyzes the
predicate, it can end up reading a different column from the one the predicate
names.
The consequence is silent rather than an exception, like the tests attached to
the first two comments, where, in the first comment, after
{{RelMdSelectivity#getSelectivity(Aggregate)}} we reach
{{RelMdSelectivity#getSelectivity(TableScan)}} and this method hands the
predicate to the {{BuiltInMetadata.Selectivity.Handler}} that the table exposes
through {{RelOptTable#unwrap}}, which reports the null fraction of a field ($1)
that is not the one the predicate names. This $1 is the Aggregate's second
group key, which is field ($2) of the scan underneath it. The second comment
shows the same thing one method over, where the
{{BuiltInMetadata.DistinctRowCount.Handler}} receives that unconverted ($1)
next to a {{groupKey}} that {{RelMdUtil.setAggChildKeys}} did convert.
Note I: [CALCITE-4414|https://issues.apache.org/jira/browse/CALCITE-4414] is
very closely related to this ticket.
Note II: in a similar situation, {{RelMdSelectivity#getSelectivity(Project)}}
uses {{RelOptUtil.pushPastProject}} and
{{RelMdSelectivity#getSelectivity(Calc)}} uses {{RelOptUtil.pushPastCalc}}
(which "Converts an expression that is based on the output fields of a Calc to
an equivalent expression on the Calc's input fields.") to convert the predicate
before passing it to the input. There is no equivalent
{{RelOptUtil.pushPastAggregate}}, even though {{FilterAggregateTransposeRule}}
already computes that same mapping inline in order to push a {{Filter}} past an
{{Aggregate}}.
> RelMdSelectivity and RelMdDistinctRowCount for Aggregate can propagate a
> predicate with wrong references
> --------------------------------------------------------------------------------------------------------
>
> Key: CALCITE-7687
> URL: https://issues.apache.org/jira/browse/CALCITE-7687
> Project: Calcite
> Issue Type: Bug
> Components: core
> Affects Versions: 1.42.0
> Reporter: Etienne Pelissier
> Assignee: Etienne Pelissier
> Priority: Minor
> Labels: in-progress, pull-request-available
>
> {{[RelMdSelectivity#getSelectivity|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdSelectivity.java#L178](Aggregate
> rel, RelMetadataQuery mq, @Nullable RexNode predicate)}} method:
> {code:java}
> public @Nullable Double getSelectivity(Aggregate rel, RelMetadataQuery mq,
> @Nullable RexNode predicate) {
> ...
> RelOptUtil.splitFilters(
> rel.getGroupSet(),
> predicate,
> pushable,
> notPushable);
> final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
> RexNode childPred =
> RexUtil.composeConjunction(rexBuilder, pushable, true);
> Double selectivity = mq.getSelectivity(rel.getInput(), childPred);
> ...
> }
> {code}
> {{[RelMdDistinctRowCount#getDistinctRowCount|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdDistinctRowCount.java#L168](Aggregate
> rel, RelMetadataQuery mq, ImmutableBitSet groupKey, @Nullable RexNode
> predicate)}} method:
> {code:java}
> public @Nullable Double getDistinctRowCount(Aggregate rel, RelMetadataQuery
> mq,
> ImmutableBitSet groupKey, @Nullable RexNode predicate) {
> ...
> RelOptUtil.splitFilters(
> ImmutableBitSet.range(rel.getGroupCount()),
> predicate,
> pushable,
> notPushable);
> final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
> RexNode childPreds =
> RexUtil.composeConjunction(rexBuilder, pushable, true);
> // set the bits as they correspond to the child input
> ImmutableBitSet.Builder childKey = ImmutableBitSet.builder();
> RelMdUtil.setAggChildKeys(groupKey, rel, childKey);
> Double distinctRowCount =
> mq.getDistinctRowCount(rel.getInput(), childKey.build(), childPreds);
> ...
> }
> {code}
> [CALCITE-4414|https://issues.apache.org/jira/browse/CALCITE-4414] fixed the
> analogous problem in RelMdSelectivity#getSelectivity for Calc: it pushes a
> predicate to its input, and RelOptUtil.pushPastCalc converts it to
> input-column references first, which is possible because Calc output fields
> are themselves expressions over the input.
> Aggregate's group-key output fields are also references to specific input
> fields, so predicates that reference only group keys can be translated the
> same way, via an analogous RelOptUtil.pushPastAggregate, which currently
> doesn't exist. Aggregate call outputs (e.g., COUNT(x)) have no equivalent
> input expression, so a predicate referencing one must not be pushed at all.
> FilterAggregateTransposeRule already computes the same group-key mapping to
> push a Filter past an Aggregate, and can be used as a basis to implement the
> missing RelOptUtil.pushPastAggregate.
> The two examples below show the problem for
> RelMdSelectivity#getSelectivity(Aggregate) and
> RelMdDistinctRowCount#getDistinctRowCount(Aggregate), respectively.
> In both cases, the predicates reach the input with the wrong field
> references, silently returning statistics for the wrong column rather than
> raising an exception.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
