[
https://issues.apache.org/jira/browse/CALCITE-7574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
leishp updated CALCITE-7574:
----------------------------
Description:
`RelDecorrelator.isFieldNotNullRecursive` throws `IndexOutOfBoundsException`
when decorrelating a correlated scalar subquery that involves an Aggregate. The
bug was introduced in CALCITE-6962 which added the
`isFieldNotNull`/`isFieldNotNullRecursive` methods.
### Root Cause
In `RelDecorrelator.isFieldNotNullRecursive`, the Aggregate branch at line 3856:
```java
} else if (rel instanceof Aggregate) {
Aggregate agg = (Aggregate) rel;
ImmutableBitSet groupSet = agg.getGroupSet();
if (index >= groupSet.size())
{ // BUG: should be agg.getGroupCount() return false; }
return isFieldNotNullRecursive(agg.getInput(),
groupSet.asList().get(index));
}
```
- `ImmutableBitSet.size()` returns the bitset {*}{{*}}capacity{{*}}{*}
(`words.length * 64`), typically 64 or more
- `groupSet.asList().get(index)` internally calls `nth(index)`, which requires
`index < cardinality()`
- `Aggregate.getGroupCount()` returns `groupSet.cardinality()`, the actual
number of group keys
When a field index corresponds to an aggregate result field (not a group
field), the bounds check `index >= groupSet.size()` is too lenient (e.g., `1 >=
64` is false), allowing execution to proceed to `nth(1)` which throws
`IndexOutOfBoundsException` because the bitset has only 1 set bit.
#
##
### Reproduction SQL
```sql
select t1.deptno, t1.total_sal,
(select count(distinct t2.total_sal) + 1
from (select deptno, sum(sal) as total_sal
from emp where deptno is not null
group by deptno) t2
where t2.deptno = t1.deptno
and t2.total_sal > t1.total_sal) as rank_sal
from (select deptno, sum(sal) as total_sal
from emp where deptno is not null
group by deptno) t1
order by t1.deptno, t1.total_sal desc
```
### Exception
```
java.lang.IndexOutOfBoundsException: index out of range: 1
at org.apache.calcite.util.ImmutableBitSet.nth(ImmutableBitSet.java:903)
at org.apache.calcite.util.ImmutableBitSet$2.get(ImmutableBitSet.java:681)
at org.apache.calcite.util.ImmutableBitSet$2.get(ImmutableBitSet.java:679)
at
org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNullRecursive(RelDecorrelator.java:3859)
at
org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNullRecursive(RelDecorrelator.java:3851)
at
org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNullRecursive(RelDecorrelator.java:3859)
at
org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNullRecursive(RelDecorrelator.java:3872)
at
org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNullRecursive(RelDecorrelator.java:3851)
at
org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNull(RelDecorrelator.java:3840)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1902)
...
```
##
was:
`RelDecorrelator.isFieldNotNullRecursive` throws `IndexOutOfBoundsException`
when decorrelating a correlated scalar subquery that involves an Aggregate. The
bug was introduced in [CALCITE-6962] which added the
`isFieldNotNull`/`isFieldNotNullRecursive` methods.
### Root Cause
In `RelDecorrelator.isFieldNotNullRecursive`, the Aggregate branch at line 3856:
```java
} else if (rel instanceof Aggregate) {
Aggregate agg = (Aggregate) rel;
ImmutableBitSet groupSet = agg.getGroupSet();
if (index >= groupSet.size())
{ // BUG: should be agg.getGroupCount() return false; }
return isFieldNotNullRecursive(agg.getInput(),
groupSet.asList().get(index));
}
```
- `ImmutableBitSet.size()` returns the bitset *{*}capacity{*}* (`words.length
* 64`), typically 64 or more
- `groupSet.asList().get(index)` internally calls `nth(index)`, which requires
`index < cardinality()`
- `Aggregate.getGroupCount()` returns `groupSet.cardinality()`, the actual
number of group keys
When a field index corresponds to an aggregate result field (not a group
field), the bounds check `index >= groupSet.size()` is too lenient (e.g., `1 >=
64` is false), allowing execution to proceed to `nth(1)` which throws
`IndexOutOfBoundsException` because the bitset has only 1 set bit.
### Reproduction SQL
```sql
select t1.deptno, t1.total_sal,
(select count(distinct t2.total_sal) + 1
from (select deptno, sum(sal) as total_sal
from emp where deptno is not null
group by deptno) t2
where t2.deptno = t1.deptno
and t2.total_sal > t1.total_sal) as rank_sal
from (select deptno, sum(sal) as total_sal
from emp where deptno is not null
group by deptno) t1
order by t1.deptno, t1.total_sal desc
```
### Exception
```
java.lang.IndexOutOfBoundsException: index out of range: 1
at org.apache.calcite.util.ImmutableBitSet.nth(ImmutableBitSet.java:903)
at org.apache.calcite.util.ImmutableBitSet$2.get(ImmutableBitSet.java:681)
at org.apache.calcite.util.ImmutableBitSet$2.get(ImmutableBitSet.java:679)
at
org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNullRecursive(RelDecorrelator.java:3859)
at
org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNullRecursive(RelDecorrelator.java:3851)
at
org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNullRecursive(RelDecorrelator.java:3859)
at
org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNullRecursive(RelDecorrelator.java:3872)
at
org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNullRecursive(RelDecorrelator.java:3851)
at
org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNull(RelDecorrelator.java:3840)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1902)
...
```
##
> RelDecorrelator.isFieldNotNullRecursive throws IndexOutOfBoundsException when
> decorrelating correlated scalar subquery with Aggregate
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CALCITE-7574
> URL: https://issues.apache.org/jira/browse/CALCITE-7574
> Project: Calcite
> Issue Type: Bug
> Components: core
> Affects Versions: 1.41.0, 1.42.0
> Reporter: leishp
> Priority: Major
>
> `RelDecorrelator.isFieldNotNullRecursive` throws `IndexOutOfBoundsException`
> when decorrelating a correlated scalar subquery that involves an Aggregate.
> The bug was introduced in CALCITE-6962 which added the
> `isFieldNotNull`/`isFieldNotNullRecursive` methods.
> ### Root Cause
> In `RelDecorrelator.isFieldNotNullRecursive`, the Aggregate branch at line
> 3856:
>
> ```java
> } else if (rel instanceof Aggregate) {
> Aggregate agg = (Aggregate) rel;
> ImmutableBitSet groupSet = agg.getGroupSet();
> if (index >= groupSet.size())
> { // BUG: should be agg.getGroupCount() return false; }
> return isFieldNotNullRecursive(agg.getInput(),
> groupSet.asList().get(index));
> }
> ```
>
> - `ImmutableBitSet.size()` returns the bitset {*}{{*}}capacity{{*}}{*}
> (`words.length * 64`), typically 64 or more
> - `groupSet.asList().get(index)` internally calls `nth(index)`, which
> requires `index < cardinality()`
> - `Aggregate.getGroupCount()` returns `groupSet.cardinality()`, the actual
> number of group keys
> When a field index corresponds to an aggregate result field (not a group
> field), the bounds check `index >= groupSet.size()` is too lenient (e.g., `1
> >= 64` is false), allowing execution to proceed to `nth(1)` which throws
> `IndexOutOfBoundsException` because the bitset has only 1 set bit.
> #
> ##
> ### Reproduction SQL
> ```sql
> select t1.deptno, t1.total_sal,
> (select count(distinct t2.total_sal) + 1
> from (select deptno, sum(sal) as total_sal
> from emp where deptno is not null
> group by deptno) t2
> where t2.deptno = t1.deptno
> and t2.total_sal > t1.total_sal) as rank_sal
> from (select deptno, sum(sal) as total_sal
> from emp where deptno is not null
> group by deptno) t1
> order by t1.deptno, t1.total_sal desc
> ```
> ### Exception
> ```
> java.lang.IndexOutOfBoundsException: index out of range: 1
> at org.apache.calcite.util.ImmutableBitSet.nth(ImmutableBitSet.java:903)
> at org.apache.calcite.util.ImmutableBitSet$2.get(ImmutableBitSet.java:681)
> at org.apache.calcite.util.ImmutableBitSet$2.get(ImmutableBitSet.java:679)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNullRecursive(RelDecorrelator.java:3859)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNullRecursive(RelDecorrelator.java:3851)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNullRecursive(RelDecorrelator.java:3859)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNullRecursive(RelDecorrelator.java:3872)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNullRecursive(RelDecorrelator.java:3851)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.isFieldNotNull(RelDecorrelator.java:3840)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1902)
> ...
> ```
> ##
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
