[jira] [Commented] (CALCITE-6261) RelBuilder.aggregate() field pruning does not use permuted field indices when used with force project and duplicate agg calls

2024-02-22 Thread Niels Pardon (Jira)


[ 
https://issues.apache.org/jira/browse/CALCITE-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819738#comment-17819738
 ] 

Niels Pardon commented on CALCITE-6261:
---

I finally got all the internal approvals I need and will submit a PR tomorrow.

> RelBuilder.aggregate() field pruning does not use permuted field indices when 
> used with force project and duplicate agg calls
> -
>
> Key: CALCITE-6261
> URL: https://issues.apache.org/jira/browse/CALCITE-6261
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.35.0, 1.36.0
>Reporter: Niels Pardon
>Assignee: Niels Pardon
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.37.0
>
>
> We were running into issues with RelBuilder.aggregate() when we tried to 
> upgrade our code from Calcite 1.32.0 to 1.36.0 and after some debugging it 
> seems this is caused by changes introduced with CALCITE-4334.
> I was able to put together this test case for RelBuilderTest reproducing the 
> issue:
> {code:java}
> @Test void testAggregateForceProject() {
>   final Function f1 = builder ->
>   builder.scan("EMP")
>   .project(
>   ImmutableList.of(
>   builder.field("EMPNO"),
>   builder.field("ENAME"),
>   builder.field("JOB"),
>   builder.field("MGR"),
>   builder.field("HIREDATE"),
>   builder.field("SAL"),
>   builder.field("COMM"),
>   builder.field("DEPTNO")),
>   ImmutableList.of(),
>   true);
>   final Function f2 = builder ->
>   builder.aggregate(
>   builder.groupKey(builder.field("MGR")),
>   builder.avg(false, "SALARY_AVG", builder.field("SAL")),
>   builder.sum(false, "SALARY_SUM", builder.field("SAL")),
>   builder.avg(false, "SALARY_MEAN", builder.field("SAL")));
>   final String expected = ""
>   + "LogicalProject(MGR=[$0], SALARY_AVG=[$1], SALARY_SUM=[$2], 
> SALARY_MEAN=[$1])\n"
>   + "  LogicalAggregate(group=[{0}], SALARY_AVG=[AVG($1)], 
> SALARY_SUM=[SUM($1)])\n"
>   + "LogicalProject(MGR=[$3], SAL=[$5])\n"
>   + "  LogicalTableScan(table=[[scott, EMP]])\n";
>   assertThat(f2.apply(f1.apply(createBuilder())).build(), hasTree(expected));
> } {code}
> With Calcite 1.36.0 you will get an AssertionError:
> {code:java}
> java.lang.AssertionError
>     at org.apache.calcite.rel.core.Aggregate.(Aggregate.java:175)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:72)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create_(LogicalAggregate.java:144)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create(LogicalAggregate.java:116)
>     at 
> org.apache.calcite.rel.core.RelFactories$AggregateFactoryImpl.createAggregate(RelFactories.java:333)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2576)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2523)
>     at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:2335)
>     at 
> org.apache.calcite.test.RelBuilderTest.lambda$testAggregateForceProject$36(RelBuilderTest.java:1509)
>     at 
> org.apache.calcite.test.RelBuilderTest.testAggregateForceProject(RelBuilderTest.java:1522)
>     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)
>     at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>     at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
>     at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>     at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>     at 
> org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
>     at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
>     at 
> 

[jira] [Commented] (CALCITE-6261) RelBuilder.aggregate() field pruning does not use permuted field indices when used with force project and duplicate agg calls

2024-02-22 Thread Niels Pardon (Jira)


[ 
https://issues.apache.org/jira/browse/CALCITE-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819528#comment-17819528
 ] 

Niels Pardon commented on CALCITE-6261:
---

[~phanimahidhar] apologies accepted. It's also my first contribution to this 
community so I guess we are both learning here.

> RelBuilder.aggregate() field pruning does not use permuted field indices when 
> used with force project and duplicate agg calls
> -
>
> Key: CALCITE-6261
> URL: https://issues.apache.org/jira/browse/CALCITE-6261
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.35.0, 1.36.0
>Reporter: Niels Pardon
>Assignee: Niels Pardon
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.37.0
>
>
> We were running into issues with RelBuilder.aggregate() when we tried to 
> upgrade our code from Calcite 1.32.0 to 1.36.0 and after some debugging it 
> seems this is caused by changes introduced with CALCITE-4334.
> I was able to put together this test case for RelBuilderTest reproducing the 
> issue:
> {code:java}
> @Test void testAggregateForceProject() {
>   final Function f1 = builder ->
>   builder.scan("EMP")
>   .project(
>   ImmutableList.of(
>   builder.field("EMPNO"),
>   builder.field("ENAME"),
>   builder.field("JOB"),
>   builder.field("MGR"),
>   builder.field("HIREDATE"),
>   builder.field("SAL"),
>   builder.field("COMM"),
>   builder.field("DEPTNO")),
>   ImmutableList.of(),
>   true);
>   final Function f2 = builder ->
>   builder.aggregate(
>   builder.groupKey(builder.field("MGR")),
>   builder.avg(false, "SALARY_AVG", builder.field("SAL")),
>   builder.sum(false, "SALARY_SUM", builder.field("SAL")),
>   builder.avg(false, "SALARY_MEAN", builder.field("SAL")));
>   final String expected = ""
>   + "LogicalProject(MGR=[$0], SALARY_AVG=[$1], SALARY_SUM=[$2], 
> SALARY_MEAN=[$1])\n"
>   + "  LogicalAggregate(group=[{0}], SALARY_AVG=[AVG($1)], 
> SALARY_SUM=[SUM($1)])\n"
>   + "LogicalProject(MGR=[$3], SAL=[$5])\n"
>   + "  LogicalTableScan(table=[[scott, EMP]])\n";
>   assertThat(f2.apply(f1.apply(createBuilder())).build(), hasTree(expected));
> } {code}
> With Calcite 1.36.0 you will get an AssertionError:
> {code:java}
> java.lang.AssertionError
>     at org.apache.calcite.rel.core.Aggregate.(Aggregate.java:175)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:72)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create_(LogicalAggregate.java:144)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create(LogicalAggregate.java:116)
>     at 
> org.apache.calcite.rel.core.RelFactories$AggregateFactoryImpl.createAggregate(RelFactories.java:333)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2576)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2523)
>     at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:2335)
>     at 
> org.apache.calcite.test.RelBuilderTest.lambda$testAggregateForceProject$36(RelBuilderTest.java:1509)
>     at 
> org.apache.calcite.test.RelBuilderTest.testAggregateForceProject(RelBuilderTest.java:1522)
>     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)
>     at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>     at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
>     at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>     at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>     at 
> org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
>     at 
> 

[jira] [Commented] (CALCITE-6261) RelBuilder.aggregate() field pruning does not use permuted field indices when used with force project and duplicate agg calls

2024-02-21 Thread Niels Pardon (Jira)


[ 
https://issues.apache.org/jira/browse/CALCITE-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819492#comment-17819492
 ] 

Niels Pardon commented on CALCITE-6261:
---

[~phanimahidhar] thanks for the PR I guess though I was already working on a PR 
and thus had assigned the issue to me.

> RelBuilder.aggregate() field pruning does not use permuted field indices when 
> used with force project and duplicate agg calls
> -
>
> Key: CALCITE-6261
> URL: https://issues.apache.org/jira/browse/CALCITE-6261
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.35.0, 1.36.0
>Reporter: Niels Pardon
>Assignee: Niels Pardon
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.37.0
>
>
> We were running into issues with RelBuilder.aggregate() when we tried to 
> upgrade our code from Calcite 1.32.0 to 1.36.0 and after some debugging it 
> seems this is caused by changes introduced with CALCITE-4334.
> I was able to put together this test case for RelBuilderTest reproducing the 
> issue:
> {code:java}
> @Test void testAggregateForceProject() {
>   final Function f1 = builder ->
>   builder.scan("EMP")
>   .project(
>   ImmutableList.of(
>   builder.field("EMPNO"),
>   builder.field("ENAME"),
>   builder.field("JOB"),
>   builder.field("MGR"),
>   builder.field("HIREDATE"),
>   builder.field("SAL"),
>   builder.field("COMM"),
>   builder.field("DEPTNO")),
>   ImmutableList.of(),
>   true);
>   final Function f2 = builder ->
>   builder.aggregate(
>   builder.groupKey(builder.field("MGR")),
>   builder.avg(false, "SALARY_AVG", builder.field("SAL")),
>   builder.sum(false, "SALARY_SUM", builder.field("SAL")),
>   builder.avg(false, "SALARY_MEAN", builder.field("SAL")));
>   final String expected = ""
>   + "LogicalProject(MGR=[$0], SALARY_AVG=[$1], SALARY_SUM=[$2], 
> SALARY_MEAN=[$1])\n"
>   + "  LogicalAggregate(group=[{0}], SALARY_AVG=[AVG($1)], 
> SALARY_SUM=[SUM($1)])\n"
>   + "LogicalProject(MGR=[$3], SAL=[$5])\n"
>   + "  LogicalTableScan(table=[[scott, EMP]])\n";
>   assertThat(f2.apply(f1.apply(createBuilder())).build(), hasTree(expected));
> } {code}
> With Calcite 1.36.0 you will get an AssertionError:
> {code:java}
> java.lang.AssertionError
>     at org.apache.calcite.rel.core.Aggregate.(Aggregate.java:175)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:72)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create_(LogicalAggregate.java:144)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create(LogicalAggregate.java:116)
>     at 
> org.apache.calcite.rel.core.RelFactories$AggregateFactoryImpl.createAggregate(RelFactories.java:333)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2576)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2523)
>     at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:2335)
>     at 
> org.apache.calcite.test.RelBuilderTest.lambda$testAggregateForceProject$36(RelBuilderTest.java:1509)
>     at 
> org.apache.calcite.test.RelBuilderTest.testAggregateForceProject(RelBuilderTest.java:1522)
>     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)
>     at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>     at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
>     at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>     at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>     at 
> org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
>     at 
> 

[jira] [Commented] (CALCITE-6261) RelBuilder.aggregate() field pruning does not use permuted field indices when used with force project and duplicate agg calls

2024-02-15 Thread Niels Pardon (Jira)


[ 
https://issues.apache.org/jira/browse/CALCITE-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817618#comment-17817618
 ] 

Niels Pardon commented on CALCITE-6261:
---

I tried recreating both scenarios in SQL and I can't reproduce the issue there. 
These are the Quidem test cases I used:
{code:java}
# test duplicate agg calls with field pruning issue 5888
select min(e.SAL), max(e.SAL), min(e.SAL), max(e.SAL) from emp as e join dept 
as d using (deptno) group by cast(NULL as integer);
++-++-+
| EXPR$0 | EXPR$1  | EXPR$2 | EXPR$3  |
++-++-+
| 800.00 | 5000.00 | 800.00 | 5000.00 |
++-++-+
(1 row)

!ok

# test duplicate agg calls with field pruning issue 6261
select mgr, min(e.SAL), max(e.SAL), min(e.SAL), max(e.SAL) from (select empno, 
ename, job, mgr, hiredate, sal, comm, deptno from emp) as e group by e.mgr;
+--+-+-+-+-+
| MGR  | EXPR$1  | EXPR$2  | EXPR$3  | EXPR$4  |
+--+-+-+-+-+
| 7566 | 3000.00 | 3000.00 | 3000.00 | 3000.00 |
| 7698 |  950.00 | 1600.00 |  950.00 | 1600.00 |
| 7782 | 1300.00 | 1300.00 | 1300.00 | 1300.00 |
| 7788 | 1100.00 | 1100.00 | 1100.00 | 1100.00 |
| 7839 | 2450.00 | 2975.00 | 2450.00 | 2975.00 |
| 7902 |  800.00 |  800.00 |  800.00 |  800.00 |
|  | 5000.00 | 5000.00 | 5000.00 | 5000.00 |
+--+-+-+-+-+
(7 rows)

!ok {code}
Not sure if there's something I'm missing in the SQL variant, to me they look 
logically the same to the RelBuilder calls.

> RelBuilder.aggregate() field pruning does not use permuted field indices when 
> used with force project and duplicate agg calls
> -
>
> Key: CALCITE-6261
> URL: https://issues.apache.org/jira/browse/CALCITE-6261
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.35.0, 1.36.0
>Reporter: Niels Pardon
>Assignee: Niels Pardon
>Priority: Major
> Fix For: 1.37.0
>
>
> We were running into issues with RelBuilder.aggregate() when we tried to 
> upgrade our code from Calcite 1.32.0 to 1.36.0 and after some debugging it 
> seems this is caused by changes introduced with CALCITE-4334.
> I was able to put together this test case for RelBuilderTest reproducing the 
> issue:
> {code:java}
> @Test void testAggregateForceProject() {
>   final Function f1 = builder ->
>   builder.scan("EMP")
>   .project(
>   ImmutableList.of(
>   builder.field("EMPNO"),
>   builder.field("ENAME"),
>   builder.field("JOB"),
>   builder.field("MGR"),
>   builder.field("HIREDATE"),
>   builder.field("SAL"),
>   builder.field("COMM"),
>   builder.field("DEPTNO")),
>   ImmutableList.of(),
>   true);
>   final Function f2 = builder ->
>   builder.aggregate(
>   builder.groupKey(builder.field("MGR")),
>   builder.avg(false, "SALARY_AVG", builder.field("SAL")),
>   builder.sum(false, "SALARY_SUM", builder.field("SAL")),
>   builder.avg(false, "SALARY_MEAN", builder.field("SAL")));
>   final String expected = ""
>   + "LogicalProject(MGR=[$0], SALARY_AVG=[$1], SALARY_SUM=[$2], 
> SALARY_MEAN=[$1])\n"
>   + "  LogicalAggregate(group=[{0}], SALARY_AVG=[AVG($1)], 
> SALARY_SUM=[SUM($1)])\n"
>   + "LogicalProject(MGR=[$3], SAL=[$5])\n"
>   + "  LogicalTableScan(table=[[scott, EMP]])\n";
>   assertThat(f2.apply(f1.apply(createBuilder())).build(), hasTree(expected));
> } {code}
> With Calcite 1.36.0 you will get an AssertionError:
> {code:java}
> java.lang.AssertionError
>     at org.apache.calcite.rel.core.Aggregate.(Aggregate.java:175)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:72)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create_(LogicalAggregate.java:144)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create(LogicalAggregate.java:116)
>     at 
> org.apache.calcite.rel.core.RelFactories$AggregateFactoryImpl.createAggregate(RelFactories.java:333)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2576)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2523)
>     at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:2335)
>     at 
> org.apache.calcite.test.RelBuilderTest.lambda$testAggregateForceProject$36(RelBuilderTest.java:1509)
>     at 
> org.apache.calcite.test.RelBuilderTest.testAggregateForceProject(RelBuilderTest.java:1522)
>     at 

[jira] [Commented] (CALCITE-6261) RelBuilder.aggregate() field pruning does not use permuted field indices when used with force project and duplicate agg calls

2024-02-14 Thread Niels Pardon (Jira)


[ 
https://issues.apache.org/jira/browse/CALCITE-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817455#comment-17817455
 ] 

Niels Pardon commented on CALCITE-6261:
---

[~pmsgd] I would use the following unit test to recreate your reproducer from 
CALCITE-5888, please tell me if you think this correctly represents your issue. 
It uses slightly less verbose syntax and the existing tables used in 
RelBuilderTest instead of your CalciqueTable classes. Though it should be 
structurally the same.
{code:java}
@Test void 
testAggregateDuplicateAggCallsAndFieldPruningWithJoinAndLiteralGroupKey() {
  final Function f1 = builder ->
  // first inner join two tables
  builder.scan("EMP").scan("DEPT")
  .join(JoinRelType.INNER, "DEPTNO")
  .aggregate(
  // null group key
  builder.groupKey(builder.cast(builder.literal(null), 
SqlTypeName.INTEGER)),
  // duplicated min/max agg calls
  builder.min(builder.field("SAL")),
  builder.max(builder.field("SAL")),
  builder.min(builder.field("SAL")),
  builder.max(builder.field("SAL")))
  .build();
  final String expected = ""
  + "LogicalProject($f11=[$0], $f1=[$1], $f2=[$2], $f10=[$1], $f20=[$2])\n"
  + "  LogicalAggregate(group=[{1}], agg#0=[MIN($0)], agg#1=[MAX($0)])\n"
  + "LogicalProject(SAL=[$5], $f11=[null:INTEGER])\n"
  + "  LogicalJoin(condition=[=($7, $8)], joinType=[inner])\n"
  + "LogicalTableScan(table=[[scott, EMP]])\n"
  + "LogicalTableScan(table=[[scott, DEPT]])\n";
  assertThat(f1.apply(createBuilder()), hasTree(expected));
} {code}

> RelBuilder.aggregate() field pruning does not use permuted field indices when 
> used with force project and duplicate agg calls
> -
>
> Key: CALCITE-6261
> URL: https://issues.apache.org/jira/browse/CALCITE-6261
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.35.0, 1.36.0
>Reporter: Niels Pardon
>Assignee: Niels Pardon
>Priority: Major
> Fix For: 1.37.0
>
>
> We were running into issues with RelBuilder.aggregate() when we tried to 
> upgrade our code from Calcite 1.32.0 to 1.36.0 and after some debugging it 
> seems this is caused by changes introduced with CALCITE-4334.
> I was able to put together this test case for RelBuilderTest reproducing the 
> issue:
> {code:java}
> @Test void testAggregateForceProject() {
>   final Function f1 = builder ->
>   builder.scan("EMP")
>   .project(
>   ImmutableList.of(
>   builder.field("EMPNO"),
>   builder.field("ENAME"),
>   builder.field("JOB"),
>   builder.field("MGR"),
>   builder.field("HIREDATE"),
>   builder.field("SAL"),
>   builder.field("COMM"),
>   builder.field("DEPTNO")),
>   ImmutableList.of(),
>   true);
>   final Function f2 = builder ->
>   builder.aggregate(
>   builder.groupKey(builder.field("MGR")),
>   builder.avg(false, "SALARY_AVG", builder.field("SAL")),
>   builder.sum(false, "SALARY_SUM", builder.field("SAL")),
>   builder.avg(false, "SALARY_MEAN", builder.field("SAL")));
>   final String expected = ""
>   + "LogicalProject(MGR=[$0], SALARY_AVG=[$1], SALARY_SUM=[$2], 
> SALARY_MEAN=[$1])\n"
>   + "  LogicalAggregate(group=[{0}], SALARY_AVG=[AVG($1)], 
> SALARY_SUM=[SUM($1)])\n"
>   + "LogicalProject(MGR=[$3], SAL=[$5])\n"
>   + "  LogicalTableScan(table=[[scott, EMP]])\n";
>   assertThat(f2.apply(f1.apply(createBuilder())).build(), hasTree(expected));
> } {code}
> With Calcite 1.36.0 you will get an AssertionError:
> {code:java}
> java.lang.AssertionError
>     at org.apache.calcite.rel.core.Aggregate.(Aggregate.java:175)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:72)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create_(LogicalAggregate.java:144)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create(LogicalAggregate.java:116)
>     at 
> org.apache.calcite.rel.core.RelFactories$AggregateFactoryImpl.createAggregate(RelFactories.java:333)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2576)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2523)
>     at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:2335)
>     at 
> org.apache.calcite.test.RelBuilderTest.lambda$testAggregateForceProject$36(RelBuilderTest.java:1509)
>     at 
> 

[jira] [Commented] (CALCITE-6261) RelBuilder.aggregate() field pruning does not use permuted field indices when used with force project and duplicate agg calls

2024-02-14 Thread Niels Pardon (Jira)


[ 
https://issues.apache.org/jira/browse/CALCITE-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817443#comment-17817443
 ] 

Niels Pardon commented on CALCITE-6261:
---

Would love to make it for 1.37.0. Just waiting for my employer right now so 
that I can push the submit PR button :) 

> RelBuilder.aggregate() field pruning does not use permuted field indices when 
> used with force project and duplicate agg calls
> -
>
> Key: CALCITE-6261
> URL: https://issues.apache.org/jira/browse/CALCITE-6261
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.35.0, 1.36.0
>Reporter: Niels Pardon
>Assignee: Niels Pardon
>Priority: Major
> Fix For: 1.37.0
>
>
> We were running into issues with RelBuilder.aggregate() when we tried to 
> upgrade our code from Calcite 1.32.0 to 1.36.0 and after some debugging it 
> seems this is caused by changes introduced with CALCITE-4334.
> I was able to put together this test case for RelBuilderTest reproducing the 
> issue:
> {code:java}
> @Test void testAggregateForceProject() {
>   final Function f1 = builder ->
>   builder.scan("EMP")
>   .project(
>   ImmutableList.of(
>   builder.field("EMPNO"),
>   builder.field("ENAME"),
>   builder.field("JOB"),
>   builder.field("MGR"),
>   builder.field("HIREDATE"),
>   builder.field("SAL"),
>   builder.field("COMM"),
>   builder.field("DEPTNO")),
>   ImmutableList.of(),
>   true);
>   final Function f2 = builder ->
>   builder.aggregate(
>   builder.groupKey(builder.field("MGR")),
>   builder.avg(false, "SALARY_AVG", builder.field("SAL")),
>   builder.sum(false, "SALARY_SUM", builder.field("SAL")),
>   builder.avg(false, "SALARY_MEAN", builder.field("SAL")));
>   final String expected = ""
>   + "LogicalProject(MGR=[$0], SALARY_AVG=[$1], SALARY_SUM=[$2], 
> SALARY_MEAN=[$1])\n"
>   + "  LogicalAggregate(group=[{0}], SALARY_AVG=[AVG($1)], 
> SALARY_SUM=[SUM($1)])\n"
>   + "LogicalProject(MGR=[$3], SAL=[$5])\n"
>   + "  LogicalTableScan(table=[[scott, EMP]])\n";
>   assertThat(f2.apply(f1.apply(createBuilder())).build(), hasTree(expected));
> } {code}
> With Calcite 1.36.0 you will get an AssertionError:
> {code:java}
> java.lang.AssertionError
>     at org.apache.calcite.rel.core.Aggregate.(Aggregate.java:175)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:72)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create_(LogicalAggregate.java:144)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create(LogicalAggregate.java:116)
>     at 
> org.apache.calcite.rel.core.RelFactories$AggregateFactoryImpl.createAggregate(RelFactories.java:333)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2576)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2523)
>     at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:2335)
>     at 
> org.apache.calcite.test.RelBuilderTest.lambda$testAggregateForceProject$36(RelBuilderTest.java:1509)
>     at 
> org.apache.calcite.test.RelBuilderTest.testAggregateForceProject(RelBuilderTest.java:1522)
>     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)
>     at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>     at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
>     at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>     at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>     at 
> org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
>     at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
>     at 
> 

[jira] [Assigned] (CALCITE-6261) RelBuilder.aggregate() field pruning does not use permuted field indices when used with force project and duplicate agg calls

2024-02-14 Thread Niels Pardon (Jira)


 [ 
https://issues.apache.org/jira/browse/CALCITE-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Pardon reassigned CALCITE-6261:
-

Assignee: Niels Pardon

> RelBuilder.aggregate() field pruning does not use permuted field indices when 
> used with force project and duplicate agg calls
> -
>
> Key: CALCITE-6261
> URL: https://issues.apache.org/jira/browse/CALCITE-6261
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.35.0, 1.36.0
>Reporter: Niels Pardon
>Assignee: Niels Pardon
>Priority: Major
>
> We were running into issues with RelBuilder.aggregate() when we tried to 
> upgrade our code from Calcite 1.32.0 to 1.36.0 and after some debugging it 
> seems this is caused by changes introduced with CALCITE-4334.
> I was able to put together this test case for RelBuilderTest reproducing the 
> issue:
> {code:java}
> @Test void testAggregateForceProject() {
>   final Function f1 = builder ->
>   builder.scan("EMP")
>   .project(
>   ImmutableList.of(
>   builder.field("EMPNO"),
>   builder.field("ENAME"),
>   builder.field("JOB"),
>   builder.field("MGR"),
>   builder.field("HIREDATE"),
>   builder.field("SAL"),
>   builder.field("COMM"),
>   builder.field("DEPTNO")),
>   ImmutableList.of(),
>   true);
>   final Function f2 = builder ->
>   builder.aggregate(
>   builder.groupKey(builder.field("MGR")),
>   builder.avg(false, "SALARY_AVG", builder.field("SAL")),
>   builder.sum(false, "SALARY_SUM", builder.field("SAL")),
>   builder.avg(false, "SALARY_MEAN", builder.field("SAL")));
>   final String expected = ""
>   + "LogicalProject(MGR=[$0], SALARY_AVG=[$1], SALARY_SUM=[$2], 
> SALARY_MEAN=[$1])\n"
>   + "  LogicalAggregate(group=[{0}], SALARY_AVG=[AVG($1)], 
> SALARY_SUM=[SUM($1)])\n"
>   + "LogicalProject(MGR=[$3], SAL=[$5])\n"
>   + "  LogicalTableScan(table=[[scott, EMP]])\n";
>   assertThat(f2.apply(f1.apply(createBuilder())).build(), hasTree(expected));
> } {code}
> With Calcite 1.36.0 you will get an AssertionError:
> {code:java}
> java.lang.AssertionError
>     at org.apache.calcite.rel.core.Aggregate.(Aggregate.java:175)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:72)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create_(LogicalAggregate.java:144)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create(LogicalAggregate.java:116)
>     at 
> org.apache.calcite.rel.core.RelFactories$AggregateFactoryImpl.createAggregate(RelFactories.java:333)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2576)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2523)
>     at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:2335)
>     at 
> org.apache.calcite.test.RelBuilderTest.lambda$testAggregateForceProject$36(RelBuilderTest.java:1509)
>     at 
> org.apache.calcite.test.RelBuilderTest.testAggregateForceProject(RelBuilderTest.java:1522)
>     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)
>     at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>     at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
>     at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>     at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>     at 
> org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
>     at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
>     at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
>     at 
> 

[jira] [Assigned] (CALCITE-5888) Assertion error in aggregate

2024-02-14 Thread Niels Pardon (Jira)


 [ 
https://issues.apache.org/jira/browse/CALCITE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Pardon reassigned CALCITE-5888:
-

Assignee: Niels Pardon

> Assertion error in aggregate
> 
>
> Key: CALCITE-5888
> URL: https://issues.apache.org/jira/browse/CALCITE-5888
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.35.0, 1.36.0
>Reporter: Petr Masopust
>Assignee: Niels Pardon
>Priority: Major
>
> We have {{relBuilder.aggregate(groupKey, aggregateCalls)}} in our code with 
> values {{[AS($31, 'a_label_d_opp_stage_id_foodmart_bd4c9c91212f9f'), 
> AS(CAST(FLOOR($34, FLAG(QUARTER))):TIMESTAMP(0), 
> 'a_label_snapshot_timestamp_quarter_foodmart_89fe710c8628d9')]}} and 
> {{[COUNT(CASE(AND(COALESCE($37, false), =($36, $13)), $15, null:NULL)), 
> SUM(CASE(AND(COALESCE($37, false), =($36, $13)), $2, null:NULL)), 
> MAX(CASE(AND(COALESCE($37, false), =($36, $13)), 0, null:NULL)), 
> MAX(CASE(AND(COALESCE($37, false), =($36, $13)), 0, null:NULL))].}}
> It works perfectly in version 1.34.0 but in 1.35.0 we got this exception:
> {{java.lang.AssertionError}}
> {{    at org.apache.calcite.rel.core.Aggregate.(Aggregate.java:175)}}
> {{    at 
> org.apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:72)}}
> {{    at 
> org.apache.calcite.rel.logical.LogicalAggregate.create_(LogicalAggregate.java:144)}}
> {{    at 
> org.apache.calcite.rel.logical.LogicalAggregate.create(LogicalAggregate.java:116)}}
> {{    at 
> org.apache.calcite.rel.core.RelFactories$AggregateFactoryImpl.createAggregate(RelFactories.java:328)}}
> {{    at 
> org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2564)}}
> {{    at 
> org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2511)}}
> {{    at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:2348)}}
>  
> I think it is either missing {{permute}} or assert should compare 
> {{cardinality}} instead of {{length.}} Because it compares field index? to 
> number of fields which looks like nonsense to me.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (CALCITE-5888) Assertion error in aggregate

2024-02-13 Thread Niels Pardon (Jira)


[ 
https://issues.apache.org/jira/browse/CALCITE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816983#comment-17816983
 ] 

Niels Pardon commented on CALCITE-5888:
---

As I mentioned in my duplicate issue CALCITE-6261 I do have a fix and will 
provide a PR soon.

> Assertion error in aggregate
> 
>
> Key: CALCITE-5888
> URL: https://issues.apache.org/jira/browse/CALCITE-5888
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.35.0, 1.36.0
>Reporter: Petr Masopust
>Priority: Major
>
> We have {{relBuilder.aggregate(groupKey, aggregateCalls)}} in our code with 
> values {{[AS($31, 'a_label_d_opp_stage_id_foodmart_bd4c9c91212f9f'), 
> AS(CAST(FLOOR($34, FLAG(QUARTER))):TIMESTAMP(0), 
> 'a_label_snapshot_timestamp_quarter_foodmart_89fe710c8628d9')]}} and 
> {{[COUNT(CASE(AND(COALESCE($37, false), =($36, $13)), $15, null:NULL)), 
> SUM(CASE(AND(COALESCE($37, false), =($36, $13)), $2, null:NULL)), 
> MAX(CASE(AND(COALESCE($37, false), =($36, $13)), 0, null:NULL)), 
> MAX(CASE(AND(COALESCE($37, false), =($36, $13)), 0, null:NULL))].}}
> It works perfectly in version 1.34.0 but in 1.35.0 we got this exception:
> {{java.lang.AssertionError}}
> {{    at org.apache.calcite.rel.core.Aggregate.(Aggregate.java:175)}}
> {{    at 
> org.apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:72)}}
> {{    at 
> org.apache.calcite.rel.logical.LogicalAggregate.create_(LogicalAggregate.java:144)}}
> {{    at 
> org.apache.calcite.rel.logical.LogicalAggregate.create(LogicalAggregate.java:116)}}
> {{    at 
> org.apache.calcite.rel.core.RelFactories$AggregateFactoryImpl.createAggregate(RelFactories.java:328)}}
> {{    at 
> org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2564)}}
> {{    at 
> org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2511)}}
> {{    at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:2348)}}
>  
> I think it is either missing {{permute}} or assert should compare 
> {{cardinality}} instead of {{length.}} Because it compares field index? to 
> number of fields which looks like nonsense to me.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (CALCITE-6261) RelBuilder.aggregate() field pruning does not use permuted field indices when used with force project and duplicate agg calls

2024-02-12 Thread Niels Pardon (Jira)


[ 
https://issues.apache.org/jira/browse/CALCITE-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816718#comment-17816718
 ] 

Niels Pardon commented on CALCITE-6261:
---

I'm planning on submitting a PR. I just need to go through some internal 
approval processes of my employer before I can submit it.

> RelBuilder.aggregate() field pruning does not use permuted field indices when 
> used with force project and duplicate agg calls
> -
>
> Key: CALCITE-6261
> URL: https://issues.apache.org/jira/browse/CALCITE-6261
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.35.0, 1.36.0
>Reporter: Niels Pardon
>Priority: Major
>
> We were running into issues with RelBuilder.aggregate() when we tried to 
> upgrade our code from Calcite 1.32.0 to 1.36.0 and after some debugging it 
> seems this is caused by changes introduced with CALCITE-4334.
> I was able to put together this test case for RelBuilderTest reproducing the 
> issue:
> {code:java}
> @Test void testAggregateForceProject() {
>   final Function f1 = builder ->
>   builder.scan("EMP")
>   .project(
>   ImmutableList.of(
>   builder.field("EMPNO"),
>   builder.field("ENAME"),
>   builder.field("JOB"),
>   builder.field("MGR"),
>   builder.field("HIREDATE"),
>   builder.field("SAL"),
>   builder.field("COMM"),
>   builder.field("DEPTNO")),
>   ImmutableList.of(),
>   true);
>   final Function f2 = builder ->
>   builder.aggregate(
>   builder.groupKey(builder.field("MGR")),
>   builder.avg(false, "SALARY_AVG", builder.field("SAL")),
>   builder.sum(false, "SALARY_SUM", builder.field("SAL")),
>   builder.avg(false, "SALARY_MEAN", builder.field("SAL")));
>   final String expected = ""
>   + "LogicalProject(MGR=[$0], SALARY_AVG=[$1], SALARY_SUM=[$2], 
> SALARY_MEAN=[$1])\n"
>   + "  LogicalAggregate(group=[{0}], SALARY_AVG=[AVG($1)], 
> SALARY_SUM=[SUM($1)])\n"
>   + "LogicalProject(MGR=[$3], SAL=[$5])\n"
>   + "  LogicalTableScan(table=[[scott, EMP]])\n";
>   assertThat(f2.apply(f1.apply(createBuilder())).build(), hasTree(expected));
> } {code}
> With Calcite 1.36.0 you will get an AssertionError:
> {code:java}
> java.lang.AssertionError
>     at org.apache.calcite.rel.core.Aggregate.(Aggregate.java:175)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:72)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create_(LogicalAggregate.java:144)
>     at 
> org.apache.calcite.rel.logical.LogicalAggregate.create(LogicalAggregate.java:116)
>     at 
> org.apache.calcite.rel.core.RelFactories$AggregateFactoryImpl.createAggregate(RelFactories.java:333)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2576)
>     at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2523)
>     at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:2335)
>     at 
> org.apache.calcite.test.RelBuilderTest.lambda$testAggregateForceProject$36(RelBuilderTest.java:1509)
>     at 
> org.apache.calcite.test.RelBuilderTest.testAggregateForceProject(RelBuilderTest.java:1522)
>     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)
>     at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>     at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
>     at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>     at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>     at 
> org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
>     at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
>     at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
>     at 
> 

[jira] [Created] (CALCITE-6261) RelBuilder.aggregate() field pruning does not use permuted field indices when used with force project and duplicate agg calls

2024-02-12 Thread Niels Pardon (Jira)
Niels Pardon created CALCITE-6261:
-

 Summary: RelBuilder.aggregate() field pruning does not use 
permuted field indices when used with force project and duplicate agg calls
 Key: CALCITE-6261
 URL: https://issues.apache.org/jira/browse/CALCITE-6261
 Project: Calcite
  Issue Type: Bug
  Components: core
Affects Versions: 1.36.0, 1.35.0
Reporter: Niels Pardon


We were running into issues with RelBuilder.aggregate() when we tried to 
upgrade our code from Calcite 1.32.0 to 1.36.0 and after some debugging it 
seems this is caused by changes introduced with CALCITE-4334.

I was able to put together this test case for RelBuilderTest reproducing the 
issue:
{code:java}
@Test void testAggregateForceProject() {
  final Function f1 = builder ->
  builder.scan("EMP")
  .project(
  ImmutableList.of(
  builder.field("EMPNO"),
  builder.field("ENAME"),
  builder.field("JOB"),
  builder.field("MGR"),
  builder.field("HIREDATE"),
  builder.field("SAL"),
  builder.field("COMM"),
  builder.field("DEPTNO")),
  ImmutableList.of(),
  true);
  final Function f2 = builder ->
  builder.aggregate(
  builder.groupKey(builder.field("MGR")),
  builder.avg(false, "SALARY_AVG", builder.field("SAL")),
  builder.sum(false, "SALARY_SUM", builder.field("SAL")),
  builder.avg(false, "SALARY_MEAN", builder.field("SAL")));
  final String expected = ""
  + "LogicalProject(MGR=[$0], SALARY_AVG=[$1], SALARY_SUM=[$2], 
SALARY_MEAN=[$1])\n"
  + "  LogicalAggregate(group=[{0}], SALARY_AVG=[AVG($1)], 
SALARY_SUM=[SUM($1)])\n"
  + "LogicalProject(MGR=[$3], SAL=[$5])\n"
  + "  LogicalTableScan(table=[[scott, EMP]])\n";
  assertThat(f2.apply(f1.apply(createBuilder())).build(), hasTree(expected));
} {code}
With Calcite 1.36.0 you will get an AssertionError:
{code:java}
java.lang.AssertionError
    at org.apache.calcite.rel.core.Aggregate.(Aggregate.java:175)
    at 
org.apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:72)
    at 
org.apache.calcite.rel.logical.LogicalAggregate.create_(LogicalAggregate.java:144)
    at 
org.apache.calcite.rel.logical.LogicalAggregate.create(LogicalAggregate.java:116)
    at 
org.apache.calcite.rel.core.RelFactories$AggregateFactoryImpl.createAggregate(RelFactories.java:333)
    at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2576)
    at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2523)
    at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:2335)
    at 
org.apache.calcite.test.RelBuilderTest.lambda$testAggregateForceProject$36(RelBuilderTest.java:1509)
    at 
org.apache.calcite.test.RelBuilderTest.testAggregateForceProject(RelBuilderTest.java:1522)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
    at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
    at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
    at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
    at 
org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
    at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
    at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
    at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
    at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
    at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
    at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
    at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
    at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
    at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
    at