[ https://issues.apache.org/jira/browse/CALCITE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ruben Q L updated CALCITE-6468: ------------------------------- Description: The problem can be reproduced with this query (a "simplified" version of TPC-DS query1): {code:sql} WITH agg_sal AS (SELECT deptno, sum(sal) AS total FROM emp GROUP BY deptno) SELECT 1 FROM agg_sal s1 WHERE s1.total > (SELECT avg(total) FROM agg_sal s2 WHERE s1.deptno = s2.deptno) {code} If we apply subquery program, FilterAggregateTransposeRule and then we call the RelDecorrelator, it will fail with: {noformat} java.lang.AssertionError at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:581) at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:495) ... {noformat} The problem appears in this assert (RelDecorrelator.java:581): {code} assert newPos == newInputOutput.size(); {code} The root cause seems to be that, a few lines before, when processing the correlating variables from {{corDefOutputs}} a certain value is inserted in {{mapNewInputToProjOutputs}}: {code} if (!frame.corDefOutputs.isEmpty()) { for (Map.Entry<CorDef, Integer> entry : frame.corDefOutputs.entrySet()) { RexInputRef.add2(projects, entry.getValue(), newInputOutput); corDefOutputs.put(entry.getKey(), newPos); mapNewInputToProjOutputs.put(entry.getValue(), newPos); // <-- HERE newPos++; } } {code} The problem is that this value was already in the map, as it had been inserted previously as part of the group key processing: {code} for (int i = 0; i < oldGroupKeyCount; i++) { final int idx = groupKeyIndices.get(i); ... // add mapping of group keys. outputMap.put(idx, newPos); int newInputPos = requireNonNull(frame.oldToNewOutputs.get(idx)); RexInputRef.add2(projects, newInputPos, newInputOutput); mapNewInputToProjOutputs.put(newInputPos, newPos); // <-- HERE added firstly newPos++; } {code} Therefore, the unnecessary insertion into {{mapNewInputToProjOutputs}} and the subsequent increment of {{newPos}} when the {{CorDef}}s are processed leads to the mismatch. Notice how, right before the assertion, when processing the remaining fields, it is verified that the value is not already contained on the {{mapNewInputToProjOutputs}}: {code} // add the remaining fields final int newGroupKeyCount = newPos; for (int i = 0; i < newInputOutput.size(); i++) { if (!mapNewInputToProjOutputs.containsKey(i)) { // <-- HERE checked RexInputRef.add2(projects, i, newInputOutput); mapNewInputToProjOutputs.put(i, newPos); newPos++; } } {code} Thus, probably the solution would be to apply the same logic when the CorDef are processed: {code} if (!frame.corDefOutputs.isEmpty()) { for (Map.Entry<CorDef, Integer> entry : frame.corDefOutputs.entrySet()) { Integer pos = mapNewInputToProjOutputs.get(entry.getValue()); // <-- HERE verify map if (pos == null) { RexInputRef.add2(projects, entry.getValue(), newInputOutput); corDefOutputs.put(entry.getKey(), newPos); mapNewInputToProjOutputs.put(entry.getValue(), newPos); newPos++; } else { corDefOutputs.put(entry.getKey(), pos); } } } {code} was: The problem can be reproduced with this query (a "simplified" version of TPC-DS query1): {code:sql} WITH agg_sal AS (SELECT deptno, sum(sal) AS total FROM emp GROUP BY deptno) SELECT 1 FROM agg_sal s1 WHERE s1.total > (SELECT avg(total) FROM agg_sal s2 WHERE s1.deptno = s2.deptno) {code} If we apply subquery program, FilterAggregateTransposeRule and then we call the RelDecorrelator, it will fail with: {noformat} java.lang.AssertionError at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:581) at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:495) ... {noformat} The problem appears in this assert (RelDecorrelator.java:581): {code} assert newPos == newInputOutput.size(); {code} The root cause seems to be that, a few lines before, when processing the correlating variables from {{corDefOutputs}} a certain value is inserted in {{mapNewInputToProjOutputs}}: {code} if (!frame.corDefOutputs.isEmpty()) { for (Map.Entry<CorDef, Integer> entry : frame.corDefOutputs.entrySet()) { RexInputRef.add2(projects, entry.getValue(), newInputOutput); corDefOutputs.put(entry.getKey(), newPos); mapNewInputToProjOutputs.put(entry.getValue(), newPos); // <-- HERE newPos++; } } {code} The problem is that this value was already in the map, as it had been inserted previously as part of the group key processing: {code} for (int i = 0; i < oldGroupKeyCount; i++) { final int idx = groupKeyIndices.get(i); ... // add mapping of group keys. outputMap.put(idx, newPos); int newInputPos = requireNonNull(frame.oldToNewOutputs.get(idx)); RexInputRef.add2(projects, newInputPos, newInputOutput); mapNewInputToProjOutputs.put(newInputPos, newPos); // <-- HERE added firstly newPos++; } {code} Therefore, the unnecessary insertion into {{mapNewInputToProjOutputs}} and the subsequent increment of {{newPos}} when the {{CorDef}}s are processed leads to the mismatch. Notice how, right before the assertion, when processing the remaining fields, it is verified that the value is not already contained on the {{mapNewInputToProjOutputs}}: {code} // add the remaining fields final int newGroupKeyCount = newPos; for (int i = 0; i < newInputOutput.size(); i++) { if (!mapNewInputToProjOutputs.containsKey(i)) { // <-- HERE checked RexInputRef.add2(projects, i, newInputOutput); mapNewInputToProjOutputs.put(i, newPos); newPos++; } } {code} Thus, probably the solution would be to apply the same logic when the CorDef are processed: {code} if (!frame.corDefOutputs.isEmpty()) { for (Map.Entry<CorDef, Integer> entry : frame.corDefOutputs.entrySet()) { Integer pos = mapNewInputToProjOutputs.get(entry.getValue()); if (pos == null) { RexInputRef.add2(projects, entry.getValue(), newInputOutput); corDefOutputs.put(entry.getKey(), newPos); mapNewInputToProjOutputs.put(entry.getValue(), newPos); newPos++; } else { corDefOutputs.put(entry.getKey(), pos); } } } {code} > RelDecorrelator throws AssertionError if correlated variable is used as > Aggregate group key > ------------------------------------------------------------------------------------------- > > Key: CALCITE-6468 > URL: https://issues.apache.org/jira/browse/CALCITE-6468 > Project: Calcite > Issue Type: Bug > Components: core > Affects Versions: 1.37.0 > Reporter: Ruben Q L > Assignee: Ruben Q L > Priority: Major > Fix For: 1.38.0 > > > The problem can be reproduced with this query (a "simplified" version of > TPC-DS query1): > {code:sql} > WITH agg_sal AS > (SELECT deptno, sum(sal) AS total FROM emp GROUP BY deptno) > SELECT 1 FROM agg_sal s1 > WHERE s1.total > (SELECT avg(total) FROM agg_sal s2 WHERE s1.deptno = > s2.deptno) > {code} > If we apply subquery program, FilterAggregateTransposeRule and then we call > the RelDecorrelator, it will fail with: > {noformat} > java.lang.AssertionError > at > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:581) > at > org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:495) > ... > {noformat} > The problem appears in this assert (RelDecorrelator.java:581): > {code} > assert newPos == newInputOutput.size(); > {code} > The root cause seems to be that, a few lines before, when processing the > correlating variables from {{corDefOutputs}} a certain value is inserted in > {{mapNewInputToProjOutputs}}: > {code} > if (!frame.corDefOutputs.isEmpty()) { > for (Map.Entry<CorDef, Integer> entry : frame.corDefOutputs.entrySet()) { > RexInputRef.add2(projects, entry.getValue(), newInputOutput); > corDefOutputs.put(entry.getKey(), newPos); > mapNewInputToProjOutputs.put(entry.getValue(), newPos); // <-- HERE > newPos++; > } > } > {code} > The problem is that this value was already in the map, as it had been > inserted previously as part of the group key processing: > {code} > for (int i = 0; i < oldGroupKeyCount; i++) { > final int idx = groupKeyIndices.get(i); > ... > // add mapping of group keys. > outputMap.put(idx, newPos); > int newInputPos = requireNonNull(frame.oldToNewOutputs.get(idx)); > RexInputRef.add2(projects, newInputPos, newInputOutput); > mapNewInputToProjOutputs.put(newInputPos, newPos); // <-- HERE added firstly > newPos++; > } > {code} > Therefore, the unnecessary insertion into {{mapNewInputToProjOutputs}} and > the subsequent increment of {{newPos}} when the {{CorDef}}s are processed > leads to the mismatch. > Notice how, right before the assertion, when processing the remaining fields, > it is verified that the value is not already contained on the > {{mapNewInputToProjOutputs}}: > {code} > // add the remaining fields > final int newGroupKeyCount = newPos; > for (int i = 0; i < newInputOutput.size(); i++) { > if (!mapNewInputToProjOutputs.containsKey(i)) { // <-- HERE checked > RexInputRef.add2(projects, i, newInputOutput); > mapNewInputToProjOutputs.put(i, newPos); > newPos++; > } > } > {code} > Thus, probably the solution would be to apply the same logic when the CorDef > are processed: > {code} > if (!frame.corDefOutputs.isEmpty()) { > for (Map.Entry<CorDef, Integer> entry : frame.corDefOutputs.entrySet()) { > Integer pos = mapNewInputToProjOutputs.get(entry.getValue()); // <-- HERE > verify map > if (pos == null) { > RexInputRef.add2(projects, entry.getValue(), newInputOutput); > corDefOutputs.put(entry.getKey(), newPos); > mapNewInputToProjOutputs.put(entry.getValue(), newPos); > newPos++; > } else { > corDefOutputs.put(entry.getKey(), pos); > } > } > } > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)