[
https://issues.apache.org/jira/browse/IMPALA-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18107026#comment-18107026
]
ASF subversion and git services commented on IMPALA-15243:
----------------------------------------------------------
Commit 218b32e2cbafdcb9013a5a2ddd613788bf524286 in impala's branch
refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=218b32e2c ]
IMPALA-15243: Fix costing for CTEs
Addresses missing elements in plan costing when
COMPUTE_PROCESSING_COST=true.
Notably TPC-DS Q39 contains fragments with CTEConsumerNodes and no
ScanNodes, some of which have no child fragments because only the first
CTEConsumerNode contains a link to its producer to avoid creating a DAG.
This may affect the efficacy of CPC with CTEs.
A bunch of other queries also visited some CTEConsumerNodes before the
fragment for their CTEProducerNode. We only want to count the
contribution of the CTEProducerNode once, so it's safe to skip these.
Updates cte-distributed to include parallel planning with costing of a
union over the same CTE twice.
Change-Id: I21b60af313070029a1fa8bf835107bef88f54912
Reviewed-on: http://gerrit.cloudera.org:8080/24665
Reviewed-by: Michael Smith <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Error while applying rule MaterializedViewFilterScanRule for TPC-DS Q39
> -----------------------------------------------------------------------
>
> Key: IMPALA-15243
> URL: https://issues.apache.org/jira/browse/IMPALA-15243
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
> Reporter: Stamatis Zampetakis
> Assignee: Michael Smith
> Priority: Major
>
> IMPALA-15238 adds planner tests for TPC-DS queries with CTE suggestions
> enabled. Currently, the TPC-DS Q39 fails during compilation while trying to
> use some suggestions.
> *Stacktrace*
> {noformat}
> Error Stack:
> java.lang.RuntimeException: java.lang.RuntimeException: Error while applying
> rule MaterializedViewFilterScanRule, args
> [rel#247256:LogicalFilter.NONE(input=RelSubset#247255,condition=AND(IS NOT
> NULL($0), IS NOT NULL($1), IS NOT NULL($3))),
> rel#246073:LogicalTableScan.NONE(table=[tpcds_partitioned_parquet_snap,
> inventory])]
> at
> org.apache.impala.service.Frontend.getTExecRequestWithFallback(Frontend.java:2493)
> at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2153)
> at
> org.apache.impala.planner.PlannerTestBase.testPlan(PlannerTestBase.java:558)
> at
> org.apache.impala.planner.PlannerTestBase.runTestCase(PlannerTestBase.java:459)
> at
> org.apache.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:990)
> at
> org.apache.impala.calcite.planner.TpcdsCtePlannerTest.testQuery(TpcdsCtePlannerTest.java:200)
> at jdk.internal.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:569)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:240)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
> at
> org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
> Caused by: java.lang.RuntimeException: Error while applying rule
> MaterializedViewFilterScanRule, args
> [rel#247256:LogicalFilter.NONE(input=RelSubset#247255,condition=AND(IS NOT
> NULL($0), IS NOT NULL($1), IS NOT NULL($3))),
> rel#246073:LogicalTableScan.NONE(table=[tpcds_partitioned_parquet_snap,
> inventory])]
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:250)
> at
> org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:61)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:525)
> at
> org.apache.impala.calcite.service.CalciteOptimizer.rewriteUsingViews(CalciteOptimizer.java:440)
> at
> org.apache.impala.calcite.service.CalciteOptimizer.runCTEProgram(CalciteOptimizer.java:374)
> at
> org.apache.impala.calcite.service.CalciteOptimizer.optimize(CalciteOptimizer.java:163)
> at
> org.apache.impala.calcite.service.CalciteSingleNodePlanner.createSingleNodePlan(CalciteSingleNodePlanner.java:77)
> at
> org.apache.impala.planner.Planner.createPlanFragments(Planner.java:144)
> at org.apache.impala.planner.Planner.createPlans(Planner.java:550)
> at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2091)
> at
> org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:3552)
> at
> org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:3234)
> at
> org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:2624)
> at
> org.apache.impala.service.Frontend.getTExecRequestWithFallback(Frontend.java:2470)
> ... 41 more
> Caused by: java.lang.RuntimeException: cannot translate
> rel#247252:ImpalaCTEConsumer.NONE(cteName=cte_suggestion_0) to MutableRel
> at
> org.apache.calcite.rel.mutable.MutableRels.toMutable(MutableRels.java:449)
> at
> org.apache.calcite.plan.SubstitutionVisitor.go(SubstitutionVisitor.java:533)
> at
> org.apache.calcite.rel.rules.MaterializedViewFilterScanRule.apply(MaterializedViewFilterScanRule.java:100)
> at
> org.apache.calcite.rel.rules.MaterializedViewFilterScanRule.onMatch(MaterializedViewFilterScanRule.java:79)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223)
> ... 54 more
> {noformat}
> At the moment the error can be reproduced by using the code in IMPALA-15238.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
