[jira] [Assigned] (CALCITE-3174) IS NOT DISTINCT FROM condition pushed from filter to join is not collapsed
[
https://issues.apache.org/jira/browse/CALCITE-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bohdan Kazydub reassigned CALCITE-3174:
---
Assignee: Bohdan Kazydub
> IS NOT DISTINCT FROM condition pushed from filter to join is not collapsed
> --
>
> Key: CALCITE-3174
> URL: https://issues.apache.org/jira/browse/CALCITE-3174
> Project: Calcite
> Issue Type: Bug
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If filter containing {{IS NOT DISTINCT FROM}} expression is pushed to join,
> this results to new join condition having this expanded {{IS NOT DISTINCT
> FROM}} expression (and not collapsed one), which can be further modified
> making impossible to identify the {{IS NOT DISTINCT FROM}} condition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3174) IS NOT DISTINCT FROM condition pushed from filter to join is not collapsed
Bohdan Kazydub created CALCITE-3174:
---
Summary: IS NOT DISTINCT FROM condition pushed from filter to join
is not collapsed
Key: CALCITE-3174
URL: https://issues.apache.org/jira/browse/CALCITE-3174
Project: Calcite
Issue Type: Bug
Reporter: Bohdan Kazydub
If filter containing {{IS NOT DISTINCT FROM}} expression is pushed to join,
this results to new join condition having this expanded {{IS NOT DISTINCT
FROM}} expression (and not collapsed one), which can be further modified making
impossible to identify the {{IS NOT DISTINCT FROM}} condition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-3121) VolcanoPlanner hangs due to removing ORDER BY from sub-query
[
https://issues.apache.org/jira/browse/CALCITE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bohdan Kazydub updated CALCITE-3121:
Affects Version/s: 1.19.0
> VolcanoPlanner hangs due to removing ORDER BY from sub-query
>
>
> Key: CALCITE-3121
> URL: https://issues.apache.org/jira/browse/CALCITE-3121
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Bohdan Kazydub
>Assignee: Danny Chan
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After the fix for CALCITE-2798 some queries hang during planning in
> VolcanoPlanner (similar issue was reported in CALCITE-2223).
> Here is a test case which should be added to the {{RelOptRulesTest}} class:
> {code:java}
> @Test public void testSubQueryWithOrderByHang() {
> String sql = "select n.n_regionkey from ( select * from "
> + "( select * from sales.customer) t order by t.n_regionkey) n where
> n.n_nationkey >1 ";
> VolcanoPlanner planner = new VolcanoPlanner(null, null);
> planner.addRelTraitDef(ConventionTraitDef.INSTANCE);
> Tester dynamicTester = createDynamicTester().withDecorrelation(true)
> .withClusterFactory(
> relOptCluster -> RelOptCluster.create(planner,
> relOptCluster.getRexBuilder()));
> RelRoot root = dynamicTester.convertSqlToRel(sql);
> String planBefore = NL + RelOptUtil.toString(root.rel);
> getDiffRepos().assertEquals("planBefore", "${planBefore}", planBefore);
> PushProjector.ExprCondition exprCondition = expr -> {
> if (expr instanceof RexCall) {
> RexCall call = (RexCall) expr;
> return
> "item".equals(call.getOperator().getName().toLowerCase(Locale.ROOT));
> }
> return false;
> };
> RuleSet ruleSet =
> RuleSets.ofList(
> FilterProjectTransposeRule.INSTANCE,
> FilterMergeRule.INSTANCE,
> ProjectMergeRule.INSTANCE,
> new ProjectFilterTransposeRule(Project.class, Filter .class,
> RelFactories.LOGICAL_BUILDER, exprCondition),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_FILTER_RULE,
> EnumerableRules.ENUMERABLE_SORT_RULE,
> EnumerableRules.ENUMERABLE_LIMIT_RULE,
> EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> root.rel.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode relAfter = program.run(planner, root.rel, toTraits,
> Collections.emptyList(), Collections.emptyList());
> String planAfter = NL + RelOptUtil.toString(relAfter);
> getDiffRepos().assertEquals("planAfter", "${planAfter}", planAfter);
> }
> {code}
> Please note that if {{LIMIT }} is added to the sub-query with order by
> (so it is not removed due to the fix for CALCITE-2798) the test succeeds.
> Though the issue with hanging is more general, I think that if it wouldn't be
> fixed, the fix for CALCITE-2798 should be reverted to reduce cases when
> planner may hang.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3121) VolcanoPlanner hangs due to removing ORDER BY from sub-query
[
https://issues.apache.org/jira/browse/CALCITE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16859862#comment-16859862
]
Bohdan Kazydub commented on CALCITE-3121:
-
Reverted CALCITE-2798 and added unit test in [pull
request|https://github.com/apache/calcite/pull/1264].
Can someone please review?
> VolcanoPlanner hangs due to removing ORDER BY from sub-query
>
>
> Key: CALCITE-3121
> URL: https://issues.apache.org/jira/browse/CALCITE-3121
> Project: Calcite
> Issue Type: Bug
>Reporter: Bohdan Kazydub
>Assignee: Danny Chan
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After the fix for CALCITE-2798 some queries hang during planning in
> VolcanoPlanner (similar issue was reported in CALCITE-2223).
> Here is a test case which should be added to the {{RelOptRulesTest}} class:
> {code:java}
> @Test public void testSubQueryWithOrderByHang() {
> String sql = "select n.n_regionkey from ( select * from "
> + "( select * from sales.customer) t order by t.n_regionkey) n where
> n.n_nationkey >1 ";
> VolcanoPlanner planner = new VolcanoPlanner(null, null);
> planner.addRelTraitDef(ConventionTraitDef.INSTANCE);
> Tester dynamicTester = createDynamicTester().withDecorrelation(true)
> .withClusterFactory(
> relOptCluster -> RelOptCluster.create(planner,
> relOptCluster.getRexBuilder()));
> RelRoot root = dynamicTester.convertSqlToRel(sql);
> String planBefore = NL + RelOptUtil.toString(root.rel);
> getDiffRepos().assertEquals("planBefore", "${planBefore}", planBefore);
> PushProjector.ExprCondition exprCondition = expr -> {
> if (expr instanceof RexCall) {
> RexCall call = (RexCall) expr;
> return
> "item".equals(call.getOperator().getName().toLowerCase(Locale.ROOT));
> }
> return false;
> };
> RuleSet ruleSet =
> RuleSets.ofList(
> FilterProjectTransposeRule.INSTANCE,
> FilterMergeRule.INSTANCE,
> ProjectMergeRule.INSTANCE,
> new ProjectFilterTransposeRule(Project.class, Filter .class,
> RelFactories.LOGICAL_BUILDER, exprCondition),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_FILTER_RULE,
> EnumerableRules.ENUMERABLE_SORT_RULE,
> EnumerableRules.ENUMERABLE_LIMIT_RULE,
> EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> root.rel.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode relAfter = program.run(planner, root.rel, toTraits,
> Collections.emptyList(), Collections.emptyList());
> String planAfter = NL + RelOptUtil.toString(relAfter);
> getDiffRepos().assertEquals("planAfter", "${planAfter}", planAfter);
> }
> {code}
> Please note that if {{LIMIT }} is added to the sub-query with order by
> (so it is not removed due to the fix for CALCITE-2798) the test succeeds.
> Though the issue with hanging is more general, I think that if it wouldn't be
> fixed, the fix for CALCITE-2798 should be reverted to reduce cases when
> planner may hang.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3121) VolcanoPlanner hangs due to removing ORDER BY from sub-query
[
https://issues.apache.org/jira/browse/CALCITE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16859747#comment-16859747
]
Bohdan Kazydub commented on CALCITE-3121:
-
[~hyuan], as was described in description, the test succeeds without
CALCITE-2798 but fails with it. The query you wrote is the same bug, which
starts to reproduce for more cases with the changes in CALCITE-2798.
> VolcanoPlanner hangs due to removing ORDER BY from sub-query
>
>
> Key: CALCITE-3121
> URL: https://issues.apache.org/jira/browse/CALCITE-3121
> Project: Calcite
> Issue Type: Bug
>Reporter: Bohdan Kazydub
>Assignee: Danny Chan
>Priority: Blocker
>
> After the fix for CALCITE-2798 some queries hang during planning in
> VolcanoPlanner (similar issue was reported in CALCITE-2223).
> Here is a test case which should be added to the {{RelOptRulesTest}} class:
> {code:java}
> @Test public void testSubQueryWithOrderByHang() {
> String sql = "select n.n_regionkey from ( select * from "
> + "( select * from sales.customer) t order by t.n_regionkey) n where
> n.n_nationkey >1 ";
> VolcanoPlanner planner = new VolcanoPlanner(null, null);
> planner.addRelTraitDef(ConventionTraitDef.INSTANCE);
> Tester dynamicTester = createDynamicTester().withDecorrelation(true)
> .withClusterFactory(
> relOptCluster -> RelOptCluster.create(planner,
> relOptCluster.getRexBuilder()));
> RelRoot root = dynamicTester.convertSqlToRel(sql);
> String planBefore = NL + RelOptUtil.toString(root.rel);
> getDiffRepos().assertEquals("planBefore", "${planBefore}", planBefore);
> PushProjector.ExprCondition exprCondition = expr -> {
> if (expr instanceof RexCall) {
> RexCall call = (RexCall) expr;
> return
> "item".equals(call.getOperator().getName().toLowerCase(Locale.ROOT));
> }
> return false;
> };
> RuleSet ruleSet =
> RuleSets.ofList(
> FilterProjectTransposeRule.INSTANCE,
> FilterMergeRule.INSTANCE,
> ProjectMergeRule.INSTANCE,
> new ProjectFilterTransposeRule(Project.class, Filter .class,
> RelFactories.LOGICAL_BUILDER, exprCondition),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_FILTER_RULE,
> EnumerableRules.ENUMERABLE_SORT_RULE,
> EnumerableRules.ENUMERABLE_LIMIT_RULE,
> EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> root.rel.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode relAfter = program.run(planner, root.rel, toTraits,
> Collections.emptyList(), Collections.emptyList());
> String planAfter = NL + RelOptUtil.toString(relAfter);
> getDiffRepos().assertEquals("planAfter", "${planAfter}", planAfter);
> }
> {code}
> Please note that if {{LIMIT }} is added to the sub-query with order by
> (so it is not removed due to the fix for CALCITE-2798) the test succeeds.
> Though the issue with hanging is more general, I think that if it wouldn't be
> fixed, the fix for CALCITE-2798 should be reverted to reduce cases when
> planner may hang.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3121) VolcanoPlanner hangs due to removing ORDER BY from sub-query
Bohdan Kazydub created CALCITE-3121:
---
Summary: VolcanoPlanner hangs due to removing ORDER BY from
sub-query
Key: CALCITE-3121
URL: https://issues.apache.org/jira/browse/CALCITE-3121
Project: Calcite
Issue Type: Bug
Reporter: Bohdan Kazydub
After the fix for CALCITE-2798 some queries hang during planning in
VolcanoPlanner (similar issue was reported in CALCITE-2223).
Here is a test case which should be added to the {{RelOptRulesTest}} class:
{code:java}
@Test public void testSubQueryWithOrderByHang() {
String sql = "select n.n_regionkey from ( select * from "
+ "( select * from sales.customer) t order by t.n_regionkey) n where
n.n_nationkey >1 ";
VolcanoPlanner planner = new VolcanoPlanner(null, null);
planner.addRelTraitDef(ConventionTraitDef.INSTANCE);
Tester dynamicTester = createDynamicTester().withDecorrelation(true)
.withClusterFactory(
relOptCluster -> RelOptCluster.create(planner,
relOptCluster.getRexBuilder()));
RelRoot root = dynamicTester.convertSqlToRel(sql);
String planBefore = NL + RelOptUtil.toString(root.rel);
getDiffRepos().assertEquals("planBefore", "${planBefore}", planBefore);
PushProjector.ExprCondition exprCondition = expr -> {
if (expr instanceof RexCall) {
RexCall call = (RexCall) expr;
return
"item".equals(call.getOperator().getName().toLowerCase(Locale.ROOT));
}
return false;
};
RuleSet ruleSet =
RuleSets.ofList(
FilterProjectTransposeRule.INSTANCE,
FilterMergeRule.INSTANCE,
ProjectMergeRule.INSTANCE,
new ProjectFilterTransposeRule(Project.class, Filter .class,
RelFactories.LOGICAL_BUILDER, exprCondition),
EnumerableRules.ENUMERABLE_PROJECT_RULE,
EnumerableRules.ENUMERABLE_FILTER_RULE,
EnumerableRules.ENUMERABLE_SORT_RULE,
EnumerableRules.ENUMERABLE_LIMIT_RULE,
EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
Program program = Programs.of(ruleSet);
RelTraitSet toTraits =
root.rel.getCluster().traitSet()
.replace(0, EnumerableConvention.INSTANCE);
RelNode relAfter = program.run(planner, root.rel, toTraits,
Collections.emptyList(), Collections.emptyList());
String planAfter = NL + RelOptUtil.toString(relAfter);
getDiffRepos().assertEquals("planAfter", "${planAfter}", planAfter);
}
{code}
Please note that if {{LIMIT }} is added to the sub-query with order by (so
it is not removed due to the fix for CALCITE-2798) the test succeeds.
Though the issue with hanging is more general, I think that if it wouldn't be
fixed, the fix for CALCITE-2798 should be reverted to reduce cases when planner
may hang.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3055) TableScan type mismatch in VolcanoPlanner
[
https://issues.apache.org/jira/browse/CALCITE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16848673#comment-16848673
]
Bohdan Kazydub commented on CALCITE-3055:
-
[~danny0405], yes, we also observed a similar problem for HepPlanner, but we
did a workaround with setting `noDag` to {{true}}, so this issue is not seen. I
think it would be better o fix this issue in the scope of other Jira. I'll
create it with the fix and unit test which reproduces it.
> TableScan type mismatch in VolcanoPlanner
> -
>
> Key: CALCITE-3055
> URL: https://issues.apache.org/jira/browse/CALCITE-3055
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Bohdan Kazydub
>Assignee: Danny Chan
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.20.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In Drill with Calcite updated to 1.19.0 we observe following problem:
> the following query
> {code}
> CREATE VIEW nation_view_testunionall_expandable_star AS select n_name,
> n_nationkey FROM cp.`tpch/nation.parquet`;
> (SELECT n_comment, n_regionkey FROM cp.`tpch/nation.parquet` LIMIT 5) UNION
> ALL (SELECT * FROM nation_view_testunionall_expandable_star LIMIT 5)
> {code}
> fails with
> {code}
> 15:46:06.005 [2333c7f1-89f3-0b80-8369-68e24895729e:foreman] ERROR
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: Type
> mismatch:
> left:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey) NOT NULL
> right:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey, ANY n_name, ANY
> n_nationkey) NOT NULL
> {code}
> when Volcano planner is registering rel nodes. The initial plan is the
> following:
> {code}
> LogicalUnion(all=[true]): rowcount = 10.0, cumulative cost = \{520.0 rows,
> 940.755032994728 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 74
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{205.0 rows,
> 365.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 68
> LogicalProject(n_comment=[$1], n_regionkey=[$2]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 67
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 29
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{305.0 rows,
> 565.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 73
> LogicalProject(n_name=[$0], n_nationkey=[$1]): rowcount = 100.0,
> cumulative cost = \{300.0 rows, 501.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 72
> LogicalProject(n_name=[$3], n_nationkey=[$4]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 71
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 34
> {code}
> It contains two EnumerableTableScan operators with different row types
> specified in the error message above. In the scope of CALCITE-2454 was
> removed rel data type from the key in the \{{VolcanoPlanner.mapDigestToRel}}
> map.
> To fix this, `{{TableScan` should include rowType}} item in
> {{AbstractRelNode#explainTerms(RelWriter)}} method because in case of dynamic
> tables there is the same digest for different operators. The same thing was
> done for Values operator and literals in conditions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3055) TableScan type mismatch in VolcanoPlanner
[
https://issues.apache.org/jira/browse/CALCITE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16846785#comment-16846785
]
Bohdan Kazydub commented on CALCITE-3055:
-
Added the RelDataType pair back and opened [pull
request|https://github.com/apache/calcite/pull/1230].
Can someone review, please?
> TableScan type mismatch in VolcanoPlanner
> -
>
> Key: CALCITE-3055
> URL: https://issues.apache.org/jira/browse/CALCITE-3055
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Bohdan Kazydub
>Assignee: Danny Chan
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.20.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In Drill with Calcite updated to 1.19.0 we observe following problem:
> the following query
> {code}
> CREATE VIEW nation_view_testunionall_expandable_star AS select n_name,
> n_nationkey FROM cp.`tpch/nation.parquet`;
> (SELECT n_comment, n_regionkey FROM cp.`tpch/nation.parquet` LIMIT 5) UNION
> ALL (SELECT * FROM nation_view_testunionall_expandable_star LIMIT 5)
> {code}
> fails with
> {code}
> 15:46:06.005 [2333c7f1-89f3-0b80-8369-68e24895729e:foreman] ERROR
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: Type
> mismatch:
> left:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey) NOT NULL
> right:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey, ANY n_name, ANY
> n_nationkey) NOT NULL
> {code}
> when Volcano planner is registering rel nodes. The initial plan is the
> following:
> {code}
> LogicalUnion(all=[true]): rowcount = 10.0, cumulative cost = \{520.0 rows,
> 940.755032994728 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 74
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{205.0 rows,
> 365.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 68
> LogicalProject(n_comment=[$1], n_regionkey=[$2]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 67
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 29
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{305.0 rows,
> 565.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 73
> LogicalProject(n_name=[$0], n_nationkey=[$1]): rowcount = 100.0,
> cumulative cost = \{300.0 rows, 501.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 72
> LogicalProject(n_name=[$3], n_nationkey=[$4]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 71
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 34
> {code}
> It contains two EnumerableTableScan operators with different row types
> specified in the error message above. In the scope of CALCITE-2454 was
> removed rel data type from the key in the \{{VolcanoPlanner.mapDigestToRel}}
> map.
> To fix this, `{{TableScan` should include rowType}} item in
> {{AbstractRelNode#explainTerms(RelWriter)}} method because in case of dynamic
> tables there is the same digest for different operators. The same thing was
> done for Values operator and literals in conditions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3055) TableScan type mismatch in VolcanoPlanner
[
https://issues.apache.org/jira/browse/CALCITE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845975#comment-16845975
]
Bohdan Kazydub commented on CALCITE-3055:
-
[~julianhyde]
Before CALCITE-2454 `VolcanoPlanner.mapDigestToRel` map field had a key
consisting of Pair of relnode's digest string and rowType. After the change the
map has RelNode's digest as key.
I see 3 more ways to handle the issue:
1) Change key back to Pair of digest and row type, with rowtype being `null`
for rel nodes other than TableScan (something intermediate between former and
current implementation);
2) Construct key based on digest and row type for TableScan (concatenate digest
and row type);
3) Introduce new method to RelOptTable (something like getDescription) to use
in 'table' item in TableScan's `explainTerm` method.
>From the proposed solutions, I prefer 1) as it is clean and retains previous
>performance.
> TableScan type mismatch in VolcanoPlanner
> -
>
> Key: CALCITE-3055
> URL: https://issues.apache.org/jira/browse/CALCITE-3055
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Bohdan Kazydub
>Assignee: Danny Chan
>Priority: Blocker
> Fix For: 1.20.0
>
>
> In Drill with Calcite updated to 1.19.0 we observe following problem:
> the following query
> {code}
> CREATE VIEW nation_view_testunionall_expandable_star AS select n_name,
> n_nationkey FROM cp.`tpch/nation.parquet`;
> (SELECT n_comment, n_regionkey FROM cp.`tpch/nation.parquet` LIMIT 5) UNION
> ALL (SELECT * FROM nation_view_testunionall_expandable_star LIMIT 5)
> {code}
> fails with
> {code}
> 15:46:06.005 [2333c7f1-89f3-0b80-8369-68e24895729e:foreman] ERROR
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: Type
> mismatch:
> left:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey) NOT NULL
> right:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey, ANY n_name, ANY
> n_nationkey) NOT NULL
> {code}
> when Volcano planner is registering rel nodes. The initial plan is the
> following:
> {code}
> LogicalUnion(all=[true]): rowcount = 10.0, cumulative cost = \{520.0 rows,
> 940.755032994728 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 74
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{205.0 rows,
> 365.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 68
> LogicalProject(n_comment=[$1], n_regionkey=[$2]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 67
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 29
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{305.0 rows,
> 565.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 73
> LogicalProject(n_name=[$0], n_nationkey=[$1]): rowcount = 100.0,
> cumulative cost = \{300.0 rows, 501.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 72
> LogicalProject(n_name=[$3], n_nationkey=[$4]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 71
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 34
> {code}
> It contains two EnumerableTableScan operators with different row types
> specified in the error message above. In the scope of CALCITE-2454 was
> removed rel data type from the key in the \{{VolcanoPlanner.mapDigestToRel}}
> map.
> To fix this, `{{TableScan` should include rowType}} item in
> {{AbstractRelNode#explainTerms(RelWriter)}} method because in case of dynamic
> tables there is the same digest for different operators. The same thing was
> done for Values operator and literals in conditions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3055) TableScan type mismatch in VolcanoPlanner
[
https://issues.apache.org/jira/browse/CALCITE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16835748#comment-16835748
]
Bohdan Kazydub commented on CALCITE-3055:
-
Here is the change:
[commit|https://github.com/KazydubB/calcite/commit/179aef66b4b50f4097a48d56613ecbf6d1bd10d8].
With this solution there are 989 test failures, because additional item was
added to *TableScan, e.g.
{code}
[ERROR] PlannerTest.testSortPlan:540
Expected: "EnumerableSort(sort0=[$1], dir0=[ASC])\n
EnumerableProject(empid=[$0], deptno=[$1], name=[$2], salary=[$3],
commission=[$4])\nEnumerableTableScan(table=[[hr, emps]])\n"
but: was "EnumerableSort(sort0=[$1], dir0=[ASC])\n
EnumerableProject(empid=[$0], deptno=[$1], name=[$2], salary=[$3],
commission=[$4])\nEnumerableTableScan(table=[[hr, emps]],
rowType=[RecordType(JavaType(int) empid, JavaType(int) deptno, JavaType(class
java.lang.String) name, JavaType(float) salary, JavaType(class
java.lang.Integer) commission)])\n"
[ERROR] PlannerTest.testTwoSortRemoveInnerSort:616->runDuplicateSortCheck:680
Expected: "EnumerableSort(sort0=[$1], dir0=[ASC])\n
EnumerableProject(EXPR$0=[+($0, $1)], deptno=[$1])\n
EnumerableTableScan(table=[[hr, emps]])\n"
but: was "EnumerableSort(sort0=[$1], dir0=[ASC])\n
EnumerableProject(EXPR$0=[+($0, $1)], deptno=[$1])\n
EnumerableTableScan(table=[[hr, emps]], rowType=[RecordType(JavaType(int)
empid, JavaType(int) deptno, JavaType(class java.lang.String) name,
JavaType(float) salary, JavaType(class java.lang.Integer) commission)])\n"
{code}
If there are no objections with the change I will proceed with fixing test
failures caused by the change.
> TableScan type mismatch in VolcanoPlanner
> -
>
> Key: CALCITE-3055
> URL: https://issues.apache.org/jira/browse/CALCITE-3055
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Bohdan Kazydub
>Priority: Major
>
> In Drill with Calcite updated to 1.19.0 we observe following problem:
> the following query
> {code}
> CREATE VIEW nation_view_testunionall_expandable_star AS select n_name,
> n_nationkey FROM cp.`tpch/nation.parquet`;
> (SELECT n_comment, n_regionkey FROM cp.`tpch/nation.parquet` LIMIT 5) UNION
> ALL (SELECT * FROM nation_view_testunionall_expandable_star LIMIT 5)
> {code}
> fails with
> {code}
> 15:46:06.005 [2333c7f1-89f3-0b80-8369-68e24895729e:foreman] ERROR
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: Type
> mismatch:
> left:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey) NOT NULL
> right:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey, ANY n_name, ANY
> n_nationkey) NOT NULL
> {code}
> when Volcano planner is registering rel nodes. The initial plan is the
> following:
> {code}
> LogicalUnion(all=[true]): rowcount = 10.0, cumulative cost = \{520.0 rows,
> 940.755032994728 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 74
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{205.0 rows,
> 365.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 68
> LogicalProject(n_comment=[$1], n_regionkey=[$2]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 67
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 29
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{305.0 rows,
> 565.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 73
> LogicalProject(n_name=[$0], n_nationkey=[$1]): rowcount = 100.0,
> cumulative cost = \{300.0 rows, 501.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 72
> LogicalProject(n_name=[$3], n_nationkey=[$4]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 71
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 34
> {code}
> It contains two EnumerableTableScan operators with different row types
> specified in the error message above. In the scope of CALCITE-2454 was
> removed rel data type from the key in the \{{VolcanoPlanner.mapDigestToRel}}
> map.
> To fix this, `{{TableScan` should include rowType}} item in
> {{AbstractRelNode#explainTerms(RelWriter)}} method because in case of dynamic
> tables there is the same digest for different operators. The same thing was
> done for Values operator and literals in conditions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3055) TableScan type mismatch in VolcanoPlanner
[
https://issues.apache.org/jira/browse/CALCITE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16835731#comment-16835731
]
Bohdan Kazydub commented on CALCITE-3055:
-
There was discussion started in
[CALCITE-3020|https://issues.apache.org/jira/browse/CALCITE-3020] regarding the
issue.
> TableScan type mismatch in VolcanoPlanner
> -
>
> Key: CALCITE-3055
> URL: https://issues.apache.org/jira/browse/CALCITE-3055
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Bohdan Kazydub
>Priority: Major
>
> In Drill with Calcite updated to 1.19.0 we observe following problem:
> the following query
> {code}
> CREATE VIEW nation_view_testunionall_expandable_star AS select n_name,
> n_nationkey FROM cp.`tpch/nation.parquet`;
> (SELECT n_comment, n_regionkey FROM cp.`tpch/nation.parquet` LIMIT 5) UNION
> ALL (SELECT * FROM nation_view_testunionall_expandable_star LIMIT 5)
> {code}
> fails with
> {code}
> 15:46:06.005 [2333c7f1-89f3-0b80-8369-68e24895729e:foreman] ERROR
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: Type
> mismatch:
> left:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey) NOT NULL
> right:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey, ANY n_name, ANY
> n_nationkey) NOT NULL
> {code}
> when Volcano planner is registering rel nodes. The initial plan is the
> following:
> {code}
> LogicalUnion(all=[true]): rowcount = 10.0, cumulative cost = \{520.0 rows,
> 940.755032994728 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 74
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{205.0 rows,
> 365.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 68
> LogicalProject(n_comment=[$1], n_regionkey=[$2]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 67
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 29
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{305.0 rows,
> 565.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 73
> LogicalProject(n_name=[$0], n_nationkey=[$1]): rowcount = 100.0,
> cumulative cost = \{300.0 rows, 501.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 72
> LogicalProject(n_name=[$3], n_nationkey=[$4]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 71
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 34
> {code}
> It contains two EnumerableTableScan operators with different row types
> specified in the error message above. In the scope of CALCITE-2454 was
> removed rel data type from the key in the \{{VolcanoPlanner.mapDigestToRel}}
> map.
> To fix this, `{{TableScan` should include rowType}} item in
> {{AbstractRelNode#explainTerms(RelWriter)}} method because in case of dynamic
> tables there is the same digest for different operators. The same thing was
> done for Values operator and literals in conditions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3055) TableScan type mismatch in VolcanoPlanner
Bohdan Kazydub created CALCITE-3055:
---
Summary: TableScan type mismatch in VolcanoPlanner
Key: CALCITE-3055
URL: https://issues.apache.org/jira/browse/CALCITE-3055
Project: Calcite
Issue Type: Bug
Affects Versions: 1.19.0
Reporter: Bohdan Kazydub
In Drill with Calcite updated to 1.19.0 we observe following problem:
the following query
{code}
CREATE VIEW nation_view_testunionall_expandable_star AS select n_name,
n_nationkey FROM cp.`tpch/nation.parquet`;
(SELECT n_comment, n_regionkey FROM cp.`tpch/nation.parquet` LIMIT 5) UNION ALL
(SELECT * FROM nation_view_testunionall_expandable_star LIMIT 5)
{code}
fails with
{code}
15:46:06.005 [2333c7f1-89f3-0b80-8369-68e24895729e:foreman] ERROR
o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: Type
mismatch:
left:
RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey) NOT NULL
right:
RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey, ANY n_name, ANY
n_nationkey) NOT NULL
{code}
when Volcano planner is registering rel nodes. The initial plan is the
following:
{code}
LogicalUnion(all=[true]): rowcount = 10.0, cumulative cost = \{520.0 rows,
940.755032994728 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 74
LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{205.0 rows,
365.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 68
LogicalProject(n_comment=[$1], n_regionkey=[$2]): rowcount = 100.0,
cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id
= 67
EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount = 100.0,
cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id
= 29
LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{305.0 rows,
565.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 73
LogicalProject(n_name=[$0], n_nationkey=[$1]): rowcount = 100.0, cumulative
cost = \{300.0 rows, 501.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 72
LogicalProject(n_name=[$3], n_nationkey=[$4]): rowcount = 100.0,
cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id
= 71
EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
memory}, id = 34
{code}
It contains two EnumerableTableScan operators with different row types
specified in the error message above. In the scope of CALCITE-2454 was removed
rel data type from the key in the \{{VolcanoPlanner.mapDigestToRel}} map.
To fix this, `{{TableScan` should include rowType}} item in
{{AbstractRelNode#explainTerms(RelWriter)}} method because in case of dynamic
tables there is the same digest for different operators. The same thing was
done for Values operator and literals in conditions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3020) throws AssertionError:Type mismatch in VolcanoPlanner
[
https://issues.apache.org/jira/browse/CALCITE-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16833777#comment-16833777
]
Bohdan Kazydub commented on CALCITE-3020:
-
[~vlsi], Calcite has DynamicRecordType which is used in validation phase, after
the validation a list of columns is transformed into 'plain' RelDataType, that
may or may not contain a dynamic star.
For example, if SALES.NATION is dynamic table then for
{code}
SELECT n_nationkey FROM SALES.NATION
{code}
there will be no DYNAMIC STAR.
> throws AssertionError:Type mismatch in VolcanoPlanner
> -
>
> Key: CALCITE-3020
> URL: https://issues.apache.org/jira/browse/CALCITE-3020
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.19.0
>Reporter: godfrey he
>Assignee: Danny Chan
>Priority: Major
>
> after [CALCITE-2454|https://issues.apache.org/jira/browse/CALCITE-2454]
> merged, an AssertionError:Type mismatch will be thrown in VolcanoPlanner when
> running the following sql:
> {code:sql}
> WITH t1 AS (SELECT CAST(a as BIGINT) AS a, SUM(b) AS b FROM x GROUP BY CAST(a
> as BIGINT)),
> t2 AS (SELECT CAST(a as DOUBLE) AS a, SUM(b) AS b FROM x GROUP BY CAST(a
> as DOUBLE))
> SELECT t1.*, t2.* FROM t1, t2 WHERE t1.b = t2.b
> {code}
> Caused by: java.lang.AssertionError: Type mismatch:
> left:
> RecordType(BIGINT a, BIGINT b) NOT NULL
> right:
> RecordType(DOUBLE a, BIGINT b) NOT NULL
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:1858)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1705)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:850)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:872)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1958)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:126)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234)
> at
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:205)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:637)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3020) throws AssertionError:Type mismatch in VolcanoPlanner
[
https://issues.apache.org/jira/browse/CALCITE-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16833696#comment-16833696
]
Bohdan Kazydub commented on CALCITE-3020:
-
[~vlsi], currently the type is included (except for INT, BOOLEAN and TIME,
TIMESTAMP, DATE with precision == 0) in Values' digest.
Regarding you proposal with enhancing digest for scans with dynamic stars only
- it can't be determined if the table is dynamic (with RelOptTable) and dynamic
star is not reliable as it may be missing in some cases.
Therefore, I think it is better to enhance digest for all tablescans to avoid
the issue.
> throws AssertionError:Type mismatch in VolcanoPlanner
> -
>
> Key: CALCITE-3020
> URL: https://issues.apache.org/jira/browse/CALCITE-3020
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.19.0
>Reporter: godfrey he
>Assignee: Danny Chan
>Priority: Major
>
> after [CALCITE-2454|https://issues.apache.org/jira/browse/CALCITE-2454]
> merged, an AssertionError:Type mismatch will be thrown in VolcanoPlanner when
> running the following sql:
> {code:sql}
> WITH t1 AS (SELECT CAST(a as BIGINT) AS a, SUM(b) AS b FROM x GROUP BY CAST(a
> as BIGINT)),
> t2 AS (SELECT CAST(a as DOUBLE) AS a, SUM(b) AS b FROM x GROUP BY CAST(a
> as DOUBLE))
> SELECT t1.*, t2.* FROM t1, t2 WHERE t1.b = t2.b
> {code}
> Caused by: java.lang.AssertionError: Type mismatch:
> left:
> RecordType(BIGINT a, BIGINT b) NOT NULL
> right:
> RecordType(DOUBLE a, BIGINT b) NOT NULL
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:1858)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1705)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:850)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:872)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1958)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:126)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234)
> at
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:205)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:637)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-3020) throws AssertionError:Type mismatch in VolcanoPlanner
[
https://issues.apache.org/jira/browse/CALCITE-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832576#comment-16832576
]
Bohdan Kazydub edited comment on CALCITE-3020 at 5/3/19 3:24 PM:
-
[~vlsi] what do you think abouth this?
was (Author: kazydubb):
@vsitnikov what do you think abouth this?
> throws AssertionError:Type mismatch in VolcanoPlanner
> -
>
> Key: CALCITE-3020
> URL: https://issues.apache.org/jira/browse/CALCITE-3020
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.19.0
>Reporter: godfrey he
>Priority: Major
>
> after [CALCITE-2454|https://issues.apache.org/jira/browse/CALCITE-2454]
> merged, an AssertionError:Type mismatch will be thrown in VolcanoPlanner when
> running the following sql:
> {code:sql}
> WITH t1 AS (SELECT CAST(a as BIGINT) AS a, SUM(b) AS b FROM x GROUP BY CAST(a
> as BIGINT)),
> t2 AS (SELECT CAST(a as DOUBLE) AS a, SUM(b) AS b FROM x GROUP BY CAST(a
> as DOUBLE))
> SELECT t1.*, t2.* FROM t1, t2 WHERE t1.b = t2.b
> {code}
> Caused by: java.lang.AssertionError: Type mismatch:
> left:
> RecordType(BIGINT a, BIGINT b) NOT NULL
> right:
> RecordType(DOUBLE a, BIGINT b) NOT NULL
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:1858)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1705)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:850)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:872)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1958)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:126)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234)
> at
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:205)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:637)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3020) throws AssertionError:Type mismatch in VolcanoPlanner
[
https://issues.apache.org/jira/browse/CALCITE-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832576#comment-16832576
]
Bohdan Kazydub commented on CALCITE-3020:
-
@vsitnikov what do you think abouth this?
> throws AssertionError:Type mismatch in VolcanoPlanner
> -
>
> Key: CALCITE-3020
> URL: https://issues.apache.org/jira/browse/CALCITE-3020
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.19.0
>Reporter: godfrey he
>Priority: Major
>
> after [CALCITE-2454|https://issues.apache.org/jira/browse/CALCITE-2454]
> merged, an AssertionError:Type mismatch will be thrown in VolcanoPlanner when
> running the following sql:
> {code:sql}
> WITH t1 AS (SELECT CAST(a as BIGINT) AS a, SUM(b) AS b FROM x GROUP BY CAST(a
> as BIGINT)),
> t2 AS (SELECT CAST(a as DOUBLE) AS a, SUM(b) AS b FROM x GROUP BY CAST(a
> as DOUBLE))
> SELECT t1.*, t2.* FROM t1, t2 WHERE t1.b = t2.b
> {code}
> Caused by: java.lang.AssertionError: Type mismatch:
> left:
> RecordType(BIGINT a, BIGINT b) NOT NULL
> right:
> RecordType(DOUBLE a, BIGINT b) NOT NULL
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:1858)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1705)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:850)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:872)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1958)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:126)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234)
> at
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:205)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:637)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3020) throws AssertionError:Type mismatch in VolcanoPlanner
[
https://issues.apache.org/jira/browse/CALCITE-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832573#comment-16832573
]
Bohdan Kazydub commented on CALCITE-3020:
-
We observe a similar problem in Drill when updating to Calcite 1.19.0:
the following query
{code}
CREATE VIEW nation_view_testunionall_expandable_star AS select n_name,
n_nationkey FROM cp.`tpch/nation.parquet`;
(SELECT n_comment, n_regionkey FROM cp.`tpch/nation.parquet` LIMIT 5) UNION ALL
(SELECT * FROM nation_view_testunionall_expandable_star LIMIT 5)
{code}
fails with
{code}
15:46:06.005 [2333c7f1-89f3-0b80-8369-68e24895729e:foreman] ERROR
o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: Type
mismatch:
left:
RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey) NOT NULL
right:
RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey, ANY n_name, ANY
n_nationkey) NOT NULL
{code}
when Volcano planner is registering rel nodes. The initial plan is the
following:
{code}
LogicalUnion(all=[true]): rowcount = 10.0, cumulative cost = {520.0 rows,
940.755032994728 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 74
LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = {205.0 rows,
365.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 68
LogicalProject(n_comment=[$1], n_regionkey=[$2]): rowcount = 100.0,
cumulative cost = {200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id
= 67
EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount = 100.0,
cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id
= 29
LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = {305.0 rows,
565.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 73
LogicalProject(n_name=[$0], n_nationkey=[$1]): rowcount = 100.0, cumulative
cost = {300.0 rows, 501.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 72
LogicalProject(n_name=[$3], n_nationkey=[$4]): rowcount = 100.0,
cumulative cost = {200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id
= 71
EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
memory}, id = 34
{code}
It contains two EnumerableTableScan operators with different row types
specified in the error message above. In the scope of CALCITE-2454 was removed
rel data type from the key in the {{VolcanoPlanner.mapDigestToRel}} map.
I think, {{RelNode}}s with no input (like scan) should include {{rowType}} item
in {{AbstractRelNode#explainTerms(RelWriter)}} method because in case of
dynamic tables there is the same digest for different operators. The same thing
was done for Values operator and literals in conditions.
> throws AssertionError:Type mismatch in VolcanoPlanner
> -
>
> Key: CALCITE-3020
> URL: https://issues.apache.org/jira/browse/CALCITE-3020
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.19.0
>Reporter: godfrey he
>Priority: Major
>
> after [CALCITE-2454|https://issues.apache.org/jira/browse/CALCITE-2454]
> merged, an AssertionError:Type mismatch will be thrown in VolcanoPlanner when
> running the following sql:
> {code:sql}
> WITH t1 AS (SELECT CAST(a as BIGINT) AS a, SUM(b) AS b FROM x GROUP BY CAST(a
> as BIGINT)),
> t2 AS (SELECT CAST(a as DOUBLE) AS a, SUM(b) AS b FROM x GROUP BY CAST(a
> as DOUBLE))
> SELECT t1.*, t2.* FROM t1, t2 WHERE t1.b = t2.b
> {code}
> Caused by: java.lang.AssertionError: Type mismatch:
> left:
> RecordType(BIGINT a, BIGINT b) NOT NULL
> right:
> RecordType(DOUBLE a, BIGINT b) NOT NULL
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:1858)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1705)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:850)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:872)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1958)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:126)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234)
> at
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:205)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:637)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2627) UnsupportedOperationException with unclear message is shown when SqlNode with unknown type is validated
[
https://issues.apache.org/jira/browse/CALCITE-2627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bohdan Kazydub updated CALCITE-2627:
Description:
If RelDataType for SqlNode is unknown then
SqlValidatorImpl#{color:#ffc66d}getValidatedNodeType{color}(SqlNode node) (see
https://github.com/apache/calcite/blob/59b76701d9c805f51b0929577ca604fdbfe00b0b/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L1555)
throws Util.needToImplement(node), which is the following:
{code:java}
public static RuntimeException needToImplement(Object o) {
String description = null;
if (o != null) {
description = o.getClass().toString() + ": " + o.toString();
}
throw new UnsupportedOperationException(description);
}
{code}
Javadoc for the method dictates that the method 'Returns [in a sense of
"throws"] a RuntimeException indicating that a particular feature has not been
implemented, but should be.'. As not every unknown type should be implemented,
call to this method should be replaced with throwing of an RuntimeException
(UnsupportedOperationException, perhaps) containing a more helpful message,
something like 'Unknown type: [node.toString()]'.
For example, currently there is following error:
{code}
0: jdbc:calcite:model=target/test-classes/mod> select cast('' as sometype) from
(values(1));
Error: Error while executing SQL "select cast('' as sometype) from
(values(1))": class org.apache.calcite.sql.SqlIdentifier: SOMETYPE
(state=,code=0)
{code}
was:
If RelDataType for SqlNode is unknown then
SqlValidatorImpl#{color:#ffc66d}getValidatedNodeType{color}(SqlNode node) (see
https://github.com/apache/calcite/blob/59b76701d9c805f51b0929577ca604fdbfe00b0b/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L1555)
throws Util.needToImplement(node), which is the following:
{code:java}
public static RuntimeException needToImplement(Object o) {
String description = null;
if (o != null) {
description = o.getClass().toString() + ": " + o.toString();
}
throw new UnsupportedOperationException(description);
}
{code}
Javadoc for the method dictates that the method 'Returns [in a sense of
"throws"] a RuntimeException indicating that a particular feature has not been
implemented, but should be.'. As not every unknown type should be implemented,
call to this method should be replaced with throwing of an RuntimeException
(UnsupportedOperationException, perhaps) containing a more helpful message,
something like 'Unknown type: [node.toString()]'.
> UnsupportedOperationException with unclear message is shown when SqlNode with
> unknown type is validated
> ---
>
> Key: CALCITE-2627
> URL: https://issues.apache.org/jira/browse/CALCITE-2627
> Project: Calcite
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Julian Hyde
>Priority: Major
>
> If RelDataType for SqlNode is unknown then
> SqlValidatorImpl#{color:#ffc66d}getValidatedNodeType{color}(SqlNode node)
> (see
> https://github.com/apache/calcite/blob/59b76701d9c805f51b0929577ca604fdbfe00b0b/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L1555)
> throws Util.needToImplement(node), which is the following:
> {code:java}
> public static RuntimeException needToImplement(Object o) {
> String description = null;
> if (o != null) {
> description = o.getClass().toString() + ": " + o.toString();
> }
> throw new UnsupportedOperationException(description);
> }
> {code}
> Javadoc for the method dictates that the method 'Returns [in a sense of
> "throws"] a RuntimeException indicating that a particular feature has not
> been implemented, but should be.'. As not every unknown type should be
> implemented, call to this method should be replaced with throwing of an
> RuntimeException (UnsupportedOperationException, perhaps) containing a more
> helpful message, something like 'Unknown type: [node.toString()]'.
> For example, currently there is following error:
> {code}
> 0: jdbc:calcite:model=target/test-classes/mod> select cast('' as sometype)
> from (values(1));
> Error: Error while executing SQL "select cast('' as sometype) from
> (values(1))": class org.apache.calcite.sql.SqlIdentifier: SOMETYPE
> (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2627) UnsupportedOperationException with unclear message is shown when SqlNode with unknown type is validated
Bohdan Kazydub created CALCITE-2627:
---
Summary: UnsupportedOperationException with unclear message is
shown when SqlNode with unknown type is validated
Key: CALCITE-2627
URL: https://issues.apache.org/jira/browse/CALCITE-2627
Project: Calcite
Issue Type: Improvement
Reporter: Bohdan Kazydub
Assignee: Julian Hyde
If RelDataType for SqlNode is unknown then
SqlValidatorImpl#{color:#ffc66d}getValidatedNodeType{color}(SqlNode node) (see
https://github.com/apache/calcite/blob/59b76701d9c805f51b0929577ca604fdbfe00b0b/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L1555)
throws Util.needToImplement(node), which is the following:
{code:java}
public static RuntimeException needToImplement(Object o) {
String description = null;
if (o != null) {
description = o.getClass().toString() + ": " + o.toString();
}
throw new UnsupportedOperationException(description);
}
{code}
Javadoc for the method dictates that the method 'Returns [in a sense of
"throws"] a RuntimeException indicating that a particular feature has not been
implemented, but should be.'. As not every unknown type should be implemented,
call to this method should be replaced with throwing of an RuntimeException
(UnsupportedOperationException, perhaps) containing a more helpful message,
something like 'Unknown type: [node.toString()]'.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
