[
https://issues.apache.org/jira/browse/CALCITE-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961995#comment-16961995
]
jin xing commented on CALCITE-2970:
-----------------------------------
I migrate my PR description from github as below:
Currently {{AbstractConvert}} is disabled for {{EnumerableConvention}}(and
other conventions). Thus no chance to apply a {{sort-merge-join (SMJ)}} when
the inputs of join are not sorted.
{{AbstractConvert}} is a way to enable SMJ. However relying on expansion of
{{AbstractConverter}} could be expensive. There might be matching explosion
using {{VolcanoPlanner}}.
I did a test on {{JdbcTest#testJoinManyWay -- checkJoinNWay(4)}}
When {{EnumerableConvention#useAbstractConvertersForConversion}} is {{false}}:
{code:java}
Timing Cost: 3 seconds
Rule Matched Times:
org.apache.calcite.rel.rules.AggregateProjectMergeRule:380
org.apache.calcite.rel.rules.ProjectFilterTransposeRule:6
org.apache.calcite.adapter.enumerable.EnumerableProjectRule:196
org.apache.calcite.adapter.enumerable.EnumerableJoinRule:23
org.apache.calcite.adapter.enumerable.EnumerableFilterRule:16
org.apache.calcite.rel.rules.FilterProjectTransposeRule:20
org.apache.calcite.rel.rules.FilterJoinRule$FilterIntoJoinRule:12
org.apache.calcite.rel.rules.JoinCommuteRule:23
org.apache.calcite.rel.rules.ProjectMergeRule:3387
org.apache.calcite.adapter.enumerable.EnumerableAggregateRule:17
org.apache.calcite.adapter.enumerable.EnumerableMergeJoinRule:18
org.apache.calcite.rel.rules.JoinPushThroughJoinRule:45
{code}
{{}}
But when {{EnumerableConvention#useAbstractConvertersForConversion}} is
{{true}}:
{code:java}
Timing Cost: 52 seconds
Rule Matched Times:
org.apache.calcite.rel.rules.AggregateProjectMergeRule:6240
org.apache.calcite.rel.rules.ProjectFilterTransposeRule:6
org.apache.calcite.adapter.enumerable.EnumerableProjectRule:3190
org.apache.calcite.adapter.enumerable.EnumerableJoinRule:739
org.apache.calcite.adapter.enumerable.EnumerableSortRule:68
org.apache.calcite.rel.rules.FilterProjectTransposeRule:20
org.apache.calcite.rel.rules.SortRemoveRule:111
org.apache.calcite.adapter.enumerable.EnumerableAggregateRule:625
org.apache.calcite.adapter.enumerable.EnumerableMergeJoinRule:738
org.apache.calcite.plan.volcano.AbstractConverter$ExpandConversionRule:70
org.apache.calcite.adapter.enumerable.EnumerableFilterRule:16
org.apache.calcite.rel.rules.FilterJoinRule$FilterIntoJoinRule:12
org.apache.calcite.rel.rules.JoinCommuteRule:763
org.apache.calcite.rel.rules.ProjectMergeRule:44769
org.apache.calcite.rel.rules.JoinPushThroughJoinRule:1827
{code}
When I test {{JdbcTest#testJoinManyWay -- checkJoinNWay(6)}} and
{{EnumerableConvention#useAbstractConvertersForConversion is true}}, it just
never return.
Note that in design&impl of {{VolcanoRuleCall#matchRecurse}}, a rule is
triggered when the root operand or child operand get matched by a new
created/registered {{RelNode}}.
If we construct the operator of {{EnumerableSort}} when creating
{{EnumerableMergeJoin}}, thus no {{AbstractConvert}} is created and save extra
matching effort when optimization.
With this change, {{JdbcTest#testJoinManyWay}} finished in 8 seconds and we can
also enable test of {{VolcanoPlannerTest#testMergeJoin}}.
> Performance issue when enabling abstract converter for EnumerableConvention
> ---------------------------------------------------------------------------
>
> Key: CALCITE-2970
> URL: https://issues.apache.org/jira/browse/CALCITE-2970
> Project: Calcite
> Issue Type: Bug
> Components: core
> Reporter: Haisheng Yuan
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> If we enable the use of abstract converter for {{EnumerableConvention}}, by
> making {{useAbstractConvertersForConversion}} return true,
> {{JDBCTest.testJoinManyWay}} will not complete.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
