[jira] [Commented] (CALCITE-4677) Different physical conventions conversion on new physical node producing
[ https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389779#comment-17389779 ] ZiLin Chen commented on CALCITE-4677: - The physical node is generated by a Implementation rule. > Different physical conventions conversion on new physical node producing > - > > Key: CALCITE-4677 > URL: https://issues.apache.org/jira/browse/CALCITE-4677 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: ZiLin Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When we enable top-down optimization, the VolcanoPlanner couldn't be > completely compatible with the old one. > Especially when we using rules matching RelNodes which convention is not > NONE, like rules that want to convert different conventions between physical > node or some rules that want to do some optimization for serval physical > nodes. It could be found that rules can't fire, because in top-down > optimization physical nodes can only be created by implementation > rules(logical node -> physical node) but not rules matching physical > node(physical node -> physical node). Try to use enforcer instead of rules to > convert node between different conventions. > > Example: > Whenever new physical nodes are on produce, we need to fire rules in case of > different physical conventions conversion. > DeriveTrait Task deal with this case, while some physical node maybe missing. > > There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. > 1. LogicalSortToTopNRule > 2. LogicalSortToLimitMemSortRule > 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) > > LogicalAggreate -> 1. RelSet1 > - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and > LogicalSortToLimitMemSortRule > - OtherNode -> 3. RelSet3 > > ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. > ApplyRuleTask LogicalSortToTopNRule still on the stack. > > when RelSet3 on Merge (Merge with other RelSets), then RelSet2 > clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed(): > subset.resetTaskState() > because RelSet1 is root so OptimizeGroup will be scheduled again > > Now the task stack > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule > Task about RelSet3 > Task about RelSet1 root schedule again > Task about RelSet2 > Task about RelSet3 (current task) > > next > Task Stack: > Task about RelSet1 > Task about RelSet2 > Task about RelSet3 > Task about RelSet1 > Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on > RelSet (LogicalSort and LimitMemSort) > > next... > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule. > Here we found when TopN is on produce, every RelSubSet can not fire > OptimizeInputTask. so TopNToSomeRelNodeRule can not fire -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4677) Different physical conventions conversion on new physical node producing
[ https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387545#comment-17387545 ] Haisheng Yuan commented on CALCITE-4677: [~chenz...@mail2.sysu.edu.com] sorry to reply you late. Is the new physical node generated by passThrough or by derive? That makes a difference. > Different physical conventions conversion on new physical node producing > - > > Key: CALCITE-4677 > URL: https://issues.apache.org/jira/browse/CALCITE-4677 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: ZiLin Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When we enable top-down optimization, the VolcanoPlanner couldn't be > completely compatible with the old one. > Especially when we using rules matching RelNodes which convention is not > NONE, like rules that want to convert different conventions between physical > node or some rules that want to do some optimization for serval physical > nodes. It could be found that rules can't fire, because in top-down > optimization physical nodes can only be created by implementation > rules(logical node -> physical node) but not rules matching physical > node(physical node -> physical node). Try to use enforcer instead of rules to > convert node between different conventions. > > Example: > Whenever new physical nodes are on produce, we need to fire rules in case of > different physical conventions conversion. > DeriveTrait Task deal with this case, while some physical node maybe missing. > > There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. > 1. LogicalSortToTopNRule > 2. LogicalSortToLimitMemSortRule > 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) > > LogicalAggreate -> 1. RelSet1 > - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and > LogicalSortToLimitMemSortRule > - OtherNode -> 3. RelSet3 > > ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. > ApplyRuleTask LogicalSortToTopNRule still on the stack. > > when RelSet3 on Merge (Merge with other RelSets), then RelSet2 > clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed(): > subset.resetTaskState() > because RelSet1 is root so OptimizeGroup will be scheduled again > > Now the task stack > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule > Task about RelSet3 > Task about RelSet1 root schedule again > Task about RelSet2 > Task about RelSet3 (current task) > > next > Task Stack: > Task about RelSet1 > Task about RelSet2 > Task about RelSet3 > Task about RelSet1 > Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on > RelSet (LogicalSort and LimitMemSort) > > next... > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule. > Here we found when TopN is on produce, every RelSubSet can not fire > OptimizeInputTask. so TopNToSomeRelNodeRule can not fire -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4677) Different physical conventions conversion on new physical node producing
[ https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385692#comment-17385692 ] ZiLin Chen commented on CALCITE-4677: - [~hyuan] , When I use PhysicalNode (the passThrough interface), I find that some passThrough optimization maybe missing, because passThrough only generate a candidate for those RelSubset which state is OPTIMIZING. However, the example I had given shows that after RelSet merged , the root will be scheduled again, finally those RelSubset state become COMPLETED. Later, when new PhysicalNode on produce, it will fail to passThrough that RelSubset. I think the problem is after RelSet merged, the ApplyRule tasks for the RelSet before merged need to re-target to the new RelSet. > Different physical conventions conversion on new physical node producing > - > > Key: CALCITE-4677 > URL: https://issues.apache.org/jira/browse/CALCITE-4677 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: ZiLin Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When we enable top-down optimization, the VolcanoPlanner couldn't be > completely compatible with the old one. > Especially when we using rules matching RelNodes which convention is not > NONE, like rules that want to convert different conventions between physical > node or some rules that want to do some optimization for serval physical > nodes. It could be found that rules can't fire, because in top-down > optimization physical nodes can only be created by implementation > rules(logical node -> physical node) but not rules matching physical > node(physical node -> physical node). Try to use enforcer instead of rules to > convert node between different conventions. > > Example: > Whenever new physical nodes are on produce, we need to fire rules in case of > different physical conventions conversion. > DeriveTrait Task deal with this case, while some physical node maybe missing. > > There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. > 1. LogicalSortToTopNRule > 2. LogicalSortToLimitMemSortRule > 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) > > LogicalAggreate -> 1. RelSet1 > - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and > LogicalSortToLimitMemSortRule > - OtherNode -> 3. RelSet3 > > ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. > ApplyRuleTask LogicalSortToTopNRule still on the stack. > > when RelSet3 on Merge (Merge with other RelSets), then RelSet2 > clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed(): > subset.resetTaskState() > because RelSet1 is root so OptimizeGroup will be scheduled again > > Now the task stack > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule > Task about RelSet3 > Task about RelSet1 root schedule again > Task about RelSet2 > Task about RelSet3 (current task) > > next > Task Stack: > Task about RelSet1 > Task about RelSet2 > Task about RelSet3 > Task about RelSet1 > Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on > RelSet (LogicalSort and LimitMemSort) > > next... > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule. > Here we found when TopN is on produce, every RelSubSet can not fire > OptimizeInputTask. so TopNToSomeRelNodeRule can not fire -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4677) Different physical conventions conversion on new physical node producing
[ https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377736#comment-17377736 ] ZiLin Chen commented on CALCITE-4677: - [~hyuan] I agree that rules matching physical nodes are not really cascades style, although it is flexible. So it become a compatible problem ? The reason why I choose to use rules matching physical nodes is that in some case, specific physical operator can provide specific property, which is unrealistic to add it to trait, but it is more efficient and easy to use a rule matching physical operator to achieve the goal. If it is a compatible problem, the piece of code in DeriveTraitTask will become misleading, because at most of time it can satisfy my requirement. > Different physical conventions conversion on new physical node producing > - > > Key: CALCITE-4677 > URL: https://issues.apache.org/jira/browse/CALCITE-4677 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: ZiLin Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Whenever new physical nodes are on produce, we need to fire rules in case of > different physical conventions conversion. > DeriveTrait Task deal with this case, while some physical node maybe missing. > > There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. > 1. LogicalSortToTopNRule > 2. LogicalSortToLimitMemSortRule > 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) > > LogicalAggreate -> 1. RelSet1 > - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and > LogicalSortToLimitMemSortRule > - OtherNode -> 3. RelSet3 > > ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. > ApplyRuleTask LogicalSortToTopNRule still on the stack. > > when RelSet3 on Merge (Merge with other RelSets), then RelSet2 > clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed(): > subset.resetTaskState() > because RelSet1 is root so OptimizeGroup will be scheduled again > > Now the task stack > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule > Task about RelSet3 > Task about RelSet1 root schedule again > Task about RelSet2 > Task about RelSet3 (current task) > > next > Task Stack: > Task about RelSet1 > Task about RelSet2 > Task about RelSet3 > Task about RelSet1 > Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on > RelSet (LogicalSort and LimitMemSort) > > next... > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule. > Here we found when TopN is on produce, every RelSubSet can not fire > OptimizeInputTask. so TopNToSomeRelNodeRule can not fire -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4677) Different physical conventions conversion on new physical node producing
[ https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377705#comment-17377705 ] Haisheng Yuan commented on CALCITE-4677: Rule matches on physical operators are prohibited. You said you have some rules to only match physical nodes, but that is not the reason that we should support. You didn't provide any rationale to justify that the problem you are dealing with can only be solved by rules to match physical relnodes. Completely compatible with old volcano optimizer is not the goal. > Different physical conventions conversion on new physical node producing > - > > Key: CALCITE-4677 > URL: https://issues.apache.org/jira/browse/CALCITE-4677 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: ZiLin Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Whenever new physical nodes are on produce, we need to fire rules in case of > different physical conventions conversion. > DeriveTrait Task deal with this case, while some physical node maybe missing. > > There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. > 1. LogicalSortToTopNRule > 2. LogicalSortToLimitMemSortRule > 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) > > LogicalAggreate -> 1. RelSet1 > - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and > LogicalSortToLimitMemSortRule > - OtherNode -> 3. RelSet3 > > ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. > ApplyRuleTask LogicalSortToTopNRule still on the stack. > > when RelSet3 on Merge (Merge with other RelSets), then RelSet2 > clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed(): > subset.resetTaskState() > because RelSet1 is root so OptimizeGroup will be scheduled again > > Now the task stack > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule > Task about RelSet3 > Task about RelSet1 root schedule again > Task about RelSet2 > Task about RelSet3 (current task) > > next > Task Stack: > Task about RelSet1 > Task about RelSet2 > Task about RelSet3 > Task about RelSet1 > Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on > RelSet (LogicalSort and LimitMemSort) > > next... > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule. > Here we found when TopN is on produce, every RelSubSet can not fire > OptimizeInputTask. so TopNToSomeRelNodeRule can not fire -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4677) Different physical conventions conversion on new physical node producing
[ https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17376299#comment-17376299 ] ZiLin Chen commented on CALCITE-4677: - OK [~julianhyde], I know what you mean, I will re-describe the problem in a more user friendly way after achieving an agreement on whether we should support physical multi-operand rule within TopDownRuleDriver mode. > Different physical conventions conversion on new physical node producing > - > > Key: CALCITE-4677 > URL: https://issues.apache.org/jira/browse/CALCITE-4677 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: ZiLin Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Whenever new physical nodes are on produce, we need to fire rules in case of > different physical conventions conversion. > DeriveTrait Task deal with this case, while some physical node maybe missing. > > There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. > 1. LogicalSortToTopNRule > 2. LogicalSortToLimitMemSortRule > 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) > > LogicalAggreate -> 1. RelSet1 > - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and > LogicalSortToLimitMemSortRule > - OtherNode -> 3. RelSet3 > > ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. > ApplyRuleTask LogicalSortToTopNRule still on the stack. > > when RelSet3 on Merge (Merge with other RelSets), then RelSet2 > clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed(): > subset.resetTaskState() > because RelSet1 is root so OptimizeGroup will be scheduled again > > Now the task stack > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule > Task about RelSet3 > Task about RelSet1 root schedule again > Task about RelSet2 > Task about RelSet3 (current task) > > next > Task Stack: > Task about RelSet1 > Task about RelSet2 > Task about RelSet3 > Task about RelSet1 > Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on > RelSet (LogicalSort and LimitMemSort) > > next... > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule. > Here we found when TopN is on produce, every RelSubSet can not fire > OptimizeInputTask. so TopNToSomeRelNodeRule can not fire -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4677) Different physical conventions conversion on new physical node producing
[ https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17376275#comment-17376275 ] ZiLin Chen commented on CALCITE-4677: - Actually, I have some rules that can only match several physical Convention RelNodes (not just like ConvertRule only match a single RelNode, enforcer can not help), so these rules can only be applied when physical RelNodes are on produce. It works pretty well on IterativeRuleDriver, however, TopDownRuleDriver can't work in some case (the example in description) . I found that DeriveTraitTask has a of code which can help. // In case some implementations use rules to convert between different physical conventions. // Note that this is deprecated and will be removed in the future. tasks.push(new ApplyRules(mExpr, group, false)); > Different physical conventions conversion on new physical node producing > - > > Key: CALCITE-4677 > URL: https://issues.apache.org/jira/browse/CALCITE-4677 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: ZiLin Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Whenever new physical nodes are on produce, we need to fire rules in case of > different physical conventions conversion. > DeriveTrait Task deal with this case, while some physical node maybe missing. > > There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. > 1. LogicalSortToTopNRule > 2. LogicalSortToLimitMemSortRule > 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) > > LogicalAggreate -> 1. RelSet1 > - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and > LogicalSortToLimitMemSortRule > - OtherNode -> 3. RelSet3 > > ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. > ApplyRuleTask LogicalSortToTopNRule still on the stack. > > when RelSet3 on Merge (Merge with other RelSets), then RelSet2 > clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed(): > subset.resetTaskState() > because RelSet1 is root so OptimizeGroup will be scheduled again > > Now the task stack > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule > Task about RelSet3 > Task about RelSet1 root schedule again > Task about RelSet2 > Task about RelSet3 (current task) > > next > Task Stack: > Task about RelSet1 > Task about RelSet2 > Task about RelSet3 > Task about RelSet1 > Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on > RelSet (LogicalSort and LimitMemSort) > > next... > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule. > Here we found when TopN is on produce, every RelSubSet can not fire > OptimizeInputTask. so TopNToSomeRelNodeRule can not fire -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4677) Different physical conventions conversion on new physical node producing
[ https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17376239#comment-17376239 ] Haisheng Yuan commented on CALCITE-4677: We prohibited rule matching on physical operators on purpose. If you want to convert between physical conventions, try enforcer instead of rules. Is there any specific case that justify your rationale? > Different physical conventions conversion on new physical node producing > - > > Key: CALCITE-4677 > URL: https://issues.apache.org/jira/browse/CALCITE-4677 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: ZiLin Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Whenever new physical nodes are on produce, we need to fire rules in case of > different physical conventions conversion. > DeriveTrait Task deal with this case, while some physical node maybe missing. > > There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. > 1. LogicalSortToTopNRuleLogicalSortToTopNRule > 2. LogicalSortToLimitMemSortRule > 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) > > LogicalAggreate -> 1. RelSet1 > - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and > LogicalSortToLimitMemSortRule > - OtherNode -> 3. RelSet3 > > ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. > ApplyRuleTask LogicalSortToTopNRule still on the stack. > > when RelSet3 on Merge (Merge with other RelSets), then RelSet2 > clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed(): > subset.resetTaskState() > because RelSet1 is root so OptimizeGroup will be scheduled again > > Now the task stack > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule > Task about RelSet3 > Task about RelSet1 root schedule again > Task about RelSet2 > Task about RelSet3 (current task) > > next > Task Stack: > Task about RelSet1 > Task about RelSet2 > Task about RelSet3 > Task about RelSet1 > Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on > RelSet (LogicalSort and LimitMemSort) > > next... > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule. > Here we found when TopN is on produce, every RelSubSet can not fire > OptimizeInputTask. so TopNToSomeRelNodeRule can not fire -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4677) Different physical conventions conversion on new physical node producing
[ https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17376013#comment-17376013 ] Julian Hyde commented on CALCITE-4677: -- [~chenz...@mail2.sysu.edu.com], Thanks for the examples. Your description still contains lot of details about the workings of the planner, and I think the problem could be described so that people don't need to read those details. (Why is this important? Because as you have currently described it, only one or two people in the world will ever understand the bug that you have encountered. For example, I wrote the planner, but I don't have the necessary two hours to read the code and understand the bug. But many people might be running into this bug, or use the functionality that you are planning to provide.) > Different physical conventions conversion on new physical node producing > - > > Key: CALCITE-4677 > URL: https://issues.apache.org/jira/browse/CALCITE-4677 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: ZiLin Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Whenever new physical nodes are on produce, we need to fire rules in case of > different physical conventions conversion. > DeriveTrait Task deal with this case, while some physical node maybe missing. > > There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. > 1. LogicalSortToTopNRuleLogicalSortToTopNRule > 2. LogicalSortToLimitMemSortRule > 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) > > LogicalAggreate -> 1. RelSet1 > - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and > LogicalSortToLimitMemSortRule > - OtherNode -> 3. RelSet3 > > ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. > ApplyRuleTask LogicalSortToTopNRule still on the stack. > > when RelSet3 on Merge (Merge with other RelSets), then RelSet2 > clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed(): > subset.resetTaskState() > because RelSet1 is root so OptimizeGroup will be scheduled again > > Now the task stack > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule > Task about RelSet3 > Task about RelSet1 root schedule again > Task about RelSet2 > Task about RelSet3 (current task) > > next > Task Stack: > Task about RelSet1 > Task about RelSet2 > Task about RelSet3 > Task about RelSet1 > Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on > RelSet (LogicalSort and LimitMemSort) > > next... > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule. > Here we found when TopN is on produce, every RelSubSet can not fire > OptimizeInputTask. so TopNToSomeRelNodeRule can not fire -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4677) Different physical conventions conversion on new physical node producing
[ https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17375233#comment-17375233 ] ZiLin Chen commented on CALCITE-4677: - Ok [~zabetak] ，I give an example. Based on this, we can discuss more details. > Different physical conventions conversion on new physical node producing > - > > Key: CALCITE-4677 > URL: https://issues.apache.org/jira/browse/CALCITE-4677 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: ZiLin Chen >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Whenever new physical nodes are on produce, we need to fire rules in case of > different physical conventions conversion. > DeriveTrait Task deal with this case, while some physical node maybe missing. > > There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. > 1. LogicalSortToTopNRuleLogicalSortToTopNRule > 2. LogicalSortToLimitMemSortRule > 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) > > LogicalAggreate -> 1. RelSet1 > - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and > LogicalSortToLimitMemSortRule > - OtherNode -> 3. RelSet3 > > ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. > ApplyRuleTask LogicalSortToTopNRule still on stack. > > when RelSet3 on Merge (Merge with other RelSets)RelSet2 clearProcessed(): > subset.resetTaskState() > RelSet1 clearProcessed(): subset.resetTaskState() > because RelSet1 is root so OptimizeGroup will be scheduled again > > Now the task stack > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRule > Task LogicalSortToTopNRule > Task about RelSet3 > Task about RelSet1 root schedule again > Task about RelSet2 > Task about RelSet3 (current task) > > next > Task Stack: > Task about RelSet1 > Task about RelSet2 > Task about RelSet3 > Task about RelSet1 > Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on > RelSet (LogicalSort and LimitMemSort) > > next... > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule. > Here we found when TopN is on produce, every RelSubSet can not fire > OptimizeInputTask. so TopNToSomeRelNodeRule can not fire -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4677) Different physical conventions conversion on new physical node producing
[ https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17374816#comment-17374816 ] Stamatis Zampetakis commented on CALCITE-4677: -- Hi [~chenz...@mail2.sysu.edu.com], can you please add more details/examples about what is the problem or what you would like to achieve? > Different physical conventions conversion on new physical node producing > - > > Key: CALCITE-4677 > URL: https://issues.apache.org/jira/browse/CALCITE-4677 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: ZiLin Chen >Priority: Major > > Whenever new physical nodes are on produce, we need to fire rules in case of > different physical conventions conversion. > DeriveTrait Task deal with this case, while some physical node maybe missing. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
