> 发件人：Vladimir Ozerov
> 日 期：2020年03月15日 04:18:53
> 收件人：Haisheng Yuan
> 抄 送：dev@calcite.apache.org (dev@calcite.apache.org) >
> 主 题：Re: Re: Re: Re: [DISCUSS] Proposal to add API to force rules matching
> specific rels
>
>
.
- Haisheng
--
发件人：Vladimir Ozerov
日 期：2020年03月15日 04:18:53
收件人：Haisheng Yuan
抄 送：dev@calcite.apache.org (dev@calcite.apache.org)
主 题：Re: Re: Re: Re: [DISCUSS] Proposal to add API to force rules matching
specific rels
Hi Haisheng,
You
C_limit, we need to get its child group
> > cardinality, compute local cost C_local, so that we can calculate the
> > child group's upper bound cost and pass it down.
> >
> > I can't figure out how we can do group pruning without shared relational
> > properties.
>
@calcite.apache.org (dev@calcite.apache.org)
主 题：Re: Re: Re: [DISCUSS] Proposal to add API to force rules matching specific
rels
Hi Roman,
In my understanding, the proposed minor changes may only decrease the total
number of rule invocations slightly, but all principal problems remain the
same. In the top-down
accurate will the statistics be, how accurate
> will the cost model be?
>
> - Haisheng
>
> --------------
> 发件人：Julian Hyde
> 日 期：2020年01月13日 08:44:54
> 收件人：dev@calcite.apache.org
> 主 题：Re: [DISCUSS] Proposal to add API to
--
发件人：Haisheng Yuan
日 期：2020年01月14日 12:06:17
收件人：dev@calcite.apache.org
主 题：Re: Re: [DISCUSS] Proposal to add API to force rules matching specific rels
The example is valid if Calcite doesn't do normalization or preprocessing
before going
：2020年01月13日 08:44:54
收件人：dev@calcite.apache.org
主 题：Re: [DISCUSS] Proposal to add API to force rules matching specific rels
> MEMO group (RelSet) represents logically equivalent expressions.
> All the expressions in one group should share the same logical
> properties, e.g. functional d
done for each individual operator.
> >
> > Without resolving those issue, space pruning won't help much.
> >
> > There are a lot of room for improvement. Hope the community can join the
> > effort to make Calcite better.
> >
> > - Haisheng
> >
> > --
>
> There are a lot of room for improvement. Hope the community can join the
> effort to make Calcite better.
>
> - Haisheng
>
> --------------
> 发件人：Roman Kondakov
> 日 期：2020年01月10日 19:39:51
> 收件人：
> 主 题：Re: [DISCUSS] P
--
发件人：Roman Kondakov
日 期：2020年01月10日 19:39:51
收件人：
主 题：Re: [DISCUSS] Proposal to add API to force rules matching specific rels
@Haisheng, could you please clarify what you mean by these points?
> - the poor-design of column pruning,
>
> - Haisheng
>
> ----------------------
> 发件人：Roman Kondakov
> 日 期：2020年01月08日 02:39:19
> 收件人：
> 主 题：Re: [DISCUSS] Proposal to add API to force rules matching specific rels
>
> I forgot t
I see. But that’s unrelated to join ordering.
> On Jan 7, 2020, at 11:29 PM, Danny Chan wrote:
>
> Internally we have a multi-inputs merge join, for each input there maybe a
> collation permutations.
>
> Best,
> Danny Chan
> 在 2020年1月8日 +0800 AM1:20，Xiening Dai ，写道：
>> Danny, I am not sure
Internally we have a multi-inputs merge join, for each input there maybe a
collation permutations.
Best,
Danny Chan
在 2020年1月8日 +0800 AM1:20，Xiening Dai ，写道：
> Danny, I am not sure how this would affect join re-order. Could you elaborate?
>
>
> > On Jan 7, 2020, at 1:29 AM, Danny Chan wrote:
>
日 02:39:19
收件人：
主 题：Re: [DISCUSS] Proposal to add API to force rules matching specific rels
I forgot to mention that this approach was inspired by Stamatis's idea [1]
[1]
https://ponymail-vm.apache.org/_GUI_/thread.html/d8f8bc0efd091c0750534ca5cd224f4dfe8940c9d0a99ce486516fd5@%3Cdev.calcite.apache
I forgot to mention that this approach was inspired by Stamatis's idea [1]
[1]
https://ponymail-vm.apache.org/_GUI_/thread.html/d8f8bc0efd091c0750534ca5cd224f4dfe8940c9d0a99ce486516fd5@%3Cdev.calcite.apache.org%3E
--
Kind Regards
Roman Kondakov
On 07.01.2020 21:14, Roman Kondakov wrote:
>
y radar.
>
> I have been busy these days, but will send out a design doc in a few days.
> But just a heads up, the change would be larger than everyone's expected.
>
> - Haisheng
>
> ----------------------
> 发件人：Danny Chan
> 日 期：
Danny, I am not sure how this would affect join re-order. Could you elaborate?
> On Jan 7, 2020, at 1:29 AM, Danny Chan wrote:
>
> Hi, guys, it seems that the discussion silent now, so do we have some
> conclusion that can contribute to current code, i.e. the suggested API change
> or new
人：
主 题：Re: [DISCUSS] Proposal to add API to force rules matching specific rels
Hi, guys, it seems that the discussion silent now, so do we have some
conclusion that can contribute to current code, i.e. the suggested API change
or new abstraction ?
Or better, can someone give a design doc so
Hi, guys, it seems that the discussion silent now, so do we have some
conclusion that can contribute to current code, i.e. the suggested API change
or new abstraction ?
Or better, can someone give a design doc so that we can push and make that
implemented ?
Personally I was always looking
HI Igor,
Thank you for the details. Meanwhile, I solved it with separation of
conversion rules from the physical optimization rules. So the first pass
creates physical nodes with unknown physical properties (top-bottom), while
subsequent processing of the leaf nodes triggers rules which convert
Vladimir,
Hope it may help you.
Currently we applied the next way (just rough description):
1) We created an API to derive possible traits permutations on the basis of
children traits (quite similar to one, described in «On Demand trait set
request» topic)
2) added a general rule that
Hi Xing,
Thanks for your suggestion. Yes, this may help, and if I get your idea
right, I already had it in my reproducer:
1) Create the converted physical input:
https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/ProjectPhysicalRule.java#L49
2) Use it in
Hi Vladimir,
I think the way PlannerTests#GoodSingleRule and EnumerableXXXRule work may
help you ~
They work by a top-down fashion, but when matching parent, they convert the
children explicitly.
You may try below steps:
1. Construct a rule LogicalParentRule to match the LogicalParent without
Hi Xiening,
I read the thread about on-demand trait requests. It seems pretty similar
to what I am trying to achieve, as it facilitates the bottom-up propagation
of physical traits. In fact, both your and my strategy propagate traits
bottom-up, but I do this through rules, which also fire
I think the Convention as a trait is something special to Calcite (not a
Volcano concept). Calcite uses it for federation queries over heterogeneous
data source. Look at Calcite paper[1] 4 Traits. It explains the idea quite well.
[1] https://arxiv.org/pdf/1802.10233.pdf
> On Nov 1, 2019, at
@Vladimir: Given that you want to make the optimizer to work in bottom-up
fashion maybe it suffices to inverse the comparator in the RuleQueue [1].
For sure there are rules which would not be compatible with this change but
maybe for your project it makes sense.
I never tried it my self but I
I think "pull-up traits" is necessary, since the physical traits are
propagated upwards. However, I'm not fully convinced "On Demand
Trait" is the best solution, as I feel it may restrict the choices the
planner may consider. Maybe after the proposed design doc is ready,
we may look into the
Actually we solved this problem in our setup using a mechanism called “Pull-Up
Traits”, which explores the possible trait set of children’s input to decide
parent’s physical properties. In order to determine child input trait, you
would have to look at child’s children, and all the way to the
Hi Vladimir,
The SubsetTransformer interface and the iterating over the RelNodes
within a RelSubset in Drill is exactly implemented to do the trait
propagation. We also had to rely on AbstractConverter to fire
necessary rule to avoid the CanNotPlan issue. At some point, Calcite
community chooses
Hi Xiening,
Yes, I think that the manual creation of converters to trigger parent rules
should be enough at the moment. I'll try to explore this possibility. Thank
you.
Regards,
Vladimir
чт, 31 окт. 2019 г. в 20:11, Xiening Dai :
> Hi Vladimir,
>
> I think for short/mid term, #2 way (using
Hi Vladimir,
I think for short/mid term, #2 way (using AbstractConverter) should work after
we fix CALCITE-2970. We already understand the root cause, now are looking at
the best way to fix it. If you cannot wait, you can also create your own
converter rule so it won’t generate logical node,
Hi Danny,
Thank you very much for the links. What is described here is pretty much
similar to the problem I describe. Especially the discussion about trait
propagation, as this is basically what I need - to explore potential traits
of children before optimizing parents. And this is basically what
Thanks Vladimir for bringing up this discussion !
Did you notice that there is a JIRA issue about this problem ? [1] Also a
discussion about how to propagate the traits [2]
[1] https://issues.apache.org/jira/browse/CALCITE-2970
[2]
Not only "fireRule" method is needed, but the way to get all parents of a
set, containing a subset, produced by the transformation. Or a way to fire
rules without traits satisfaction check.
чт, 31 окт. 2019 г., 10:56 Vladimir Ozerov :
> Hi colleagues,
>
> I would like to discuss with the
34 matches
Mail list logo
