Re: Assigning jira issue

[XING JIN]

Thanks a lot ~ Francis

Francis Chuang  于2019年8月9日周五 上午11:13写道：

> Hi Xing Jin,
>
> I've added you to the contributor role.
>
> Francis
>
> On 9/08/2019 12:52 pm, XING JIN wrote:
> > Hi Francis,
> > I have the same request with Sahith.
> > Could you please add me to the contributor list as well ?
> > My user name is -- jin xing
> > Thanks a lot !
> >
> > Francis Chuang  于2019年8月9日周五 上午5:58写道：
> >
> >> Hey Sahith,
> >>
> >> I've added you to the contributor role and assigned the issue to you.
> >>
> >> Francis
> >>
> >> On 8/08/2019 11:54 pm, sahith.re...@gmail.com wrote:
> >>> Hello,
> >>>
> >>> I created issue CALCITE-3237,
> >> https://issues.apache.org/jira/browse/CALCITE-3237, and I submitted a
> PR
> >> for it as well. Can this issue be assigned to me? My jira username is
> >> snallapa. Thank you!
> >>>
> >>> Thanks,
> >>>
> >>> Sahith
> >>>
> >>
> >
>

Re: Match Converter Rule based on Child Nodes

[XING JIN]

I guess in Rahul's case, the child node of join should be converted first,
thus physical properties will be appended.
But RelNode doesn't have a reference to parent, thus an independent
RelOptRule is needed to convert LogicalJoin.
Just as Michael said, you need to override the method of RelOptRule#matches
and check the properties of child node.

Best,
Jin

Stamatis Zampetakis  于2019年8月21日周三 下午5:50写道：

> The Volcano planner works in a top-down fashion. It starts by transforming
> the root and move towards the leafs of the plan. Due to this when
> transforming a logical join its inputs are still in the logical convention
> so in principle they should not have any physical properties.
>
> Normally when the physical join algorithm is chosen the respective rule
> should be responsible for demanding the inputs of the join to have certain
> properties.
>
> Long story short, I think in your use case you should not make the rule
> match based on the properties of the child nodes but it should match
> unconditionally and if necessary demand some properties in its inputs. If I
> remember well the EnumerableMergeJoinRule follows this approach so you can
> have a look there.
>
> Best,
> Stamatis
>
> On Tue, Aug 20, 2019, 5:07 PM Michael Mior  wrote:
>
> > If you just want to control whether the rule gets applied, you can
> > override RelOptRule#matches which canreturns a boolean indicating
> > whether the rule should be applied.
> > --
> > Michael Mior
> > mm...@apache.org
> >
> > Le ven. 9 août 2019 à 08:48, rahul patwari
> >  a écrit :
> > >
> > > Hi,
> > >
> > > We want to create a ConverterRule which converts the default calling
> > > Convention to external storage-specific calling convention depending on
> > the
> > > Children nodes, like RelOptRule.
> > >
> > > For example, depending on the properties of the child nodes, we want to
> > > convert LogicalJoin to external system's specific Join implementation.
> > >
> > > Currently, ConverterRule
> > > <
> >
> https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75
> > >
> > > cannot take Children and Child Policy is
> > RelOptRuleOperandChildPolicy.ANY.
> > >
> > > What is the preferred way to achieve this task?
> > >
> > > Thanks,
> > > Rahul
> >
>

Re: Mapping arithmetic '+' operator with Date and Timestamp operands to target dialect functions

[XING JIN]

I guess in your approach, you need to maintain a customized
`SqlImplementor` ?
How about add a RelOptRule and detect pattern of "date + interval '1' DAY"
and use a RexShuttle to convert to "target_func(date, interval '1' DAY)"
Thus only a RelOptRule needs to be maintained.

Sachin Rohaj  于2019年8月23日周五 下午11:32写道：

> Hello,
>
> In one of my use case i am working on DATE and TIMESTAMP based operands. We
> have
> select date + interval '1' DAY from table;
>
> I need to map this to target dialect.
> select target_func(date, interval '1' DAY) from table;
>
> *Approach*: I am planning to intercept the '+' operator in the
> 'SqlImplementor' class using 'call.type.getSqlTypeName()' and delegate that
> responsibility to get the 'target_func' from the specific dialect.
> For exampe: 'dialect.getTargetFunction(SqlTypeName)'.
>
> Thanks,
> Sachin
>

Re: Match Converter Rule based on Child Nodes

[XING JIN]

Hmm,, I get it
So maybe you can try HepPlanner with match order of BOTTOM_UP;
Or if VolcanoPlanner is necessary, how about call the optimization multiple
times, e.g. wrap it with a loop

Jin

rahul patwari  于2019年8月23日周五 上午10:59写道：

> Hi Jin,
>
> We wanted to transform LogicalJoin to different Join Types in the external
> system depending on the LogicalJoin Child Properties(which are in external
> convention).
> As LogicalJoin can be a child of LogicalJoin, because of VolcanoPlanner's
> top-down approach, the child LogicalJoin is not converted first and we end
> up in "Not enough rules to find the Cheapest Plan".
>
> Regards,
> Rahul
>
> On Fri, Aug 23, 2019 at 8:17 AM XING JIN  wrote:
>
> > If I understand correctly, you can try below steps:
> > 1. Create a rule of Converter to convert child nodes to append external
> > Convention;
> > 2. Create a rule of RelOptRule to convert the parent node -- check the
> > Convention of child node when matching
> >
> > Is it applicable ?
> >
> > rahul patwari  于2019年8月22日周四 下午11:18写道：
> >
> > > Hi,
> > >
> > > The properties of the child nodes correspond to the external
> Convention.
> > > So, the child nodes have to be converted before the parent can be
> > > transformed.
> > > If the property doesn't match (or) if the property cannot be
> > > determined(child node not converted), the rule cannot be applied. So,
> we
> > > end up in "Not enough Rules to find the Cheapest Plan".
> > > Is there a way to convert the Child nodes before the parent?
> > > Can VolcanoPlanner be configured to work in bottom-up fashion?
> > >
> > > Regards,
> > > Rahul
> > >
> > > On Thu, Aug 22, 2019 at 3:33 PM XING JIN 
> > wrote:
> > >
> > > > I guess in Rahul's case, the child node of join should be converted
> > > first,
> > > > thus physical properties will be appended.
> > > > But RelNode doesn't have a reference to parent, thus an independent
> > > > RelOptRule is needed to convert LogicalJoin.
> > > > Just as Michael said, you need to override the method of
> > > RelOptRule#matches
> > > > and check the properties of child node.
> > > >
> > > > Best,
> > > > Jin
> > > >
> > > > Stamatis Zampetakis  于2019年8月21日周三 下午5:50写道：
> > > >
> > > > > The Volcano planner works in a top-down fashion. It starts by
> > > > transforming
> > > > > the root and move towards the leafs of the plan. Due to this when
> > > > > transforming a logical join its inputs are still in the logical
> > > > convention
> > > > > so in principle they should not have any physical properties.
> > > > >
> > > > > Normally when the physical join algorithm is chosen the respective
> > rule
> > > > > should be responsible for demanding the inputs of the join to have
> > > > certain
> > > > > properties.
> > > > >
> > > > > Long story short, I think in your use case you should not make the
> > rule
> > > > > match based on the properties of the child nodes but it should
> match
> > > > > unconditionally and if necessary demand some properties in its
> > inputs.
> > > > If I
> > > > > remember well the EnumerableMergeJoinRule follows this approach so
> > you
> > > > can
> > > > > have a look there.
> > > > >
> > > > > Best,
> > > > > Stamatis
> > > > >
> > > > > On Tue, Aug 20, 2019, 5:07 PM Michael Mior 
> wrote:
> > > > >
> > > > > > If you just want to control whether the rule gets applied, you
> can
> > > > > > override RelOptRule#matches which canreturns a boolean indicating
> > > > > > whether the rule should be applied.
> > > > > > --
> > > > > > Michael Mior
> > > > > > mm...@apache.org
> > > > > >
> > > > > > Le ven. 9 août 2019 à 08:48, rahul patwari
> > > > > >  a écrit :
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > We want to create a ConverterRule which converts the default
> > > calling
> > > > > > > Convention to external storage-specific calling convention
> > > depending
> > > > on
> > > > > > the
> > > > > > > Children nodes, like RelOptRule.
> > > > > > >
> > > > > > > For example, depending on the properties of the child nodes, we
> > > want
> > > > to
> > > > > > > convert LogicalJoin to external system's specific Join
> > > > implementation.
> > > > > > >
> > > > > > > Currently, ConverterRule
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75
> > > > > > >
> > > > > > > cannot take Children and Child Policy is
> > > > > > RelOptRuleOperandChildPolicy.ANY.
> > > > > > >
> > > > > > > What is the preferred way to achieve this task?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Rahul
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Match Converter Rule based on Child Nodes

[XING JIN]

If I understand correctly, you can try below steps:
1. Create a rule of Converter to convert child nodes to append external
Convention;
2. Create a rule of RelOptRule to convert the parent node -- check the
Convention of child node when matching

Is it applicable ?

rahul patwari  于2019年8月22日周四 下午11:18写道：

> Hi,
>
> The properties of the child nodes correspond to the external Convention.
> So, the child nodes have to be converted before the parent can be
> transformed.
> If the property doesn't match (or) if the property cannot be
> determined(child node not converted), the rule cannot be applied. So, we
> end up in "Not enough Rules to find the Cheapest Plan".
> Is there a way to convert the Child nodes before the parent?
> Can VolcanoPlanner be configured to work in bottom-up fashion?
>
> Regards,
> Rahul
>
> On Thu, Aug 22, 2019 at 3:33 PM XING JIN  wrote:
>
> > I guess in Rahul's case, the child node of join should be converted
> first,
> > thus physical properties will be appended.
> > But RelNode doesn't have a reference to parent, thus an independent
> > RelOptRule is needed to convert LogicalJoin.
> > Just as Michael said, you need to override the method of
> RelOptRule#matches
> > and check the properties of child node.
> >
> > Best,
> > Jin
> >
> > Stamatis Zampetakis  于2019年8月21日周三 下午5:50写道：
> >
> > > The Volcano planner works in a top-down fashion. It starts by
> > transforming
> > > the root and move towards the leafs of the plan. Due to this when
> > > transforming a logical join its inputs are still in the logical
> > convention
> > > so in principle they should not have any physical properties.
> > >
> > > Normally when the physical join algorithm is chosen the respective rule
> > > should be responsible for demanding the inputs of the join to have
> > certain
> > > properties.
> > >
> > > Long story short, I think in your use case you should not make the rule
> > > match based on the properties of the child nodes but it should match
> > > unconditionally and if necessary demand some properties in its inputs.
> > If I
> > > remember well the EnumerableMergeJoinRule follows this approach so you
> > can
> > > have a look there.
> > >
> > > Best,
> > > Stamatis
> > >
> > > On Tue, Aug 20, 2019, 5:07 PM Michael Mior  wrote:
> > >
> > > > If you just want to control whether the rule gets applied, you can
> > > > override RelOptRule#matches which canreturns a boolean indicating
> > > > whether the rule should be applied.
> > > > --
> > > > Michael Mior
> > > > mm...@apache.org
> > > >
> > > > Le ven. 9 août 2019 à 08:48, rahul patwari
> > > >  a écrit :
> > > > >
> > > > > Hi,
> > > > >
> > > > > We want to create a ConverterRule which converts the default
> calling
> > > > > Convention to external storage-specific calling convention
> depending
> > on
> > > > the
> > > > > Children nodes, like RelOptRule.
> > > > >
> > > > > For example, depending on the properties of the child nodes, we
> want
> > to
> > > > > convert LogicalJoin to external system's specific Join
> > implementation.
> > > > >
> > > > > Currently, ConverterRule
> > > > > <
> > > >
> > >
> >
> https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75
> > > > >
> > > > > cannot take Children and Child Policy is
> > > > RelOptRuleOperandChildPolicy.ANY.
> > > > >
> > > > > What is the preferred way to achieve this task?
> > > > >
> > > > > Thanks,
> > > > > Rahul
> > > >
> > >
> >
>

Re: Is it possible to retrieve materialized view query rewrite from Calcite Plan

[XING JIN]

I think it will work. I once did similar things and was playing around with
VolcanoPlanner and RelOptMaterializations.



Shubham Kumar  于2019年9月16日周一 下午8:32写道：

> Hey everyone,
>
> Thank you for the help.
>
> @Stamatis, @Danny : Thanks for pointing out that  the reltosql api should
> be used after volcano planner execution which considers materializations
> and lattices, I understand it now.
>
> @Xing: Yeah, it works perfectly fine, I am able to see simple substitutions
> in action done by Calcite Planner!
>
> Although I guess "*org.apache.calcite.tes*t" API can't be accessed in some
> other project. To extract out the materialized query,  I was playing around
> with Prepare and optimize of Calcite Prepare class[1]. Let me know if I am
> moving in the correct direction.
>
> Since, now I am able to get the rewrite query, I wanted to share the exact
> use case which I am trying to implement and get your insights:
>
> I have explicitly defined materialized views which are stored in data
> source and defining it in *root.schema.materializations *of model file
> (keeping the "*view*" field empty as its already exists). Now, I will be
> completely responsible for maintaining this materialized view table in my
> datasource as various datasources don't have built in Materialized View
> like Mysql etc. Finally, I aim the rewrite query that I retrieve from
> Calcite to execute it using Spark SQL (data of tables will be stored in
> HDFS). So basically query rewrite of Calcite will be a layer before Spark
> SQL and it uses the optimized query using materialized views for making the
> actual Spark plan and executing it in a distributed manner.
>
> I would love to have your insights on this on whether this is will be
> feasible and if it would improve query performance.
>
> Problems that I am facing currently is that while using "
> *org.apache.calcite.adapter.jdbc.JdbcSchema$Factory*" as the factory method
> I get the following exception:
>
>
> *Cannot define materialization; parent schema 'test_calcite' is not a
> SemiMutableSchema*
>
> Currently I am using CloneSchema$Factory for testing things out but I guess
> it maintains in memory tables which eventually I won't be able to use in
> case of large tables. I was wondering if I would have to write my own
> Factory classes and implementations for my use case or is there something
> already present.
>
> [1]
>
>
> https://github.com/apache/calcite/blob/73023148e7f37d494f6caf92b01b090f6dde13cd/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L320
>
> Thanks
> Shubham
>
>
> On Mon, Sep 16, 2019 at 11:32 AM XING JIN  wrote:
>
> > Hi,  Shubham Kumar
> > If I understand correctly, you want get the optimized(by materialized
> view)
> > SQL String. I wrote a simple test as below, please check if it's helpful
> > for you.
> >
> > @Test public void testDEV() {
> >   final String m = "select \"deptno\", \"empid\", \"name\""
> >   + "from \"emps\" where \"deptno\" = 10";
> >   final String q = "select \"empid\" + 1 as x, \"name\""
> >   + "from \"emps\" where \"deptno\" = 10";
> >
> >   CalciteAssert.that()
> >   .withMaterializations(HR_FKUK_MODEL, "m0", m)
> >   .query(q)
> >   .withHook(Hook.SUB, (Consumer) r ->
> >   {
> > RelToSqlConverter converter =
> > new RelToSqlConverter(CalciteSqlDialect.DEFAULT);
> > final SqlNode sqlNode = converter.visitChild(0, r).asStatement();
> >
> >
> System.out.println(sqlNode.toSqlString(CalciteSqlDialect.DEFAULT).getSql());
> >   })
> >   .enableMaterializations(true)
> >   .explainContains("hr, m0");
> > }
> >
> > The output is as below:
> >
> > SELECT "empid" + 1 AS "X", "name"
> > FROM "hr"."m0"
> >
> >
> > Danny Chan  于2019年9月16日周一 下午1:44写道：
> >
> > > Hi, Shubham Kumar ~
> > >
> > > > However, I wanted the optimized rewrite SQL query if possible, not
> just
> > > the
> > > > plan. So far, I tried to use the rel2sql api but when I print
> > > > RelNode.toString(), it gives me a plan which involves scanning the
> raw
> > > > tables and hence the SQL generated by rel2sql is not the one which
> > > utilized
> > > > the materialized view. Any pointers to get the rewrite query.
> > >
> > > Did you try the RelNode.toString() after the volcano planner promotion,
> > it
> > > is not expected to involve scanning the raw tables.
> > >
> > > Best,
> > > Danny Chan
> > > 在 2019年9月11日 +0800 PM9:42，dev@calcite.apache.org，写道：
> > > >
> > > > However, I wanted the optimized rewrite SQL query if possible, not
> just
> > > the
> > > > plan. So far, I tried to use the rel2sql api but when I print
> > > > RelNode.toString(), it gives me a plan which involves scanning the
> raw
> > > > tables and hence the SQL generated by rel2sql is not the one which
> > > utilized
> > > > the materialized view. Any pointers to get the rewrite query.
> > >
> >
>
>
> --
> Thanks & Regards
>
> Shubham Kumar
>

Re: [ANNOUNCE] New committer: Muhammad Gelbana

[XING JIN]

Congrats, Muhammad !

王炎林 <1989yanlinw...@163.com> 于2019年9月18日周三 上午10:38写道：

> Congratulations, Muhammad!
>
>
> Best,
> Yanlin
>
>
>
>
>
>
> At 2019-09-18 05:58:53, "Francis Chuang"  wrote:
> >Apache Calcite's Project Management Committee (PMC) has invited Muhammad
> >Gelbana to become a committer, and we are pleased to announce that he
> >has accepted.
> >
> >Muhammad is an active contributor and has contributed numerous patches
> >to Calcite. He has also been extremely active on the mailing list,
> >helping out new users and participating in design discussions.
> >
> >Muhammad, welcome, thank you for your contributions, and we look forward
> >your further interactions with the community! If you wish, please feel
> >free to tell us more about yourself and what you are working on.
> >
> >Francis (on behalf of the Apache Calcite PMC)
>

Re: Re: [ANNOUNCE] New committer: Julian Feinauer

[XING JIN]

Congrats, Julian !
You are well deserved ~

Haisheng Yuan  于2019年9月18日周三 上午10:38写道：

> Congrats, Julian!
>
> - Haisheng
>
> --
> 发件人：Chunwei Lei
> 日 期：2019年09月18日 10:30:31
> 收件人：
> 主 题：Re: [ANNOUNCE] New committer: Julian Feinauer
>
> Congratulations, Julian!
>
>
>
> Best,
> Chunwei
>
>
> On Wed, Sep 18, 2019 at 9:24 AM Danny Chan  wrote:
>
> > Congratulations, Muhammad ! Welcome to join us ! Thanks for your huge
> > contribution for the Match Recognize.
> >
> > Best,
> > Danny Chan
> > 在 2019年9月18日 +0800 AM5:55，Francis Chuang ，写道：
> > > Apache Calcite's Project Management Committee (PMC) has invited Julian
> > > Feinauer to become a committer, and we are pleased to announce that he
> > > has accepted.
> > >
> > > Julian is an active contributor to the Calcite code base and has been
> > > active on the mailing list answering questions, participating in
> > > discussions and voting for releases.
> > >
> > > Julian, welcome, thank you for your contributions, and we look forward
> > > your further interactions with the community! If you wish, please feel
> > > free to tell us more about yourself and what you are working on.
> > >
> > > Francis (on behalf of the Apache Calcite PMC)
> >
>
>

Re: Is it possible to retrieve materialized view query rewrite from Calcite Plan

[XING JIN]

Hi,  Shubham Kumar
If I understand correctly, you want get the optimized(by materialized view)
SQL String. I wrote a simple test as below, please check if it's helpful
for you.

@Test public void testDEV() {
  final String m = "select \"deptno\", \"empid\", \"name\""
  + "from \"emps\" where \"deptno\" = 10";
  final String q = "select \"empid\" + 1 as x, \"name\""
  + "from \"emps\" where \"deptno\" = 10";

  CalciteAssert.that()
  .withMaterializations(HR_FKUK_MODEL, "m0", m)
  .query(q)
  .withHook(Hook.SUB, (Consumer) r ->
  {
RelToSqlConverter converter =
new RelToSqlConverter(CalciteSqlDialect.DEFAULT);
final SqlNode sqlNode = converter.visitChild(0, r).asStatement();

System.out.println(sqlNode.toSqlString(CalciteSqlDialect.DEFAULT).getSql());
  })
  .enableMaterializations(true)
  .explainContains("hr, m0");
}

The output is as below:

SELECT "empid" + 1 AS "X", "name"
FROM "hr"."m0"


Danny Chan  于2019年9月16日周一 下午1:44写道：

> Hi, Shubham Kumar ~
>
> > However, I wanted the optimized rewrite SQL query if possible, not just
> the
> > plan. So far, I tried to use the rel2sql api but when I print
> > RelNode.toString(), it gives me a plan which involves scanning the raw
> > tables and hence the SQL generated by rel2sql is not the one which
> utilized
> > the materialized view. Any pointers to get the rewrite query.
>
> Did you try the RelNode.toString() after the volcano planner promotion, it
> is not expected to involve scanning the raw tables.
>
> Best,
> Danny Chan
> 在 2019年9月11日 +0800 PM9:42，dev@calcite.apache.org，写道：
> >
> > However, I wanted the optimized rewrite SQL query if possible, not just
> the
> > plan. So far, I tried to use the rel2sql api but when I print
> > RelNode.toString(), it gives me a plan which involves scanning the raw
> > tables and hence the SQL generated by rel2sql is not the one which
> utilized
> > the materialized view. Any pointers to get the rewrite query.
>

[DISCUSS] Refinement for Substitution-Based MV Matching

[XING JIN]

Hi Contributors,

I'm writing this email and hope to start a discussion about
https://issues.apache.org/jira/browse/CALCITE-3334 .

The approach of substitution-based materialized view matching is effective
for its simplicity and extensibility. But now and then we confront
materialization matching failures especially for SQLs with multiple nested
levels. We keep thinking that how many more rules still need to be added
and can we enumerate all the common matching patterns.

The existing rules in SubstitutionVisitor &
MaterializedViewSubstitutionVisitor are created by heuristic and
experience. They are quite straightforward. But from my understanding the
design of rule set is not systematic enough. Especially compensating
Project or Filter are not given enough good care.

I raised a doc --
https://docs.google.com/document/d/1JpwGNFE3hw3yXb7W3-95-jXKClZC5UFPKbuhgYDuEu4
to
propose refinement for substitution-based materialized view matching from
two aspects:
1. Canonicalize before materialization matching;
2. Separate matching rules into two categories and enumerate matching
patterns which need to be covered by rules.

It's great if you can share some comments on this.

Best,
Jin

Re: [DISCUSS] Refinement for Substitution-Based MV Matching

[XING JIN]

Thanks a lot Danny and Haisheng for comments on the doc ~
And more comments are welcome ~

Best,
Jin

XING JIN  于2019年9月16日周一 下午9:17写道：

> Hi Contributors,
>
> I'm writing this email and hope to start a discussion about
> https://issues.apache.org/jira/browse/CALCITE-3334 .
>
> The approach of substitution-based materialized view matching is effective
> for its simplicity and extensibility. But now and then we confront
> materialization matching failures especially for SQLs with multiple nested
> levels. We keep thinking that how many more rules still need to be added
> and can we enumerate all the common matching patterns.
>
> The existing rules in SubstitutionVisitor &
> MaterializedViewSubstitutionVisitor are created by heuristic and
> experience. They are quite straightforward. But from my understanding the
> design of rule set is not systematic enough. Especially compensating
> Project or Filter are not given enough good care.
>
> I raised a doc --
> https://docs.google.com/document/d/1JpwGNFE3hw3yXb7W3-95-jXKClZC5UFPKbuhgYDuEu4
>  to
> propose refinement for substitution-based materialized view matching from
> two aspects:
> 1. Canonicalize before materialization matching;
> 2. Separate matching rules into two categories and enumerate matching
> patterns which need to be covered by rules.
>
> It's great if you can share some comments on this.
>
> Best,
> Jin
>
>

Re: Assigning jira issue

[XING JIN]

Hi Francis,
I have the same request with Sahith.
Could you please add me to the contributor list as well ?
My user name is -- jin xing
Thanks a lot !

Francis Chuang  于2019年8月9日周五 上午5:58写道：

> Hey Sahith,
>
> I've added you to the contributor role and assigned the issue to you.
>
> Francis
>
> On 8/08/2019 11:54 pm, sahith.re...@gmail.com wrote:
> > Hello,
> >
> > I created issue CALCITE-3237,
> https://issues.apache.org/jira/browse/CALCITE-3237, and I submitted a PR
> for it as well. Can this issue be assigned to me? My jira username is
> snallapa. Thank you!
> >
> > Thanks,
> >
> > Sahith
> >
>

Re: How to get columnName as `COUNT(*)` , not `EXPR$0`

[XING JIN]

You can check the below doc of SqlValidatorUtil#getAlias for explanation:

  /**
   * Derives an alias for a node, and invents a mangled identifier if it
   * cannot.
   *
   * Examples:
   *
   * 
   * Alias: "1 + 2 as foo" yields "foo"
   * Identifier: "foo.bar.baz" yields "baz"
   * Anything else yields "expr$ordinal"
   * 
   *
   * @return An alias, if one can be derived; or a synthetic alias
   * "expr$ordinal" if ordinal  0; otherwise null
   */
  public static String getAlias(SqlNode node, int ordinal)

But from my experience, you'd better not rely on above logic heavily. If
you really care about the output name, just give it an alias explicitly.

Juan Pan  于2019年9月29日周日 下午1:27写道：

> That means Calcite can only return real columnName or columnLabel from
> simple column or alias. And any aggregate function, or calculate expression
> without alias, parsing expression, i.e, `EXPR$0` will be returned?
>
>
>  Juan Pan
>
>
> panj...@apache.org
> Juan Pan(Trista), Apache ShardingSphere
>
>
> On 09/29/2019 13:16，XING JIN wrote：
> If no column name given explicitly, e.g. by alias or simple identifier,
> Calcite will derive one but not from the aggregate function.
>
> Juan Pan  于2019年9月29日周日 下午1:12写道：
>
> Thank for your reply. It is a indirect way to get columnName.
>
>
> Calcite can not return the real columnName from SQL, is it right?
>
>
> Juan Pan
>
>
> panj...@apache.org
> Juan Pan(Trista), Apache ShardingSphere
>
>
> On 09/29/2019 12:21，XING JIN wrote：
> You can try to give an alias for the selected column.
>
> Juan Pan  于2019年9月29日周日 上午11:39写道：
>
>
>
> Hi everyone,
>
>
> I executed SQL `select count(*) from tb1` through Calcite and
> resultSet.getMetaData().getColumnName(i) in my project. But the result is
> `EXPR$0` not `COUNT(*)`.
>
>
> Is there any way to get real columnName?
>
>
> Thanks for your attention.
>
>
> Regard,
> Trista
>
>
>
>
> Juan Pan
>
>
> panj...@apache.org
> Juan Pan(Trista), Apache ShardingSphere
>
>
>
>

Re: How to get columnName as `COUNT(*)` , not `EXPR$0`

[XING JIN]

If no column name given explicitly, e.g. by alias or simple identifier,
Calcite will derive one but not from the aggregate function.

Juan Pan  于2019年9月29日周日 下午1:12写道：

> Thank for your reply. It is a indirect way to get columnName.
>
>
> Calcite can not return the real columnName from SQL, is it right?
>
>
>  Juan Pan
>
>
> panj...@apache.org
> Juan Pan(Trista), Apache ShardingSphere
>
>
> On 09/29/2019 12:21，XING JIN wrote：
> You can try to give an alias for the selected column.
>
> Juan Pan  于2019年9月29日周日 上午11:39写道：
>
>
>
> Hi everyone,
>
>
> I executed SQL `select count(*) from tb1` through Calcite and
> resultSet.getMetaData().getColumnName(i) in my project. But the result is
> `EXPR$0` not `COUNT(*)`.
>
>
> Is there any way to get real columnName?
>
>
> Thanks for your attention.
>
>
> Regard,
> Trista
>
>
>
>
> Juan Pan
>
>
> panj...@apache.org
> Juan Pan(Trista), Apache ShardingSphere
>
>
>

Re: How to get columnName as `COUNT(*)` , not `EXPR$0`

[XING JIN]

It's my pleasure, you are welcome ~

Juan Pan  于2019年9月29日周日 下午2:24写道：

> Hi XING,
> I appreciate your kindness. :-D Your detailed and prompt replies really
> helped me a lot.
> I will review the java doc you mentioned.
>
>
> Best wishes,
> Trista
>
>
>  Juan Pan
>
>
> panj...@apache.org
> Juan Pan(Trista), Apache ShardingSphere
>
>
> On 09/29/2019 13:58，XING JIN wrote：
> You can check the below doc of SqlValidatorUtil#getAlias for explanation:
>
> /**
> * Derives an alias for a node, and invents a mangled identifier if it
> * cannot.
> *
> * Examples:
> *
> * 
> * Alias: "1 + 2 as foo" yields "foo"
> * Identifier: "foo.bar.baz" yields "baz"
> * Anything else yields "expr$ordinal"
> * 
> *
> * @return An alias, if one can be derived; or a synthetic alias
> * "expr$ordinal" if ordinal  0; otherwise null
> */
> public static String getAlias(SqlNode node, int ordinal)
>
> But from my experience, you'd better not rely on above logic heavily. If
> you really care about the output name, just give it an alias explicitly.
>
> Juan Pan  于2019年9月29日周日 下午1:27写道：
>
> That means Calcite can only return real columnName or columnLabel from
> simple column or alias. And any aggregate function, or calculate expression
> without alias, parsing expression, i.e, `EXPR$0` will be returned?
>
>
> Juan Pan
>
>
> panj...@apache.org
> Juan Pan(Trista), Apache ShardingSphere
>
>
> On 09/29/2019 13:16，XING JIN wrote：
> If no column name given explicitly, e.g. by alias or simple identifier,
> Calcite will derive one but not from the aggregate function.
>
> Juan Pan  于2019年9月29日周日 下午1:12写道：
>
> Thank for your reply. It is a indirect way to get columnName.
>
>
> Calcite can not return the real columnName from SQL, is it right?
>
>
> Juan Pan
>
>
> panj...@apache.org
> Juan Pan(Trista), Apache ShardingSphere
>
>
> On 09/29/2019 12:21，XING JIN wrote：
> You can try to give an alias for the selected column.
>
> Juan Pan  于2019年9月29日周日 上午11:39写道：
>
>
>
> Hi everyone,
>
>
> I executed SQL `select count(*) from tb1` through Calcite and
> resultSet.getMetaData().getColumnName(i) in my project. But the result is
> `EXPR$0` not `COUNT(*)`.
>
>
> Is there any way to get real columnName?
>
>
> Thanks for your attention.
>
>
> Regard,
> Trista
>
>
>
>
> Juan Pan
>
>
> panj...@apache.org
> Juan Pan(Trista), Apache ShardingSphere
>
>
>
>
>

Re: Trivial query simplification

[XING JIN]

"v = 1 and v is null"
cannot be simplified to "v = 1" not matter v is nullable or not nullable

If you really mean that "v is not null", I made below test case in
RelOptRulesTest.java for illustration:


// mgr is nullable
  @Test public void testDEV() throws Exception {
HepProgram program = new HepProgramBuilder()
  .addRuleInstance(ReduceExpressionsRule.FILTER_INSTANCE)
  .build();

final String sql = "select deptno"
  + " from emp"
  + " where mgr = 10 and mgr is not null";
checkPlanning(new HepPlanner(program), sql);
  }

The plan is
LogicalProject(DEPTNO=[$7])
  LogicalFilter(condition=[=($3, 10)])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])

Enrico ~ you may try ReduceExpressionsRule.FILTER_INSTANCE


Feng Zhu  于2019年9月24日周二 下午5:50写道：

> Hi, Enrico,
> I'm a little confused about your expectations. Could you clarify it?
> Moreover, is it right for the below simplification (do you mean v is not
> null)?
> (v=1 and v is null) -> v=1
> (do you mean v is not null?)
>
> Best regards
>
> Enrico Olivelli  于2019年9月24日周二 下午5:41写道：
>
> > Hi,
> > I have a query like
> > SELECT * FROM MYTABLE WHERE v = 1 and v is null
> >
> > I am expecting Calcite to simplify it to
> > SELECT * FROM MYTABLE WHERE v = 1
> >
> > but this does not happen.
> >
> > Is any rule I should enable in order to make it happen ?
> >
> > This is the configuration of my Volcano planner:
> >
> >   final FrameworkConfig config = Frameworks.newConfigBuilder()
> > .parserConfig()
> > .defaultSchema(...)
> > .traitDefs()
> > .programs(Programs.ofRules(Programs.RULE_SET))
> > .build();
> >
> > Best regards
> > Enrico
> >
>

Re: How to get columnName as `COUNT(*)` , not `EXPR$0`

[XING JIN]

You can try to give an alias for the selected column.

Juan Pan  于2019年9月29日周日 上午11:39写道：

>
>
> Hi everyone,
>
>
> I executed SQL `select count(*) from tb1` through Calcite and
> resultSet.getMetaData().getColumnName(i) in my project. But the result is
> `EXPR$0` not `COUNT(*)`.
>
>
> Is there any way to get real columnName?
>
>
> Thanks for your attention.
>
>
> Regard,
> Trista
>
>
>
>
>  Juan Pan
>
>
> panj...@apache.org
> Juan Pan(Trista), Apache ShardingSphere
>
>

Re: Question about Interpreter and Corelations

[XING JIN]

You may modify your sql and operator Collect should not reference correlate
variables.

XING JIN  于2019年11月9日周六 下午12:32写道：

> Currently RelDecorrelator doesn't support a decorrelateRel(Collect),
> that's why decorrelate failed
>
> Zoltan Farkas  于2019年11月9日周六 上午5:46写道：
>
>> Done: https://issues.apache.org/jira/browse/CALCITE-3488 <
>> https://issues.apache.org/jira/browse/CALCITE-3488>
>> let me know if you need any more info.
>>
>> thank you
>>
>> —Z
>>
>> > On Nov 8, 2019, at 2:17 PM, Julian Hyde  wrote:
>> >
>> > As I said, I think you should log a bug.
>> >
>> > On Fri, Nov 8, 2019 at 10:23 AM Haisheng Yuan 
>> wrote:
>> >>
>> >> I am afraid this query can't be easily decorrelated.
>> >>
>> >> - Haisheng
>> >>
>> >> --
>> >> 发件人：Zoltan Farkas
>> >> 日 期：2019年11月08日 22:46:53
>> >> 收件人：
>> >> 主 题：Re: Question about Interpreter and Corelations
>> >>
>> >> Thanks Julian,
>> >>
>> >> Any idea how could I de-corelate the query?
>> >> RelDecorrelator.decorrelateQuery does not seem to alter the plan.
>> >>
>> >> Will log a jira for the interpreter issue.
>> >>
>> >> thank you
>> >>
>> >> —Z
>> >>
>> >>
>> >>
>> >>> On Nov 7, 2019, at 5:08 PM, Julian Hyde 
>> wrote:
>> >>>
>> >>> I don’t recall what the interpreter is currently capable of, but you
>> should log a bug.
>> >>>
>> >>> I wonder whether if you could get the query to work in the
>> interpreter if you decorrelated the query first.
>> >>>
>> >>> Julian
>> >>>
>> >>>> On Nov 7, 2019, at 11:16, Zoltan Farkas 
>> wrote:
>> >>>>
>> >>>> for a test query with the following plan:
>> >>>>
>> >>>> LogicalProject(name=[$1], friends=[$4])
>> >>>> LogicalCorrelate(correlation=[$cor0], joinType=[inner],
>> requiredColumns=[{0}])
>> >>>>  LogicalTableScan(table=[[characters]])
>> >>>>  Collect(field=[EXPR$0])
>> >>>>LogicalProject(name=[$2])
>> >>>>  LogicalJoin(condition=[=($0, $1)], joinType=[inner])
>> >>>>LogicalProject(characterId2=[$1])
>> >>>>  LogicalFilter(condition=[=($0, $cor0.characterId)])
>> >>>>   LogicalTableScan(table=[[friendships]])
>> >>>>LogicalProject(characterId=[$0], name=[$1])
>> >>>>  LogicalTableScan(table=[[characters]])
>> >>>>
>> >>>> I get :
>> >>>>
>> >>>> java.lang.nsupportedOperationException
>> >>>>  at
>> o.a.c.i.JaninoRexCompiler.lambda$compile$0(JaninoRexCompiler.java:94)[calcite-core-1.21.0.jar:1.21.0]
>> >>>>  at
>> o.a.c.a.e.RexToLixTranslator.translate0(RexToLixTranslator.java:714)[^]
>> >>>>  at ^.translate(^:199)[^]
>> >>>>  at ^.translate0(^:684)[^]
>> >>>>  at ^.translate(^:199)[^]
>> >>>>  at ^.translate(^:194)[^]
>> >>>>  ...
>> >>>>
>> >>>> when trying to use the Interpreter.
>> >>>>
>> >>>> Is this a current limitation?
>> >>>>
>> >>>> the query is :
>> >>>>
>> >>>> select name,
>> >>>>ARRAY(select c2.name from friendships f, characters c2
>> >>>>   where f.characterId1 = c.characterId and
>> f.characterId2 = c2.characterId) as friends
>> >>>> from characters c
>> >>>>
>> >>>> let me know
>> >>>>
>> >>>> thank you
>> >>>>
>> >>>> —Z
>>
>>

Re: Question about Interpreter and Corelations

[XING JIN]

Currently RelDecorrelator doesn't support a decorrelateRel(Collect), that's
why decorrelate failed

Zoltan Farkas  于2019年11月9日周六 上午5:46写道：

> Done: https://issues.apache.org/jira/browse/CALCITE-3488 <
> https://issues.apache.org/jira/browse/CALCITE-3488>
> let me know if you need any more info.
>
> thank you
>
> —Z
>
> > On Nov 8, 2019, at 2:17 PM, Julian Hyde  wrote:
> >
> > As I said, I think you should log a bug.
> >
> > On Fri, Nov 8, 2019 at 10:23 AM Haisheng Yuan 
> wrote:
> >>
> >> I am afraid this query can't be easily decorrelated.
> >>
> >> - Haisheng
> >>
> >> --
> >> 发件人：Zoltan Farkas
> >> 日 期：2019年11月08日 22:46:53
> >> 收件人：
> >> 主 题：Re: Question about Interpreter and Corelations
> >>
> >> Thanks Julian,
> >>
> >> Any idea how could I de-corelate the query?
> >> RelDecorrelator.decorrelateQuery does not seem to alter the plan.
> >>
> >> Will log a jira for the interpreter issue.
> >>
> >> thank you
> >>
> >> —Z
> >>
> >>
> >>
> >>> On Nov 7, 2019, at 5:08 PM, Julian Hyde 
> wrote:
> >>>
> >>> I don’t recall what the interpreter is currently capable of, but you
> should log a bug.
> >>>
> >>> I wonder whether if you could get the query to work in the interpreter
> if you decorrelated the query first.
> >>>
> >>> Julian
> >>>
>  On Nov 7, 2019, at 11:16, Zoltan Farkas 
> wrote:
> 
>  for a test query with the following plan:
> 
>  LogicalProject(name=[$1], friends=[$4])
>  LogicalCorrelate(correlation=[$cor0], joinType=[inner],
> requiredColumns=[{0}])
>   LogicalTableScan(table=[[characters]])
>   Collect(field=[EXPR$0])
> LogicalProject(name=[$2])
>   LogicalJoin(condition=[=($0, $1)], joinType=[inner])
> LogicalProject(characterId2=[$1])
>   LogicalFilter(condition=[=($0, $cor0.characterId)])
>    LogicalTableScan(table=[[friendships]])
> LogicalProject(characterId=[$0], name=[$1])
>   LogicalTableScan(table=[[characters]])
> 
>  I get :
> 
>  java.lang.nsupportedOperationException
>   at
> o.a.c.i.JaninoRexCompiler.lambda$compile$0(JaninoRexCompiler.java:94)[calcite-core-1.21.0.jar:1.21.0]
>   at
> o.a.c.a.e.RexToLixTranslator.translate0(RexToLixTranslator.java:714)[^]
>   at ^.translate(^:199)[^]
>   at ^.translate0(^:684)[^]
>   at ^.translate(^:199)[^]
>   at ^.translate(^:194)[^]
>   ...
> 
>  when trying to use the Interpreter.
> 
>  Is this a current limitation?
> 
>  the query is :
> 
>  select name,
> ARRAY(select c2.name from friendships f, characters c2
>    where f.characterId1 = c.characterId and
> f.characterId2 = c2.characterId) as friends
>  from characters c
> 
>  let me know
> 
>  thank you
> 
>  —Z
>
>

Re: [ANNOUNCE] Danny Chan joins Calcite PMC

[XING JIN]

Congratulations ! Danny ~

OpenInx  于2019年10月31日周四 上午9:33写道：

> Congrats, Danny!  Well deserve.
>
> On Thu, Oct 31, 2019 at 9:22 AM Leonard Xu  wrote:
>
> > Congratulation！ Danny
> >
> >
> > > On 2019年10月31日, at 上午9:06, Andrei Sereda  wrote:
> > >
> > > Congrats, Danny!
> > >
> > > On Wed, Oct 30, 2019 at 8:08 PM Julian Hyde  wrote:
> > >
> > >> Congratulations, Danny! Thank you for all of your hard work on code,
> > >> reviewing others' work, answering questions, and generally making our
> > >> community welcoming to all!
> > >>
> > >> Julian
> > >>
> > >> On Wed, Oct 30, 2019 at 4:32 PM Michael Mior 
> wrote:
> > >>>
> > >>> Welcome on board Danny! Glad to have you.
> > >>> --
> > >>> Michael Mior
> > >>> mm...@apache.org
> > >>>
> > >>> Le mer. 30 oct. 2019 à 17:22, Francis Chuang
> > >>>  a écrit :
> > 
> >  I'm pleased to announce that Danny has accepted an invitation to
> >  join the Calcite PMC. Danny has been a consistent and helpful
> >  figure in the Calcite community for which we are very grateful. We
> >  look forward to the continued contributions and support.
> > 
> >  Please join me in congratulating Danny!
> > 
> >  - Francis (on behalf of the Calcite PMC)
> > >>
> >
> >
>

Re: Re: [DISCUSS] On-demand traitset request

[XING JIN]

Hi Haisheng,

Thanks a lot for sharing this great proposal ~
For short I understand your idea as below:
1. Derive the distributions/collations that children COULD/MIGHT offer
2. Decide the best distributions/collations by first point and computing
logic of operator, say gropuings in Aggregate;

It comes to me that another important part is that children operator should
also provide the corresponding COST for the POSSIBLE distribution/collation.
The COST is not for the final plan, but a hypothesis.

Take below example
SELECT DISTINCT c, b FROM
  ( SELECT R.c c, S.b b FROM R, S
WHERE R.a=S.a and R.b=S.b and R.c=S.c) t;
Suppose R is ordered by (c, b, a), and S is ordered by (b, c, a).
Aggregate
+--- InnerJoin
 |--- TableScan on R
 +--- TableScan on S

InnerJoin should deliver that its possible collations and corresponding
costs at the same time.
- If ordered by (c, b, a) my cost is ...
- If ordered by (b, c, a) my cost is ...
- If ordered by (a, b, c) my cost is ...
By which Aggregate decide the 'best' required collation.
By this way we can better limit the searching space and also target the
relatively optimized (if not best) plan.

Also when you say "I didn't say adding to RelNode, but a new API/interface
for physical operator only.", I'm not so clear;
Currently the physical operators in Calcite like EnumerableHashJoin,
EnumerableMergeJoin, when created, their physical behavior(like real
collations) are determined.
So I belive you intend to add new API at upper layer, but there's no
physical optimizing phase in Calcite at this moment. Where do you want to
add the new API, can you specify ?

Thanks,
Jin

Jinfeng Ni  于2019年11月6日周三 上午1:56写道：

> @Haisheng, @Xiening,
>
> Thanks for pointing that previous email out.  Overall, I agree that
> the physical trait enforcement should be done in the engine, not in
> the rule. For the rule, it should only specify the request, and the
> corresponding transformation, and let the engine to explore the search
> space. It will be great if we can revamp the Volcano optimizer
> framework, to do that way.
>
> In terms of search space, it's always a tradeoff between the space
> searched and the optimality of the plan found. I think it's fine for
> the engine to explore a potential big search space, as long as it has
> effective "bound-and-prune" strategy. In the original Volcano paper,
> there is a way to prune the search space based on the best plan found
> so far, using the parameter "limit".  When an implementable plan is
> found, a "real" cost is obtained, which could be used to prune
> un-necessary search space.  That's actually the advantage of Volcano's
> "top-down" approach. However,  seems to me that Calcite's Volcano did
> not apply that approach effectively, because of the existence of
> AbstractConverter.
>
>
> On Sun, Nov 3, 2019 at 10:12 PM Haisheng Yuan 
> wrote:
> >
> > Hi Jinfeng,
> >
> > I think you might have missed the email about proposed API for physical
> operators I sent out previously in [1].
> >
> > We don't need request all the permutation, which is also impossible in
> practice, the search space is going to explode.
> >
> > In the example in email [1], I already talked about your concen on
> passing down parent request into children may lead to less optimal plan.
> Besically join operator can send 2 collation optimization requests, one is
> to pass request down, the other one is ignore the parent's request.
> >
> > Using AbstractConverter to enforce properties is inapporpriate, which
> handles all the optimization work to physical operator providers, meaning
> there is almost no physical level optimization mechanism in Calcite. SQL
> Server and Greenplum's optimizer, which are Cascades framework based,
> implemented the property enforcement in the core optimizer engine, not
> through AbstractConverter and rules, physical operators just need to
> implement those methods (or similar) I mentioned in email [1]. My goal is
> completely abolishing AbstractConverter.
> >
> > [1]
> http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e
> >
> > - Haisheng
> >
> > --
> > 发件人：Jinfeng Ni
> > 日 期：2019年11月01日 14:10:30
> > 收件人：
> > 主 题：Re: [DISCUSS] On-demand traitset request
> >
> > Hi Xiening,
> >
> > "Let say if R and S doesn’t have sorting properties at all. In your
> > case, we would end up adding enforcers for LHS and RHS to get
> > collation (a, b, c). Then we would need another enforcer to get
> > collation (b, c). This is a sub optimal plan as we could have use (b,
> > c, a) for join."
> >
> > In such case, for step 2 when MergeJoin request a permutation match of
> > (a, b,c) on both it's input, it is not necessary to end up with
> > collation (a, b, c) only. Since it request "permutation", MJ could ask
> > all possible satisfying collations, which include (b, c, a). In other
> > words, the steps I 

Re: [DISCUSS] Proposal to add API to force rules matching specific rels

[XING JIN]

Hi Vladimir,

I think the way PlannerTests#GoodSingleRule and EnumerableXXXRule work may
help you ~
They work by a top-down fashion, but when matching parent, they convert the
children explicitly.
You may try below steps:
1. Construct a rule LogicalParentRule to match the LogicalParent without
distribution/physical requirement for the LogicalChild;
2. In this rule, you call 'planner.changeTraits' on the LogicalChild to
build a new child with physical convention. Note that at this moment only
an empty RelSubset is created and no PhysicalChild exists.
3. Then set the RelNode to be the new input of LogicalParent;

By above steps, you can build a parent-child relationship between
LogicalParent and PhysicalChild, and at last the PhysicalParentRule will be
fired based on this relationship.

I have a commit to illustrate my idea, check VolcanoPlannerTest#testDEV in
below link, hope it may help you ~
https://github.com/jinxing64/calcite/tree/demo

Also I'm +1 with Seliverstov that to get all parents of a set, which
against the current check in RelSubset#getParentRels

Best,
Jin

Vladimir Ozerov  于2019年11月5日周二 下午6:41写道：

> Hi Xiening,
>
> I read the thread about on-demand trait requests. It seems pretty similar
> to what I am trying to achieve, as it facilitates the bottom-up propagation
> of physical traits. In fact, both your and my strategy propagate traits
> bottom-up, but I do this through rules, which also fire bottom-up, while in
> your case only the traits are propagated bottom-up, while rules continue
> working in a top-down fashion.
>
> However, I am thinking of how I would potentially implement my optimizer
> with your approach, and it feels like with on-demand traits resulting
> implementation of metadata queries may become very complex to that point
> that it will look like another set of rules, parallel to the already
> existing ruleset. For example, consider that I have a couple of distributed
> tables in an OLTP application. These tables have a number of indexes, and I
> would like to join them. First, I have a number of choices on how to join
> tables with respect to distribution. Then, I have a number of choices on
> which access method to use. Because sometimes it is beneficial to pick
> index scans instead of table scans even without index conditions, for
> example, to preserve a comfortable collation. So when my logical scan
> receives such metadata request, it typically cannot return all possible
> combinations, because there are too many of them. Instead, some heuristical
> or cost-based logic will be used to calculate a couple of most prospective
> ones. But it seems that we will have to duplicate the same logic in the
> corresponding rule, aren't we?
>
> I would love to read your design because this is a really interesting
> topic, and it is of great importance for the distributed engines developed
> on top of Calcite since proper use of distribution and collation is the key
> success factor for efficient query optimization.
>
> Regards,
> Vladimir.
>
> пт, 1 нояб. 2019 г. в 00:40, Xiening Dai :
>
> > Actually we solved this problem in our setup using a mechanism called
> > “Pull-Up Traits”, which explores the possible trait set of children’s
> input
> > to decide parent’s physical properties. In order to determine child input
> > trait, you would have to look at child’s children, and all the way to the
> > leaves nodes or a barrier. A barrier is a rel node which cannot derive
> any
> > traits regardless the input. A good example would be a user define
> function
> > which would throw off any distribution or collation. Then we realize just
> > pulling up is not enough, sometimes we would need to look at parent’s
> > requirement as well. So we try to solve this in a unified framework,
> which
> > we call “On Demand Trait” and implement it as part of the framework so
> > anyone can be benefited. I hope Haisheng can share a design doc once we
> > have more concrete ideas.
> >
> >
> > > On Oct 31, 2019, at 11:37 AM, Jinfeng Ni  wrote:
> > >
> > > Hi Vladimir,
> > >
> > > The SubsetTransformer interface and the iterating over the RelNodes
> > > within a RelSubset in Drill  is exactly implemented to do the trait
> > > propagation. We also had to rely on AbstractConverter to fire
> > > necessary rule to avoid the CanNotPlan issue. At some point, Calcite
> > > community chooses to remove AbstractConverter and Drill had to add it
> > > back, which is probably one of the main reasons for us to continue
> > > using a Calcite fork.  I still remember we constantly had to deal with
> > > the dilemma between "CanNotPlan" and long planing time due to explored
> > > search space.
> > >
> > > Glad to see more people are joining the effort to solve this long
> > > overdue issue, something missing in Calcite's core optimizer framework
> > > "since before Calcite was Calcite" (Jacques's words).
> > >
> > > Jinfeng
> > >
> > >
> > > On Thu, Oct 31, 2019 at 3:38 AM Vladimir Ozerov 
> > wrote:
> > >>
> > >> Hi 

CI passed error tests

[XING JIN]

Hi guys,
I made a PR and run continuous integration tests. [1]
A error test contained in the PR and tagged with @slowTest.
The tests should be failed but CI passed by mistake.
I doubt our current CI is not running with 'testSlow' configuration. Isn't
it ?
I'm not sure if I should create a JIRA.

Best,
Jin

[1] https://github.com/apache/calcite/pull/1653

Re: CI passed error tests

[XING JIN]

Thanks a lot Ruben ~
I'm working on "[CALCITE-3478] Restructure tests for materialized views" (
https://github.com/apache/calcite/pull/1560) , which contains "slow tests".
But I'm not sure how to add the label to PR. Is it only available for
committers ?
BTW, It's great if someone can help review
https://github.com/apache/calcite/pull/1560 . I've rebased the PR to
resolve conflicts for several times.

Best,
Jin

Ruben Q L  于2019年12月13日周五 下午4:00写道：

> Hi Jin,
>
> this is the expected behavior: slow tests are not executed by default in PR
> CI.
> If you want slow tests to be executed in your PR, you need to explicitly
> add the label "slow-tests-needed", as specified by [1].
> You can see a PR example with this label here [2].
>
> Best regards,
> Ruben
>
> [1] https://issues.apache.org/jira/browse/CALCITE-3141
> [2] https://github.com/apache/calcite/pull/1651
>
>
> Le ven. 13 déc. 2019 à 08:41, XING JIN  a écrit :
>
> > Hi guys,
> > I made a PR and run continuous integration tests. [1]
> > A error test contained in the PR and tagged with @slowTest.
> > The tests should be failed but CI passed by mistake.
> > I doubt our current CI is not running with 'testSlow' configuration.
> Isn't
> > it ?
> > I'm not sure if I should create a JIRA.
> >
> > Best,
> > Jin
> >
> > [1] https://github.com/apache/calcite/pull/1653
> >
>

Re: CI passed error tests

[XING JIN]

Thanks a lot, Stamatis~
I will run slow tests from my local.

Best,
Jin

在 2019年12月14日星期六，Stamatis Zampetakis  写道：

> Hi Jin,
>
> If the test run fine locally at your machine (e.g., ./gradlew testSlow)
> then they should be fine in the CI.
> Labels are only available for committers yes!
>
> Best,
> Stamatis
>
> On Fri, Dec 13, 2019 at 4:52 PM XING JIN  wrote:
>
> > Thanks a lot Ruben ~
> > I'm working on "[CALCITE-3478] Restructure tests for materialized views"
> (
> > https://github.com/apache/calcite/pull/1560) , which contains "slow
> > tests".
> > But I'm not sure how to add the label to PR. Is it only available for
> > committers ?
> > BTW, It's great if someone can help review
> > https://github.com/apache/calcite/pull/1560 . I've rebased the PR to
> > resolve conflicts for several times.
> >
> > Best,
> > Jin
> >
> > Ruben Q L  于2019年12月13日周五 下午4:00写道：
> >
> > > Hi Jin,
> > >
> > > this is the expected behavior: slow tests are not executed by default
> in
> > PR
> > > CI.
> > > If you want slow tests to be executed in your PR, you need to
> explicitly
> > > add the label "slow-tests-needed", as specified by [1].
> > > You can see a PR example with this label here [2].
> > >
> > > Best regards,
> > > Ruben
> > >
> > > [1] https://issues.apache.org/jira/browse/CALCITE-3141
> > > [2] https://github.com/apache/calcite/pull/1651
> > >
> > >
> > > Le ven. 13 déc. 2019 à 08:41, XING JIN  a
> > écrit :
> > >
> > > > Hi guys,
> > > > I made a PR and run continuous integration tests. [1]
> > > > A error test contained in the PR and tagged with @slowTest.
> > > > The tests should be failed but CI passed by mistake.
> > > > I doubt our current CI is not running with 'testSlow' configuration.
> > > Isn't
> > > > it ?
> > > > I'm not sure if I should create a JIRA.
> > > >
> > > > Best,
> > > > Jin
> > > >
> > > > [1] https://github.com/apache/calcite/pull/1653
> > > >
> > >
> >
>

Re: Problems to validate ddl commands

[XING JIN]

Hi Anthony ~
Calcite supports to validate DDL;
I cannot open your link, would you please give some details for your
problem ?

Best,
Jin

Anthony Louis  于2019年12月17日周二
上午4:08写道：

> When I try to validate a sql statement that contains "CREATE TABLE" it
> throws an error that is described in this gist
> https://gist.github.com/anthonylouisbsb/5f7d57886836cfac1742366722afcf41.
> I
> would like to know if is possible to validate a query using ddl statements?
>

Re: [DISCUSS] Tests vs multiline strings

[XING JIN]

Before coming to Calcite, I works quite some time on Scala. The code style
shares some similarities with Kotlin -- It's much simple and easier to read
when you write a test.
But we should think twice whether to bring in another language.
To answer Haisheng's question:
Because default lex config is Lex.ORACLE.

Best,
Jin

Chunwei Lei  于2019年12月16日周一 上午10:34写道：

> I agree with Julian.
>
> Changing to Kotlin needs lots of error, but gets a little gain. Besides, It
> costs much more
> time to write a test if developers are not familiar with Kotlin. I
> prefer to use Java as it is now.
>
>
> Best,
> Chunwei
>
>
> On Mon, Dec 16, 2019 at 9:02 AM Julian Hyde  wrote:
>
> > I don't think we should do this.
> >
> > Multi-line strings are a bit unwieldy, but they're not a major
> > problem. Transitioning our tests to a different language (Kotlin) is a
> > drastic solution. It requires developers to understand a new language,
> > and it loses all history from the source files.
> >
> > Julian
> >
> > On Sun, Dec 15, 2019 at 4:37 AM Vladimir Sitnikov
> >  wrote:
> > >
> > > I've filed two PRs to evaluate the impact of the replacement.
> > >
> > > $ to _: https://github.com/apache/calcite/pull/1659
> > > 203.3sec, 5510 completed,   3 failed,  91 skipped, Gradle Test Run
> > > :core:test
> > >
> > > $ to #: https://github.com/apache/calcite/pull/1660
> > > 196.7sec, 5510 completed,  53 failed,  91 skipped, Gradle Test Run
> > > :core:test
> > >
> > > There are test failures, however, both of them are almost ready.
> > >
> > > Both $ and _ are valid "Java identifier start" characters, so variable
> > name
> > > like _3 is valid.
> > > If we use # instead of $, then code generator needs to be updated as
> well
> > > as it sometimes uses Java variables like $3, and #3 would become
> invalid.
> > >
> > > Any thoughts?
> > >
> > > Vladimir
> >
>

Re: [DISCUSSION] Extension of Metadata Query

[XING JIN]

+1 on Danny's comment.
If we use MedataFactory to customize and use RelMetadataQuery for
convenience, that will make user confused.

Danny Chan  于2019年10月18日周五 下午12:33写道：

> That is the point, we should supply a way to extend the RelMetadataQuery
> conveniently for Calcite, because in most of the RelOptRules, user would
> use the code like:
>
> RelOptRuleCall.getMetadataQuery
>
> To get a RMQ instead of using AbstractRelNode.metadata() to fetch a
> MedataFactory.
>
> We should at lest unity the metadata query entrance/interfaces, or it
> would confuse a lot.
>
> Best,
> Danny Chan
> 在 2019年10月18日 +0800 AM12:23，Seliverstov Igor ，写道：
> > At least in our project (Apache Ignite) we use
> AbstractRelNode.metadata().
> >
> > But it so because there is no way to put our metadata type into
> > RelMetadataQuery without changes in Calcite.
> >
> > Regards,
> > Igor
> >
> > чт, 17 окт. 2019 г., 19:16 Xiening Dai :
> >
> > > MetadataFactory is still useful. It provides a way to access Metadata
> > > directly. If someone creates a new type of Metadata class, it can be
> > > accessed through AbstractRelNode.metadata(). This way you don’t need to
> > > update RelMetadataQuery interface to include the getter for this new
> meta.
> > > Although I don’t see this pattern being used often, but I do think it
> is
> > > still useful and shouldn’t be removed.
> > >
> > >
> > > For your second point, I think you would still need a way to keep
> > > RelMetadataQuery object during a rule call. If you choose to create new
> > > instance, you will have to pass it around while applying the rule. That
> > > actually complicates things a lot.
> > >
> > >
> > > > On Oct 17, 2019, at 12:49 AM, XING JIN 
> wrote:
> > > >
> > > > 1. RelMetadataQuery covers the functionality of MetadataFactory, why
> > > should
> > > > we keep/maintain both of them ? shall we just deprecate
> MetadataFactory.
> > > I
> > > > see MetadataFactory is rarely used in current code. Also I
> > > > think MetadataFactory is not good place to offering customized
> metadata,
> > > > which will make user confused for the difference between
> RelMetadataQuery
> > > > and MetadataFactory.
> > > >
> > > > > Customized RelMetadataQuery with code generated meta handler for
> > > > customized metadata, also can provide convenient way to get metadata.
> > > > It makes sense for me.
> > > >
> > > > 2. If the natural lifespan of a RelMetadataQuery is a RelOptCall,
> shall
> > > we
> > > > deprecate RelOptCluster#getMetadataQuery ? If a user wants to get the
> > > > metadata but without a RelOptCall, he/she will need to create a new
> > > > instance of RelMetadataQuery.
> > > >
> > > > Xiening Dai  于2019年10月17日周四 上午2:27写道：
> > > >
> > > > > I have seen both patterns in current code base. In most places, for
> > > > > example SubQueryRemoveRule, AggregateUnionTrasposeRule
> > > > > SortJoinTransposeRule, etc., RelOptCluster.getMetadataQuery() is
> used.
> > > And
> > > > > there are a few other places where new RelMetadataQuery instance is
> > > > > created, which Haisheng attempts to fix.
> > > > >
> > > > > Currently RelOptCluster.invalidateMetadataQuery() is called at the
> end
> > > of
> > > > > RelOptRuleCall.transformTo(). So the lifespan of RelMetadataQuery
> is
> > > > > guaranteed to be within a RelOptCall. I think Haisheng’s fix is
> safe.
> > > > >
> > > > >
> > > > > > On Oct 16, 2019, at 1:53 AM, Danny Chan 
> wrote:
> > > > > >
> > > > > > This is the reason I was struggling for the discussion.
> > > > > >
> > > > > > Best,
> > > > > > Danny Chan
> > > > > > 在 2019年10月16日 +0800 AM11:23，dev@calcite.apache.org，写道：
> > > > > > >
> > > > > > > RelMetadataQuery
> > > > >
> > > > >
> > >
> > >
>

Re: [DISCUSS] State of the project 2019

[XING JIN]

Thanks a lot for your work Francis and +1 for Stamatis to be the next PMC
chair ~ ~

Calcite is a great project and it offers as a strong SQL engine to help
users to build data systems.
My group uses Calcite to accelerate SQLs from multiple data engines by
heuristic based, cost based and materialization based optimization.
As a developer, I really benefit a lot from Calcite and the help from this
great community.

Calcite is not well known ? Yes, it might be less heard by the end users
(who focus on data analysis), but most of data engine developers (AFAIK)
know that Calcite is always a good choice when they
want to build a strong SQL engine for their data systems. This is the charm
of Calcite.

Finally, I agree with Muhammad Gelbana to strenghten documentation for
Calcite.

Best,
Jin

Alessandro Solimando  于2019年10月24日周四
下午4:41写道：

> Hello,
> I have the impression too that Calcite is lesser known than other Apache
> projects, but this is not that surprising to me given the nature of the
> project itself, which requires in-depth knowledge of a rather complex
> topic, which makes it is less prone to a casual use and plug adoption.
> But IMO this is a point of strength and not a weakness, and it's what makes
> Calcite so fascinating! This been said, spreading the word never hurts.
>
> I also agree that the community is much more active now than in the past
> two years, the effort has always been high but now there are more people
> involved, which makes it easier.
>
> On my side I mostly follow the "high-level" discussions more than the
> individual tickets. Such discussions are constructive, pleasant to read and
> instructive.
>
> A big plus also for the high-quality research articles directly stemming
> from the efforts of this community.
>
> Last but not least, Stamatis would make a great PMC chair!
>
> I take the chance to thank you all for the great deal of effort in keeping
> the project alive at such high standards.
>
> Best regards,
> Alessandro
>
>
>
> On Wed, 23 Oct 2019 at 17:00, Ruben Q L  wrote:
>
> > Hello everyone,
> >
> > First of all, thanks for your work, Francis.
> >
> > While I am relatively new in this community I must admit that I totally
> > agree with Danny Chan when he says that we had some *"interesting
> > discussions on the mailing list and we did have some valuable
> > conclusion(like the Join expression rework, the trait sets propagation,
> the
> > metadata etc.)"*. I think we must continue in this direction and keep on
> > our good work together.
> >
> > Apart from that, I agree with Muhammad Gelbana too: ideally we could have
> > more technical documentation of the main modules; but I am also aware
> that
> > it can be very time consuming to generate such documentation, and also to
> > maintain it, and we could easily have out-of-date material on our hands.
> > So, I am not sure how we could proceed.
> > Despite being a great project, I also have the impression that Calcite is
> > not so well-known, and we need to "spread the word" to reach a bigger
> > number of users.
> >
> > I cannot compare with previous years, but I what I have seen so far is a
> > healthy community, with many people collaborating and willing to help
> > beginner (and advanced) users.
> >
> > Finally, I think Stamatis is a great choice for PMC chair :)
> >
> > Best,
> > Ruben Quesada Lopez
> >
> >
> >
> > Le mer. 23 oct. 2019 à 15:09, Michael Mior  a écrit :
> >
> > > Thanks for the summary Francis and +1 for Stamatis as the next PMC
> chair
> > :)
> > >
> > > --
> > > Michael Mior
> > > mm...@apache.org
> > >
> > > Le mer. 23 oct. 2019 à 02:36, Stamatis Zampetakis 
> a
> > > écrit :
> > > >
> > > > It's been a great year so far. Thanks for your great work Francis,
> you
> > > did
> > > > an excellent job!
> > > >
> > > > I also like a lot the way that the mailing list works and the fact
> that
> > > we
> > > > are very active. Apart from helping each other it also centralizes a
> > lot
> > > of
> > > > information, where people may find an answer to their problem quite
> > > easily.
> > > > When I first started to learn about Calcite, the list and the
> archives
> > > were
> > > > indispensable.
> > > >
> > > > It is also very positive that we don't encounter very often
> regressions
> > > > (and most of those encountered were minor), which shows that the
> > quality
> > > > standards of the project are really high and reviewers are really
> > paying
> > > > attention on what is going in. I believe a lot in the motto of
> quality
> > > over
> > > > quantity.
> > > >
> > > > Although we are managing PRs a bit better than the previous years, we
> > > have
> > > > to improve more. I had in my mind that if people assigned to
> themselves
> > > the
> > > > review of the JIRA/PR the feeling of responsibility would make things
> > > > advance faster. I tried applying this to myself but it didn't really
> > work
> > > > much better since there is certainly a lack of time. Other than lack
> of
> > > > time 

Re: [DISCUSSION] Extension of Metadata Query

[XING JIN]

BTW, I think one JIRA number discussed in the thread would be
https://issues.apache.org/jira/browse/CALCITE-2855 not CALCITE-2885

Best,
Jin

XING JIN  于2019年10月17日周四 下午3:49写道：

> 1. RelMetadataQuery covers the functionality of MetadataFactory, why
> should we keep/maintain both of them ? shall we just
> deprecate MetadataFactory. I see MetadataFactory is rarely used in current
> code. Also I think MetadataFactory is not good place to offering customized
> metadata, which will make user confused for the difference
> between RelMetadataQuery and MetadataFactory.
>
> > Customized RelMetadataQuery with code generated meta handler for
> customized metadata, also can provide convenient way to get metadata.
> It makes sense for me.
>
> 2. If the natural lifespan of a RelMetadataQuery is a RelOptCall, shall we
> deprecate RelOptCluster#getMetadataQuery ? If a user wants to get the
> metadata but without a RelOptCall, he/she will need to create a new
> instance of RelMetadataQuery.
>
> Xiening Dai  于2019年10月17日周四 上午2:27写道：
>
>> I have seen both patterns in current code base. In most places, for
>> example SubQueryRemoveRule, AggregateUnionTrasposeRule
>> SortJoinTransposeRule, etc., RelOptCluster.getMetadataQuery() is used. And
>> there are a few other places where new RelMetadataQuery instance is
>> created, which Haisheng attempts to fix.
>>
>> Currently RelOptCluster.invalidateMetadataQuery() is called at the end of
>> RelOptRuleCall.transformTo(). So the lifespan of RelMetadataQuery is
>> guaranteed to be within a RelOptCall. I think Haisheng’s fix is safe.
>>
>>
>> > On Oct 16, 2019, at 1:53 AM, Danny Chan  wrote:
>> >
>> > This is the reason I was struggling for the discussion.
>> >
>> > Best,
>> > Danny Chan
>> > 在 2019年10月16日 +0800 AM11:23，dev@calcite.apache.org，写道：
>> >>
>> >> RelMetadataQuery
>>
>>

Re: [DISCUSSION] Extension of Metadata Query

[XING JIN]

1. RelMetadataQuery covers the functionality of MetadataFactory, why should
we keep/maintain both of them ? shall we just deprecate MetadataFactory. I
see MetadataFactory is rarely used in current code. Also I
think MetadataFactory is not good place to offering customized metadata,
which will make user confused for the difference between RelMetadataQuery
and MetadataFactory.

> Customized RelMetadataQuery with code generated meta handler for
customized metadata, also can provide convenient way to get metadata.
It makes sense for me.

2. If the natural lifespan of a RelMetadataQuery is a RelOptCall, shall we
deprecate RelOptCluster#getMetadataQuery ? If a user wants to get the
metadata but without a RelOptCall, he/she will need to create a new
instance of RelMetadataQuery.

Xiening Dai  于2019年10月17日周四 上午2:27写道：

> I have seen both patterns in current code base. In most places, for
> example SubQueryRemoveRule, AggregateUnionTrasposeRule
> SortJoinTransposeRule, etc., RelOptCluster.getMetadataQuery() is used. And
> there are a few other places where new RelMetadataQuery instance is
> created, which Haisheng attempts to fix.
>
> Currently RelOptCluster.invalidateMetadataQuery() is called at the end of
> RelOptRuleCall.transformTo(). So the lifespan of RelMetadataQuery is
> guaranteed to be within a RelOptCall. I think Haisheng’s fix is safe.
>
>
> > On Oct 16, 2019, at 1:53 AM, Danny Chan  wrote:
> >
> > This is the reason I was struggling for the discussion.
> >
> > Best,
> > Danny Chan
> > 在 2019年10月16日 +0800 AM11:23，dev@calcite.apache.org，写道：
> >>
> >> RelMetadataQuery
>
>

Re: [QUESTION] Pushing up evaluations from LogicalProjects

[XING JIN]

Hi Rommel ~
I'm very happy you tried the patch ~
Well the stacktrace you provided  seems outdated ~
I updated the the code several hours ago. The current commit id is
05d0e2d9fbadec060e54c622824b3725df36aab0
Could you please try the case again and send me the exception and related
information ?
Or you can just reply on https://github.com/apache/calcite/pull/1500 or
https://issues.apache.org/jira/browse/CALCITE-3405
I will try to fix ASAP

Best,
Jin


Rommel Quintanilla  于2019年10月15日周二 下午7:22写道：

> Hi Jin, your contribution is awesome! thanks.
>
> I tested it and works like a charm in most cases. Definitely clearer than
> the approach that I was trying. However, on the application I'm working on,
> I found a weird issue that I wasn't able to reproduce as a unit test on the
> calcite-core project. Btw, I'm using your changes after Haisheng's revision.
>
> So, it is as follows, I have this simple query: select l_extendedprice-1
> from lineitem
> where the type of l_extendedprice is DOUBLE. Before applying
> ProjectTableScanRule the logical plan is:
>
> LogicalProject(EXPR$0=[-($5, 1)])
>   LogicalTableScan(table=[[main, lineitem]])
>
> So far all is fine, but when the ProjectTableScanRule was applied I got:
>
> java.lang.AssertionError: type mismatch:
> ref:
> DOUBLE
> input:
> BIGINT
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:2000)
> at
> org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:125)
> at
> org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:57)
> at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
> at org.apache.calcite.rex.RexChecker.visitCall(RexChecker.java:140)
> at org.apache.calcite.rex.RexChecker.visitCall(RexChecker.java:57)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:191)
> at org.apache.calcite.rel.core.Project.isValid(Project.java:192)
> at org.apache.calcite.rel.core.Project.(Project.java:83)
> at
> org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:62)
> at
> org.apache.calcite.rel.logical.LogicalProject.create(LogicalProject.java:112)
> at
> org.apache.calcite.rel.logical.LogicalProject.create(LogicalProject.java:100)
> at
> org.apache.calcite.rel.core.RelFactories$ProjectFactoryImpl.createProject(RelFactories.java:158)
> at
> org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:1414)
> at
> org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:1252)
> at
> org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:1241)
> at
> com.blazingdb.calcite.rules.ProjectTableScanRule.apply(ProjectTableScanRule.java:147)
> at
> com.blazingdb.calcite.rules.ProjectTableScanRule$1.onMatch(ProjectTableScanRule.java:65)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:319)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:560)
> at
> org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:374)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:436)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:256)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:215)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:202)
> at
> com.blazingdb.calcite.application.RelationalAlgebraGenerator.getOptimizedRelationalAlgebra(RelationalAlgebraGenerator.java:202)
> at
> com.blazingdb.calcite.catalog.BlazingRulesTest.generateSQLTest(BlazingRulesTest.java:274)
>
> Any idea why could this be happening?
>
> Thank you in advance.
>
> On 2019/10/12 09:12:20, XING JIN  wrote:
> > Hi Stamatis,
> > In current code, BindableTableScan is only created by rules of
> > ProjectTableScanRule and FilterTableScanRule. I think it's better to keep
> > as it is?
> > I made a PR for CALCITE-3405  --
> https://github.com/apache/calcite/pull/1500
> >
> > The idea of the PR is quite straightforward:
> > 1. Analyze the parent Project -- collect all the needed fields;
> > 2. Column pruning by pushing down the needed fields to BindableTableScan;
> > 3. Adjust RexInputRefs in parent Project
> >
> > @Haisheng @Stamatis It would be great if you can give a review when you
> > have time ~~~ Thanks a lot !
> >
> > Best,
> > Jin
> >
> >
> > Stamati

Re: lex parameter does not get taken into account for views

[XING JIN]

Filed a JIRA: https://issues.apache.org/jira/browse/CALCITE-3549

Danny Chan  于2019年12月2日周一 上午9:06写道：

> Dear Erin ~
>
> You are right, Calcite now always hard code the parser config for JDBC
> query[1], that means, Lex config for view expanding is not supported yet,
> can you log a issue and probably fire a patch to support that?
>
> [1]
> https://github.com/apache/calcite/blob/ab136b5f76a4cb951e847fcba6b414c5e80dbbe6/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L385
>
> Best,
> Danny Chan
> 在 2019年12月2日 +0800 AM4:19，Erin Drummond ，写道：
> > Hello,
> >
> > I am currently attempting to test out the views feature of Calcite.
> >
> > I am using a connection string like so:
> > jdbc:calcite:lex=JAVA;model=inline:
> >
> > I have created a JdbcSchema that is attached to an actual database - lets
> > call it "t1". I can successfully run the query "select * from t1.table
> > limit 10"
> >
> > I then created another schema, defined like so:
> > {
> > "name": "viewtest",
> > "tables": [
> > { "name": "test_view", "type": "view", "sql": "select * from
> > t1.table limit 10" }
> > ]
> > }
> >
> > The idea is to set up connections to a bunch of physical data sources and
> > then have a "views" schema that just contains views over them.
> >
> > However, when attempting to run the query "select * from
> > viewtest.test_view", I get the following exception:
> >
> > StatementCallback; uncategorized SQLException for SQL [select * from
> > viewtest.test_view]; SQL state [null]; error code [0];
> > Error while executing SQL \"select * from viewtest.test_view\": From line
> > 1, column 15 to line 1, column 32: Object 'T1' not found; did you mean
> 't1'?
> >
> > I can clearly see that lex=JAVA is not being used for the view and the
> > default lex=ORACLE is being used. If I change the view SQL to 'select *
> > from "t1".table limit 10' it works - but I don't want to have to quote
> the
> > identifiers oracle style, I would like the lex=JAVA that I specified on
> the
> > connection string to flow down to views as well.
> >
> > What am I missing here?
> >
> > Cheers,
> > Erin
>

Re: [ANNOUNCE] Haisheng Yuan joins Calcite PMC

[XING JIN]

Congratulations Haisheng ~
You well deserved !

Kevin Risden  于2019年11月12日周二 上午3:13写道：

> Congrats and welcome!
>
> Kevin Risden
>
>
> On Mon, Nov 11, 2019 at 2:10 PM Chunhui Shi  wrote:
>
> > Congratulations!
> >
> > On Mon, Nov 11, 2019 at 10:09 AM Jinfeng Ni  wrote:
> >
> > > Congratulations!
> > >
> > >
> > > On Tue, Nov 12, 2019 at 1:23 AM Rui Wang  wrote:
> > > >
> > > > Congrats HaiSheng!
> > > >
> > > >
> > > > -Rui
> > > >
> > > > On Mon, Nov 11, 2019 at 8:05 AM Stamatis Zampetakis <
> zabe...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Congrats Haisheng!
> > > > >
> > > > > Reviews, code contributions, design discussions, helping users, and
> > > many
> > > > > more things for improving the project.
> > > > >
> > > > > Personally, I also learn a lot from our interactions.
> > > > >
> > > > > All these are much appreciated; keep it up!!
> > > > >
> > > > > Best,
> > > > > Stamatis
> > > > >
> > > > > On Mon, Nov 11, 2019, 4:17 PM Michael Mior 
> wrote:
> > > > >
> > > > > > Welcome and congratulations HaiSheng!
> > > > > > --
> > > > > > Michael Mior
> > > > > > mm...@apache.org
> > > > > >
> > > > > > Le dim. 10 nov. 2019 à 22:45, Francis Chuang
> > > > > >  a écrit :
> > > > > > >
> > > > > > > I'm pleased to announce that Haisheng has accepted an
> invitation
> > to
> > > > > > > join the Calcite PMC. Haisheng has been a consistent and
> helpful
> > > > > > > figure in the Calcite community for which we are very grateful.
> > We
> > > > > > > look forward to the continued contributions and support.
> > > > > > >
> > > > > > > Please join me in congratulating Haisheng!
> > > > > > >
> > > > > > > - Francis (on behalf of the Calcite PMC)
> > > > > >
> > > > >
> > >
> >
>

Re: Adding RelOptMaterializations to a planner

[XING JIN]

Hi Shubham,

In my understanding, same RelOptPlanner is the way to go, as one
VolcanoPlanner#registerMaterializations calls
RelOptMaterializations#useMaterializedViews and picks the best algebra
expression tree.
I'd suggest to create all the materializations into a
VolcanoPlanner and then "findBestExp".
Please note that there are two kinds of strategies for materialized view
optimization:
1. Substitution based materialized view optimization
2. Rewriting using plan structural information

For the first one, only RelOptMaterializations#useMaterializedViews is
enough, you can even don't rely on VolcanoPlanner, i.e. call
RelOptMaterializations#useMaterializedViews explicitly and pick the best
one by yourself. But for the second one, you need to to rely on
VolcanoPlanner and register corresponding rules from
AbstractMaterializedViewRule. The second one tend to be smarter but only
supports SPJA pattern.

In short, when you enable config of "materializationEnabled" for connection
property, both of the two strategies above are enabled.

In addition, I'd suggest to do canonicalization before materialized view
optimization, which helps a lot for materialized view matching.

I'm also doing some work for materialized view optimization. It would be
great to have more discussion on this :)
https://issues.apache.org/jira/browse/CALCITE-3334
https://docs.google.com/document/d/1JpwGNFE3hw3yXb7W3-95-jXKClZC5UFPKbuhgYDuEu4/edit#heading=h.bmvjxz1h5evc

Best,
Jin

Re: [QUESTION] Pushing up evaluations from LogicalProjects

[XING JIN]

Filed a JIRA:
https://issues.apache.org/jira/browse/CALCITE-3405

Haisheng Yuan  于2019年10月12日周六 上午4:34写道：

> Yes, definitely.
>
> You can go through the project expression with InputFinder to find all the
> used columns, create a  logical project with those columns, and remap the
> top project with new column indexes.
>
> On the other hand, instead of creating a new intermidiate pogical project,
> we can also update ProjectTableScanRule to accept LogicalProject that is
> not a simple mapping, and do the same task I mentioned above.
>
> - Haisheng
>
> --
> 发件人：Rommel Quintanilla
> 日 期：2019年10月12日 03:15:31
> 收件人：
> 主 题：[QUESTION] Pushing up evaluations from LogicalProjects
>
> Hi, maybe you can help me.
> I have this portion from a larger logical plan:
> ..
> LogicalProject(l_orderkey=[$0], l_suppkey=[$2], *=[*($5, -(1,
> $6))])
> LogicalTableScan(table=[[main, lineitem]])
> ..
>
> Because the LogicalProject above contains an evaluation, the
> ProjectTableScanRule can't convert it to a BindableTableScan.
>
> I wonder if somehow the evaluation could be pushed up more or less like
> this:
> ..
>  LogicalProject(l_orderkey=[$0], l_suppkey=[$1], *=[*($2, -(1, $3))])
>  LogicalProject(l_orderkey=[$0], l_suppkey=[$2], l_extendedprice=[$5],
> l_discount=[$6]])
>  LogicalTableScan(table=[[main, lineitem]])
> ..
>
> Regards.
>

Re: [QUESTION] Pushing up evaluations from LogicalProjects

[XING JIN]

Hi Stamatis,
In current code, BindableTableScan is only created by rules of
ProjectTableScanRule and FilterTableScanRule. I think it's better to keep
as it is?
I made a PR for CALCITE-3405  -- https://github.com/apache/calcite/pull/1500

The idea of the PR is quite straightforward:
1. Analyze the parent Project -- collect all the needed fields;
2. Column pruning by pushing down the needed fields to BindableTableScan;
3. Adjust RexInputRefs in parent Project

@Haisheng @Stamatis It would be great if you can give a review when you
have time ~~~ Thanks a lot !

Best,
Jin


Stamatis Zampetakis  于2019年10月12日周六 下午3:00写道：

> Hi Rommel,
>
> I was hoping that this could be done at least by RelFieldTrimmer [1]. Are
> you using it already?
>
> Best,
> Stamatis
>
> [1]
>
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java
>
> On Sat, Oct 12, 2019 at 6:06 AM XING JIN  wrote:
>
> > Filed a JIRA:
> > https://issues.apache.org/jira/browse/CALCITE-3405
> >
> > Haisheng Yuan  于2019年10月12日周六 上午4:34写道：
> >
> > > Yes, definitely.
> > >
> > > You can go through the project expression with InputFinder to find all
> > the
> > > used columns, create a  logical project with those columns, and remap
> the
> > > top project with new column indexes.
> > >
> > > On the other hand, instead of creating a new intermidiate pogical
> > project,
> > > we can also update ProjectTableScanRule to accept LogicalProject that
> is
> > > not a simple mapping, and do the same task I mentioned above.
> > >
> > > - Haisheng
> > >
> > > --
> > > 发件人：Rommel Quintanilla
> > > 日 期：2019年10月12日 03:15:31
> > > 收件人：
> > > 主 题：[QUESTION] Pushing up evaluations from LogicalProjects
> > >
> > > Hi, maybe you can help me.
> > > I have this portion from a larger logical plan:
> > > ..
> > > LogicalProject(l_orderkey=[$0], l_suppkey=[$2], *=[*($5, -(1,
> > > $6))])
> > > LogicalTableScan(table=[[main, lineitem]])
> > > ..
> > >
> > > Because the LogicalProject above contains an evaluation, the
> > > ProjectTableScanRule can't convert it to a BindableTableScan.
> > >
> > > I wonder if somehow the evaluation could be pushed up more or less like
> > > this:
> > > ..
> > >  LogicalProject(l_orderkey=[$0], l_suppkey=[$1], *=[*($2, -(1, $3))])
> > >  LogicalProject(l_orderkey=[$0], l_suppkey=[$2], l_extendedprice=[$5],
> > > l_discount=[$6]])
> > >  LogicalTableScan(table=[[main, lineitem]])
> > > ..
> > >
> > > Regards.
> > >
> >
>

Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects

[XING JIN]

Sure we can ~
If we use BindableTableScanRule to derive a BindableTableScan from
ProjectableFilterableTable, that would happen during a stage of
optimization run by RelOptPlanner. But RelFieldTrimmer works right during
conversion of Sql to Rel.

Wang Yanlin <1989yanlinw...@163.com> 于2019年10月12日周六 下午5:56写道：

> Can we just use  Bindables.BINDABLE_TABLE_SCAN_RULE to translate  the
> table scan to BindableTableScan ?
>
>
>
> --
>
> Best,
> Wang Yanlin
>
>
>
> At 2019-10-12 17:12:20, "XING JIN"  wrote:
> >Hi Stamatis,
> >In current code, BindableTableScan is only created by rules of
> >ProjectTableScanRule and FilterTableScanRule. I think it's better to keep
> >as it is?
> >I made a PR for CALCITE-3405  --
> https://github.com/apache/calcite/pull/1500
> >
> >The idea of the PR is quite straightforward:
> >1. Analyze the parent Project -- collect all the needed fields;
> >2. Column pruning by pushing down the needed fields to BindableTableScan;
> >3. Adjust RexInputRefs in parent Project
> >
> >@Haisheng @Stamatis It would be great if you can give a review when you
> >have time ~~~ Thanks a lot !
> >
> >Best,
> >Jin
> >
> >
> >Stamatis Zampetakis  于2019年10月12日周六 下午3:00写道：
> >
> >> Hi Rommel,
> >>
> >> I was hoping that this could be done at least by RelFieldTrimmer [1].
> Are
> >> you using it already?
> >>
> >> Best,
> >> Stamatis
> >>
> >> [1]
> >>
> >>
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java
> >>
> >> On Sat, Oct 12, 2019 at 6:06 AM XING JIN 
> wrote:
> >>
> >> > Filed a JIRA:
> >> > https://issues.apache.org/jira/browse/CALCITE-3405
> >> >
> >> > Haisheng Yuan  于2019年10月12日周六 上午4:34写道：
> >> >
> >> > > Yes, definitely.
> >> > >
> >> > > You can go through the project expression with InputFinder to find
> all
> >> > the
> >> > > used columns, create a  logical project with those columns, and
> remap
> >> the
> >> > > top project with new column indexes.
> >> > >
> >> > > On the other hand, instead of creating a new intermidiate pogical
> >> > project,
> >> > > we can also update ProjectTableScanRule to accept LogicalProject
> that
> >> is
> >> > > not a simple mapping, and do the same task I mentioned above.
> >> > >
> >> > > - Haisheng
> >> > >
> >> > > --
> >> > > 发件人：Rommel Quintanilla
> >> > > 日 期：2019年10月12日 03:15:31
> >> > > 收件人：
> >> > > 主 题：[QUESTION] Pushing up evaluations from LogicalProjects
> >> > >
> >> > > Hi, maybe you can help me.
> >> > > I have this portion from a larger logical plan:
> >> > > ..
> >> > > LogicalProject(l_orderkey=[$0], l_suppkey=[$2], *=[*($5,
> -(1,
> >> > > $6))])
> >> > > LogicalTableScan(table=[[main, lineitem]])
> >> > > ..
> >> > >
> >> > > Because the LogicalProject above contains an evaluation, the
> >> > > ProjectTableScanRule can't convert it to a BindableTableScan.
> >> > >
> >> > > I wonder if somehow the evaluation could be pushed up more or less
> like
> >> > > this:
> >> > > ..
> >> > >  LogicalProject(l_orderkey=[$0], l_suppkey=[$1], *=[*($2, -(1,
> $3))])
> >> > >  LogicalProject(l_orderkey=[$0], l_suppkey=[$2],
> l_extendedprice=[$5],
> >> > > l_discount=[$6]])
> >> > >  LogicalTableScan(table=[[main, lineitem]])
> >> > > ..
> >> > >
> >> > > Regards.
> >> > >
> >> >
> >>
>

Re: Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects

[XING JIN]

Hi, Stamatis, Danny~

Thanks for explain ~

> "The consumer in the case of P1 is the project which only needs $0, $2,
$5,
$6 so the trimmer could slim down the scan by projecting only these fields."

I think RelFieldTrimmer is already doing this by [1].
But why the final BindableTableScan is not pruned ? I think the answer is
that RelFieldTrimmer merges projects when trimming fields for Project:
[2][3]

> "RelFieldTrimmer can be used to transform the plan P1 to
plan P2 and then ProjectTableScanRule can be used to introduce the
BindableTableScan."

If we wanna go by this approach, shall we avoid the project merging as
mentioned in RelFieldTrimmer [2][3] ?

Well, as an independent functionality for column pruning, I still suggest
ProjectTableScanRule have the the ability to handle such complex scenarios.

How do you think?

Best,
Jin



[1]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java#L1054
[2]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java#L416

[3]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1327

Danny Chan  于2019年10月14日周一 上午9:42写道：

> +1, RelFieldTrimmer is the role to trim the unused fields.
>
> Best,
> Danny Chan
> 在 2019年10月13日 +0800 AM6:25，dev@calcite.apache.org，写道：
> >
> > RelFieldTrimmer
>

Re: Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects

[XING JIN]

Also regarding that Projects merging is common in Calcite optimization
rules, we should always remember to avoid merging for cases like the one
given by Rommel.
I think that would be hard.

Best,
Jin

XING JIN  于2019年10月14日周一 上午11:51写道：

> Hi, Stamatis, Danny~
>
> Thanks for explain ~
>
> > "The consumer in the case of P1 is the project which only needs $0, $2,
> $5,
> $6 so the trimmer could slim down the scan by projecting only these
> fields."
>
> I think RelFieldTrimmer is already doing this by [1].
> But why the final BindableTableScan is not pruned ? I think the answer is
> that RelFieldTrimmer merges projects when trimming fields for Project:
> [2][3]
>
> > "RelFieldTrimmer can be used to transform the plan P1 to
> plan P2 and then ProjectTableScanRule can be used to introduce the
> BindableTableScan."
>
> If we wanna go by this approach, shall we avoid the project merging as
> mentioned in RelFieldTrimmer [2][3] ?
>
> Well, as an independent functionality for column pruning, I still suggest
> ProjectTableScanRule have the the ability to handle such complex scenarios.
>
> How do you think?
>
> Best,
> Jin
>
>
>
> [1]
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java#L1054
> [2]
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java#L416
>
> [3]
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1327
>
> Danny Chan  于2019年10月14日周一 上午9:42写道：
>
>> +1, RelFieldTrimmer is the role to trim the unused fields.
>>
>> Best,
>> Danny Chan
>> 在 2019年10月13日 +0800 AM6:25，dev@calcite.apache.org，写道：
>> >
>> > RelFieldTrimmer
>>
>

Re: Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects

[XING JIN]

Yes, that's how testPushNonSimpleMappingProject [1] works

[1]
https://github.com/apache/calcite/pull/1500/files#diff-3c834a32d46b821b5241e132f2ae6bfaR324

Danny Chan  于2019年10月14日周一 下午3:36写道：

> > But why the final BindableTableScan is not pruned ?
>
> The RelFieldTrimmer default is turned off, you should open it explicitly.
>
> Best,
> Danny Chan
> 在 2019年10月14日 +0800 AM11:51，dev@calcite.apache.org，写道：
> >
> > BINDABLE_TABLE_SCAN_RULE
>

Re: Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects

[XING JIN]

I mean, testPushNonSimpleMappingProject [1] runs with RelFieldTrimmer
enabled, which is done by [2]

[1]
https://github.com/apache/calcite/pull/1500/files#diff-3c834a32d46b821b5241e132f2ae6bfaR324
[2]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L249

XING JIN  于2019年10月14日周一 下午4:03写道：

> Yes, that's how testPushNonSimpleMappingProject [1] works
>
> [1]
> https://github.com/apache/calcite/pull/1500/files#diff-3c834a32d46b821b5241e132f2ae6bfaR324
>
> Danny Chan  于2019年10月14日周一 下午3:36写道：
>
>> > But why the final BindableTableScan is not pruned ?
>>
>> The RelFieldTrimmer default is turned off, you should open it explicitly.
>>
>> Best,
>> Danny Chan
>> 在 2019年10月14日 +0800 AM11:51，dev@calcite.apache.org，写道：
>> >
>> > BINDABLE_TABLE_SCAN_RULE
>>
>

Re: [DISCUSS] Move CalciteAssert and similar "test framework" classes to its own testlib module

[XING JIN]

Hi, Vladimir
> I suggest we do not publish testlib artifacts to Nexus (who needs
> Calcite-related assertions and things like that?)

+1 to have the module be published.
Some of my users write tests based on functionalities provided from Calcite
test framework.

Best,
Jin

Michael Mior  于2019年12月18日周三 上午7:47写道：

> Sounds good to me. Although I'd still like to see the module be
> published. I'm currently using it my notebooks project
> (https://github.com/michaelmior/calcite-notebooks) because some of the
> test code removes boilerplate when writing examples.
> --
> Michael Mior
> mm...@apache.org
>
> Le mar. 17 déc. 2019 à 15:49, Vladimir Sitnikov
>  a écrit :
> >
> > Hi,
> >
> > Currently
> > org.apache.calcite.test.CalciteAssert, org.apache.calcite.util.TestUtil,
> > and probably other classes are hard to find.
> > I suggest we create a testlib module (or something like that), that would
> > contain "test framework" code.
> >
> > That would make test framework easier to discover (one could open the
> full
> > set of packages in the testlib),
> > and it would make "autocomplete" better (especially for adapters).
> >
> > I suggest we do not publish testlib artifacts to Nexus (who needs
> > Calcite-related assertions and things like that?)
> >
> > Currently testImplementation(project(":core", "testClasses")) is used in
> > lots of places, and it causes "core"
> > test clases appear in autocomplete menu for adapters.
> > For example, typing "new Sql..." in GeodeAssertions suggests
> > SqlAdvisorJdbcTest which is valid, yet it is weird.
> >
> > What do you think?
> >
> > Just in case you did not know: file movement in Git keeps history (even
> in
> > case there are slight modifications).
> > However, the key idea here is to keep files in their current packages,
> but
> > move them into teslib module.
> >
> > Vladimir
>

Re: Re: [ANNOUNCE] New Calcite PMC chair: Stamatis Zampetakis

[XING JIN]

Congratulations Stamatis!

-Jin

Chunwei Lei  于2019年12月19日周四 上午10:33写道：

> Congratulations Stamatis!
>
>
> Best,
> Chunwei
>
>
> On Thu, Dec 19, 2019 at 9:36 AM Danny Chan  wrote:
>
> > Congratulations Stamatis!
> >
> > Best,
> > Danny Chan
> > 在 2019年12月19日 +0800 AM7:37，dev@calcite.apache.org，写道：
> > >
> > > Congratulations Stamatis!
> >
>

Re: Cannot parse query that contains colons for casting in postgres sql

[XING JIN]

Hi, Rick ~
Babel parser supports casting by double colon [1]

[1] https://issues.apache.org/jira/browse/CALCITE-2843

-Jin

Rick Liao  于2020年1月9日周四 上午6:05写道：

> Hello all,
>
> I'm trying to parse a postgres sql query that contains :: for casting. The
> parser fails when it reaches the colon. I'm using version 1.21.0. This is
> what I'm running (with a simple query to showcase the error):
>
> SqlParser.Config parserConfig =
>
> SqlParser.configBuilder().setConformance(SqlConformanceEnum.LENIENT).build();
> SqlParser parser1 = SqlParser.create("SELECT 'test'::text t from
> countries", parserConfig);
> SqlNode parsedQuery1 = parser1.parseQuery();
>
> The error:
>
> org.apache.calcite.sql.parser.SqlParseException: Encountered ":" at line 1,
> column 14.
>
> Was expecting one of:
>
> 
>
> "AS" ...
>
> "EXCEPT" ...
>
> "FETCH" ...
>
> "FROM" ...
>
> "INTERSECT" ...
>
> "LIMIT" ...
>
> "OFFSET" ...
>
> "ORDER" ...
>
> "MINUS" ...
>
> "UESCAPE" ...
>
> "UNION" ...
>
>  ...
>
> "," ...
>
>  ...
>
>  ...
>
>  ...
>
>  ...
>
>  ...
>
> "." ...
>
> "NOT" ...
>
> "IN" ...
>
> "<" ...
>
> "<=" ...
>
> ">" ...
>
> ">=" ...
>
> "=" ...
>
> "<>" ...
>
> "!=" ...
>
> "BETWEEN" ...
>
> "LIKE" ...
>
> "SIMILAR" ...
>
> "+" ...
>
> "-" ...
>
> "*" ...
>
> "/" ...
>
> "%" ...
>
> "||" ...
>
> "AND" ...
>
> "OR" ...
>
> "IS" ...
>
> "MEMBER" ...
>
> "SUBMULTISET" ...
>
> "CONTAINS" ...
>
> "OVERLAPS" ...
>
> "EQUALS" ...
>
> "PRECEDES" ...
>
> "SUCCEEDS" ...
>
> "MULTISET" ...
>
> "[" ...
>
>
>
> at
>
> org.apache.calcite.sql.parser.impl.SqlParserImpl.convertException(SqlParserImpl.java:343)
>
> at
>
> org.apache.calcite.sql.parser.impl.SqlParserImpl.normalizeException(SqlParserImpl.java:142)
>
> at
> org.apache.calcite.sql.parser.SqlParser.handleException(SqlParser.java:147)
>
> at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:162)
>
> Perhaps, I have just not found the right conformance for postgres sql.
>
> Thanks for your help!
>
> Rick
>

Re: [DISCUSS] CALCITE-3661, CALCITE-3665, MaterializationTest vs HR schema statistics

[XING JIN]

Hi, Vladimir ~

In ReflectiveSchema, Statistics of FieldTable is given as UNKNOWN[1][2].
When reading a table's row count, if no statistics given, a default value
of 100 will be returned [3] -- this is relatively a bigger value compared
with the fields defined in HRFKUKSchema.
When a materialized view gets matched, view-sql is executed and the values
are wrapped in an ArrayTable and accurate row count is given [4].
So I'm not sure when a materialized view containing JOIN gets matched but
cannot help reduce cost of the plan.

HRFKUKSchema is only used in MaterializationTest. There's no existing test
checking content of the query result. Most of them checks whether same
results are returned no matter if materialized view is used or not. If we
add rows to existing emps table, how can tests be invalidated ?

Best,
Jin

[1]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/java/ReflectiveSchema.java#L369
[2]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/schema/Statistics.java#L40
[3]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/RelOptTableImpl.java#L239
[4]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/clone/ArrayTable.java#L82

Vladimir Sitnikov  于2020年1月4日周六 上午4:21写道：

> Hi,
>
> It looks like MaterializationTest heavily relies on inaccurate statistics
> for hr.emps and hr.depts tables.
>
> I was trying to improve statistic estimation for better join planning (see
> https://github.com/apache/calcite/pull/1712 ),
> and it looks like better estimates open the eyes of the optimizer, and now
> it realizes it does not really need to use materialized view
> for 4-row long table.
>
> In other words, the cost of the table access is more-or-less the same as
> the cost of the materialized view access.
>
> It looks like the way to go here is to add hr_with_extra_rows scheme so it
> contains the same emps and depts tables, but it should
> have bigger tables.
> Adding rows to the existing emps table is not an option because it would
> invalidate lots of tests.
>
> Does anybody have better ideas?
>
> Vladimir
>

Re: How to get the used columns (categorized by tables) from a SQL

[XING JIN]

You can use SqlValidator.getFieldOrigins for a SqlNode, or use
RelMetadataQuery.getColumnOrigins for a RelNode.

- Jin


Julian Hyde  于2020年4月17日周五 上午7:29写道：

> At the RelNode level, you can use RelMetadataQuery.getColumnOrigins.
>
> But I’m sure that there’s something available at the SqlNode level, after
> validation. PreparedExplain.getFieldOrigins() is what the JDBC driver uses
> to provide ResultSetMetadata.getColumnName(int) etc.
>
> Julian
>
>
>
> > On Apr 16, 2020, at 4:17 PM, Feng Zhu  wrote:
> >
> > You can customize a RelVisitor to visit the RelNode, and collect the
> > columns from TableScan's rowtype.
> >
> > 王驰  于2020年4月16日周四 下午11:23写道：
> >
> >> Hi folks,
> >>
> >>
> >> We're using CalCite in our project for SQL parsing and validating. We
> have
> >> a question: is there any way to get all used columns from a SQL?
> >>
> >>
> >> Consider the following case:
> >>
> >> we have two tables `user` and `user_region`. And the SQL is like
> >>
> >>
> >> ```
> >>
> >> SELECT
> >>
> >>id, name, age, country, province, city
> >>
> >> FROM
> >>
> >>user
> >>
> >>INNER JOIN user_region ON user.id = user_region.user_id
> >>
> >> WHERE age > 18;
> >>
> >> ```
> >>
> >>
> >> The result will be a Map with two keys:
> >>
> >>
> >>   - 'user' --> List('id', 'name', 'age')
> >>   - 'user_region' --> List('country', 'province', 'city', 'user_id')
> >>
> >>
> >> ==
> >>
> >> I've tried with SqlValidator (along with SqlValidatorScope,
> >> SqlValidatorNamespace) but found no easy ways to do this.
> >> I also tried to replay the validate process in our code but it seems
> >> impossible since most classes used are package private.
> >>
> >>
> >> Could you please give us some suggestions? Thanks!
> >>
> >>
>
>

Re: The Travis CI build failure

[XING JIN]

I have force pushed several times, but it keeps failing.

Chunwei Lei  于2020年4月20日周一 上午10:49写道：

> I am a little confused too. Sometimes the traivs ci build of master fails
> when I merge
> the pr which passes all tests.
>
>
>
> Best,
> Chunwei
>
>
> On Mon, Apr 20, 2020 at 10:33 AM XING JIN  wrote:
>
> > In PR https://github.com/apache/calcite/pull/1926 and
> > https://github.com/apache/calcite/pull/1925, The Travis CI build keeps
> > failing with below messages. The exception has nothing to do with the
> > change in PR. There seems to be something wrong with the building
> > environment.
> >
> > ...
> >
> >
> /home/travis/build/apache/calcite/spark/src/main/java/org/apache/calcite/adapter/spark/HttpServer.java:42:
> > error: cannot access java.lang
> > class HttpServer {
> > ^
> >   zip END header not found
> >
> >
> /home/travis/build/apache/calcite/spark/src/main/java/org/apache/calcite/adapter/spark/HttpServer.java:43:
> > error: cannot find symbol
> >   private static String localIpAddress;
> >  ^
> >   symbol:   class String
> >   location: class HttpServer
> >
> >
> /home/travis/build/apache/calcite/spark/src/main/java/org/apache/calcite/adapter/spark/HttpServer.java:45:
> > error: cannot find symbol
> >   private final File resourceBase;
> > ^
> >   symbol:   class File
> >   location: class HttpServer
> > ...
> >
> > Jin
> >
>

The Travis CI build failure

[XING JIN]

In PR https://github.com/apache/calcite/pull/1926 and
https://github.com/apache/calcite/pull/1925, The Travis CI build keeps
failing with below messages. The exception has nothing to do with the
change in PR. There seems to be something wrong with the building
environment.

...
/home/travis/build/apache/calcite/spark/src/main/java/org/apache/calcite/adapter/spark/HttpServer.java:42:
error: cannot access java.lang
class HttpServer {
^
  zip END header not found
/home/travis/build/apache/calcite/spark/src/main/java/org/apache/calcite/adapter/spark/HttpServer.java:43:
error: cannot find symbol
  private static String localIpAddress;
 ^
  symbol:   class String
  location: class HttpServer
/home/travis/build/apache/calcite/spark/src/main/java/org/apache/calcite/adapter/spark/HttpServer.java:45:
error: cannot find symbol
  private final File resourceBase;
^
  symbol:   class File
  location: class HttpServer
...

Jin

Re: How to give type in case of type mismatch while doing union

[XING JIN]

Hi, Anjali ~
Are you doing the UNION by Sql ? If so, can you give the Sql content ?
Are you doing the UNION on RelNodes ?, If so, you need to do type CAST.

Jin

Anjali Shrishrimal  于2020年4月27日周一
下午4:25写道：

> Hi,
>
> While doing union of 2 RelNodes with different types, I am getting NPE. (I
> am using calcite 1.21.0)
> java.lang.NullPointerException: at index 0
> at
> com.google.common.collect.ObjectArrays.checkElementNotNull(ObjectArrays.java:225)
> at
> com.google.common.collect.ObjectArrays.checkElementsNotNull(ObjectArrays.java:215)
> at
> com.google.common.collect.ObjectArrays.checkElementsNotNull(ObjectArrays.java:209)
> at
> com.google.common.collect.ImmutableList.construct(ImmutableList.java:346)
> at
> com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:258)
> at
> org.apache.calcite.rel.type.RelDataTypeFactoryImpl.canonize(RelDataTypeFactoryImpl.java:373)
> at
> org.apache.calcite.rel.type.RelDataTypeFactoryImpl.createStructType(RelDataTypeFactoryImpl.java:155)
> at
> org.apache.calcite.rel.type.RelDataTypeFactoryImpl.createStructType(RelDataTypeFactoryImpl.java:146)
> at
> org.apache.calcite.rel.type.RelDataTypeFactory$Builder.build(RelDataTypeFactory.java:569)
> at
> org.apache.calcite.rel.type.RelDataTypeFactoryImpl.leastRestrictiveStructuredType(RelDataTypeFactoryImpl.java:257)
> at
> org.apache.calcite.sql.type.SqlTypeFactoryImpl.leastRestrictiveSqlType(SqlTypeFactoryImpl.java:285)
> at
> org.apache.calcite.sql.type.SqlTypeFactoryImpl.leastRestrictive(SqlTypeFactoryImpl.java:156)
> at
> org.apache.calcite.rel.core.SetOp.deriveRowType(SetOp.java:107)
>
> If the column types (family types) are different, currently the derived
> type is null. Is there any way to control that?
> Where can I define the type in case of mismatch ?
>
>
> Thank you,
> Anjali Shrishrimal
>

Re: [ANNOUNCE] New committer: Vineet Garg

[XING JIN]

Congrats, Vineet!

Best,
Jin

Fan Liya  于2020年4月26日周日 上午11:45写道：

> Congratulations, Vineet!
>
> Best,
> Liya Fan
>
> On Sun, Apr 26, 2020 at 10:30 AM Aman Sinha  wrote:
>
> > Congratulations Vineet !
> >
> > -Aman
> >
> > On Sat, Apr 25, 2020 at 7:15 PM Feng Zhu  wrote:
> >
> > >  Congratulations, well deserved!
> > >
> > > best,
> > > Feng
> > >
> > > Chunwei Lei  于2020年4月26日周日 上午10:12写道：
> > >
> > > > Congrats, Vineet!
> > > >
> > > >
> > > > Best,
> > > > Chunwei
> > > >
> > > >
> > > > On Sun, Apr 26, 2020 at 8:24 AM Haisheng Yuan 
> > wrote:
> > > >
> > > > > Congrats, Vineet!
> > > > >
> > > > > On 2020/04/25 22:18:35, Forward Xu  wrote:
> > > > > > Congratulations
> > > > > >
> > > > > > best,
> > > > > > Forward
> > > > > >
> > > > > > Francis Chuang  于2020年4月26日周日
> 上午6:04写道：
> > > > > >
> > > > > > > Congrats, Vineet!
> > > > > > >
> > > > > > > On 26/04/2020 7:52 am, Stamatis Zampetakis wrote:
> > > > > > > > Apache Calcite's Project Management Committee (PMC) has
> invited
> > > > > Vineet
> > > > > > > > Garg to become a committer, and we are pleased to announce
> that
> > > he
> > > > > > > > has accepted.
> > > > > > > >
> > > > > > > > With the first code contribution in Calcite back in 2017,
> > Vineet
> > > is
> > > > > > > > definitely
> > > > > > > > not new to the project. Since then he has contributed many
> > > patches,
> > > > > > > > fixing and improving various modules of Calcite, notably
> things
> > > > > around
> > > > > > > > subqueries.
> > > > > > > >
> > > > > > > > Vineet, welcome, thank you for your contributions, and we
> look
> > > > > forward
> > > > > > > > your further interactions with the community! If you wish,
> > please
> > > > > feel
> > > > > > > > free to tell us more about yourself and what you are working
> > on.
> > > > > > > >
> > > > > > > > Stamatis (on behalf of the Apache Calcite PMC)
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Understanding annotations of SqlGroupingFunction

[XING JIN]

Hi Vineet ~
+1 on your analysis.
Checking below case in agg.iq. We can see that the behavior of GROUPING
function in Calcite is the same as Hive.

# GROUPING in SELECT clause of CUBE query
select deptno, job, count(*) as c, grouping(deptno) as d,
  grouping(job) j, grouping(deptno, job) as x
from "scott".emp
group by cube(deptno, job);
++---++---+---+---+
| DEPTNO | JOB   | C  | D | J | X |
++---++---+---+---+
| 10 | CLERK |  1 | 0 | 0 | 0 |
| 10 | MANAGER   |  1 | 0 | 0 | 0 |
| 10 | PRESIDENT |  1 | 0 | 0 | 0 |
| 10 |   |  3 | 0 | 1 | 1 |
| 20 | ANALYST   |  2 | 0 | 0 | 0 |
| 20 | CLERK |  2 | 0 | 0 | 0 |
| 20 | MANAGER   |  1 | 0 | 0 | 0 |
| 20 |   |  5 | 0 | 1 | 1 |
| 30 | CLERK |  1 | 0 | 0 | 0 |
| 30 | MANAGER   |  1 | 0 | 0 | 0 |
| 30 | SALESMAN  |  4 | 0 | 0 | 0 |
| 30 |   |  6 | 0 | 1 | 1 |
|| ANALYST   |  2 | 1 | 0 | 2 |
|| CLERK |  4 | 1 | 0 | 2 |
|| MANAGER   |  3 | 1 | 0 | 2 |
|| PRESIDENT |  1 | 1 | 0 | 2 |
|| SALESMAN  |  4 | 1 | 0 | 2 |
||   | 14 | 1 | 1 | 3 |
++---++---+---+---+

IMHO, we might rectify the doc of SqlGroupingFunction as below:

* 0 if both deptno and gender are being grouped,
* 1 if only deptno is being grouped,
* 2 if only gender is being groped,
* 3 if neither deptno nor gender are being grouped.


- Jin


Vineet G  于2020年4月22日周三 上午5:18写道：

> I expect that the user behavior for the GROUPING in both hive and calcite
> is same. It’s just the documentation which is a bit confusing.
> e.g.  comment line on grouping : if both deptno and gender are being
> grouped
>
> should really mean that the row which represents the grand total i.e
> without group by expression. will return 3.
>
> FYI Hive had its behavior fixed with
> https://issues.apache.org/jira/browse/HIVE-16102 <
> https://issues.apache.org/jira/browse/HIVE-16102> and currently uses
> GROUPING ID function.
>
> Vineet Garg
>
>
> > On Apr 21, 2020, at 10:42 AM, Julian Hyde 
> wrote:
> >
> > Suppose we have one row that represents the total for department 10, and
> another that represents the grand total of all departments. Which row would
> we say that department is “grouped” (in Calcite’s parlance) or “aggregated”
> in (Hive’s parlance)?
> >
> > I find the terms confusing. It’s possible that Calcite has them “wrong”.
> >
> > I would simply run a query like
> >
> >  SELECT deptno, job, GROUPING(deptno, job), COUNT(*)
> >  FROM emp
> >  GROUP BY CUBE (deptno, job)
> >
> > and see whether Hive and Calcite return the same result.
> >
> >> On Apr 20, 2020, at 6:58 PM, ZZY  wrote:
> >>
> >> Hi, Hyde:
> >> It's confused me that some annotations in
> >> Calcite(org.apache.calcite.sql.fun.SqlGroupingFunction.java) :
> >> /**
> >> * The {@code GROUPING} function.
> >> *
> >> * Accepts 1 or more arguments.
> >> * Example: {@code GROUPING(deptno, gender)} returns
> >> * 3 if both deptno and gender are being grouped,
> >> * 2 if only deptno is being grouped,
> >> * 1 if only gender is being groped,
> >> * 0 if neither deptno nor gender are being grouped.
> >> *
> >> * This function is defined in the SQL standard.
> >> * {@code GROUPING_ID} is a non-standard synonym.
> >> *
> >> * Some examples are in {@code agg.iq}.
> >> */
> >>
> >> The annotations above seems conflicts with other implementations like
> Hive(
> >>
> https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C+Grouping+and+Rollup?spm=ata.13261165.0.0.528c6dfcXalQFy#EnhancedAggregation,Cube,GroupingandRollup-Groupingfunction
> >> )
> >>
> >> Notice that: "The grouping function indicates whether an expression in a
> >> GROUP BY clause is aggregated or not for a given row. The value 0
> >> represents a column that is part of the grouping set, while the value 1
> >> represents a column that is not part of the grouping set. "
> >>
> >>
> >> It is clearly that 0 and 1 bit have different interpretation  between
> >> annotations in Calcite and in Hive. And I did not figure out why...
> >>
> >> Any feedback can give me on this would be highly appreciated.
> >>
> >> Best regards!
>
>

Re: Understanding annotations of SqlGroupingFunction

[XING JIN]

Filed a JIRA: https://issues.apache.org/jira/browse/CALCITE-3950

XING JIN  于2020年4月22日周三 下午2:51写道：

> Hi Vineet ~
> +1 on your analysis.
> Checking below case in agg.iq. We can see that the behavior of GROUPING
> function in Calcite is the same as Hive.
>
> # GROUPING in SELECT clause of CUBE query
> select deptno, job, count(*) as c, grouping(deptno) as d,
>   grouping(job) j, grouping(deptno, job) as x
> from "scott".emp
> group by cube(deptno, job);
> ++---++---+---+---+
> | DEPTNO | JOB   | C  | D | J | X |
> ++---++---+---+---+
> | 10 | CLERK |  1 | 0 | 0 | 0 |
> | 10 | MANAGER   |  1 | 0 | 0 | 0 |
> | 10 | PRESIDENT |  1 | 0 | 0 | 0 |
> | 10 |   |  3 | 0 | 1 | 1 |
> | 20 | ANALYST   |  2 | 0 | 0 | 0 |
> | 20 | CLERK |  2 | 0 | 0 | 0 |
> | 20 | MANAGER   |  1 | 0 | 0 | 0 |
> | 20 |   |  5 | 0 | 1 | 1 |
> | 30 | CLERK |  1 | 0 | 0 | 0 |
> | 30 | MANAGER   |  1 | 0 | 0 | 0 |
> | 30 | SALESMAN  |  4 | 0 | 0 | 0 |
> | 30 |   |  6 | 0 | 1 | 1 |
> || ANALYST   |  2 | 1 | 0 | 2 |
> || CLERK |  4 | 1 | 0 | 2 |
> || MANAGER   |  3 | 1 | 0 | 2 |
> || PRESIDENT |  1 | 1 | 0 | 2 |
> || SALESMAN  |  4 | 1 | 0 | 2 |
> ||   | 14 | 1 | 1 | 3 |
> ++---++---+---+---+
>
> IMHO, we might rectify the doc of SqlGroupingFunction as below:
>
> * 0 if both deptno and gender are being grouped,
> * 1 if only deptno is being grouped,
> * 2 if only gender is being groped,
> * 3 if neither deptno nor gender are being grouped.
>
>
> - Jin
>
>
> Vineet G  于2020年4月22日周三 上午5:18写道：
>
>> I expect that the user behavior for the GROUPING in both hive and calcite
>> is same. It’s just the documentation which is a bit confusing.
>> e.g.  comment line on grouping : if both deptno and gender are being
>> grouped
>>
>> should really mean that the row which represents the grand total i.e
>> without group by expression. will return 3.
>>
>> FYI Hive had its behavior fixed with
>> https://issues.apache.org/jira/browse/HIVE-16102 <
>> https://issues.apache.org/jira/browse/HIVE-16102> and currently uses
>> GROUPING ID function.
>>
>> Vineet Garg
>>
>>
>> > On Apr 21, 2020, at 10:42 AM, Julian Hyde 
>> wrote:
>> >
>> > Suppose we have one row that represents the total for department 10,
>> and another that represents the grand total of all departments. Which row
>> would we say that department is “grouped” (in Calcite’s parlance) or
>> “aggregated” in (Hive’s parlance)?
>> >
>> > I find the terms confusing. It’s possible that Calcite has them
>> “wrong”.
>> >
>> > I would simply run a query like
>> >
>> >  SELECT deptno, job, GROUPING(deptno, job), COUNT(*)
>> >  FROM emp
>> >  GROUP BY CUBE (deptno, job)
>> >
>> > and see whether Hive and Calcite return the same result.
>> >
>> >> On Apr 20, 2020, at 6:58 PM, ZZY  wrote:
>> >>
>> >> Hi, Hyde:
>> >> It's confused me that some annotations in
>> >> Calcite(org.apache.calcite.sql.fun.SqlGroupingFunction.java) :
>> >> /**
>> >> * The {@code GROUPING} function.
>> >> *
>> >> * Accepts 1 or more arguments.
>> >> * Example: {@code GROUPING(deptno, gender)} returns
>> >> * 3 if both deptno and gender are being grouped,
>> >> * 2 if only deptno is being grouped,
>> >> * 1 if only gender is being groped,
>> >> * 0 if neither deptno nor gender are being grouped.
>> >> *
>> >> * This function is defined in the SQL standard.
>> >> * {@code GROUPING_ID} is a non-standard synonym.
>> >> *
>> >> * Some examples are in {@code agg.iq}.
>> >> */
>> >>
>> >> The annotations above seems conflicts with other implementations like
>> Hive(
>> >>
>> https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C+Grouping+and+Rollup?spm=ata.13261165.0.0.528c6dfcXalQFy#EnhancedAggregation,Cube,GroupingandRollup-Groupingfunction
>> >> )
>> >>
>> >> Notice that: "The grouping function indicates whether an expression in
>> a
>> >> GROUP BY clause is aggregated or not for a given row. The value 0
>> >> represents a column that is part of the grouping set, while the value 1
>> >> represents a column that is not part of the grouping set. "
>> >>
>> >>
>> >> It is clearly that 0 and 1 bit have different interpretation  between
>> >> annotations in Calcite and in Hive. And I did not figure out why...
>> >>
>> >> Any feedback can give me on this would be highly appreciated.
>> >>
>> >> Best regards!
>>
>>

Re: [ANNOUNCE] New committer: Jin Xing

[XING JIN]

Hi, Julian ~
For introduction of my company [1]. It's under Alibaba Group and doing
financial business.
Regarding "powered by" page[2], it will be great if we could add an entry
and logo on it ;D
Thanks a lot !

Jin

[1]  https://en.wikipedia.org/wiki/Ant_Financial
<https://en.wikipedia.org/wiki/Ant_Financial>
[2]  https://calcite.apache.org/docs/powered_by.html
<https://calcite.apache.org/docs/powered_by.html#alibaba-maxcompute>

XING JIN  于2020年5月1日周五 上午10:54写道：

> Thanks a lot, Julian ~
> I'm not from MaxCompute team, but from big data platform in Alibaba Ant
> Financial Group.
> Actually we cooperate a lot with MaxCompute, it's our sister team.
>
> Jin
>
> Julian Hyde  于2020年5月1日周五 上午1:48写道：
>
>> Welcome Jin! Thanks for your contributions so far, looking forward to
>> more!
>>
>> Are you on the MaxCompute project? It’s already on our “powered by”
>> page[1], so I think people are familiar with it.
>>
>> Julian
>>
>> [1] https://calcite.apache.org/docs/powered_by.html#alibaba-maxcompute <
>> https://calcite.apache.org/docs/powered_by.html#alibaba-maxcompute>
>>
>>
>> > On Apr 29, 2020, at 5:06 AM, XING JIN  wrote:
>> >
>> > Thanks a lot ~
>> > Calcite is a great project and it's great honor for me to work with you
>> > guys. I really appreciate the help from community.
>> > I'm working in Alibaba. My team builds big data system to optimize batch
>> > and streaming jobs. We use Calcite to process Sql queries and
>> accommodate
>> > to different physical engines.
>> > I'm very excited to become Calcite committer and looking forward to make
>> > more contributions.
>> >
>> > Best regards,
>> > Jin
>> >
>> >
>> > Zoltan Haindrich  于2020年4月29日周三 下午1:58写道：
>> >
>> >> Congrats!
>> >>
>> >> On 4/29/20 7:32 AM, Enrico Olivelli wrote:
>> >>> Congratulations!
>> >>>
>> >>> Enrico
>> >>>
>> >>> Il Mer 29 Apr 2020, 04:51 Feng Zhu  ha
>> scritto:
>> >>>
>> >>>>  Congrations!
>> >>>>
>> >>>> best,
>> >>>> Feng
>> >>>>
>> >>>> Chunwei Lei  于2020年4月29日周三 上午10:16写道：
>> >>>>
>> >>>>> Congrats, Jin!
>> >>>>>
>> >>>>>
>> >>>>> Best,
>> >>>>> Chunwei
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Apr 29, 2020 at 10:07 AM Forward Xu > >
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Congrats
>> >>>>>>
>> >>>>>>
>> >>>>>> best,
>> >>>>>>
>> >>>>>> Forward
>> >>>>>>
>> >>>>>> 953396112 <953396...@qq.com> 于2020年4月29日周三 上午8:21写道：
>> >>>>>>
>> >>>>>>> Congrats, Jin Xing!
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> ---Original---
>> >>>>>>> From: "Stamatis Zampetakis"> >>>>>>> Date: Wed, Apr 29, 2020 05:47 AM
>> >>>>>>> To: "dev"> >>>>>>> Subject: [ANNOUNCE] New committer: Jin Xing
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Apache Calcite's Project Management Committee (PMC) has invited
>> Jin
>> >>>>> Xing
>> >>>>>> to
>> >>>>>>> become a committer, and we are pleased to announce that he has
>> >>>>> accepted.
>> >>>>>>>
>> >>>>>>> Jin has contributed a lot of code in the project and many
>> >>>>>>> recent improvements in
>> >>>>>>> materialized view matching have his signature on them. Apart from
>> >>>> code
>> >>>>>>> contributions, Jin provides valuable help to the community by
>> doing
>> >>>>>> reviews
>> >>>>>>> and
>> >>>>>>> answering questions in the devlist.
>> >>>>>>>
>> >>>>>>> Jin, welcome, thank you for your contributions, and we look
>> forward
>> >>>> to
>> >>>>>> your
>> >>>>>>> further interactions with the community! If you wish, please feel
>> >>>> free
>> >>>>> to
>> >>>>>>> tell
>> >>>>>>> us more about yourself and what you are working on.
>> >>>>>>>
>> >>>>>>> Stamatis (on behalf of the Apache Calcite PMC)
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>>
>>

Re: [ANNOUNCE] New committer: Jin Xing

[XING JIN]

Thanks a lot, Julian ~
I'm not from MaxCompute team, but from big data platform in Alibaba Ant
Financial Group.
Actually we cooperate a lot with MaxCompute, it's our sister team.

Jin

Julian Hyde  于2020年5月1日周五 上午1:48写道：

> Welcome Jin! Thanks for your contributions so far, looking forward to more!
>
> Are you on the MaxCompute project? It’s already on our “powered by”
> page[1], so I think people are familiar with it.
>
> Julian
>
> [1] https://calcite.apache.org/docs/powered_by.html#alibaba-maxcompute <
> https://calcite.apache.org/docs/powered_by.html#alibaba-maxcompute>
>
>
> > On Apr 29, 2020, at 5:06 AM, XING JIN  wrote:
> >
> > Thanks a lot ~
> > Calcite is a great project and it's great honor for me to work with you
> > guys. I really appreciate the help from community.
> > I'm working in Alibaba. My team builds big data system to optimize batch
> > and streaming jobs. We use Calcite to process Sql queries and accommodate
> > to different physical engines.
> > I'm very excited to become Calcite committer and looking forward to make
> > more contributions.
> >
> > Best regards,
> > Jin
> >
> >
> > Zoltan Haindrich  于2020年4月29日周三 下午1:58写道：
> >
> >> Congrats!
> >>
> >> On 4/29/20 7:32 AM, Enrico Olivelli wrote:
> >>> Congratulations!
> >>>
> >>> Enrico
> >>>
> >>> Il Mer 29 Apr 2020, 04:51 Feng Zhu  ha scritto:
> >>>
> >>>>  Congrations!
> >>>>
> >>>> best,
> >>>> Feng
> >>>>
> >>>> Chunwei Lei  于2020年4月29日周三 上午10:16写道：
> >>>>
> >>>>> Congrats, Jin!
> >>>>>
> >>>>>
> >>>>> Best,
> >>>>> Chunwei
> >>>>>
> >>>>>
> >>>>> On Wed, Apr 29, 2020 at 10:07 AM Forward Xu 
> >>>>> wrote:
> >>>>>
> >>>>>> Congrats
> >>>>>>
> >>>>>>
> >>>>>> best,
> >>>>>>
> >>>>>> Forward
> >>>>>>
> >>>>>> 953396112 <953396...@qq.com> 于2020年4月29日周三 上午8:21写道：
> >>>>>>
> >>>>>>> Congrats, Jin Xing!
> >>>>>>>
> >>>>>>>
> >>>>>>> ---Original---
> >>>>>>> From: "Stamatis Zampetakis" >>>>>>> Date: Wed, Apr 29, 2020 05:47 AM
> >>>>>>> To: "dev" >>>>>>> Subject: [ANNOUNCE] New committer: Jin Xing
> >>>>>>>
> >>>>>>>
> >>>>>>> Apache Calcite's Project Management Committee (PMC) has invited Jin
> >>>>> Xing
> >>>>>> to
> >>>>>>> become a committer, and we are pleased to announce that he has
> >>>>> accepted.
> >>>>>>>
> >>>>>>> Jin has contributed a lot of code in the project and many
> >>>>>>> recent improvements in
> >>>>>>> materialized view matching have his signature on them. Apart from
> >>>> code
> >>>>>>> contributions, Jin provides valuable help to the community by doing
> >>>>>> reviews
> >>>>>>> and
> >>>>>>> answering questions in the devlist.
> >>>>>>>
> >>>>>>> Jin, welcome, thank you for your contributions, and we look forward
> >>>> to
> >>>>>> your
> >>>>>>> further interactions with the community! If you wish, please feel
> >>>> free
> >>>>> to
> >>>>>>> tell
> >>>>>>> us more about yourself and what you are working on.
> >>>>>>>
> >>>>>>> Stamatis (on behalf of the Apache Calcite PMC)
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>

Re: [ANNOUNCE] New committer: Jin Xing

[XING JIN]

Thanks a lot ~
Calcite is a great project and it's great honor for me to work with you
guys. I really appreciate the help from community.
I'm working in Alibaba. My team builds big data system to optimize batch
and streaming jobs. We use Calcite to process Sql queries and accommodate
to different physical engines.
I'm very excited to become Calcite committer and looking forward to make
more contributions.

Best regards,
Jin


Zoltan Haindrich  于2020年4月29日周三 下午1:58写道：

> Congrats!
>
> On 4/29/20 7:32 AM, Enrico Olivelli wrote:
> > Congratulations!
> >
> > Enrico
> >
> > Il Mer 29 Apr 2020, 04:51 Feng Zhu  ha scritto:
> >
> >>   Congrations!
> >>
> >> best,
> >> Feng
> >>
> >> Chunwei Lei  于2020年4月29日周三 上午10:16写道：
> >>
> >>> Congrats, Jin!
> >>>
> >>>
> >>> Best,
> >>> Chunwei
> >>>
> >>>
> >>> On Wed, Apr 29, 2020 at 10:07 AM Forward Xu 
> >>> wrote:
> >>>
>  Congrats
> 
> 
>  best,
> 
>  Forward
> 
>  953396112 <953396...@qq.com> 于2020年4月29日周三 上午8:21写道：
> 
> > Congrats, Jin Xing!
> >
> >
> > ---Original---
> > From: "Stamatis Zampetakis" > Date: Wed, Apr 29, 2020 05:47 AM
> > To: "dev" > Subject: [ANNOUNCE] New committer: Jin Xing
> >
> >
> > Apache Calcite's Project Management Committee (PMC) has invited Jin
> >>> Xing
>  to
> > become a committer, and we are pleased to announce that he has
> >>> accepted.
> >
> > Jin has contributed a lot of code in the project and many
> > recent improvements in
> > materialized view matching have his signature on them. Apart from
> >> code
> > contributions, Jin provides valuable help to the community by doing
>  reviews
> > and
> > answering questions in the devlist.
> >
> > Jin, welcome, thank you for your contributions, and we look forward
> >> to
>  your
> > further interactions with the community! If you wish, please feel
> >> free
> >>> to
> > tell
> > us more about yourself and what you are working on.
> >
> > Stamatis (on behalf of the Apache Calcite PMC)
> 
> >>>
> >>
> >
>

Re: [VOTE] Release apache-calcite-1.23.0 (release candidate 1)

[XING JIN]

Thanks, Haisheng ~

Local Calcite build with tests enabled on Linux: *OK*
Calcite-based system test suite: *OK*

Vote:
+1 (non-binding)

Francis Chuang  于2020年5月18日周一 下午2:00写道：

> Thanks for making this release available for voting, Haisheng!
>
> Verified GPG signature - OK
> Verified SHA512 - OK
> Ran tests per HOWTO (./gradlew check) - OK
> Quickly skimmed release notes - OK
> Spotted checked a few JARs in the Maven repository - OK
>
> Environment (OpenJDK:latest docker container):
> Gradle 6.3 (via gradlew)
> Debian GNU/Linux 8
> openjdk version "1.8.0_111"
> OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2~bpo8+1-b14)
> OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
>
> My vote is: +1 (binding)
>
> Francis
>
> On 16/05/2020 2:02 pm, Haisheng Yuan wrote:
> > Hi all,
> >
> > I have created a build for Apache Calcite 1.23.0, release
> > candidate 1.
> >
> > Thanks to everyone who has contributed to this release.
> >
> > You can read the release notes here:
> >
> https://github.com/apache/calcite/blob/calcite-1.23.0-rc1/site/_docs/history.md
> >
> > The commit to be voted upon:
> >
> https://gitbox.apache.org/repos/asf?p=calcite.git;a=commit;h=b708fdc46d4c5fd4c5a6c7a398823318a7b4dce3
> >
> > Its hash is b708fdc46d4c5fd4c5a6c7a398823318a7b4dce3
> >
> > Tag:
> > https://github.com/apache/calcite/tree/calcite-1.23.0-rc1
> >
> > The artifacts to be voted on are located here:
> > https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.23.0-rc1
> > (revision 39622)
> >
> > The hashes of the artifacts are as follows:
> >
> 961c4f13199e199c669a6168ba655a9492bdd80d644da375a684b732c0b628b8a2ffacea5da97c82e8702a8e3bf7a1f58784baa49509fb3c48ef593259e11f46
> > *apache-calcite-1.23.0-src.tar.gz
> >
> > A staged Maven repository is available for review at:
> >
> https://repository.apache.org/content/repositories/orgapachecalcite-1089/org/apache/calcite/
> >
> > Release artifacts are signed with the following key:
> > https://dist.apache.org/repos/dist/release/calcite/KEYS
> >
> > N.B.
> > To create the jars and test Apache Calcite: "./gradlew build".
> >
> > If you do not have a Java environment available, you can run the tests
> > using docker. To do so, install docker and docker-compose, then run
> > "docker-compose run test" from the root of the directory.
> >
> > Please vote on releasing this package as Apache Calcite 1.23.0.
> >
> > The vote is open for the next 72 hours and passes if a majority of at
> > least three +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Calcite 1.23.0
> > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> > [ ] -1 Do not release this package because...
> >
> >
> > Here is my vote:
> >
> > +1 (binding)
> >
> > Thanks,
> > Haisheng Yuan
> >
>

Re: [ANNOUNCE] Apache Calcite 1.23.0 released

[XING JIN]

Thanks a lot for driving this, Haisheng!

Best,
Jin

Stamatis Zampetakis  于2020年5月28日周四 上午7:14写道：

> Many thanks again Haisheng, the release has your signature on it and it is
> in many places :)
>
> On Wed, May 27, 2020 at 9:34 PM Haisheng Yuan  wrote:
>
> > The Apache Calcite team is pleased to announce the release of Apache
> > Calcite 1.23.0.
> >
> > Calcite is a dynamic data management framework. Its cost-based
> > optimizer converts queries, represented in relational algebra, into
> > executable plans. Calcite supports many front-end languages and
> > back-end data engines, and includes an SQL parser and, as a
> > sub-project, the Avatica JDBC driver.
> >
> > This release comes two months after 1.22.0. It includes more than
> > 100 resolved issues, comprising of a few new features as well as
> > general improvements and bug-fixes. It includes support for top
> > down trait request and trait enforcement without abstract converter,
> > ClickHouse dialect, SESSION and HOP Table functions, and many
> > more bug fixes and improvements.
> >
> > You can start using it in Maven by simply updating your dependency to:
> >
> >   
> > org.apache.calcite
> > calcite-core
> > 1.23.0
> >   
> >
> > If you'd like to download the source release, you can find it here:
> >
> >   https://calcite.apache.org/downloads/
> >
> > You can read more about the release (including release notes) here:
> >
> >   https://calcite.apache.org/news/2020/05/23/release-1.23.0/
> >
> > We welcome your help and feedback. For more information on how to
> > report problems, and to get involved, visit the project website at:
> >
> >   https://calcite.apache.org/
> >
> > Thanks to everyone involved!
> >
>

Re: [ANNOUNCE] Ruben Quesada Lopez joins Calcite PMC

[XING JIN]

Congrats, Ruben!

953396112 <953396...@qq.com> 于2020年8月12日周三 上午7:47写道：

> Congratulations,Ruben!
>
>
> xzh
> --原始邮件--
> 发件人:
>   "dev"
> <
> zabe...@gmail.com;
> 发送时间:2020年8月12日(星期三) 凌晨5:53
> 收件人:"dev"
> 主题:[ANNOUNCE] Ruben Quesada Lopez joins Calcite PMC
>
>
>
> I'm pleased to announce that Ruben has accepted an invitation to
> join the Calcite PMC. Ruben has been a consistent and helpful
> figure in the Calcite community for which we are very grateful. We
> look forward to the continued contributions and support.
>
> Please join me in congratulating Ruben!
>
> - Stamatis (on behalf of the Calcite PMC)

Re: [ANNOUNCE] New committer: Vladimir Ozerov

[XING JIN]

Congratulations ~

Best,
Jin

guangyuan wang  于2021年6月24日周四 上午9:50写道：

> Congratulations!
>
> Francis Chuang  于2021年6月24日周四 上午6:39写道：
>
> > Congrats, Vladimir!
> >
> > Francis
> >
> > On 24/06/2021 7:48 am, Haisheng Yuan wrote:
> > > Congratulations and thanks for your contributions, Vladimir!
> > >
> > > Regards,
> > > Haisheng
> > >
> > > On 2021/06/23 21:34:40, Stamatis Zampetakis  wrote:
> > >> Apache Calcite's Project Management Committee (PMC) has invited
> Vladimir
> > >> Ozerov to
> > >> become a committer, and we are pleased to announce that he has
> accepted.
> > >>
> > >> Vladimir is among the few people who know very well the internal
> > workings
> > >> of the
> > >> Calcite optimizer. He started and participated in many discussions
> about
> > >> the core engine and contributed ideas and code for making it better.
> > >> Moreover, Vladimir has blogged and talked about Calcite in various
> > >> conferences and meetups giving publicity and showcasing the
> > capabilities of
> > >> the project.
> > >>
> > >> Vladimir, welcome, thank you for your contributions, and we look
> > forward to
> > >> your
> > >> further interactions with the community! If you wish, please feel free
> > to
> > >> tell
> > >> us more about yourself and what you are working on.
> > >>
> > >> Stamatis (on behalf of the Apache Calcite PMC)
> > >>
> >
>

Re: [ANNOUNCE] New committer: Zhaohui Xu

[XING JIN]

Congratulations !

Jin

Michael Mior  于2021年10月15日周五 上午12:56写道：

> Welcome Zhaohui!
> --
> Michael Mior
> mm...@apache.org
>
> Le mer. 6 oct. 2021 à 16:48, Stamatis Zampetakis  a
> écrit :
> >
> > Apache Calcite's Project Management Committee (PMC) has invited Zhaohui
> Xu
> > to
> > become a committer, and we are pleased to announce that they have
> accepted.
> >
> > Numbers speak for themselves and Zhaohui has over 30 commits already in
> > master
> > and more than 20 open pull requests waiting to get in. Great record so
> far
> > including
> > (but not limited to) improvements and fixes in the view based
> > rewriting modules,
> > JSON serialization, metadata, and field trimming.
> >
> > Zhaohui, welcome, thank you for your contributions, and we look forward
> to
> > your
> > further interactions with the community! If you wish, please feel free to
> > tell
> > us more about yourself and what you are working on.
> >
> > Stamatis (on behalf of the Apache Calcite PMC)
>

Re: [ANNOUNCE] New committer: Xiong Duan

[XING JIN]

Congratulations !

Best,
Jin

Chunwei Lei  于2021年10月25日周一 下午4:58写道：

> Congratulations, Xiong!
>
>
> Best,
> Chunwei
>
>
> On Sun, Oct 24, 2021 at 6:34 AM Haisheng Yuan  wrote:
>
> > Congrats, Xiong!
> >
> > On 2021/10/23 21:23:59, Francis Chuang  wrote:
> > > Congratulations!
> > >
> > > On 24/10/2021 12:03 am, Stamatis Zampetakis wrote:
> > > > Apache Calcite's Project Management Committee (PMC) has invited Xiong
> > Duan
> > > > to
> > > > become a committer, and we are pleased to announce that they have
> > accepted.
> > > >
> > > > Xiong has pushed a lot of high quality patches, fixing and improving
> > code
> > > > around
> > > > aggregations and sub-queries,  in a rather short period of time.
> Apart
> > from
> > > > code
> > > > contributions, Xiong has been regularly reviewing PRs in GitHub and
> > helping
> > > > out
> > > > others in various JIRA issues.
> > > >
> > > > Xiong, welcome, thank you for your contributions, and we look forward
> > to
> > > > your
> > > > further interactions with the community! If you wish, please feel
> free
> > to
> > > > tell
> > > > us more about yourself and what you are working on.
> > > >
> > > > Stamatis (on behalf of the Apache Calcite PMC)
> > > >
> > >
> >
>