Calcite-Master - Build # 1312 - Failure

[Apache Jenkins Server]

The Apache Jenkins build system has built Calcite-Master (build #1312)

Status: Failure

Check console output at https://builds.apache.org/job/Calcite-Master/1312/ to 
view the results.

[jira] [Created] (CALCITE-3286) In LatticeSuggester, allow join conditions that use expressions

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-3286:


 Summary: In LatticeSuggester, allow join conditions that use 
expressions
 Key: CALCITE-3286
 URL: https://issues.apache.org/jira/browse/CALCITE-3286
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


In {{LatticeSuggester}}, allow join conditions that use expressions. For 
example, the query
{code}
select *
from emp
join dept on emp.deptno + 1 = dept.deptno
{code}
uses an expression "emp.deptno + 1" on the "emp" side of the join key.

Currently this query throws:
{noformat}
java.lang.ClassCastException: class 
org.apache.calcite.materialize.LatticeSuggester$DerivedColRef cannot be cast to 
class org.apache.calcite.materialize.LatticeSuggester$BaseColRef
at 
org.apache.calcite.materialize.LatticeSuggester.frame(LatticeSuggester.java:478)
at 
org.apache.calcite.materialize.LatticeSuggester.frame(LatticeSuggester.java:420)
at 
org.apache.calcite.materialize.LatticeSuggester.frame(LatticeSuggester.java:399)
at 
org.apache.calcite.materialize.LatticeSuggester.addQuery(LatticeSuggester.java:124)
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: Match Converter Rule based on Child Nodes

[rahul patwari]

Hi Jin,

We wanted to transform LogicalJoin to different Join Types in the external
system depending on the LogicalJoin Child Properties(which are in external
convention).
As LogicalJoin can be a child of LogicalJoin, because of VolcanoPlanner's
top-down approach, the child LogicalJoin is not converted first and we end
up in "Not enough rules to find the Cheapest Plan".

Regards,
Rahul

On Fri, Aug 23, 2019 at 8:17 AM XING JIN  wrote:

> If I understand correctly, you can try below steps:
> 1. Create a rule of Converter to convert child nodes to append external
> Convention;
> 2. Create a rule of RelOptRule to convert the parent node -- check the
> Convention of child node when matching
>
> Is it applicable ?
>
> rahul patwari  于2019年8月22日周四 下午11:18写道：
>
> > Hi,
> >
> > The properties of the child nodes correspond to the external Convention.
> > So, the child nodes have to be converted before the parent can be
> > transformed.
> > If the property doesn't match (or) if the property cannot be
> > determined(child node not converted), the rule cannot be applied. So, we
> > end up in "Not enough Rules to find the Cheapest Plan".
> > Is there a way to convert the Child nodes before the parent?
> > Can VolcanoPlanner be configured to work in bottom-up fashion?
> >
> > Regards,
> > Rahul
> >
> > On Thu, Aug 22, 2019 at 3:33 PM XING JIN 
> wrote:
> >
> > > I guess in Rahul's case, the child node of join should be converted
> > first,
> > > thus physical properties will be appended.
> > > But RelNode doesn't have a reference to parent, thus an independent
> > > RelOptRule is needed to convert LogicalJoin.
> > > Just as Michael said, you need to override the method of
> > RelOptRule#matches
> > > and check the properties of child node.
> > >
> > > Best,
> > > Jin
> > >
> > > Stamatis Zampetakis  于2019年8月21日周三 下午5:50写道：
> > >
> > > > The Volcano planner works in a top-down fashion. It starts by
> > > transforming
> > > > the root and move towards the leafs of the plan. Due to this when
> > > > transforming a logical join its inputs are still in the logical
> > > convention
> > > > so in principle they should not have any physical properties.
> > > >
> > > > Normally when the physical join algorithm is chosen the respective
> rule
> > > > should be responsible for demanding the inputs of the join to have
> > > certain
> > > > properties.
> > > >
> > > > Long story short, I think in your use case you should not make the
> rule
> > > > match based on the properties of the child nodes but it should match
> > > > unconditionally and if necessary demand some properties in its
> inputs.
> > > If I
> > > > remember well the EnumerableMergeJoinRule follows this approach so
> you
> > > can
> > > > have a look there.
> > > >
> > > > Best,
> > > > Stamatis
> > > >
> > > > On Tue, Aug 20, 2019, 5:07 PM Michael Mior  wrote:
> > > >
> > > > > If you just want to control whether the rule gets applied, you can
> > > > > override RelOptRule#matches which canreturns a boolean indicating
> > > > > whether the rule should be applied.
> > > > > --
> > > > > Michael Mior
> > > > > mm...@apache.org
> > > > >
> > > > > Le ven. 9 août 2019 à 08:48, rahul patwari
> > > > >  a écrit :
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We want to create a ConverterRule which converts the default
> > calling
> > > > > > Convention to external storage-specific calling convention
> > depending
> > > on
> > > > > the
> > > > > > Children nodes, like RelOptRule.
> > > > > >
> > > > > > For example, depending on the properties of the child nodes, we
> > want
> > > to
> > > > > > convert LogicalJoin to external system's specific Join
> > > implementation.
> > > > > >
> > > > > > Currently, ConverterRule
> > > > > > <
> > > > >
> > > >
> > >
> >
> https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75
> > > > > >
> > > > > > cannot take Children and Child Policy is
> > > > > RelOptRuleOperandChildPolicy.ANY.
> > > > > >
> > > > > > What is the preferred way to achieve this task?
> > > > > >
> > > > > > Thanks,
> > > > > > Rahul
> > > > >
> > > >
> > >
> >
>

Re: Match Converter Rule based on Child Nodes

[XING JIN]

If I understand correctly, you can try below steps:
1. Create a rule of Converter to convert child nodes to append external
Convention;
2. Create a rule of RelOptRule to convert the parent node -- check the
Convention of child node when matching

Is it applicable ?

rahul patwari  于2019年8月22日周四 下午11:18写道：

> Hi,
>
> The properties of the child nodes correspond to the external Convention.
> So, the child nodes have to be converted before the parent can be
> transformed.
> If the property doesn't match (or) if the property cannot be
> determined(child node not converted), the rule cannot be applied. So, we
> end up in "Not enough Rules to find the Cheapest Plan".
> Is there a way to convert the Child nodes before the parent?
> Can VolcanoPlanner be configured to work in bottom-up fashion?
>
> Regards,
> Rahul
>
> On Thu, Aug 22, 2019 at 3:33 PM XING JIN  wrote:
>
> > I guess in Rahul's case, the child node of join should be converted
> first,
> > thus physical properties will be appended.
> > But RelNode doesn't have a reference to parent, thus an independent
> > RelOptRule is needed to convert LogicalJoin.
> > Just as Michael said, you need to override the method of
> RelOptRule#matches
> > and check the properties of child node.
> >
> > Best,
> > Jin
> >
> > Stamatis Zampetakis  于2019年8月21日周三 下午5:50写道：
> >
> > > The Volcano planner works in a top-down fashion. It starts by
> > transforming
> > > the root and move towards the leafs of the plan. Due to this when
> > > transforming a logical join its inputs are still in the logical
> > convention
> > > so in principle they should not have any physical properties.
> > >
> > > Normally when the physical join algorithm is chosen the respective rule
> > > should be responsible for demanding the inputs of the join to have
> > certain
> > > properties.
> > >
> > > Long story short, I think in your use case you should not make the rule
> > > match based on the properties of the child nodes but it should match
> > > unconditionally and if necessary demand some properties in its inputs.
> > If I
> > > remember well the EnumerableMergeJoinRule follows this approach so you
> > can
> > > have a look there.
> > >
> > > Best,
> > > Stamatis
> > >
> > > On Tue, Aug 20, 2019, 5:07 PM Michael Mior  wrote:
> > >
> > > > If you just want to control whether the rule gets applied, you can
> > > > override RelOptRule#matches which canreturns a boolean indicating
> > > > whether the rule should be applied.
> > > > --
> > > > Michael Mior
> > > > mm...@apache.org
> > > >
> > > > Le ven. 9 août 2019 à 08:48, rahul patwari
> > > >  a écrit :
> > > > >
> > > > > Hi,
> > > > >
> > > > > We want to create a ConverterRule which converts the default
> calling
> > > > > Convention to external storage-specific calling convention
> depending
> > on
> > > > the
> > > > > Children nodes, like RelOptRule.
> > > > >
> > > > > For example, depending on the properties of the child nodes, we
> want
> > to
> > > > > convert LogicalJoin to external system's specific Join
> > implementation.
> > > > >
> > > > > Currently, ConverterRule
> > > > > <
> > > >
> > >
> >
> https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75
> > > > >
> > > > > cannot take Children and Child Policy is
> > > > RelOptRuleOperandChildPolicy.ANY.
> > > > >
> > > > > What is the preferred way to achieve this task?
> > > > >
> > > > > Thanks,
> > > > > Rahul
> > > >
> > >
> >
>

[jira] [Created] (CALCITE-3285) EnumerableMergeJoin should support non-equi join conditions

[Haisheng Yuan (Jira)]

Haisheng Yuan created CALCITE-3285:
--

 Summary: EnumerableMergeJoin should support non-equi join 
conditions
 Key: CALCITE-3285
 URL: https://issues.apache.org/jira/browse/CALCITE-3285
 Project: Calcite
  Issue Type: Improvement
Reporter: Haisheng Yuan


Calcite should be able to generate EnumerableMergeJoin with non-equi join 
conditions, as long as there are equi-join conditions.




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (CALCITE-3284) Enumerable hash semijoin support non-equi join conditions

[Haisheng Yuan (Jira)]

Haisheng Yuan created CALCITE-3284:
--

 Summary: Enumerable hash semijoin support non-equi join conditions
 Key: CALCITE-3284
 URL: https://issues.apache.org/jira/browse/CALCITE-3284
 Project: Calcite
  Issue Type: Improvement
Reporter: Haisheng Yuan


Calcite should be able to generate enumerable hash semijoin with non-equi join 
conditions, as long as there are equi-join condtions, so that we can do hash 
look up.




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: Applying to be contributor

[Rui Wang]

Big thanks!


-Rui

On Wed, Aug 21, 2019 at 2:53 PM Francis Chuang 
wrote:

> Hi Rui,
>
> I've added your jira account to the contributor role.
>
> Francis
>
> On 22/08/2019 3:04 am, Rui Wang wrote:
> > Hi,
> >
> > Can I have contributor permission such that I can take some JIRA? Here is
> > my JIRA id: amaliujia
> >
> >
> >
> > -Rui
> >
>

Re: Apply for contributor

[Francis Chuang]

Hi Xiening,

I've added you to the contributor role in jira.

Francis

On 23/08/2019 7:47 am, Xiening Dai wrote:

My user name on jira is xndai. Thanks.


On Aug 22, 2019, at 2:36 PM, Xiening Dai  wrote:

Hi all,

May I have the contributor permission for Calcite project? I’d like to work on 
some issue related to VolcanoPlanner. Thanks.

Re: Apply for contributor

[Xiening Dai]

My user name on jira is xndai. Thanks.

> On Aug 22, 2019, at 2:36 PM, Xiening Dai  wrote:
> 
> Hi all,
> 
> May I have the contributor permission for Calcite project? I’d like to work 
> on some issue related to VolcanoPlanner. Thanks.

Apply for contributor

[Xiening Dai]

Hi all,

May I have the contributor permission for Calcite project? I’d like to work on 
some issue related to VolcanoPlanner. Thanks.

[jira] [Created] (CALCITE-3283) RelSubSet's best is not existed in the set

[Xiening Dai (Jira)]

Xiening Dai created CALCITE-3283:


 Summary: RelSubSet's best is not existed in the set
 Key: CALCITE-3283
 URL: https://issues.apache.org/jira/browse/CALCITE-3283
 Project: Calcite
  Issue Type: Bug
Reporter: Xiening Dai


To repro this, add below code in VolcanoPlanner.isValid() which verifies 
RelSubset.best indeed belongs to its set -

{code:java}
// Make sure best RelNode is valid
if (subset.best != null && !subset.set.rels.contains(subset.best)) {
  return litmus.fail("RelSubSet [{}] has best RelNode [{}] which is not 
existed in its set.",
  subset.getDescription(), subset.best.getDescription());
}
{code}

Run JDBCTest.testVarcharEquals, you will get below exception -

java.lang.AssertionError: RelSubSet [rel#6828:Subset#2.JDBC.foodmart.[]] has 
best RelNode 
[rel#6871:JdbcProject.JDBC.foodmart.[](input=RelSubset#6844,lname=$2)] which is 
not existed in its set.

at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:888)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:851)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1939)
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
at 
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
at 
org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
at 
org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286)
at 
org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346)
at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:320)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472)
at 
org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231)
at 
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
at 
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
at 
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
at 
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
at 
org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1447)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1410)
at 
org.apache.calcite.test.JdbcTest.testVarcharEquals(JdbcTest.java:4420)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
   

Re: [DISCUSS] ANTLR4 parse template for Calcite ?

[Julian Feinauer]

Hi,

there are some SQL dialect grammars online here (for ANTLR4)

https://github.com/antlr/grammars-v4/tree/master/mysql
https://github.com/antlr/grammars-v4/tree/master/plsql
https://github.com/antlr/grammars-v4/tree/master/sqlite
https://github.com/antlr/grammars-v4/tree/master/tsql

They could be a starting point for your work?

I used these as I wrote an ANTLR grammar for an SQL Like syntax the last time.

JulianF

Am 22.08.19, 19:39 schrieb "Julian Hyde" :

If you are going to do all that work to translate to ANTLR, one thing that 
may help is to re-use SqlParserTest.java. (Shouldn’t be hard to translate that 
into javascript, or your could use a harness that calls the javascript code 
from java.) Your code may be entirely different, but the tests will ensure that 
it gives the same result.

> On Aug 22, 2019, at 4:04 AM, Michael Franzkowiak  
wrote:
> 
> It is not using ANTLR. Since our goal is specifically to support parsing
> and manipulation of SQL in the frontend, we use
> https://sap.github.io/chevrotain/docs/ . We're quite happy with that. We
> have some pretty big ANTLR grammars for other (non-SQL) use cases and this
> approach definitely feels more lightweight.
> 
> On Thu, Aug 22, 2019 at 12:22 PM Danny Chan  wrote:
> 
>> Create ！ Do you have the ANTLR.g4 file that can be shared ?
>> 
>> Best,
>> Danny Chan
>> 在 2019年8月22日 +0800 PM5:45，Michael Franzkowiak ，写道：
>>> Danny, what is your web / frontend use case exactly?
>>> We've started to create some frontend helpers which you can find at
>>> https://github.com/contiamo/rhombic . It's all in a very early state but
>>> we'll likely spend some more time on it in the next months. Parsing is
>> here
>>> https://github.com/contiamo/rhombic/blob/master/src/SqlParser.ts .
>>> 
>>> On Thu, Aug 22, 2019 at 11:38 AM Muhammad Gelbana 
>>> wrote:
>>> 
 I once needed to fix this issue [1] but the fix was rejected because it
 introduced worse performance than it ideally should. As mentioned in
>> the
 comments, the current approach followed in the current parser is the
>> reason
 for that. I mean if we designed the grammar differently, we could've
>> had
 fixed the linked issue a long time ago as Julian already attempted to
>> fix
 it.
 
 Having that said, we might go with *antlr* only to have that "better"
 approach for our parsers. We don't have to dump our current parser of
 course as *antlr* can be optionally activated.
 
 [1] https://issues.apache.org/jira/browse/CALCITE-35
 
 Thanks,
 Gelbana
 
 
 On Thu, Aug 22, 2019 at 10:05 AM Danny Chan 
>> wrote:
 
> Thanks, Julian.
> 
> I agree this would be a huge work, but I have to do this, I’m just
> wondering if any fellows here have the similar requests.
> 
> Best,
> Danny Chan
> 在 2019年8月22日 +0800 PM2:15，Julian Hyde ，写道：
>> ANTLR isn’t significantly better than, or worse than, JavaCC, but
>> it’s
> different. So translating to ANTLR would be a rewrite, and would be a
 HUGE
> amount of work.
>> 
>> 
>> 
>>> On Aug 21, 2019, at 8:01 PM, Danny Chan 
 wrote:
>>> 
>>> Now some of our fellows want to do the syntax promote in the WEB
 page,
> and they what a parser in the front-page; The ANTLR4 can generate JS
 parser
> directly but JAVACC couldn’t.
>>> 
>>> So I’m wondering do you have the similar requests ? And do you
>> think
> there is necessity to support ANTLR4 g4 file in Calcite ?
>>> 
>>> 
>>> Best,
>>> Danny Chan
>> 
> 
 
>>> 
>>> 
>> 

Re: [DISCUSS] ANTLR4 parse template for Calcite ?

[Julian Hyde]

If you are going to do all that work to translate to ANTLR, one thing that may 
help is to re-use SqlParserTest.java. (Shouldn’t be hard to translate that into 
javascript, or your could use a harness that calls the javascript code from 
java.) Your code may be entirely different, but the tests will ensure that it 
gives the same result.

> On Aug 22, 2019, at 4:04 AM, Michael Franzkowiak  wrote:
> 
> It is not using ANTLR. Since our goal is specifically to support parsing
> and manipulation of SQL in the frontend, we use
> https://sap.github.io/chevrotain/docs/ . We're quite happy with that. We
> have some pretty big ANTLR grammars for other (non-SQL) use cases and this
> approach definitely feels more lightweight.
> 
> On Thu, Aug 22, 2019 at 12:22 PM Danny Chan  wrote:
> 
>> Create ！ Do you have the ANTLR.g4 file that can be shared ?
>> 
>> Best,
>> Danny Chan
>> 在 2019年8月22日 +0800 PM5:45，Michael Franzkowiak ，写道：
>>> Danny, what is your web / frontend use case exactly?
>>> We've started to create some frontend helpers which you can find at
>>> https://github.com/contiamo/rhombic . It's all in a very early state but
>>> we'll likely spend some more time on it in the next months. Parsing is
>> here
>>> https://github.com/contiamo/rhombic/blob/master/src/SqlParser.ts .
>>> 
>>> On Thu, Aug 22, 2019 at 11:38 AM Muhammad Gelbana 
>>> wrote:
>>> 
 I once needed to fix this issue [1] but the fix was rejected because it
 introduced worse performance than it ideally should. As mentioned in
>> the
 comments, the current approach followed in the current parser is the
>> reason
 for that. I mean if we designed the grammar differently, we could've
>> had
 fixed the linked issue a long time ago as Julian already attempted to
>> fix
 it.
 
 Having that said, we might go with *antlr* only to have that "better"
 approach for our parsers. We don't have to dump our current parser of
 course as *antlr* can be optionally activated.
 
 [1] https://issues.apache.org/jira/browse/CALCITE-35
 
 Thanks,
 Gelbana
 
 
 On Thu, Aug 22, 2019 at 10:05 AM Danny Chan 
>> wrote:
 
> Thanks, Julian.
> 
> I agree this would be a huge work, but I have to do this, I’m just
> wondering if any fellows here have the similar requests.
> 
> Best,
> Danny Chan
> 在 2019年8月22日 +0800 PM2:15，Julian Hyde ，写道：
>> ANTLR isn’t significantly better than, or worse than, JavaCC, but
>> it’s
> different. So translating to ANTLR would be a rewrite, and would be a
 HUGE
> amount of work.
>> 
>> 
>> 
>>> On Aug 21, 2019, at 8:01 PM, Danny Chan 
 wrote:
>>> 
>>> Now some of our fellows want to do the syntax promote in the WEB
 page,
> and they what a parser in the front-page; The ANTLR4 can generate JS
 parser
> directly but JAVACC couldn’t.
>>> 
>>> So I’m wondering do you have the similar requests ? And do you
>> think
> there is necessity to support ANTLR4 g4 file in Calcite ?
>>> 
>>> 
>>> Best,
>>> Danny Chan
>> 
> 
 
>>> 
>>> 
>> 

Re: Match Converter Rule based on Child Nodes

[rahul patwari]

Hi,

The properties of the child nodes correspond to the external Convention.
So, the child nodes have to be converted before the parent can be
transformed.
If the property doesn't match (or) if the property cannot be
determined(child node not converted), the rule cannot be applied. So, we
end up in "Not enough Rules to find the Cheapest Plan".
Is there a way to convert the Child nodes before the parent?
Can VolcanoPlanner be configured to work in bottom-up fashion?

Regards,
Rahul

On Thu, Aug 22, 2019 at 3:33 PM XING JIN  wrote:

> I guess in Rahul's case, the child node of join should be converted first,
> thus physical properties will be appended.
> But RelNode doesn't have a reference to parent, thus an independent
> RelOptRule is needed to convert LogicalJoin.
> Just as Michael said, you need to override the method of RelOptRule#matches
> and check the properties of child node.
>
> Best,
> Jin
>
> Stamatis Zampetakis  于2019年8月21日周三 下午5:50写道：
>
> > The Volcano planner works in a top-down fashion. It starts by
> transforming
> > the root and move towards the leafs of the plan. Due to this when
> > transforming a logical join its inputs are still in the logical
> convention
> > so in principle they should not have any physical properties.
> >
> > Normally when the physical join algorithm is chosen the respective rule
> > should be responsible for demanding the inputs of the join to have
> certain
> > properties.
> >
> > Long story short, I think in your use case you should not make the rule
> > match based on the properties of the child nodes but it should match
> > unconditionally and if necessary demand some properties in its inputs.
> If I
> > remember well the EnumerableMergeJoinRule follows this approach so you
> can
> > have a look there.
> >
> > Best,
> > Stamatis
> >
> > On Tue, Aug 20, 2019, 5:07 PM Michael Mior  wrote:
> >
> > > If you just want to control whether the rule gets applied, you can
> > > override RelOptRule#matches which canreturns a boolean indicating
> > > whether the rule should be applied.
> > > --
> > > Michael Mior
> > > mm...@apache.org
> > >
> > > Le ven. 9 août 2019 à 08:48, rahul patwari
> > >  a écrit :
> > > >
> > > > Hi,
> > > >
> > > > We want to create a ConverterRule which converts the default calling
> > > > Convention to external storage-specific calling convention depending
> on
> > > the
> > > > Children nodes, like RelOptRule.
> > > >
> > > > For example, depending on the properties of the child nodes, we want
> to
> > > > convert LogicalJoin to external system's specific Join
> implementation.
> > > >
> > > > Currently, ConverterRule
> > > > <
> > >
> >
> https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75
> > > >
> > > > cannot take Children and Child Policy is
> > > RelOptRuleOperandChildPolicy.ANY.
> > > >
> > > > What is the preferred way to achieve this task?
> > > >
> > > > Thanks,
> > > > Rahul
> > >
> >
>

Re: [DISCUSS] ANTLR4 parse template for Calcite ?

[Michael Franzkowiak]

It is not using ANTLR. Since our goal is specifically to support parsing
and manipulation of SQL in the frontend, we use
https://sap.github.io/chevrotain/docs/ . We're quite happy with that. We
have some pretty big ANTLR grammars for other (non-SQL) use cases and this
approach definitely feels more lightweight.

On Thu, Aug 22, 2019 at 12:22 PM Danny Chan  wrote:

> Create ！ Do you have the ANTLR.g4 file that can be shared ?
>
> Best,
> Danny Chan
> 在 2019年8月22日 +0800 PM5:45，Michael Franzkowiak ，写道：
> > Danny, what is your web / frontend use case exactly?
> > We've started to create some frontend helpers which you can find at
> > https://github.com/contiamo/rhombic . It's all in a very early state but
> > we'll likely spend some more time on it in the next months. Parsing is
> here
> > https://github.com/contiamo/rhombic/blob/master/src/SqlParser.ts .
> >
> > On Thu, Aug 22, 2019 at 11:38 AM Muhammad Gelbana 
> > wrote:
> >
> > > I once needed to fix this issue [1] but the fix was rejected because it
> > > introduced worse performance than it ideally should. As mentioned in
> the
> > > comments, the current approach followed in the current parser is the
> reason
> > > for that. I mean if we designed the grammar differently, we could've
> had
> > > fixed the linked issue a long time ago as Julian already attempted to
> fix
> > > it.
> > >
> > > Having that said, we might go with *antlr* only to have that "better"
> > > approach for our parsers. We don't have to dump our current parser of
> > > course as *antlr* can be optionally activated.
> > >
> > > [1] https://issues.apache.org/jira/browse/CALCITE-35
> > >
> > > Thanks,
> > > Gelbana
> > >
> > >
> > > On Thu, Aug 22, 2019 at 10:05 AM Danny Chan 
> wrote:
> > >
> > > > Thanks, Julian.
> > > >
> > > > I agree this would be a huge work, but I have to do this, I’m just
> > > > wondering if any fellows here have the similar requests.
> > > >
> > > > Best,
> > > > Danny Chan
> > > > 在 2019年8月22日 +0800 PM2:15，Julian Hyde ，写道：
> > > > > ANTLR isn’t significantly better than, or worse than, JavaCC, but
> it’s
> > > > different. So translating to ANTLR would be a rewrite, and would be a
> > > HUGE
> > > > amount of work.
> > > > >
> > > > >
> > > > >
> > > > > > On Aug 21, 2019, at 8:01 PM, Danny Chan 
> > > wrote:
> > > > > >
> > > > > > Now some of our fellows want to do the syntax promote in the WEB
> > > page,
> > > > and they what a parser in the front-page; The ANTLR4 can generate JS
> > > parser
> > > > directly but JAVACC couldn’t.
> > > > > >
> > > > > > So I’m wondering do you have the similar requests ? And do you
> think
> > > > there is necessity to support ANTLR4 g4 file in Calcite ?
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Danny Chan
> > > > >
> > > >
> > >
> >
> >
>

[jira] [Created] (CALCITE-3282) make every SqlDialect parse their own data type

[feng huang (Jira)]

feng huang created CALCITE-3282:
---

 Summary: make every SqlDialect parse their own data type
 Key: CALCITE-3282
 URL: https://issues.apache.org/jira/browse/CALCITE-3282
 Project: Calcite
  Issue Type: Bug
  Components: core
Reporter: feng huang


Every database might have different type or same type but different type name, 
therefore making every SqlDialect parse their own data type is a suitable way.
for example, “select cast(col as int) from table” change to hive sql "select 
cast(col as integer) from table", but "integer" is not allowed in hive.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [DISCUSS] ANTLR4 parse template for Calcite ?

[Danny Chan]

Create ！ Do you have the ANTLR.g4 file that can be shared ?

Best,
Danny Chan
在 2019年8月22日 +0800 PM5:45，Michael Franzkowiak ，写道：
> Danny, what is your web / frontend use case exactly?
> We've started to create some frontend helpers which you can find at
> https://github.com/contiamo/rhombic . It's all in a very early state but
> we'll likely spend some more time on it in the next months. Parsing is here
> https://github.com/contiamo/rhombic/blob/master/src/SqlParser.ts .
>
> On Thu, Aug 22, 2019 at 11:38 AM Muhammad Gelbana 
> wrote:
>
> > I once needed to fix this issue [1] but the fix was rejected because it
> > introduced worse performance than it ideally should. As mentioned in the
> > comments, the current approach followed in the current parser is the reason
> > for that. I mean if we designed the grammar differently, we could've had
> > fixed the linked issue a long time ago as Julian already attempted to fix
> > it.
> >
> > Having that said, we might go with *antlr* only to have that "better"
> > approach for our parsers. We don't have to dump our current parser of
> > course as *antlr* can be optionally activated.
> >
> > [1] https://issues.apache.org/jira/browse/CALCITE-35
> >
> > Thanks,
> > Gelbana
> >
> >
> > On Thu, Aug 22, 2019 at 10:05 AM Danny Chan  wrote:
> >
> > > Thanks, Julian.
> > >
> > > I agree this would be a huge work, but I have to do this, I’m just
> > > wondering if any fellows here have the similar requests.
> > >
> > > Best,
> > > Danny Chan
> > > 在 2019年8月22日 +0800 PM2:15，Julian Hyde ，写道：
> > > > ANTLR isn’t significantly better than, or worse than, JavaCC, but it’s
> > > different. So translating to ANTLR would be a rewrite, and would be a
> > HUGE
> > > amount of work.
> > > >
> > > >
> > > >
> > > > > On Aug 21, 2019, at 8:01 PM, Danny Chan 
> > wrote:
> > > > >
> > > > > Now some of our fellows want to do the syntax promote in the WEB
> > page,
> > > and they what a parser in the front-page; The ANTLR4 can generate JS
> > parser
> > > directly but JAVACC couldn’t.
> > > > >
> > > > > So I’m wondering do you have the similar requests ? And do you think
> > > there is necessity to support ANTLR4 g4 file in Calcite ?
> > > > >
> > > > >
> > > > > Best,
> > > > > Danny Chan
> > > >
> > >
> >
>
>
> --
> *Michael Franzkowiak*
> Managing Director
>
> *Contiamo* - the fastest way from data to results
>
> Address: Stresemannstraße 123 | 10963 Berlin | Germany
> 
> Web: www.contiamo.com
>
> Company Information (EN):
> Contiamo GmbH, Registered office of the company: Berlin
> HR Berlin-Charlottenburg, HRB no. 156569
> Managing directors: Michael Franzkowiak, Lucia Hegenbartová
>
> Unternehmensinformationen (DE):
> Contiamo GmbH, Sitz der Gesellschaft: Berlin
> HR Berlin-Charlottenburg, HRB Nr. 156569
> Geschäftsführer: Michael Franzkowiak, Lucia Hegenbartová

Re: Match Converter Rule based on Child Nodes

[XING JIN]

I guess in Rahul's case, the child node of join should be converted first,
thus physical properties will be appended.
But RelNode doesn't have a reference to parent, thus an independent
RelOptRule is needed to convert LogicalJoin.
Just as Michael said, you need to override the method of RelOptRule#matches
and check the properties of child node.

Best,
Jin

Stamatis Zampetakis  于2019年8月21日周三 下午5:50写道：

> The Volcano planner works in a top-down fashion. It starts by transforming
> the root and move towards the leafs of the plan. Due to this when
> transforming a logical join its inputs are still in the logical convention
> so in principle they should not have any physical properties.
>
> Normally when the physical join algorithm is chosen the respective rule
> should be responsible for demanding the inputs of the join to have certain
> properties.
>
> Long story short, I think in your use case you should not make the rule
> match based on the properties of the child nodes but it should match
> unconditionally and if necessary demand some properties in its inputs. If I
> remember well the EnumerableMergeJoinRule follows this approach so you can
> have a look there.
>
> Best,
> Stamatis
>
> On Tue, Aug 20, 2019, 5:07 PM Michael Mior  wrote:
>
> > If you just want to control whether the rule gets applied, you can
> > override RelOptRule#matches which canreturns a boolean indicating
> > whether the rule should be applied.
> > --
> > Michael Mior
> > mm...@apache.org
> >
> > Le ven. 9 août 2019 à 08:48, rahul patwari
> >  a écrit :
> > >
> > > Hi,
> > >
> > > We want to create a ConverterRule which converts the default calling
> > > Convention to external storage-specific calling convention depending on
> > the
> > > Children nodes, like RelOptRule.
> > >
> > > For example, depending on the properties of the child nodes, we want to
> > > convert LogicalJoin to external system's specific Join implementation.
> > >
> > > Currently, ConverterRule
> > > <
> >
> https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75
> > >
> > > cannot take Children and Child Policy is
> > RelOptRuleOperandChildPolicy.ANY.
> > >
> > > What is the preferred way to achieve this task?
> > >
> > > Thanks,
> > > Rahul
> >
>

Re: [DISCUSS] ANTLR4 parse template for Calcite ?

[Michael Franzkowiak]

Danny, what is your web / frontend use case exactly?
We've started to create some frontend helpers which you can find at
https://github.com/contiamo/rhombic . It's all in a very early state but
we'll likely spend some more time on it in the next months. Parsing is here
https://github.com/contiamo/rhombic/blob/master/src/SqlParser.ts .

On Thu, Aug 22, 2019 at 11:38 AM Muhammad Gelbana 
wrote:

> I once needed to fix this issue [1] but the fix was rejected because it
> introduced worse performance than it ideally should. As mentioned in the
> comments, the current approach followed in the current parser is the reason
> for that. I mean if we designed the grammar differently, we could've had
> fixed the linked issue a long time ago as Julian already attempted to fix
> it.
>
> Having that said, we might go with *antlr* only to have that "better"
> approach for our parsers. We don't have to dump our current parser of
> course as *antlr* can be optionally activated.
>
> [1] https://issues.apache.org/jira/browse/CALCITE-35
>
> Thanks,
> Gelbana
>
>
> On Thu, Aug 22, 2019 at 10:05 AM Danny Chan  wrote:
>
> > Thanks, Julian.
> >
> > I agree this would be a huge work, but I have to do this, I’m just
> > wondering if any fellows here have the similar requests.
> >
> > Best,
> > Danny Chan
> > 在 2019年8月22日 +0800 PM2:15，Julian Hyde ，写道：
> > > ANTLR isn’t significantly better than, or worse than, JavaCC, but it’s
> > different. So translating to ANTLR would be a rewrite, and would be a
> HUGE
> > amount of work.
> > >
> > >
> > >
> > > > On Aug 21, 2019, at 8:01 PM, Danny Chan 
> wrote:
> > > >
> > > > Now some of our fellows want to do the syntax promote in the WEB
> page,
> > and they what a parser in the front-page; The ANTLR4 can generate JS
> parser
> > directly but JAVACC couldn’t.
> > > >
> > > > So I’m wondering do you have the similar requests ? And do you think
> > there is necessity to support ANTLR4 g4 file in Calcite ?
> > > >
> > > >
> > > > Best,
> > > > Danny Chan
> > >
> >
>


-- 
*Michael Franzkowiak*
Managing Director

*Contiamo* - the fastest way from data to results

Address: Stresemannstraße 123 | 10963 Berlin | Germany

Web: www.contiamo.com

Company Information (EN):
Contiamo GmbH, Registered office of the company: Berlin
HR Berlin-Charlottenburg, HRB no. 156569
Managing directors: Michael Franzkowiak, Lucia Hegenbartová

Unternehmensinformationen (DE):
Contiamo GmbH, Sitz der Gesellschaft: Berlin
HR Berlin-Charlottenburg, HRB Nr. 156569
Geschäftsführer: Michael Franzkowiak, Lucia Hegenbartová

Re: [DISCUSS] ANTLR4 parse template for Calcite ?

[Muhammad Gelbana]

I once needed to fix this issue [1] but the fix was rejected because it
introduced worse performance than it ideally should. As mentioned in the
comments, the current approach followed in the current parser is the reason
for that. I mean if we designed the grammar differently, we could've had
fixed the linked issue a long time ago as Julian already attempted to fix
it.

Having that said, we might go with *antlr* only to have that "better"
approach for our parsers. We don't have to dump our current parser of
course as *antlr* can be optionally activated.

[1] https://issues.apache.org/jira/browse/CALCITE-35

Thanks,
Gelbana


On Thu, Aug 22, 2019 at 10:05 AM Danny Chan  wrote:

> Thanks, Julian.
>
> I agree this would be a huge work, but I have to do this, I’m just
> wondering if any fellows here have the similar requests.
>
> Best,
> Danny Chan
> 在 2019年8月22日 +0800 PM2:15，Julian Hyde ，写道：
> > ANTLR isn’t significantly better than, or worse than, JavaCC, but it’s
> different. So translating to ANTLR would be a rewrite, and would be a HUGE
> amount of work.
> >
> >
> >
> > > On Aug 21, 2019, at 8:01 PM, Danny Chan  wrote:
> > >
> > > Now some of our fellows want to do the syntax promote in the WEB page,
> and they what a parser in the front-page; The ANTLR4 can generate JS parser
> directly but JAVACC couldn’t.
> > >
> > > So I’m wondering do you have the similar requests ? And do you think
> there is necessity to support ANTLR4 g4 file in Calcite ?
> > >
> > >
> > > Best,
> > > Danny Chan
> >
>

Re: [DISCUSS] ANTLR4 parse template for Calcite ?

[Danny Chan]

Thanks, Julian.

I agree this would be a huge work, but I have to do this, I’m just wondering if 
any fellows here have the similar requests.

Best,
Danny Chan
在 2019年8月22日 +0800 PM2:15，Julian Hyde ，写道：
> ANTLR isn’t significantly better than, or worse than, JavaCC, but it’s 
> different. So translating to ANTLR would be a rewrite, and would be a HUGE 
> amount of work.
>
>
>
> > On Aug 21, 2019, at 8:01 PM, Danny Chan  wrote:
> >
> > Now some of our fellows want to do the syntax promote in the WEB page, and 
> > they what a parser in the front-page; The ANTLR4 can generate JS parser 
> > directly but JAVACC couldn’t.
> >
> > So I’m wondering do you have the similar requests ? And do you think there 
> > is necessity to support ANTLR4 g4 file in Calcite ?
> >
> >
> > Best,
> > Danny Chan
>