Re: [DISCUSS] Release apache-calcite-1.21.0 (release candidate 1)

[Stamatis Zampetakis]

Done, thanks for noticing.

I will add it also to the RM instructions along with some other minor
improvements.

Best,
Stamatis

On Tue, Sep 17, 2019 at 11:26 PM Julian Hyde  wrote:

> Stamatis,
>
> Can you mark 1.21 as released in JIRA [1].
>
> I don’t recall whether we ever made officially added this task to an RM’s
> responsibilities. I don’t mind doing it if you don’t have karma.
>
> Thank you, again, for being RM.
>
> Julian
>
> [1]
> https://issues.apache.org/jira/projects/CALCITE?selectedItem=com.atlassian.jira.jira-projects-plugin%3Arelease-page=released-unreleased
> <
> https://issues.apache.org/jira/projects/CALCITE?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page=released-unreleased
> >
>
> > On Sep 15, 2019, at 7:38 PM, Danny Chan  wrote:
> >
> > Nice summary of the release problems, Stamatis ! Maybe we should put the
> precautions to the website so that other RM can follow in the next releases
> ~
> >
> > Best,
> > Danny Chan
> > 在 2019年9月12日 +0800 AM7:54，Stamatis Zampetakis ，写道：
> >> The release process for apache-calcite-1.21.0 is now complete.
> >>
> >> You can now commit again to the master.
> >>
> >> Once again, I would like to thank again all the members of the community
> >> that helped in getting 1.21.0 out the door. A special mention for the
> >> reviewers who helped getting in some great features dedicating a
> >> significant amount of their time.
> >>
> >> During the vote various issues were raised.
> >>
> >> Spurious .xml files were present in release candidate 0; I have the
> >> impression that they came up due to the dry run that I did just before
> the
> >> release (without performing a git clean). As it is not indicated in the
> >> documentation I am thinking to add a small notice.
> >>
> >> The build fails for certain locales due to CALCITE-2816; we agree that
> the
> >> build problem must be solved for 1.22.0.
> >>
> >> There are intermittent failures in the Pig module; CALCITE-3336 was
> logged
> >> and we should continue further discussions there.
> >>
> >> There were some small issues with the release notes (passive voice, and
> >> backticks); I took care of them in
> >>
> https://github.com/apache/calcite/commit/034bd7942c35d5b6c948dc6863a9a086fb82386c
> >> .
> >>
> >> There was a comment that adc1532de does not have tag calcite-1.21.0 but
> if
> >> I am not looking at the wrong place I think its there.
> >>
> >> The checksum hash that was communicated in the vote email was wrong;
> given
> >> that the correct one was send along with the artifacts and people used
> this
> >> for the checks I assume there is no problem.
> >>
> >> There are some minor problems with README and README.md files; I will
> >> update them in the following days.
> >>
> >> The instructions in "Publishing a release" related with the release
> >> announcement, and the Javadoc generation are slightly confusing. In
> order
> >> to generate the Javadoc we must be in the tag calcite-1.21.0 and not in
> the
> >> branch-1.21 and on the other hand the release announcement should be
> >> committed in the branch-1.21. It may be worth adding a few more details
> >> there.
> >>
> >> If I missed something else worth mentioning feel free to include it.
> >>
> >> Best,
> >> Stamatis
>
>

Re: [ANNOUNCE] New committer: Julian Feinauer

[Julian Feinauer]

Hi all,

thanks all of you for the warm welcome and the honor to be a committer on this 
awesome project!
To present myself a bit... I'm currently mostly working on the MATCH_RECOGNIZE 
stuff and am planning to do a Calcite integration soon for the IoTDB podling.
Overall I'm highly interested in "traces" or "timeseries" and how we can map 
them well to the relational world.

Best
Julian

Am 17.09.19, 22:26 schrieb "Amit Chavan" :

Congrats, Julian !!

On Tue, Sep 17, 2019 at 8:12 PM XING JIN  wrote:

> Congrats, Julian !
> You are well deserved ~
>
> Haisheng Yuan  于2019年9月18日周三 上午10:38写道：
>
> > Congrats, Julian!
> >
> > - Haisheng
> >
> > --
> > 发件人：Chunwei Lei
> > 日 期：2019年09月18日 10:30:31
> > 收件人：
> > 主 题：Re: [ANNOUNCE] New committer: Julian Feinauer
> >
> > Congratulations, Julian!
> >
> >
> >
> > Best,
> > Chunwei
> >
> >
> > On Wed, Sep 18, 2019 at 9:24 AM Danny Chan  wrote:
> >
> > > Congratulations, Muhammad ! Welcome to join us ! Thanks for your huge
> > > contribution for the Match Recognize.
> > >
> > > Best,
> > > Danny Chan
> > > 在 2019年9月18日 +0800 AM5:55，Francis Chuang  >，写道：
> > > > Apache Calcite's Project Management Committee (PMC) has invited
> Julian
> > > > Feinauer to become a committer, and we are pleased to announce that
> he
> > > > has accepted.
> > > >
> > > > Julian is an active contributor to the Calcite code base and has 
been
> > > > active on the mailing list answering questions, participating in
> > > > discussions and voting for releases.
> > > >
> > > > Julian, welcome, thank you for your contributions, and we look
> forward
> > > > your further interactions with the community! If you wish, please
> feel
> > > > free to tell us more about yourself and what you are working on.
> > > >
> > > > Francis (on behalf of the Apache Calcite PMC)
> > >
> >
> >
>

[jira] [Created] (CALCITE-3358) Make Function DDLs implement SqlExecutableStatement

[Zhenqiu Huang (Jira)]

Zhenqiu Huang created CALCITE-3358:
--

 Summary: Make Function DDLs implement SqlExecutableStatement
 Key: CALCITE-3358
 URL: https://issues.apache.org/jira/browse/CALCITE-3358
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: Zhenqiu Huang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3357) Trivial null checking in RelSet#addAbstractConverters

[jin xing (Jira)]

jin xing created CALCITE-3357:
-

 Summary: Trivial null checking in RelSet#addAbstractConverters
 Key: CALCITE-3357
 URL: https://issues.apache.org/jira/browse/CALCITE-3357
 Project: Calcite
  Issue Type: Bug
  Components: core
Reporter: jin xing


In current code of *RelSet#addAbstractConverters*, null checking of 
*curRelTrait* happens after clause of assert *curRelTrait.getTraitDef() == 
traitDef*;

It makes more sense to adjust the order;

In my understanding, this issue was not found previously for two reasons:
 # AbstractConverter is not enabled by default 
([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableConvention.java#L65])
 # It rarely happens that two algebra expression operators have identical 
semantics but different *RelTraitDef*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New committer: Muhammad Gelbana

[Amit Chavan]

Congratulations, Muhammad !!

On Tue, Sep 17, 2019 at 8:10 PM XING JIN  wrote:

> Congrats, Muhammad !
>
> 王炎林 <1989yanlinw...@163.com> 于2019年9月18日周三 上午10:38写道：
>
> > Congratulations, Muhammad!
> >
> >
> > Best,
> > Yanlin
> >
> >
> >
> >
> >
> >
> > At 2019-09-18 05:58:53, "Francis Chuang" 
> wrote:
> > >Apache Calcite's Project Management Committee (PMC) has invited Muhammad
> > >Gelbana to become a committer, and we are pleased to announce that he
> > >has accepted.
> > >
> > >Muhammad is an active contributor and has contributed numerous patches
> > >to Calcite. He has also been extremely active on the mailing list,
> > >helping out new users and participating in design discussions.
> > >
> > >Muhammad, welcome, thank you for your contributions, and we look forward
> > >your further interactions with the community! If you wish, please feel
> > >free to tell us more about yourself and what you are working on.
> > >
> > >Francis (on behalf of the Apache Calcite PMC)
> >
>

Re: Re: [ANNOUNCE] New committer: Julian Feinauer

[Amit Chavan]

Congrats, Julian !!

On Tue, Sep 17, 2019 at 8:12 PM XING JIN  wrote:

> Congrats, Julian !
> You are well deserved ~
>
> Haisheng Yuan  于2019年9月18日周三 上午10:38写道：
>
> > Congrats, Julian!
> >
> > - Haisheng
> >
> > --
> > 发件人：Chunwei Lei
> > 日 期：2019年09月18日 10:30:31
> > 收件人：
> > 主 题：Re: [ANNOUNCE] New committer: Julian Feinauer
> >
> > Congratulations, Julian!
> >
> >
> >
> > Best,
> > Chunwei
> >
> >
> > On Wed, Sep 18, 2019 at 9:24 AM Danny Chan  wrote:
> >
> > > Congratulations, Muhammad ! Welcome to join us ! Thanks for your huge
> > > contribution for the Match Recognize.
> > >
> > > Best,
> > > Danny Chan
> > > 在 2019年9月18日 +0800 AM5:55，Francis Chuang  >，写道：
> > > > Apache Calcite's Project Management Committee (PMC) has invited
> Julian
> > > > Feinauer to become a committer, and we are pleased to announce that
> he
> > > > has accepted.
> > > >
> > > > Julian is an active contributor to the Calcite code base and has been
> > > > active on the mailing list answering questions, participating in
> > > > discussions and voting for releases.
> > > >
> > > > Julian, welcome, thank you for your contributions, and we look
> forward
> > > > your further interactions with the community! If you wish, please
> feel
> > > > free to tell us more about yourself and what you are working on.
> > > >
> > > > Francis (on behalf of the Apache Calcite PMC)
> > >
> >
> >
>

Re: Re: [ANNOUNCE] New committer: Julian Feinauer

[XING JIN]

Congrats, Julian !
You are well deserved ~

Haisheng Yuan  于2019年9月18日周三 上午10:38写道：

> Congrats, Julian!
>
> - Haisheng
>
> --
> 发件人：Chunwei Lei
> 日 期：2019年09月18日 10:30:31
> 收件人：
> 主 题：Re: [ANNOUNCE] New committer: Julian Feinauer
>
> Congratulations, Julian!
>
>
>
> Best,
> Chunwei
>
>
> On Wed, Sep 18, 2019 at 9:24 AM Danny Chan  wrote:
>
> > Congratulations, Muhammad ! Welcome to join us ! Thanks for your huge
> > contribution for the Match Recognize.
> >
> > Best,
> > Danny Chan
> > 在 2019年9月18日 +0800 AM5:55，Francis Chuang ，写道：
> > > Apache Calcite's Project Management Committee (PMC) has invited Julian
> > > Feinauer to become a committer, and we are pleased to announce that he
> > > has accepted.
> > >
> > > Julian is an active contributor to the Calcite code base and has been
> > > active on the mailing list answering questions, participating in
> > > discussions and voting for releases.
> > >
> > > Julian, welcome, thank you for your contributions, and we look forward
> > > your further interactions with the community! If you wish, please feel
> > > free to tell us more about yourself and what you are working on.
> > >
> > > Francis (on behalf of the Apache Calcite PMC)
> >
>
>

Re: [ANNOUNCE] New committer: Muhammad Gelbana

[XING JIN]

Congrats, Muhammad !

王炎林 <1989yanlinw...@163.com> 于2019年9月18日周三 上午10:38写道：

> Congratulations, Muhammad!
>
>
> Best,
> Yanlin
>
>
>
>
>
>
> At 2019-09-18 05:58:53, "Francis Chuang"  wrote:
> >Apache Calcite's Project Management Committee (PMC) has invited Muhammad
> >Gelbana to become a committer, and we are pleased to announce that he
> >has accepted.
> >
> >Muhammad is an active contributor and has contributed numerous patches
> >to Calcite. He has also been extremely active on the mailing list,
> >helping out new users and participating in design discussions.
> >
> >Muhammad, welcome, thank you for your contributions, and we look forward
> >your further interactions with the community! If you wish, please feel
> >free to tell us more about yourself and what you are working on.
> >
> >Francis (on behalf of the Apache Calcite PMC)
>

Re:[ANNOUNCE] New committer: Muhammad Gelbana

[王炎林]

Congratulations, Muhammad!


Best,
Yanlin






At 2019-09-18 05:58:53, "Francis Chuang"  wrote:
>Apache Calcite's Project Management Committee (PMC) has invited Muhammad 
>Gelbana to become a committer, and we are pleased to announce that he 
>has accepted.
>
>Muhammad is an active contributor and has contributed numerous patches 
>to Calcite. He has also been extremely active on the mailing list, 
>helping out new users and participating in design discussions.
>
>Muhammad, welcome, thank you for your contributions, and we look forward 
>your further interactions with the community! If you wish, please feel 
>free to tell us more about yourself and what you are working on.
>
>Francis (on behalf of the Apache Calcite PMC)

Re: Re: [ANNOUNCE] New committer: Julian Feinauer

[Haisheng Yuan]

Congrats, Julian!

- Haisheng

--
发件人：Chunwei Lei
日　期：2019年09月18日 10:30:31
收件人：
主　题：Re: [ANNOUNCE] New committer: Julian Feinauer

Congratulations, Julian!



Best,
Chunwei


On Wed, Sep 18, 2019 at 9:24 AM Danny Chan  wrote:

> Congratulations, Muhammad ! Welcome to join us ! Thanks for your huge
> contribution for the Match Recognize.
>
> Best,
> Danny Chan
> 在 2019年9月18日 +0800 AM5:55，Francis Chuang ，写道：
> > Apache Calcite's Project Management Committee (PMC) has invited Julian
> > Feinauer to become a committer, and we are pleased to announce that he
> > has accepted.
> >
> > Julian is an active contributor to the Calcite code base and has been
> > active on the mailing list answering questions, participating in
> > discussions and voting for releases.
> >
> > Julian, welcome, thank you for your contributions, and we look forward
> > your further interactions with the community! If you wish, please feel
> > free to tell us more about yourself and what you are working on.
> >
> > Francis (on behalf of the Apache Calcite PMC)
>

Re:[ANNOUNCE] New committer: Julian Feinauer

[王炎林]

Congratulations, Julian!




Best,
Wang Yanlin






At 2019-09-18 05:55:12, "Francis Chuang"  wrote:
>Apache Calcite's Project Management Committee (PMC) has invited Julian 
>Feinauer to become a committer, and we are pleased to announce that he 
>has accepted.
>
>Julian is an active contributor to the Calcite code base and has been 
>active on the mailing list answering questions, participating in 
>discussions and voting for releases.
>
>Julian, welcome, thank you for your contributions, and we look forward 
>your further interactions with the community! If you wish, please feel 
>free to tell us more about yourself and what you are working on.
>
>Francis (on behalf of the Apache Calcite PMC)

Re: [ANNOUNCE] New committer: Muhammad Gelbana

[Chunwei Lei]

Congratulations
,
Muhammad!


Best,
Chunwei


On Wed, Sep 18, 2019 at 9:23 AM Danny Chan  wrote:

> Congratulations, Muhammad ! Welcome to join us !
>
> Best,
> Danny Chan
> 在 2019年9月18日 +0800 AM5:58，Francis Chuang ，写道：
> > Apache Calcite's Project Management Committee (PMC) has invited Muhammad
> > Gelbana to become a committer, and we are pleased to announce that he
> > has accepted.
> >
> > Muhammad is an active contributor and has contributed numerous patches
> > to Calcite. He has also been extremely active on the mailing list,
> > helping out new users and participating in design discussions.
> >
> > Muhammad, welcome, thank you for your contributions, and we look forward
> > your further interactions with the community! If you wish, please feel
> > free to tell us more about yourself and what you are working on.
> >
> > Francis (on behalf of the Apache Calcite PMC)
>

Re: [ANNOUNCE] New committer: Julian Feinauer

[Chunwei Lei]

Congratulations, Julian!



Best,
Chunwei


On Wed, Sep 18, 2019 at 9:24 AM Danny Chan  wrote:

> Congratulations, Muhammad ! Welcome to join us ! Thanks for your huge
> contribution for the Match Recognize.
>
> Best,
> Danny Chan
> 在 2019年9月18日 +0800 AM5:55，Francis Chuang ，写道：
> > Apache Calcite's Project Management Committee (PMC) has invited Julian
> > Feinauer to become a committer, and we are pleased to announce that he
> > has accepted.
> >
> > Julian is an active contributor to the Calcite code base and has been
> > active on the mailing list answering questions, participating in
> > discussions and voting for releases.
> >
> > Julian, welcome, thank you for your contributions, and we look forward
> > your further interactions with the community! If you wish, please feel
> > free to tell us more about yourself and what you are working on.
> >
> > Francis (on behalf of the Apache Calcite PMC)
>

Re: [ANNOUNCE] New committer: Julian Feinauer

[Danny Chan]

Congratulations, Muhammad ! Welcome to join us ! Thanks for your huge 
contribution for the Match Recognize.

Best,
Danny Chan
在 2019年9月18日 +0800 AM5:55，Francis Chuang ，写道：
> Apache Calcite's Project Management Committee (PMC) has invited Julian
> Feinauer to become a committer, and we are pleased to announce that he
> has accepted.
>
> Julian is an active contributor to the Calcite code base and has been
> active on the mailing list answering questions, participating in
> discussions and voting for releases.
>
> Julian, welcome, thank you for your contributions, and we look forward
> your further interactions with the community! If you wish, please feel
> free to tell us more about yourself and what you are working on.
>
> Francis (on behalf of the Apache Calcite PMC)

Re: [ANNOUNCE] New committer: Muhammad Gelbana

[Danny Chan]

Congratulations, Muhammad ! Welcome to join us !

Best,
Danny Chan
在 2019年9月18日 +0800 AM5:58，Francis Chuang ，写道：
> Apache Calcite's Project Management Committee (PMC) has invited Muhammad
> Gelbana to become a committer, and we are pleased to announce that he
> has accepted.
>
> Muhammad is an active contributor and has contributed numerous patches
> to Calcite. He has also been extremely active on the mailing list,
> helping out new users and participating in design discussions.
>
> Muhammad, welcome, thank you for your contributions, and we look forward
> your further interactions with the community! If you wish, please feel
> free to tell us more about yourself and what you are working on.
>
> Francis (on behalf of the Apache Calcite PMC)

Github Actions for CI

[Francis Chuang]

A month or so ago, I started a thread on Github Actions on the list. I 
have just replaced Travis with Github Actions for Avatica-Go (CALCITE-3356).


For the workflows in Avatica-Go (currently just 1 for CI), see: 
https://github.com/apache/calcite-avatica-go/tree/master/.github/workflows


The latest run is here: 
https://github.com/apache/calcite-avatica-go/runs/226087805


The VMs provided by Github are a lot beefier (2 core CPUs, 7GB RAM, 14GB 
SSD disk). The tests are slightly slower compare to Travis (in the order 
of seconds), but this is most likely due to us having to check out the 
code during run time and the need to compile the Github Action for 
checking out the code for each run. I don't think this is too much of an 
issue and the direct integration with Github is much nicer.


I think it shouldn't be too difficult to move Avatica and Calcite over. 
We can probably get rid of Appveyor as well since Github Actions 
provides Windows and OS X nodes as well.


One thing that would be interesting would be to move the integration 
tests for Calcite to docker images and run each test for each backend in 
its own job to maximize parallelism and allow each job to maximize the 
available hardware resources.


Francis

[jira] [Created] (CALCITE-3356) Use Github Actions for continuous integration

[Francis Chuang (Jira)]

Francis Chuang created CALCITE-3356:
---

 Summary: Use Github Actions for continuous integration
 Key: CALCITE-3356
 URL: https://issues.apache.org/jira/browse/CALCITE-3356
 Project: Calcite
  Issue Type: Improvement
  Components: avatica-go
Reporter: Francis Chuang
Assignee: Francis Chuang
 Fix For: avatica-go-5.0.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3355) SQLCase nullability check

[Kirill Kozlov (Jira)]

Kirill Kozlov created CALCITE-3355:
--

 Summary: SQLCase nullability check
 Key: CALCITE-3355
 URL: https://issues.apache.org/jira/browse/CALCITE-3355
 Project: Calcite
  Issue Type: Bug
  Components: core
Affects Versions: 1.20.0
Reporter: Kirill Kozlov


When executing queries like: 
"SELECT COALESCE(name, 'unknown') as name_out FROM PCOLLECTION"
(input 'name' is nullable)

There is a need to know whether the result is nullable or not (in this case - 
not). During validation stage "COALESCE" is being transformed (via 
SqlValidatorImpl.performUnconditionalRewrites) into a "CASE" statement, which 
currently does not determine nullability of a result.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[ANNOUNCE] New committer: Muhammad Gelbana

[Francis Chuang]

Apache Calcite's Project Management Committee (PMC) has invited Muhammad 
Gelbana to become a committer, and we are pleased to announce that he 
has accepted.


Muhammad is an active contributor and has contributed numerous patches 
to Calcite. He has also been extremely active on the mailing list, 
helping out new users and participating in design discussions.


Muhammad, welcome, thank you for your contributions, and we look forward 
your further interactions with the community! If you wish, please feel 
free to tell us more about yourself and what you are working on.


Francis (on behalf of the Apache Calcite PMC)

[ANNOUNCE] New committer: Julian Feinauer

[Francis Chuang]

Apache Calcite's Project Management Committee (PMC) has invited Julian 
Feinauer to become a committer, and we are pleased to announce that he 
has accepted.


Julian is an active contributor to the Calcite code base and has been 
active on the mailing list answering questions, participating in 
discussions and voting for releases.


Julian, welcome, thank you for your contributions, and we look forward 
your further interactions with the community! If you wish, please feel 
free to tell us more about yourself and what you are working on.


Francis (on behalf of the Apache Calcite PMC)

Re: [DISCUSS] Release apache-calcite-1.21.0 (release candidate 1)

[Julian Hyde]

Stamatis,

Can you mark 1.21 as released in JIRA [1].

I don’t recall whether we ever made officially added this task to an RM’s 
responsibilities. I don’t mind doing it if you don’t have karma. 

Thank you, again, for being RM.

Julian

[1] 
https://issues.apache.org/jira/projects/CALCITE?selectedItem=com.atlassian.jira.jira-projects-plugin%3Arelease-page=released-unreleased
 


> On Sep 15, 2019, at 7:38 PM, Danny Chan  wrote:
> 
> Nice summary of the release problems, Stamatis ! Maybe we should put the 
> precautions to the website so that other RM can follow in the next releases ~
> 
> Best,
> Danny Chan
> 在 2019年9月12日 +0800 AM7:54，Stamatis Zampetakis ，写道：
>> The release process for apache-calcite-1.21.0 is now complete.
>> 
>> You can now commit again to the master.
>> 
>> Once again, I would like to thank again all the members of the community
>> that helped in getting 1.21.0 out the door. A special mention for the
>> reviewers who helped getting in some great features dedicating a
>> significant amount of their time.
>> 
>> During the vote various issues were raised.
>> 
>> Spurious .xml files were present in release candidate 0; I have the
>> impression that they came up due to the dry run that I did just before the
>> release (without performing a git clean). As it is not indicated in the
>> documentation I am thinking to add a small notice.
>> 
>> The build fails for certain locales due to CALCITE-2816; we agree that the
>> build problem must be solved for 1.22.0.
>> 
>> There are intermittent failures in the Pig module; CALCITE-3336 was logged
>> and we should continue further discussions there.
>> 
>> There were some small issues with the release notes (passive voice, and
>> backticks); I took care of them in
>> https://github.com/apache/calcite/commit/034bd7942c35d5b6c948dc6863a9a086fb82386c
>> .
>> 
>> There was a comment that adc1532de does not have tag calcite-1.21.0 but if
>> I am not looking at the wrong place I think its there.
>> 
>> The checksum hash that was communicated in the vote email was wrong; given
>> that the correct one was send along with the artifacts and people used this
>> for the checks I assume there is no problem.
>> 
>> There are some minor problems with README and README.md files; I will
>> update them in the following days.
>> 
>> The instructions in "Publishing a release" related with the release
>> announcement, and the Javadoc generation are slightly confusing. In order
>> to generate the Javadoc we must be in the tag calcite-1.21.0 and not in the
>> branch-1.21 and on the other hand the release announcement should be
>> committed in the branch-1.21. It may be worth adding a few more details
>> there.
>> 
>> If I missed something else worth mentioning feel free to include it.
>> 
>> Best,
>> Stamatis

Re: Query Compilation happening more often then expected

[Vladimir Sitnikov]

Stamatis>Out of curiosity does anybody know if popular DBMS (Postgres)
support "hoisting"?

PostgreSQL does support it, and here's a reproducible case when that
feature makes the query 300 times slower:
https://gist.github.com/vlsi/df08cbef370b2e86a5c1

Vladimir

Re: Query Compilation happening more often then expected

[Julian Hyde]

I’m just saying the approach isn’t perfect. And it isn’t implemented. If you 
are able to change your application to use bind variables, that is superior in 
every way.

You mention “a quick recheck of cost function”. This is a very nice use case 
for multi-objective parametric query optimization[1], which is a very elegant 
extension to the query optimization and which I think will gradually become the 
standard approach. The idea would be to generate plans assuming that the 
selectivity of a condition is a parameter varies between 0 and 1, and figure 
out at planning time what is the selectivity value at which plan A ceases to be 
the optimal plan and plan B becomes the optimal plan.

Julian

[1] 
https://cacm.acm.org/magazines/2017/10/221322-multi-objective-parametric-query-optimization/abstract
 


> On Sep 17, 2019, at 9:11 AM, Julian Feinauer  
> wrote:
> 
> Hi,
> 
> this is a good point Julian. So an implementation should consider a 
> re-planning (possibly triggered by a quick recheck of cost function with the 
> given Literal values). But this should not be a general issue with the 
> approach, or?
> 
> JulianF
> 
> Am 16.09.19, 23:36 schrieb "Julian Hyde" :
> 
>I found evidence that MSSQL[1] and Sybase ASE[2] do it.
> 
>I agree, it's not a free lunch. For instance, if a column has a
>non-uniform distribution, some values might be much more selective
>than others, and it would be much better to know which value you are
>dealing with at planning time, rather than execution time.
> 
>Julian
> 
>[1] 
> https://docs.microsoft.com/en-us/sql/relational-databases/performance/specify-query-parameterization-behavior-by-using-plan-guides?view=sql-server-2017
> 
>[2] 
> http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc00743.1570/html/queryprocessing/BIIIBEJJ.htm
> 
>On Mon, Sep 16, 2019 at 3:36 PM Stamatis Zampetakis  
> wrote:
>> 
>> Out of curiosity does anybody know if popular DBMS (Postgres, Oracle, SQL
>> Server, etc.) support "hoisting"?
>> 
>> Performing it all the time does not seem a very good idea (constant
>> reduction, histograms, and other optimization techniques would be
>> impossible)
>> while leaving its configuration to the end-user may not be a
>> straightforward decision.
>> 
>> On Sat, Sep 14, 2019 at 4:29 PM Julian Hyde  wrote:
>> 
>>> The idea of converting literals into bind variables is called “hoisting”.
>>> We had the idea a while ago but have not implemented it.
>>> 
>>> https://issues.apache.org/jira/browse/CALCITE-963
>>> 
>>> Until that feature is implemented, you will need to create bind variables
>>> explicitly, and bind them before executing the query.
>>> 
>>> Julian
>>> 
 On Sep 13, 2019, at 4:39 PM, Scott Reynolds 
>>> wrote:
 
 Hi,
 
 Spent a bunch of time researching and staring at code today to understand
 the code compilation path within Calcite. I started down this path
>>> because
 we noticed whenever we changed the `startDate` or `endDate` for the query
 it went through compilation process again. We expected it to use the
 previous classes `bind` it with the new RexLiterals. I was *hoping*  the
 RexLiterals were passed into the `bind()` method but that does not appear
 to be the main goal of `DataContext` objects.
 
 We also found the trick Kylin did to improve their query compilation with
 prepared statements:
 https://issues.apache.org/jira/browse/KYLIN-3434 but PreparedStatement
>>> is
 stateful and I don't believe a good way to solve this issue.
 
 I would like to propose a change to Calcite so that Filters are passed
>>> into
 the `bind()` call alongside or within DataContext. This would allow the
 `EnumerableRel` implementations to reference the `Filters` as arguments.
 This -- I believe -- would cause any change to the filters to use
 the previously compiled class instead of generating a brand new one.
 
 I am emailing everyone on this list for two reasons:
 1. Is this a bad idea ?
 2. I don't have a design yet so would love any ideas. Should we stick
>>> more
 stuff into `DataContext`? Should `EnumerableRel` have another method that
 is used to gather these RexLiterals?
>>> 
> 
> 

Re: Query Compilation happening more often then expected

[Julian Feinauer]

Hi,

this is a good point Julian. So an implementation should consider a re-planning 
(possibly triggered by a quick recheck of cost function with the given Literal 
values). But this should not be a general issue with the approach, or?

JulianF

Am 16.09.19, 23:36 schrieb "Julian Hyde" :

I found evidence that MSSQL[1] and Sybase ASE[2] do it.

I agree, it's not a free lunch. For instance, if a column has a
non-uniform distribution, some values might be much more selective
than others, and it would be much better to know which value you are
dealing with at planning time, rather than execution time.

Julian

[1] 
https://docs.microsoft.com/en-us/sql/relational-databases/performance/specify-query-parameterization-behavior-by-using-plan-guides?view=sql-server-2017

[2] 
http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc00743.1570/html/queryprocessing/BIIIBEJJ.htm

On Mon, Sep 16, 2019 at 3:36 PM Stamatis Zampetakis  
wrote:
>
> Out of curiosity does anybody know if popular DBMS (Postgres, Oracle, SQL
> Server, etc.) support "hoisting"?
>
> Performing it all the time does not seem a very good idea (constant
> reduction, histograms, and other optimization techniques would be
> impossible)
> while leaving its configuration to the end-user may not be a
> straightforward decision.
>
> On Sat, Sep 14, 2019 at 4:29 PM Julian Hyde  
wrote:
>
> > The idea of converting literals into bind variables is called 
“hoisting”.
> > We had the idea a while ago but have not implemented it.
> >
> > https://issues.apache.org/jira/browse/CALCITE-963
> >
> > Until that feature is implemented, you will need to create bind 
variables
> > explicitly, and bind them before executing the query.
> >
> > Julian
> >
> > > On Sep 13, 2019, at 4:39 PM, Scott Reynolds 
> > wrote:
> > >
> > > Hi,
> > >
> > > Spent a bunch of time researching and staring at code today to 
understand
> > > the code compilation path within Calcite. I started down this path
> > because
> > > we noticed whenever we changed the `startDate` or `endDate` for the 
query
> > > it went through compilation process again. We expected it to use the
> > > previous classes `bind` it with the new RexLiterals. I was *hoping*  
the
> > > RexLiterals were passed into the `bind()` method but that does not 
appear
> > > to be the main goal of `DataContext` objects.
> > >
> > > We also found the trick Kylin did to improve their query compilation 
with
> > > prepared statements:
> > > https://issues.apache.org/jira/browse/KYLIN-3434 but PreparedStatement
> > is
> > > stateful and I don't believe a good way to solve this issue.
> > >
> > > I would like to propose a change to Calcite so that Filters are passed
> > into
> > > the `bind()` call alongside or within DataContext. This would allow 
the
> > > `EnumerableRel` implementations to reference the `Filters` as 
arguments.
> > > This -- I believe -- would cause any change to the filters to use
> > > the previously compiled class instead of generating a brand new one.
> > >
> > > I am emailing everyone on this list for two reasons:
> > > 1. Is this a bad idea ?
> > > 2. I don't have a design yet so would love any ideas. Should we stick
> > more
> > > stuff into `DataContext`? Should `EnumerableRel` have another method 
that
> > > is used to gather these RexLiterals?
> >

[jira] [Created] (CALCITE-3354) RelToSqlConverter does not support LogicalTableFunctionScan

[Anton Haidai (Jira)]

Anton Haidai created CALCITE-3354:
-

 Summary: RelToSqlConverter does not support 
LogicalTableFunctionScan
 Key: CALCITE-3354
 URL: https://issues.apache.org/jira/browse/CALCITE-3354
 Project: Calcite
  Issue Type: Bug
Affects Versions: 1.21.0
Reporter: Anton Haidai


Here is the test in *RelBuilderTest* to reproduce the isse (the right place for 
this test is RelToSqlConverterTest, but it does not have test table functions 
configured):
{code}
  @Test
  public void testFailToConvertLogicalTableFunctionScanToSql() {
final RelBuilder builder = RelBuilder.create(config().build());
final SqlOperator rampFunction = new MockSqlOperatorTable.RampFunction();
RelNode root = builder.functionScan(rampFunction, 0, builder.literal(3))
.build();
toSql(root, CalciteSqlDialect.DEFAULT);
  }

  private static String toSql(RelNode root, SqlDialect dialect) {
final RelToSqlConverter converter = new RelToSqlConverter(dialect);
final SqlNode sqlNode = converter.visitChild(0, root).asStatement();
return sqlNode.toSqlString(dialect).getSql();
  }
{code}

Result: 
{code}
java.lang.RuntimeException: While invoking method 'public 
org.apache.calcite.rel.rel2sql.SqlImplementor$Result 
org.apache.calcite.rel.rel2sql.RelToSqlConverter.visit(org.apache.calcite.rel.RelNode)'

at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:527)
at 
org.apache.calcite.rel.rel2sql.RelToSqlConverter.dispatch(RelToSqlConverter.java:122)
at 
org.apache.calcite.rel.rel2sql.RelToSqlConverter.visitChild(RelToSqlConverter.java:128)
at org.apache.calcite.test.RelBuilderTest.toSql(RelBuilderTest.java:331)
at 
org.apache.calcite.test.RelBuilderTest.testFailToConvertLogicalTableFunctionScanToSql(RelBuilderTest.java:326)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at 
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524)
... 26 more
Caused by: java.lang.AssertionError: Need to implement 
org.apache.calcite.rel.logical.LogicalTableFunctionScan
at 
org.apache.calcite.rel.rel2sql.RelToSqlConverter.visit(RelToSqlConverter.java:136)
... 31 more
{code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (CALCITE-3353) ProjectJoinTransposeRule caused AssertionError when creating a new Join

[TANG Wen-hui (Jira)]

TANG Wen-hui created CALCITE-3353:
-

 Summary: ProjectJoinTransposeRule caused AssertionError when 
creating a new Join
 Key: CALCITE-3353
 URL: https://issues.apache.org/jira/browse/CALCITE-3353
 Project: Calcite
  Issue Type: Bug
  Components: core
Affects Versions: 1.20.0
Reporter: TANG Wen-hui
Assignee: TANG Wen-hui


Trying to add ProjectJoinTransposeRule to the rule set to optimize sql (using 
VolcanoPlanner).

Get the following exception：

Caused by: java.lang.AssertionErrorCaused by: java.lang.AssertionError at 
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.(EnumerableMergeJoin.java:72)
 at 
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:112)
 at 
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:59)
 at 
org.apache.calcite.rel.rules.ProjectJoinTransposeRule.onMatch(ProjectJoinTransposeRule.java:155)
 at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
 at 
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:703)
 at org.apache.calcite.tools.Programs.lambda$standard$7(Programs.java:466) at 
org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:528) at 
org.apache.calcite.prepare.Prepare.optimize(Prepare.java:196) at 
org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:416) at 
org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:310) at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:830)
 at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:610)
 at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:580)
 at 
org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:247)
 at 
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:711)
 
... 22 more

This is because the ProjectJoinTransposeRule can match multiple types of Join 
and ProjectJoinTransposeRule creates a new Join using the join.copy() method. 
When the rule applied to EnumerableMergeJoin, and the new Join created by this 
rule may not meet the requirements of EnumerableMergeJoin.

{code:java}
  EnumerableMergeJoin(
  RelOptCluster cluster,
  RelTraitSet traits,
  RelNode left,
  RelNode right,
  RexNode condition,
  Set variablesSet,
  JoinRelType joinType) {
super(cluster, traits, left, right, condition, variablesSet, joinType);
final List collations =
traits.getTraits(RelCollationTraitDef.INSTANCE);
assert collations == null || RelCollations.contains(collations, 
joinInfo.leftKeys);
  }
{code}




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (CALCITE-3352) ProjectToWindowRule sets wrong collation on generated Window

[Pavel Gubin (Jira)]

Pavel Gubin created CALCITE-3352:


 Summary: ProjectToWindowRule sets wrong collation on generated 
Window
 Key: CALCITE-3352
 URL: https://issues.apache.org/jira/browse/CALCITE-3352
 Project: Calcite
  Issue Type: Bug
  Components: core
Affects Versions: 1.21.0
Reporter: Pavel Gubin


{{ProjectToWindowRule}} sets the collocation of the {{Project}} it applies to 
on the generated {{LogicalWindow}} which is wrong, because the collation of the 
input can be different and Window doesn't change the collation. So the 
collation of the generated {{LogicalWindow}} should be the same as on the input 
(not the output).

The following test in {{RelOptRulesTest}} reproduces the issue:
{code:java}
  @Test public void testWindowOnSortedInput() {
// Create a customized test with RelCollation trait in the test cluster.
Tester tester = new TesterImpl(getDiffRepos(), true, true, false, false,
true, null, null) {
  @Override public RelOptPlanner createPlanner() {
return new MockRelOptPlanner(Contexts.empty()) {
  @Override public List getRelTraitDefs() {
return ImmutableList.of(RelCollationTraitDef.INSTANCE);
  }
  @Override public RelTraitSet emptyTraitSet() {
return RelTraitSet.createEmpty().plus(
RelCollationTraitDef.INSTANCE.getDefault());
  }
};
  }
};

final HepProgram preProgram = new HepProgramBuilder()
.addRuleInstance(SortProjectTransposeRule.INSTANCE)
.build();
final HepProgram program = HepProgram.builder()
.addRuleInstance(ProjectToWindowRule.PROJECT)
.build();

final String sql = "select mgr, deptno, sum(sal) over (partition by 
deptno)\n"
+ "from emp\n"
+ "order by mgr, deptno";

RelNode r = checkPlanning(tester, preProgram, new HepPlanner(program), sql);
RelCollation c = 
r.getInput(0).getTraitSet().getTrait(RelCollationTraitDef.INSTANCE);
assertEquals("Collation is incorrect", "[3, 7]", c.toString());
  }
{code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)