[RESULT] [VOTE] Release Apache Calcite 1.28.0 (release candidate 0)

[Julian Hyde]

Thanks to everyone who has tested the release candidate and given
their comments and votes.

The tally is as follows.

6 binding +1s:
Francis Chuang, Haisheng Yuan, Jacques Nadeau, Julian Hyde, Ruben Q L, Vladimir 
Sitnikov

4 non-binding +1s:
Xiong Duan, Chunwei Lei, Sergey Nuyanzin, Zhaohui Xu

No 0s or -1s.

(As you're probably aware, only PMC members' votes are binding;
committers' and other contributors' votes are non-binding but are
much appreciated, because everyone can help ensure the quality
of the release.)

Therefore, I am delighted to announce that the proposal to release
Apache Calcite 1.28.0 has passed.

Thanks everyone. We’ll now roll the release out to the mirrors, and
I'll make an official announcement tomorrow.

Master branch will be open in 2 hours, after I have pushed out the new
web site, including fixes to the release notes requested during the vote.
 
Julian


On 2021/10/17 22:25:49, Francis Chuang  wrote: 
> My vote is: +1 (binding)
> 
> - Verified GPG signature - OK
> - Verified SHA512 - OK
> - Checked release notes 
> (https://github.com/apache/calcite/blob/calcite-1.28.0-rc0/site/_docs/history.md)
>  
> - OK
> - Ran tests (gradle check) - OK
> - Spot checked Nexus artifacts - OK
> 
> Environment:
> Gradle 6.9.1 docker container in WSL2 (Ubuntu 20.04) on Windows 10 21h1
> 
>  > docker version
> Client: Docker Engine - Community
>   Cloud integration: 1.0.17
>   Version:   20.10.8
>   API version:   1.41
>   Go version:go1.16.6
>   Git commit:3967b7d
>   Built: Fri Jul 30 19:54:22 2021
>   OS/Arch:   linux/amd64
>   Context:   default
>   Experimental:  true
> 
> Server: Docker Engine - Community
>   Engine:
>Version:  20.10.8
>API version:  1.41 (minimum version 1.12)
>Go version:   go1.16.6
>Git commit:   75249d8
>Built:Fri Jul 30 19:52:31 2021
>OS/Arch:  linux/amd64
>Experimental: false
>   containerd:
>Version:  1.4.9
>GitCommit:e25210fe30a0a703442421b0f60afac609f950a3
>   runc:
>Version:  1.0.1
>GitCommit:v1.0.1-0-g4144b63
>   docker-init:
>Version:  0.19.0
>GitCommit:de40ad0
> 
>  > gradle -v
> 
> 
> Gradle 6.9.1
> 
> 
> Build time:   2021-08-20 11:15:18 UTC
> Revision: f0ddb54aaae0e44f0a7209c3c0274d506ea742a0
> 
> Kotlin:   1.4.20
> Groovy:   2.5.12
> Ant:  Apache Ant(TM) version 1.10.9 compiled on September 27 2020
> JVM:  1.8.0_292 (AdoptOpenJDK 25.292-b10)
> OS:   Linux 5.10.16.3-microsoft-standard-WSL2 amd64
> 
>  > java -version
> openjdk version "1.8.0_292"
> OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
> OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)
> 
> Francis
> 
> On 16/10/2021 5:01 am, Julian Hyde wrote:
> > Hi all,
> > 
> > I have created a build for Apache Calcite 1.28.0, release
> > candidate 0.
> > 
> > Thanks to everyone who has contributed to this release.
> > 
> > You can read the release notes here:
> > https://github.com/apache/calcite/blob/calcite-1.28.0-rc0/site/_docs/history.md
> > 
> > The commit to be voted upon:
> > https://gitbox.apache.org/repos/asf?p=calcite.git;a=commit;h=dec167ac18272c0cd8be477d6b162d7a31a62114
> > 
> > Its hash is dec167ac18272c0cd8be477d6b162d7a31a62114
> > 
> > Tag:
> > https://gitbox.apache.org/repos/asf?p=calcite.git;a=tag;h=refs/tags/calcite-1.28.0-rc0
> > 
> > The artifacts to be voted on are located here:
> > https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.28.0-rc0
> > (revision 50450)
> > 
> > The hashes of the artifacts are as follows:
> > d61c4935d7d3b66425ff6e0c5e6e506b56f734d562f0fc6e69dc1d484da1d92d7d922930fd7b545abe0dc7b0178f6847ff4239ad52c52c8fa155668aa3d3fc38
> > *apache-calcite-1.28.0-src.tar.gz
> > 
> > A staged Maven repository is available for review at:
> > https://repository.apache.org/content/repositories/orgapachecalcite-1121/org/apache/calcite/
> > 
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/jhyde.asc
> > https://www.apache.org/dist/calcite/KEYS
> > 
> > Please vote on releasing this package as Apache Calcite 1.28.0.
> > 
> > The vote is open for the next 72 hours and passes if a majority of at
> > least three +1 PMC votes are cast.
> > 
> > [ ] +1 Release this package as Apache Calcite 1.28.0
> > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> > [ ] -1 Do not release this package because...
> > 
> > Here is my vote:
> > 
> > +1 (binding)
> > 
> 

Re: [MongoDB Adapter] How to filter collection based on field in deep nested data

[Julian Hyde]

I believe that the JSON_EXISTS function [1] can do this kind of filtering, and 
Calcite supports it [2].

Julian

[1] 
https://docs.oracle.com/en/database/oracle/oracle-database/12.2/adjsn/condition-JSON_EXISTS.html#GUID-8A0043D5-95F8-4918-9126-F86FB0E203F0
 

 

[2] https://calcite.apache.org/docs/reference.html#json-functions 
 


> On Oct 17, 2021, at 6:45 AM, Justin Huang <121760...@qq.com.INVALID> wrote:
> 
> Hi Calcite developers,
> 
> 
> As we know, MongoDB can store complex JSON objects with deep nested 
> object/array data. I'd like to know whether calcite SQL parser allow filter 
> condition with hierarchical element reference like this?
> 
> 
>select * from doc where key0.key1.key2[0].key3[0] 
>  1
> 
> 
> Here is an example entry of the collection:
> 
> 
> {
> 
>  "key0": {
> 
>   "key1": {
> 
>"key2": [
> 
> {
> 
>  "key3": [
> 
>   1,
> 
>   2,
> 
>   3
> 
>  ]
> 
> }
> 
>]
> 
>   }
> 
>  }
> 
> }
> 
> 
> 
> BTW, I see that there are some commercial tools like Studio 3T has good SQL 
> query capability, not sure if the SQL syntax is their own or some standard 
> SQL dialect.
> 
> 
> 
> 
> Thanks,
> Justin

[jira] [Created] (CALCITE-4859) Tableau connector for Calcite

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4859:


 Summary: Tableau connector for Calcite
 Key: CALCITE-4859
 URL: https://issues.apache.org/jira/browse/CALCITE-4859
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


Create a Tableau connector for Calcite. This will allow Tableau to connect to a 
Calcite server via Avatica.

The format will be a directory of files, similar to [Tableau's Postgres JDBC 
example|https://github.com/tableau/connector-plugin-sdk/tree/master/samples/plugins/postgres_jdbc].
 Other projects that use Calcite/Avatica will be able to fork this connector 
and customize it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release Apache Calcite 1.28.0 (release candidate 0)

[Julian Hyde]

Since this vote overlaps a weekend, I'll give you all some extra time.
The vote will last for 96 hours (or until there is a majority of 3).
It will end at 12 noon pacific (1900 UTC) on Tuesday 19th October.
Still, early votes are appreciated.

Julian


On Fri, Oct 15, 2021 at 11:01 AM Julian Hyde  wrote:
>
> Hi all,
>
> I have created a build for Apache Calcite 1.28.0, release
> candidate 0.
>
> Thanks to everyone who has contributed to this release.
>
> You can read the release notes here:
> https://github.com/apache/calcite/blob/calcite-1.28.0-rc0/site/_docs/history.md
>
> The commit to be voted upon:
> https://gitbox.apache.org/repos/asf?p=calcite.git;a=commit;h=dec167ac18272c0cd8be477d6b162d7a31a62114
>
> Its hash is dec167ac18272c0cd8be477d6b162d7a31a62114
>
> Tag:
> https://gitbox.apache.org/repos/asf?p=calcite.git;a=tag;h=refs/tags/calcite-1.28.0-rc0
>
> The artifacts to be voted on are located here:
> https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.28.0-rc0
> (revision 50450)
>
> The hashes of the artifacts are as follows:
> d61c4935d7d3b66425ff6e0c5e6e506b56f734d562f0fc6e69dc1d484da1d92d7d922930fd7b545abe0dc7b0178f6847ff4239ad52c52c8fa155668aa3d3fc38
> *apache-calcite-1.28.0-src.tar.gz
>
> A staged Maven repository is available for review at:
> https://repository.apache.org/content/repositories/orgapachecalcite-1121/org/apache/calcite/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jhyde.asc
> https://www.apache.org/dist/calcite/KEYS
>
> Please vote on releasing this package as Apache Calcite 1.28.0.
>
> The vote is open for the next 72 hours and passes if a majority of at
> least three +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Calcite 1.28.0
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
> Here is my vote:
>
> +1 (binding)

[VOTE] Release Apache Calcite 1.28.0 (release candidate 0)

[Julian Hyde]

Hi all,

I have created a build for Apache Calcite 1.28.0, release
candidate 0.

Thanks to everyone who has contributed to this release.

You can read the release notes here:
https://github.com/apache/calcite/blob/calcite-1.28.0-rc0/site/_docs/history.md

The commit to be voted upon:
https://gitbox.apache.org/repos/asf?p=calcite.git;a=commit;h=dec167ac18272c0cd8be477d6b162d7a31a62114

Its hash is dec167ac18272c0cd8be477d6b162d7a31a62114

Tag:
https://gitbox.apache.org/repos/asf?p=calcite.git;a=tag;h=refs/tags/calcite-1.28.0-rc0

The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.28.0-rc0
(revision 50450)

The hashes of the artifacts are as follows:
d61c4935d7d3b66425ff6e0c5e6e506b56f734d562f0fc6e69dc1d484da1d92d7d922930fd7b545abe0dc7b0178f6847ff4239ad52c52c8fa155668aa3d3fc38
*apache-calcite-1.28.0-src.tar.gz

A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachecalcite-1121/org/apache/calcite/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/jhyde.asc
https://www.apache.org/dist/calcite/KEYS

Please vote on releasing this package as Apache Calcite 1.28.0.

The vote is open for the next 72 hours and passes if a majority of at
least three +1 PMC votes are cast.

[ ] +1 Release this package as Apache Calcite 1.28.0
[ ]  0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...

Here is my vote:

+1 (binding)

[jira] [Created] (CALCITE-4856) Gradle prepareVote fails with 'not authorized'

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4856:


 Summary: Gradle prepareVote fails with 'not authorized'
 Key: CALCITE-4856
 URL: https://issues.apache.org/jira/browse/CALCITE-4856
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


Gradle {{prepareVote}} fails with 'not authorized' error.

{noformat}
$ ./gradlew prepareVote -Prc=0 -Pasf

> Configure project :
Building Apache Calcite 1.28.0

> Task :gitProps
Execution optimizations have been disabled for task ':gitProps' to ensure 
correctness due to the following reasons:
  - Gradle detected a problem with the following location: 
'/home/jhyde/dev/release/calcite'. Reason: Task ':gitProps' uses this output of 
task ':babel:fmppMain' without declaring an explicit or implicit dependency. 
This can lead to incorrect results being produced, depending on what order the 
tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
...
> Task :pushRcTag
Pushing tag to Git remote release-origin: https://github.com/apache/calcite.git

> Task :pushRcTag FAILED

Build calcite FAILURE reason:
Execution failed for task ':pushRcTag':
Caused by: org.eclipse.jgit.api.errors.TransportException: 
https://github.com/apache/calcite.git: not authorized
at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:180)
at 
com.github.vlsi.gradle.release.jgit.dsl.GitExtensionsKt.push(GitExtensions.kt:132)
at 
com.github.vlsi.gradle.release.GitPushTask$pushTag$1.invoke(GitPushTask.kt:54)
at 
com.github.vlsi.gradle.release.GitPushTask$pushTag$1.invoke(GitPushTask.kt:30)
at 
com.github.vlsi.gradle.release.DefaultGitTask.jgit(DefaultGitTask.kt:45)
at 
com.github.vlsi.gradle.release.GitPushTask.pushTag(GitPushTask.kt:50)
at 
org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
at 
org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
...
at 
org.eclipse.jgit.transport.Transport.push(Transport.java:1368)
at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:170)
... 89 more


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':pushRcTag'.
> org.eclipse.jgit.api.errors.TransportException: 
> https://github.com/apache/calcite.git: not authorized

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

See 
https://docs.gradle.org/7.2/userguide/command_line_interface.html#sec:command_line_warnings

Execution optimizations have been disabled for 29 invalid unit(s) of work 
during this build to ensure correctness.
Please consult deprecation warnings for more details.

BUILD FAILED in 2m 32s
268 actionable tasks: 187 executed, 20 from cache, 61 up-to-date
S3 cache 5s wasted (3s wasted on hits, 2s wasted on misses), reads: 43, hits: 
20, elapsed: 5s, received: 8 KiB

{noformat}

I had tried previously without the {{-Pasf}} flag but that ran into other 
errors, sooner in the process, that seemed to be about my environment. I 
concluded that the dry-run was a waste of time.

I'm not sure why the command is trying to push to github rather than gitbox.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Next releases

[Julian Hyde]

I'm ready to produce the first release candidate, but I am blocked by
a Gradle issue. Vladimir, I'd be grateful if you could take a look.

https://issues.apache.org/jira/browse/CALCITE-4853

Julian

On Thu, Oct 14, 2021 at 10:37 AM Julian Hyde  wrote:
>
> I have written a first draft of the release notes and opened a pull
> request. Can people please review the release notes and check
> contributor names.
>
> https://github.com/julianhyde/calcite/blob/4835-release-1.28/site/_docs/history.md
>
> https://github.com/apache/calcite/pull/2584
>
> Master branch is still open for commits, but please be careful not to
> break the build. I'll probably start a vote today or tomorrow.
>
> On Sat, Oct 9, 2021 at 11:24 PM Julian Hyde  wrote:
> >
> > Now that Avatica’s release 1.19 is final, the master branch is open for 
> > commits. I’ll make the official release announcement on Monday.
> >
> > I’m targeting Wednesday for the first release candidate for Calcite version 
> > 1.28.
> >
> > Julian
> >
> >
> > > On Oct 7, 2021, at 12:32 PM, Jacques Nadeau  wrote:
> > >
> > > I merged the last patch and have linked the release ticket to a release
> > > note gist for the changes. Thanks.
> > >
> > > On Wed, Oct 6, 2021, 10:21 PM Jacques Nadeau  wrote:
> > >
> > >> FYI, the last patch for immutables is up and passing all checks. I'll do
> > >> another pass in the morning to make sure everything looks good and then
> > >> will merge if I don't hear any concerns.
> > >>
> > >> https://github.com/apache/calcite/pull/2568
> > >>
> > >>
> > >> On Wed, Oct 6, 2021, 1:04 PM Jacques Nadeau  wrote:
> > >>
> > >>> Definitely. I'm also trying to add helpful comments in a couple of key
> > >>> javadoc locations.
> > >>>
> > >>> On Wed, Oct 6, 2021 at 1:02 PM Julian Hyde 
> > >>> wrote:
> > >>>
> > >>>> Thanks for following through on this large change, Jacques. Be sure to
> > >>>> document the changes people will need to make if they have defined 
> > >>>> their
> > >>>> own rules.
> > >>>>
> > >>>>
> > >>>>> On Oct 6, 2021, at 12:59 PM, Jacques Nadeau 
> > >>>> wrote:
> > >>>>>
> > >>>>> I'm hoping (fingers crossed) to have the final Immutables patch up
> > >>>> today
> > >>>>> and if all goes well, get that merged before the release.
> > >>>>>
> > >>>>> On Wed, Oct 6, 2021 at 11:34 AM Julian Hyde  wrote:
> > >>>>>
> > >>>>>> So, the vote for Avatica 1.19 is underway, and we will likely have a
> > >>>>>> release in 4 or 5 days.
> > >>>>>>
> > >>>>>> Next up, Calcite release 1.28. I'll be release manager for that too,
> > >>>> and
> > >>>>>> I'd like to start the vote in a week. I have three asks:
> > >>>>>>
> > >>>>>> 1. Start stabilizing the build. If you have a change that might cause
> > >>>>>> build instability, ask on the dev list before you commit it. The
> > >>>> recent
> > >>>>>> changes for Gradle, JDK 16/17 and Immutables have been destabilizing,
> > >>>> so I
> > >>>>>> ask those authors
> > >>>>>>
> > >>>>>> 2. I have logged https://issues.apache.org/jira/browse/CALCITE-4835.
> > >>>> If
> > >>>>>> you have made breaking changes since 1.27, please document them in
> > >>>> that
> > >>>>>> case, and I will add them to the release notes. If there are bugs
> > >>>> that MUST
> > >>>>>> be fixed in 1.28, link to them from that case.
> > >>>>>>
> > >>>>>> 3. Can I have a few volunteers to go through the PRs and merge any
> > >>>> that
> > >>>>>> are ready? (Please reply to this email, so we know you are working on
> > >>>> it.)
> > >>>>>>
> > >>>>>> Julian
> > >>>>>>
> > >>>>>>
> > >>>>>> On 2021/09/27 19:21:51, Alessandro Solimando <
> > >>>>>> alessandro.solima...@gmail.c

[jira] [Created] (CALCITE-4853) Gradle could not determine the dependencies of task ':syncPreviewSiteRepo'

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4853:


 Summary: Gradle could not determine the dependencies of task 
':syncPreviewSiteRepo'
 Key: CALCITE-4853
 URL: https://issues.apache.org/jira/browse/CALCITE-4853
 Project: Calcite
  Issue Type: Bug
  Components: build
Reporter: Julian Hyde


I got the following error while following the instructions to [make a Calcite 
release 
candidate|https://calcite.apache.org/docs/howto.html#making-a-release-candidate].

{noformat}
$ ./gradlew prepareVote -Prc=0
Starting a Gradle Daemon, 1 incompatible Daemon could not be reused, use 
--status for details

> Configure project :
Building Apache Calcite 1.28.0

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine the dependencies of task ':syncPreviewSiteRepo'.
> Could not resolve all dependencies for configuration ':previewSite'.
   > Querying the mapped value of flatmap(provider(task 'distTar', class 
org.gradle.api.tasks.bundling.Tar)) before task ':release:distTar' has 
completed is not supported

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Draft board report for Oct 2021

[Julian Hyde]

The contributors are the people who have made commits that appear in
this release. You can generate the list as follows:

$ ./sqlsh "select distinct author from git_commits where
commit_timestamp > DATE '2021-06-03' order by 1 "


On Thu, Oct 14, 2021 at 9:57 AM Michael Mior  wrote:
>
> LGTM, although we have new committers to add now.
> --
> Michael Mior
> mm...@apache.org
>
> Le mer. 6 oct. 2021 à 03:15, Haisheng Yuan  a écrit :
> >
> > Attached below is a draft of this month's board report. I plan to submit it 
> > on Oct 12. Please let me know if you have additions or corrections.
> >
> > ## Description:
> > The mission of Calcite is the creation and maintenance of software related 
> > to
> > Dynamic data management framework
> >
> > ## Issues:
> > There are no issues requiring board attention.
> >
> > ## Membership Data:
> > Apache Calcite was founded 2015-10-21 (6 years ago)
> > There are currently 53 committers and 23 PMC members in this project.
> > The Committer-to-PMC ratio is roughly 7:3.
> >
> > Community changes, past quarter:
> > - No new PMC members. Last addition was Ruben Q L on 2020-08-09.
> > - No new committers. Last addition was Vladimir Ozerov on 2021-06-23.
> >
> > ## Project Activity:
> > No new version was released.
> >
> > At ApacheCon 2021, September 22, 2021, Vladimir Ozerov gave a talk on 
> > building
> > modern SQL query optimizers with Apache Calcite.
> >
> > At Strange Loop 2021, September 30, 2021, Julian Hyde gave a talk on Morel,
> > a functional query language.
> >
> > ## Community Health:
> > The overall activity in the community has increased slightly in the past
> > few months, specifically 47% more opened PRs, 25% more closed PRs on GitHub.
> > But the number of active reviewers still remains small.
> >
> > Thanks,
> > Haisheng Yuan

Re: [DISCUSS] Next releases

[Julian Hyde]

I have written a first draft of the release notes and opened a pull
request. Can people please review the release notes and check
contributor names.

https://github.com/julianhyde/calcite/blob/4835-release-1.28/site/_docs/history.md

https://github.com/apache/calcite/pull/2584

Master branch is still open for commits, but please be careful not to
break the build. I'll probably start a vote today or tomorrow.

On Sat, Oct 9, 2021 at 11:24 PM Julian Hyde  wrote:
>
> Now that Avatica’s release 1.19 is final, the master branch is open for 
> commits. I’ll make the official release announcement on Monday.
>
> I’m targeting Wednesday for the first release candidate for Calcite version 
> 1.28.
>
> Julian
>
>
> > On Oct 7, 2021, at 12:32 PM, Jacques Nadeau  wrote:
> >
> > I merged the last patch and have linked the release ticket to a release
> > note gist for the changes. Thanks.
> >
> > On Wed, Oct 6, 2021, 10:21 PM Jacques Nadeau  wrote:
> >
> >> FYI, the last patch for immutables is up and passing all checks. I'll do
> >> another pass in the morning to make sure everything looks good and then
> >> will merge if I don't hear any concerns.
> >>
> >> https://github.com/apache/calcite/pull/2568
> >>
> >>
> >> On Wed, Oct 6, 2021, 1:04 PM Jacques Nadeau  wrote:
> >>
> >>> Definitely. I'm also trying to add helpful comments in a couple of key
> >>> javadoc locations.
> >>>
> >>> On Wed, Oct 6, 2021 at 1:02 PM Julian Hyde 
> >>> wrote:
> >>>
> >>>> Thanks for following through on this large change, Jacques. Be sure to
> >>>> document the changes people will need to make if they have defined their
> >>>> own rules.
> >>>>
> >>>>
> >>>>> On Oct 6, 2021, at 12:59 PM, Jacques Nadeau 
> >>>> wrote:
> >>>>>
> >>>>> I'm hoping (fingers crossed) to have the final Immutables patch up
> >>>> today
> >>>>> and if all goes well, get that merged before the release.
> >>>>>
> >>>>> On Wed, Oct 6, 2021 at 11:34 AM Julian Hyde  wrote:
> >>>>>
> >>>>>> So, the vote for Avatica 1.19 is underway, and we will likely have a
> >>>>>> release in 4 or 5 days.
> >>>>>>
> >>>>>> Next up, Calcite release 1.28. I'll be release manager for that too,
> >>>> and
> >>>>>> I'd like to start the vote in a week. I have three asks:
> >>>>>>
> >>>>>> 1. Start stabilizing the build. If you have a change that might cause
> >>>>>> build instability, ask on the dev list before you commit it. The
> >>>> recent
> >>>>>> changes for Gradle, JDK 16/17 and Immutables have been destabilizing,
> >>>> so I
> >>>>>> ask those authors
> >>>>>>
> >>>>>> 2. I have logged https://issues.apache.org/jira/browse/CALCITE-4835.
> >>>> If
> >>>>>> you have made breaking changes since 1.27, please document them in
> >>>> that
> >>>>>> case, and I will add them to the release notes. If there are bugs
> >>>> that MUST
> >>>>>> be fixed in 1.28, link to them from that case.
> >>>>>>
> >>>>>> 3. Can I have a few volunteers to go through the PRs and merge any
> >>>> that
> >>>>>> are ready? (Please reply to this email, so we know you are working on
> >>>> it.)
> >>>>>>
> >>>>>> Julian
> >>>>>>
> >>>>>>
> >>>>>> On 2021/09/27 19:21:51, Alessandro Solimando <
> >>>>>> alessandro.solima...@gmail.com> wrote:
> >>>>>>> Hi everyone,
> >>>>>>> I have prepared the PR, you can find it here:
> >>>>>>> https://github.com/apache/calcite-avatica/pull/154.
> >>>>>>>
> >>>>>>> Best regards,
> >>>>>>> Alessandro
> >>>>>>>
> >>>>>>> On Mon, 27 Sept 2021 at 20:27, Alessandro Solimando <
> >>>>>>> alessandro.solima...@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> Hi Julian,
> >>>>>>>> I can help with PR-143
> >>>>>>>>

Downloads page

[Julian Hyde]

Can someone please volunteer to fix
https://issues.apache.org/jira/browse/CALCITE-4639, to fix broken
links on the downloads page. Because this case concerns the web site,
it doesn't need to be fixed BEFORE 1.28 but it needs to be fixed VERY
SOON AFTERWARDS.

Julian

Re: Request Update the Jirs ISSUE lABEL

[Julian Hyde]

Done; see https://issues.apache.org/jira/projects/CALCITE/versions/12350203.

On Tue, Oct 12, 2021 at 2:47 PM xiong duan  wrote:

> As we know. The Avatica-1.19.0 has been released. So the label needs to
> update.
>
> [image: 截屏2021-10-13 05.42.04.png]
>

Re: Disable the subquery

[Julian Hyde]

Have you considered disabling JoinConditionPushRule?

> On Oct 12, 2021, at 12:26 AM, JiaTao Tao  wrote:
> 
> Do you use the RelToSqlConverter to  convert plan to sql?
> 
> Regards!
> 
> Aron Tao
> 
> 
> luoc  于2021年10月11日周一 下午4:30写道：
> 
>> Hello Calcite team,
>> 
>> 
>>  I am going to develop a new feature on my project with the Calcite.
>> But got the following issues :
>> 
>> 
>> Input SQL :
>> 
>> 
>> SELECT A.N_NAME, B.R_NAME FROM V1.NATION A LEFT JOIN V1.REGION B ON
>> A.N_REGIONKEY = B.R_REGIONKEY WHERE B.R_NAME = 'ASIA';
>> 
>> 
>> Actual SQL (send to DB) :
>> 
>> 
>> SELECT "NATION0"."N_NAME" AS "n_name", "t"."R_NAME" AS "r_name"
>> FROM "V1"."NATION" AS "NATION0"
>> INNER JOIN (SELECT *
>> FROM "V1"."REGION"
>> WHERE "R_NAME" = 'ASIA') AS "t" ON "NATION0"."N_REGIONKEY" =
>> "t"."R_REGIONKEY";
>> 
>> 
>> If possible, How can I disable the join auto-convert to the subquery
>> syntax? Because the backend DB does not support the subquery (but the `left
>> join` is supported). Add the specified rule or rewrite the `JdbcJoinRule`
>> class ?
>> 
>> 
>> Thanks for your time.

[jira] [Created] (CALCITE-4847) Parse SQL with BigQuery-style quoted identifiers and character literals

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4847:


 Summary: Parse SQL with BigQuery-style quoted identifiers and 
character literals
 Key: CALCITE-4847
 URL: https://issues.apache.org/jira/browse/CALCITE-4847
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


Parse SQL with BigQuery-style quoted identifiers and character literals.

BigQuery quotes identifiers using backticks, escaping interior backticks using 
backslash. In CALCITE-4767 we added {{Quoting.BACK_TICK_BACKSLASH}} to 
distinguish this style from what MySQL does, namely {{Quoting.BACK_TICK}}.

BigQuery quotes character literals using double quotes, escaping interior 
double quotes using backslash. In CALCITE-4767 we added 
{{Quoting.DOUBLE_QUOTE_BACKSLASH}} to distinguish this style from 
{{Quoting.DOUBLE_QUOTE}}.

After this change, we should be able to parse the following query if we invoke 
the parser with {{lex=BIG_QUERY}} or {{dialect=BIG_QUERY}}:
{code}
SELECT "a \"quoted\" char literal"
FROM `a \`quoted\` table`
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[ANNOUNCE] Apache Calcite Avatica 1.19.0 released

[Julian Hyde]

The Apache Calcite team is pleased to announce the release of Apache
Calcite Avatica 1.19.0.

Avatica is a framework for building database drivers. Avatica defines a
wire API and serialization mechanism for clients to communicate with a
server as a proxy to a database. The reference Avatica client and server
are implemented in Java and communicate over HTTP. Avatica is a
sub-project of Apache Calcite.

Apache Calcite Avatica 1.19.0 adds support for BIT and NULL data types,
fixes issues with values of type ARRAY, and includes a few dependency
updates. For a full list of changes, please see the release notes:

   https://calcite.apache.org/avatica/docs/history.html#v1-19-0

The release is available here:

   https://calcite.apache.org/avatica/downloads/avatica.html

We welcome your help and feedback. For more information on how to report
problems and get involved, visit the project website at:

https://calcite.apache.org/avatica/

or the Apache Calcite project website:

https://calcite.apache.org/

Thanks to everyone involved!

Julian Hyde, on behalf of the Apache Calcite team

Re: [DISCUSS] Next releases

[Julian Hyde]

Now that Avatica’s release 1.19 is final, the master branch is open for 
commits. I’ll make the official release announcement on Monday.

I’m targeting Wednesday for the first release candidate for Calcite version 
1.28. 

Julian
 

> On Oct 7, 2021, at 12:32 PM, Jacques Nadeau  wrote:
> 
> I merged the last patch and have linked the release ticket to a release
> note gist for the changes. Thanks.
> 
> On Wed, Oct 6, 2021, 10:21 PM Jacques Nadeau  wrote:
> 
>> FYI, the last patch for immutables is up and passing all checks. I'll do
>> another pass in the morning to make sure everything looks good and then
>> will merge if I don't hear any concerns.
>> 
>> https://github.com/apache/calcite/pull/2568
>> 
>> 
>> On Wed, Oct 6, 2021, 1:04 PM Jacques Nadeau  wrote:
>> 
>>> Definitely. I'm also trying to add helpful comments in a couple of key
>>> javadoc locations.
>>> 
>>> On Wed, Oct 6, 2021 at 1:02 PM Julian Hyde 
>>> wrote:
>>> 
>>>> Thanks for following through on this large change, Jacques. Be sure to
>>>> document the changes people will need to make if they have defined their
>>>> own rules.
>>>> 
>>>> 
>>>>> On Oct 6, 2021, at 12:59 PM, Jacques Nadeau 
>>>> wrote:
>>>>> 
>>>>> I'm hoping (fingers crossed) to have the final Immutables patch up
>>>> today
>>>>> and if all goes well, get that merged before the release.
>>>>> 
>>>>> On Wed, Oct 6, 2021 at 11:34 AM Julian Hyde  wrote:
>>>>> 
>>>>>> So, the vote for Avatica 1.19 is underway, and we will likely have a
>>>>>> release in 4 or 5 days.
>>>>>> 
>>>>>> Next up, Calcite release 1.28. I'll be release manager for that too,
>>>> and
>>>>>> I'd like to start the vote in a week. I have three asks:
>>>>>> 
>>>>>> 1. Start stabilizing the build. If you have a change that might cause
>>>>>> build instability, ask on the dev list before you commit it. The
>>>> recent
>>>>>> changes for Gradle, JDK 16/17 and Immutables have been destabilizing,
>>>> so I
>>>>>> ask those authors
>>>>>> 
>>>>>> 2. I have logged https://issues.apache.org/jira/browse/CALCITE-4835.
>>>> If
>>>>>> you have made breaking changes since 1.27, please document them in
>>>> that
>>>>>> case, and I will add them to the release notes. If there are bugs
>>>> that MUST
>>>>>> be fixed in 1.28, link to them from that case.
>>>>>> 
>>>>>> 3. Can I have a few volunteers to go through the PRs and merge any
>>>> that
>>>>>> are ready? (Please reply to this email, so we know you are working on
>>>> it.)
>>>>>> 
>>>>>> Julian
>>>>>> 
>>>>>> 
>>>>>> On 2021/09/27 19:21:51, Alessandro Solimando <
>>>>>> alessandro.solima...@gmail.com> wrote:
>>>>>>> Hi everyone,
>>>>>>> I have prepared the PR, you can find it here:
>>>>>>> https://github.com/apache/calcite-avatica/pull/154.
>>>>>>> 
>>>>>>> Best regards,
>>>>>>> Alessandro
>>>>>>> 
>>>>>>> On Mon, 27 Sept 2021 at 20:27, Alessandro Solimando <
>>>>>>> alessandro.solima...@gmail.com> wrote:
>>>>>>> 
>>>>>>>> Hi Julian,
>>>>>>>> I can help with PR-143
>>>>>>>> <https://github.com/apache/calcite-avatica/pull/143>, since I have
>>>>>>>> contributed to the modified classes in the past.
>>>>>>>> 
>>>>>>>> I will still need a committer to merge but I can rebase on master
>>>> and
>>>>>> fix
>>>>>>>> issues, if any.
>>>>>>>> 
>>>>>>>> Best regards,
>>>>>>>> Alessandro
>>>>>>>> 
>>>>>>>> On Mon, 27 Sept 2021 at 07:52, Julian Hyde 
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Can people please try to review and merge the Avatica PRs this
>>>> week?
>>>>>>>>> Several look ready:
>>>>&g

[RESULT] [VOTE] Release apache-calcite-avatica-1.19.0 (release candidate 0)

[Julian Hyde]

t; > 
> > > Gradle 6.8.1
> > > 
> > > Build time:   2021-01-22 13:20:08 UTC
> > > Revision: 31f14a87d93945024ab7a78de84102a3400fa5b2
> > > Kotlin:   1.4.20
> > > Groovy:   2.5.12
> > > Ant:  Apache Ant(TM) version 1.10.9 compiled on September 27 2020
> > > JVM:  1.8.0_292 (AdoptOpenJDK 25.292-b10)
> > > OS:   Mac OS X 10.16 x86_64
> > >
> > > Best regards,
> > > Alessandro
> > >
> > > On Wed, 6 Oct 2021 at 04:42, Francis Chuang 
> > > wrote:
> > >
> > > > My vote is: +1 (Thanks Julian!)
> > > >
> > > > - Verified GPG signature - OK
> > > > - Verified SHA512 - OK
> > > > - Checked release notes on stage branch
> > > > (
> > > >
> > >
> > https://github.com/apache/calcite-avatica/blob/stage/site/_docs/history.md
> > > )
> > > >
> > > > - OK
> > > > - Ran tests (gradle test) - OK
> > > > - Spot checked Nexus artifacts - OK
> > > >
> > > > Notes:
> > > > - The build fails when using Gradle 7.2, maybe we should upgrade for
> > the
> > > > next release (1.20.0).
> > > > - I noticed the use of the stage branch for this release. Can you
> > > > document it for the next release manager after this release is
> > finalized?
> > > >
> > > > Environment:
> > > > Gradle 6.9.1 docker container in WSL2 (Ubuntu 20.04) on Windows 10 21h1
> > > >
> > > >  > docker version
> > > > Client: Docker Engine - Community
> > > >   Cloud integration: 1.0.17
> > > >   Version:   20.10.8
> > > >   API version:   1.41
> > > >   Go version:go1.16.6
> > > >   Git commit:3967b7d
> > > >   Built: Fri Jul 30 19:54:22 2021
> > > >   OS/Arch:   linux/amd64
> > > >   Context:   default
> > > >   Experimental:  true
> > > >
> > > > Server: Docker Engine - Community
> > > >   Engine:
> > > >Version:  20.10.8
> > > >API version:  1.41 (minimum version 1.12)
> > > >Go version:   go1.16.6
> > > >Git commit:   75249d8
> > > >Built:Fri Jul 30 19:52:31 2021
> > > >OS/Arch:  linux/amd64
> > > >Experimental: false
> > > >   containerd:
> > > >Version:  1.4.9
> > > >GitCommit:e25210fe30a0a703442421b0f60afac609f950a3
> > > >   runc:
> > > >Version:  1.0.1
> > > >GitCommit:v1.0.1-0-g4144b63
> > > >   docker-init:
> > > >Version:  0.19.0
> > > >GitCommit:de40ad0
> > > >
> > > >  > gradle -v
> > > >
> > > > 
> > > > Gradle 6.9.1
> > > > 
> > > >
> > > > Build time:   2021-08-20 11:15:18 UTC
> > > > Revision: f0ddb54aaae0e44f0a7209c3c0274d506ea742a0
> > > >
> > > > Kotlin:   1.4.20
> > > > Groovy:   2.5.12
> > > > Ant:  Apache Ant(TM) version 1.10.9 compiled on September 27
> > 2020
> > > > JVM:  1.8.0_292 (AdoptOpenJDK 25.292-b10)
> > > > OS:   Linux 5.10.16.3-microsoft-standard-WSL2 amd64
> > > >
> > > >  > java -version
> > > > openjdk version "1.8.0_292"
> > > > OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
> > > > OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)
> > > >
> > > > Francis
> > > >
> > > > On 6/10/2021 1:15 pm, Julian Hyde wrote:
> > > > > Hi all,
> > > > >
> > > > > I have created a build for Apache Calcite Avatica 1.19.0, release
> > > > > candidate 0.
> > > > >
> > > > > Thanks to everyone who has contributed to this release. And thank you
> > > > > to Francis, Vladimir and Stamatis for assistance with the release
> > > > process.
> > > > >
> > > > > You can read t

Re: [DISCUSS] Next releases

[Julian Hyde]

Thanks for following through on this large change, Jacques. Be sure to document 
the changes people will need to make if they have defined their own rules.
 

> On Oct 6, 2021, at 12:59 PM, Jacques Nadeau  wrote:
> 
> I'm hoping (fingers crossed) to have the final Immutables patch up today
> and if all goes well, get that merged before the release.
> 
> On Wed, Oct 6, 2021 at 11:34 AM Julian Hyde  wrote:
> 
>> So, the vote for Avatica 1.19 is underway, and we will likely have a
>> release in 4 or 5 days.
>> 
>> Next up, Calcite release 1.28. I'll be release manager for that too, and
>> I'd like to start the vote in a week. I have three asks:
>> 
>> 1. Start stabilizing the build. If you have a change that might cause
>> build instability, ask on the dev list before you commit it. The recent
>> changes for Gradle, JDK 16/17 and Immutables have been destabilizing, so I
>> ask those authors
>> 
>> 2. I have logged https://issues.apache.org/jira/browse/CALCITE-4835. If
>> you have made breaking changes since 1.27, please document them in that
>> case, and I will add them to the release notes. If there are bugs that MUST
>> be fixed in 1.28, link to them from that case.
>> 
>> 3. Can I have a few volunteers to go through the PRs and merge any that
>> are ready? (Please reply to this email, so we know you are working on it.)
>> 
>> Julian
>> 
>> 
>> On 2021/09/27 19:21:51, Alessandro Solimando <
>> alessandro.solima...@gmail.com> wrote:
>>> Hi everyone,
>>> I have prepared the PR, you can find it here:
>>> https://github.com/apache/calcite-avatica/pull/154.
>>> 
>>> Best regards,
>>> Alessandro
>>> 
>>> On Mon, 27 Sept 2021 at 20:27, Alessandro Solimando <
>>> alessandro.solima...@gmail.com> wrote:
>>> 
>>>> Hi Julian,
>>>> I can help with PR-143
>>>> <https://github.com/apache/calcite-avatica/pull/143>, since I have
>>>> contributed to the modified classes in the past.
>>>> 
>>>> I will still need a committer to merge but I can rebase on master and
>> fix
>>>> issues, if any.
>>>> 
>>>> Best regards,
>>>> Alessandro
>>>> 
>>>> On Mon, 27 Sept 2021 at 07:52, Julian Hyde 
>> wrote:
>>>> 
>>>>> Can people please try to review and merge the Avatica PRs this week?
>>>>> Several look ready:
>>>>> 
>>>>> https://github.com/apache/calcite-avatica/pulls
>>>>> 
>>>>> Julian
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Sep 24, 2021, at 6:22 PM, Haisheng Yuan 
>> wrote:
>>>>>> 
>>>>>> Sounds good!
>>>>>> 
>>>>>> On 2021/09/25 01:13:26, Julian Hyde  wrote:
>>>>>>> I propose to start a vote for Avatica 1.19 one week from now, and a
>>>>>>> vote for Calcite 1.28 two weeks from now. I'll be RM for both. How
>>>>>>> does that timing sound?
>>>>>>> 
>>>>>>> Julian
>>>>>>> 
>>>>>>> On Fri, Sep 24, 2021 at 10:12 AM James Starr >> 
>>>>> wrote:
>>>>>>>> 
>>>>>>>> When is calcite planning on going to 2.0?
>>>>>>>> 
>>>>>>>> On Fri, Sep 24, 2021 at 3:35 AM Stamatis Zampetakis <
>>>>> zabe...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> From the last discussion [1], the list was the following for
>> Calcite:
>>>>>>>>> 
>>>>>>>>> Julian Hyde for 1.28.0
>>>>>>>>> Rui Wang for 1.29.0
>>>>>>>>> Andrei Sereda for 1.30.0
>>>>>>>>> Liya Fan for 1.31.0
>>>>>>>>> 
>>>>>>>>> [1]
>>>>>>>>> 
>>>>>>>>> 
>>>>> 
>> https://lists.apache.org/thread.html/r86eead4dc703200c0f7182632bad79198d9c3fab006358c6e7dae8e8%40%3Cdev.calcite.apache.org%3E
>>>>>>>>> 
>>>>>>>>> On Fri, Sep 24, 2021 at 6:36 AM Julian Hyde 
>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> We're about due for releases of Avatica and Calcite. Who's next
>> up
>>>>> on
>>>>>>>>>> the release manager schedule?
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>> 

Re: Can`t build master of calcite.

[Julian Hyde]

Vladimir,

I think that's too harsh. In practice people aren't on the latest JDK version. 
(CI environments usually are, and therefore we'll tend to miss these issues.) 
We should make the process as painless as possible for everyone.

Julian

On 2021/10/06 10:54:17, Vladimir Sitnikov  wrote: 
> >Does no other build it locally for such a time ?:)
> 
> Evgeny, it is very likely you hit the bug because you use a stale version
> of OpenJDK.
> What Stamatis merged is not a fix, but it is a workaround.
> 
> In any case, if you use an outdated Java version, it is up to you to figure
> out why the build does not work.
> 
> Vladimir
> 

Re: Problem using MySQL with JDBC

[Julian Hyde]

What happens if you remove the line

  jdbcDriver: 'com.mysql.cj.jdbc.Driver’,

from your Calcite model? Hopefully, it just works. Most drivers load 
automatically these days, and if you don’t specify the class name, Calcite 
won’t try to load it manually.

> On Oct 6, 2021, at 12:07 PM, Justin Swanhart  wrote:
> 
> Hi,
> 
> So the example from MySQL is:
> [justin@localhost calcite.old]$ cat LoadDriver.java
> import java.sql.Connection;
> import java.sql.DriverManager;
> import java.sql.SQLException;
> 
> // Notice, do not import com.mysql.jdbc.*
> // or you will have problems!
> 
> public class LoadDriver {
>public static void main(String[] args) {
>// The newInstance() call is a work around for some
>// broken Java implementations
>  try{
>  Class.forName("com.mysql.jdbc.Driver").newInstance();
>  } catch(Exception ex) {
>}
>  }
> }
> 
> The code ignores an exception that is thrown when using .newInstance().
> The program outputs:
> Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver
> class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered
> via the SPI and manual loading of the driver class is generally unnecessary.
> (it outputs nothing if the new driver name is used)
> 
> So it seems that the MySQL driver throws an exception when used this way
> that has to be ignored, and Calcite is not ignoring it?
> 
> If that is the case, it seems like a problem with the MySQL driver, but I
> just want to confirm with you before I go report a bug and look silly :)
> 
> On Wed, Oct 6, 2021 at 2:59 PM Justin Swanhart  wrote:
> 
>> Hi,
>> 
>> The class.forName doesn't work for me though (tried
>> org.mysql.cj.jdbc.Driver and org.mysql.jdbc.Driver), but just using a
>> "jdbc:mysql://" connection string does work.
>> 
>> It makes sense that class.forName throws the same exception for me as
>> Calcite, but I don't understand why just using a connection string works,
>> so I am really confused.  I assume some other class name is being used when
>> I just use the connection string, but I don't know how to figure out what
>> that class name is.
>> 
>> On Wed, Oct 6, 2021 at 2:55 PM Julian Hyde  wrote:
>> 
>>> Does your environment use shading? Maybe Class.forName with a constant
>>> argument is handled by the shading, but Calcite is calling Class.forName
>>> with a dynamic argument.
>>> 
>>>> On Oct 6, 2021, at 11:41 AM, Justin Swanhart 
>>> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> The jar is in the classpath, and I can run the java command that the
>>>> sqlline script runs directly, and I get the same error.
>>>> 
>>>> The following works in a test java program with the CLASSPATH set:
>>>> 
>>>>   conn =
>>>>  DriverManager.getConnection("jdbc:mysql://localhost/ssb?" +
>>>>  "user=root");
>>>> I can create a resultset from a SELECT statement and fetch the results.
>>>> 
>>>> But this does not (throws ClassNotFound exception):
>>>> Class.forName("com.mysql.cj.jdbc.Driver").newInstance();
>>>> neither does
>>>> Class.forName("com.mysql.cj.jdbc.Driver").newInstance();
>>>> 
>>>> I am confused, because the JDBC connection works as long as I don't use
>>>> Class.forName.  I don't know how to figure out the name of the class
>>> that
>>>> is actually being used.  I'm sorry if this is silly, but I do not have
>>> much
>>>> java experience.  The MySQL documentation says to use
>>>> com.mysql.cj.jdbc.Driver:
>>>> 
>>> https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-usagenotes-connect-drivermanager.html
>>>> 
>>>> I suppose this is some kind of MySQL JDBC problem, so this may not be
>>> the
>>>> right mailing list, but if you have any suggestions before I start
>>> looking
>>>> elsewhere, I would appreciate it.
>>>> 
>>>> On Wed, Oct 6, 2021 at 1:59 PM Julian Hyde 
>>> wrote:
>>>> 
>>>>> It looks as if com.mysql.cj.jdbc.Driver is not on your class path.
>>>>> 
>>>>> If you are launching via SQLLine, you will need to edit the sqlline
>>> shell
>>>>> script to add a jar (or jars) to your class path.
>>>>> 
>>>>> Julian
>>>>> 
>>>>> 
>>>>>&g

Re: Problem using MySQL with JDBC

[Julian Hyde]

Does your environment use shading? Maybe Class.forName with a constant argument 
is handled by the shading, but Calcite is calling Class.forName with a dynamic 
argument.

> On Oct 6, 2021, at 11:41 AM, Justin Swanhart  wrote:
> 
> Hi,
> 
> The jar is in the classpath, and I can run the java command that the
> sqlline script runs directly, and I get the same error.
> 
> The following works in a test java program with the CLASSPATH set:
> 
>conn =
>   DriverManager.getConnection("jdbc:mysql://localhost/ssb?" +
>   "user=root");
> I can create a resultset from a SELECT statement and fetch the results.
> 
> But this does not (throws ClassNotFound exception):
> Class.forName("com.mysql.cj.jdbc.Driver").newInstance();
> neither does
> Class.forName("com.mysql.cj.jdbc.Driver").newInstance();
> 
> I am confused, because the JDBC connection works as long as I don't use
> Class.forName.  I don't know how to figure out the name of the class that
> is actually being used.  I'm sorry if this is silly, but I do not have much
> java experience.  The MySQL documentation says to use
> com.mysql.cj.jdbc.Driver:
> https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-usagenotes-connect-drivermanager.html
> 
> I suppose this is some kind of MySQL JDBC problem, so this may not be the
> right mailing list, but if you have any suggestions before I start looking
> elsewhere, I would appreciate it.
> 
> On Wed, Oct 6, 2021 at 1:59 PM Julian Hyde  wrote:
> 
>> It looks as if com.mysql.cj.jdbc.Driver is not on your class path.
>> 
>> If you are launching via SQLLine, you will need to edit the sqlline shell
>> script to add a jar (or jars) to your class path.
>> 
>> Julian
>> 
>> 
>>> On Oct 6, 2021, at 10:43 AM, Justin Swanhart 
>> wrote:
>>> 
>>> I am probably making some obvious mistake, but I am having a problem
>>> getting a simple MySQL JDBC connection working.
>>> 
>>> I have the latest version of the Connector/J MySQL java client driver.  I
>>> have a MySQL 8 server running on the local machine, and the following
>> model
>>> JSON:
>>> {
>>> version: '1.0',
>>> defaultSchema: 'ssb',
>>> schemas: [
>>>   {
>>> name: 'ssb',
>>> type: 'custom',
>>> factory: 'org.apache.calcite.adapter.jdbc.JdbcSchema$Factory',
>>> operand: {
>>>   jdbcDriver: 'com.mysql.cj.jdbc.Driver',
>>>   jdbcUrl: 'jdbc:mysql://localhost/ssb',
>>>   jdbcUser: 'root',
>>>   jdbcPassword: ''
>>> }
>>>   }
>>> ]
>>> }
>>> 
>>> I have the MySQL driver in my CLASSPATH. I can compile a simple test
>>> program which verifies that the class com.mysql.jdbc.Driver exists.
>>> 
>>> While the test program works, sqlline does not:
>>> $ ./sqlline -d com.mysql.cj.jdbc.Driver -u jdbc:mysql://root@localhost
>> /ssb
>>> Building Apache Calcite 1.28.0-SNAPSHOT
>>> scan complete in 1ms
>>> Could not find driver com.mysql.cj.jdbc.Driver
>>> 
>>> Any suggestions about what I might be doing wrong?
>>> 
>>> Using the !connect command yields a large backtrace (while probably not
>>> useful, included for completeness):
>>> sqlline version 1.11.0
>>> sqlline> !connect jdbc:calcite:model=test.json admin admin
>>> WARNING: An illegal reflective access operation has occurred
>>> WARNING: Illegal reflective access by com.google.protobuf.UnsafeUtil
>>> 
>> (file:/home/justin/.gradle/caches/modules-2/files-2.1/com.google.protobuf/protobuf-java/3.6.1/d06d46ecfd92ec6d0f3b423b4cd81cb38d8b924/protobuf-java-3.6.1.jar)
>>> to field java.nio.Buffer.address
>>> WARNING: Please consider reporting this to the maintainers of
>>> com.google.protobuf.UnsafeUtil
>>> WARNING: Use --illegal-access=warn to enable warnings of further illegal
>>> reflective access operations
>>> WARNING: All illegal access operations will be denied in a future release
>>> java.lang.RuntimeException: Error instantiating
>> JsonCustomSchema(name=ssb)
>>> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:277)
>>> at
>>> 
>> org.apache.calcite.model.JsonCustomSchema.accept(JsonCustomSchema.java:66)
>>> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:200)
>>> at org.apache.calcite.model.ModelHandler.(ModelHandler.java:106)
>>> at org.apache.calcite.jdbc.Driver$1.onConnectionInit(Driver.java:101)

Re: [DISCUSS] Next releases

[Julian Hyde]

So, the vote for Avatica 1.19 is underway, and we will likely have a release in 
4 or 5 days.

Next up, Calcite release 1.28. I'll be release manager for that too, and I'd 
like to start the vote in a week. I have three asks:

1. Start stabilizing the build. If you have a change that might cause build 
instability, ask on the dev list before you commit it. The recent changes for 
Gradle, JDK 16/17 and Immutables have been destabilizing, so I ask those 
authors 

2. I have logged https://issues.apache.org/jira/browse/CALCITE-4835. If you 
have made breaking changes since 1.27, please document them in that case, and I 
will add them to the release notes. If there are bugs that MUST be fixed in 
1.28, link to them from that case.

3. Can I have a few volunteers to go through the PRs and merge any that are 
ready? (Please reply to this email, so we know you are working on it.)

Julian


On 2021/09/27 19:21:51, Alessandro Solimando  
wrote: 
> Hi everyone,
> I have prepared the PR, you can find it here:
> https://github.com/apache/calcite-avatica/pull/154.
> 
> Best regards,
> Alessandro
> 
> On Mon, 27 Sept 2021 at 20:27, Alessandro Solimando <
> alessandro.solima...@gmail.com> wrote:
> 
> > Hi Julian,
> > I can help with PR-143
> > <https://github.com/apache/calcite-avatica/pull/143>, since I have
> > contributed to the modified classes in the past.
> >
> > I will still need a committer to merge but I can rebase on master and fix
> > issues, if any.
> >
> > Best regards,
> > Alessandro
> >
> > On Mon, 27 Sept 2021 at 07:52, Julian Hyde  wrote:
> >
> >> Can people please try to review and merge the Avatica PRs this week?
> >> Several look ready:
> >>
> >> https://github.com/apache/calcite-avatica/pulls
> >>
> >> Julian
> >>
> >>
> >>
> >> > On Sep 24, 2021, at 6:22 PM, Haisheng Yuan  wrote:
> >> >
> >> > Sounds good!
> >> >
> >> > On 2021/09/25 01:13:26, Julian Hyde  wrote:
> >> >> I propose to start a vote for Avatica 1.19 one week from now, and a
> >> >> vote for Calcite 1.28 two weeks from now. I'll be RM for both. How
> >> >> does that timing sound?
> >> >>
> >> >> Julian
> >> >>
> >> >> On Fri, Sep 24, 2021 at 10:12 AM James Starr 
> >> wrote:
> >> >>>
> >> >>> When is calcite planning on going to 2.0?
> >> >>>
> >> >>> On Fri, Sep 24, 2021 at 3:35 AM Stamatis Zampetakis <
> >> zabe...@gmail.com>
> >> >>> wrote:
> >> >>>
> >> >>>> From the last discussion [1], the list was the following for Calcite:
> >> >>>>
> >> >>>> Julian Hyde for 1.28.0
> >> >>>> Rui Wang for 1.29.0
> >> >>>> Andrei Sereda for 1.30.0
> >> >>>> Liya Fan for 1.31.0
> >> >>>>
> >> >>>> [1]
> >> >>>>
> >> >>>>
> >> https://lists.apache.org/thread.html/r86eead4dc703200c0f7182632bad79198d9c3fab006358c6e7dae8e8%40%3Cdev.calcite.apache.org%3E
> >> >>>>
> >> >>>> On Fri, Sep 24, 2021 at 6:36 AM Julian Hyde 
> >> wrote:
> >> >>>>
> >> >>>>> We're about due for releases of Avatica and Calcite. Who's next up
> >> on
> >> >>>>> the release manager schedule?
> >> >>>>>
> >> >>>>
> >> >>
> >>
> >>
> 

[jira] [Created] (CALCITE-4835) Release Calcite 1.28.0

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4835:


 Summary: Release Calcite 1.28.0
 Key: CALCITE-4835
 URL: https://issues.apache.org/jira/browse/CALCITE-4835
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


Release Calcite 1.28.0.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Problem using MySQL with JDBC

[Julian Hyde]

It looks as if com.mysql.cj.jdbc.Driver is not on your class path.

If you are launching via SQLLine, you will need to edit the sqlline shell 
script to add a jar (or jars) to your class path. 

Julian


> On Oct 6, 2021, at 10:43 AM, Justin Swanhart  wrote:
> 
> I am probably making some obvious mistake, but I am having a problem
> getting a simple MySQL JDBC connection working.
> 
> I have the latest version of the Connector/J MySQL java client driver.  I
> have a MySQL 8 server running on the local machine, and the following model
> JSON:
> {
>  version: '1.0',
>  defaultSchema: 'ssb',
>  schemas: [
>{
>  name: 'ssb',
>  type: 'custom',
>  factory: 'org.apache.calcite.adapter.jdbc.JdbcSchema$Factory',
>  operand: {
>jdbcDriver: 'com.mysql.cj.jdbc.Driver',
>jdbcUrl: 'jdbc:mysql://localhost/ssb',
>jdbcUser: 'root',
>jdbcPassword: ''
>  }
>}
>  ]
> }
> 
> I have the MySQL driver in my CLASSPATH. I can compile a simple test
> program which verifies that the class com.mysql.jdbc.Driver exists.
> 
> While the test program works, sqlline does not:
> $ ./sqlline -d com.mysql.cj.jdbc.Driver -u jdbc:mysql://root@localhost/ssb
> Building Apache Calcite 1.28.0-SNAPSHOT
> scan complete in 1ms
> Could not find driver com.mysql.cj.jdbc.Driver
> 
> Any suggestions about what I might be doing wrong?
> 
> Using the !connect command yields a large backtrace (while probably not
> useful, included for completeness):
> sqlline version 1.11.0
> sqlline> !connect jdbc:calcite:model=test.json admin admin
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by com.google.protobuf.UnsafeUtil
> (file:/home/justin/.gradle/caches/modules-2/files-2.1/com.google.protobuf/protobuf-java/3.6.1/d06d46ecfd92ec6d0f3b423b4cd81cb38d8b924/protobuf-java-3.6.1.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> java.lang.RuntimeException: Error instantiating JsonCustomSchema(name=ssb)
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:277)
> at
> org.apache.calcite.model.JsonCustomSchema.accept(JsonCustomSchema.java:66)
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:200)
> at org.apache.calcite.model.ModelHandler.(ModelHandler.java:106)
> at org.apache.calcite.jdbc.Driver$1.onConnectionInit(Driver.java:101)
> at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:139)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
> at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
> at sqlline.Commands.connect(Commands.java:1481)
> at sqlline.Commands.connect(Commands.java:1355)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:44)
> at sqlline.SqlLine.dispatch(SqlLine.java:818)
> at sqlline.SqlLine.begin(SqlLine.java:596)
> at sqlline.SqlLine.start(SqlLine.java:269)
> at sqlline.SqlLine.main(SqlLine.java:208)
> Caused by: com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException: java.sql.SQLException: Cannot load JDBC driver
> class 'com.mysql.cj.jdbc.Driver'
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051)
> at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4964)
> at
> org.apache.calcite.adapter.jdbc.JdbcUtils$DialectPool.get(JdbcUtils.java:116)
> at
> org.apache.calcite.adapter.jdbc.JdbcSchema.createDialect(JdbcSchema.java:200)
> at org.apache.calcite.adapter.jdbc.JdbcSchema.create(JdbcSchema.java:136)
> at org.apache.calcite.adapter.jdbc.JdbcSchema.create(JdbcSchema.java:123)
> at org.apache.calcite.adapter.jdbc.JdbcSchema.create(JdbcSchema.java:175)
> at
> org.apache.calcite.adapter.jdbc.JdbcSchema$Factory.create(JdbcSchema.java:570)
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:272)
> ... 18 more
> Caused by: java.lang.RuntimeException: java.sql.SQLException: Cannot load
> JDBC driver class 'com.mysql.cj.jdbc.Driver'
> at
> org.apache.calcite.adapter.jdbc.JdbcUtils$DialectPool.dialect(JdbcUtils.java:101)
> at
> 

[VOTE] Release apache-calcite-avatica-1.19.0 (release candidate 0)

[Julian Hyde]

Hi all,

I have created a build for Apache Calcite Avatica 1.19.0, release
candidate 0.

Thanks to everyone who has contributed to this release. And thank you
to Francis, Vladimir and Stamatis for assistance with the release process.

You can read the release notes here:
https://gitbox.apache.org/repos/asf/calcite-avatica-site-preview.git/avatica/docs/history.html

The commit to be voted upon:
https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=commit;h=d9f437424709fc6d7944dba174a658a7ca83e792

Its hash is d9f437424709fc6d7944dba174a658a7ca83e792

Tag:
https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=tag;h=refs/tags/avatica-1.19.0-rc0

The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-1.19.0-rc0
(revision 50284)

RAT report:
https://gitbox.apache.org/repos/asf/calcite-avatica-site-preview.git/avatica/rat/rat-report.txt

Site preview is here:
https://gitbox.apache.org/repos/asf/calcite-avatica-site-preview.git/avatica/

JavaDoc API preview is here:
https://gitbox.apache.org/repos/asf/calcite-avatica-site-preview.git/avatica/api

The hashes of the artifacts are as follows:
eb097a4638e521ea58e0736ba2b0546bef58ec602c021b381191fdea10ca8855b297b645bf58488698aa21aa30c691e00219bb5dbdf737655ca23f35b83e6aa5
*apache-calcite-avatica-1.19.0-src.tar.gz

A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachecalcite-1119/org/apache/calcite/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/jhyde.asc
https://www.apache.org/dist/calcite/KEYS

To create the jars and test Apache Calcite Avatica: "gradle build
-Prelease -PskipSign".

If you do not have a Java/Gradle environment available, you can run the tests
using docker. To do so, install docker and docker-compose, then run
"docker-compose run test" from the root of the directory.

Please vote on releasing this package as Apache Calcite Avatica 1.19.0.

The vote is open until at least October 9th at 0300 UTC (72 hours from now)
and passes if a majority of at least three +1 PMC votes are cast.

[ ] +1 Release this package as Apache Calcite Avatica 1.19.0
[ ]  0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...


Here is my vote:

+1 (binding)

Julian

Re: (CALCITE-4292) Wrong results in ElasticSearch when query contains NOT EQUAL

[Julian Hyde]

I support Stamatis’ position. Our SQL adapter must implement SQL semantics. 
That is what users of a SQL interface want and expect. 

If Elasticsearch’s data model has nuances that cannot be captured in SQL, feel 
free to add extra operators. If name is a multi-valued attribute, then some 
ideas are IS_EMPTY(name), VALUE_COUNT(name), VALUE_SET(name). 

I’ve added these comments to 
https://issues.apache.org/jira/browse/CALCITE-4292. 

> On Oct 5, 2021, at 2:57 AM, Stamatis Zampetakis  wrote:
> 
> Hi all,
> 
> The question is basically how the following SQL statement should behave for
> rows where the name is NULL in the ElasticSearch adapter.
> 
> SELECT * from zips WHERE name <> "NMAX"
> 
> I did add my comments in the JIRA case but it would be good if somebody
> also expresses an opinion so that we resolve/close the issue.
> 
> Best,
> Stamatis
> 
>> On Tue, Sep 28, 2021 at 2:52 PM Shlok Srivastava
>>  wrote:
>> 
>> Hi Community,
>> 
>> Issue id -CALCITE-4292
>> Issue name -Wrong results in ElasticSearch when query contains NOT EQUAL
>> 
>> We made a change to modify not equals criteria in elasticsearch adapters
>> as it was not in sync with Elasticsearch behavior.
>> 
>> In-case of Not_Equals condition, the default behaviour of ElasticSearch as
>> well as JSON path is to select records in which the mentioned field is
>> missing, but as calcite prefers SQL semantics it adds EXISTS condition as
>> well.
>> Adding additional EXISTS condition in NOT_EQUALS criteria deviates from
>> ElasticSearch behaviour. As the adapter is for ElasticSearch it should
>> support ES behaviour. If someone requires exists along with NO_EQUALS
>> condition it can be explicitly added in where condition but it can't be
>> removed unless the code is fixed.
>> 
>> So, this defect should be fixed to support default ElasticSearch behavior.
>> 
>> For this change we have been informed that we require community feedback,
>> so please share your thoughts/approval for same.
>> 
>> Thanks,
>> Shlok
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>> 

Re: Avatica dry-run

[Julian Hyde]

I thought the purpose of using Docker was to create a simpler, more
reproducible environment. I don't use Docker regularly (in fact, my
employer bans it from my work machine) and so, from my perspective,
any use of Docker makes the process more complicated.

I also hate having to put all of my most important passwords into a
single text file to make this process work.

Julian

On Tue, Oct 5, 2021 at 1:39 AM Francis Chuang  wrote:
>
> Hey Julian,
>
> Thanks for reporting that. I think the instructions in the howto needs
> to be clearer regarding the steps to start asflike-release-environment.
> I think we should put the actual commands used to start the mock servers
> directly in there.
>
> Hmm, the docker build should not write to your home directory at all. It
> creates a separate volume to mount into the container and uses that as
> the home directory in the container, so it should never touch your real
> home directory.
>
> I have been using it for the past few releases and it's always worked
> for me. My only suggestion if you're still willing to try is run
> `docker-compose down` from the root directory of the git repo to clear
> all containers and to remove the `gradle-cache` from docker, then
> starting over from there.
>
> As for pushing to gitbox, I think you can use
> -Pasf.git.pushRepositoryProvider=GITBOX, that's what I have for docker:
> https://github.com/apache/calcite-avatica/blob/89e0deb510311b85b8c8bacde6d2ff70c309930e/docker.sh#L251
>
> Francis
>
> On 5/10/2021 7:24 pm, Julian Hyde wrote:
> > Francis,
> >
> > That helped somewhat. I was able to get the Docker-powered server
> > running using your instructions. (From reading the HOWTO, I had no
> > idea that I was creating a server in a separate terminal.) I was
> > running Tomcat on port 80, and so I had to shut that down to prevent
> > port clashes.
> >
> > But then:
> >
> > $ ./gradlew prepareVote -Prc=0
> > Starting a Gradle Daemon (subsequent builds will be faster)
> >
> > FAILURE: Build failed with an exception.
> >
> > * What went wrong:
> > Gradle could not start your build.
> >> Cannot create service of type DefaultConfigurationCache using 
> >> DefaultConfigurationCache constructor as there is a problem with parameter 
> >> #10 of type ConfigurationCacheFingerprintController.
> > > Cannot create service of type
> > ConfigurationCacheFingerprintController using
> > ConfigurationCacheFingerprintController constructor as there is a
> > problem with parameter #5 of type FileCollectionFingerprinterRegistry.
> >> Cannot create service of type
> > FileCollectionFingerprinterRegistry using method
> > VirtualFileSystemServices$BuildSessionServices.createFileCollectionFingerprinterRegistry()
> > as there is a problem with parameter #1 of type
> > List.
> >   > Could not create service of type CrossBuildFileHashCache
> > using BuildSessionServices.createCrossBuildFileHashCache().
> >  > Failed to create parent directory
> > '/home/jhyde/dev/calcite-avatica/.gradle/6.8.1' when creating
> > directory '/home/jhyde/dev/calcite-avatica/.gradle/6.8.1/fileHashes'
> >
> >
> > I think the Docker build had created a .gradle directory as root, and
> > when I ran as myself, I couldn't write to it. And then I gave up
> > trying to do 'dry runs' and added the -Pasf flag to do it for real:
> >
> > $ ./gradlew prepareVote -Prc=0 -Pasf
> >
> >> Task :pushRcTag FAILED
> >
> > Build calcite-avatica FAILURE reason:
> >  Execution failed for task ':pushRcTag':
> >  Caused by: org.eclipse.jgit.api.errors.TransportException:
> > https://github.com/apache/calcite-avatica.git: not authorized
> >  at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:180)
> >  at 
> > com.github.vlsi.gradle.release.jgit.dsl.GitExtensionsKt.push(GitExtensions.kt:132)
> >  at 
> > com.github.vlsi.gradle.release.GitPushTask$pushTag$1.invoke(GitPushTask.kt:54)
> >
> > After an hour checking my github username and password, I found this:
> >
> > $ git push origin test-tag-1
> > Username for 'https://github.com': julianhyde
> > Password for 'https://julianh...@github.com':
> > remote: Support for password authentication was removed on August 13,
> > 2021. Please use a personal access token instead.
> > remote: Please see
> > https://github.blog/2020-12-15-token-authentication-requirements-for-git-operations/
> > for more information.
> > fatal: Authentication failed for 
> > 'https://github.com/apache/calc

Re: Avatica dry-run

[Julian Hyde]

kis wrote:
> > I haven't done an avatica release so I don't know if using docker is must
> > but maybe running directly the prepareVote task could work:
> >
> > ./gradlew prepareVote -Prc=0 -Pasf
> >
> > It may complain about missing properties. I have the following on my
> > ~/.gradle/gradle.properties file:
> >
> > useGpgCmd=true
> > signing.gnupg.executable=gpg
> > signing.gnupg.useLegacyGpg=false
> > signing.gnupg.keyName=D77C3383F1927570
> > signing.gnupg.passphrase=
> > asfSvnUsername=zabetak
> > asfSvnPassword=YYY
> > asfGitSourceUsername=zabetak
> > asfGitSourcePassword=
> > asfNexusUsername=zabetak
> > asfNexusPassword=YYY
> >
> > Best,
> > Stamatis
> >
> > On Mon, Oct 4, 2021 at 10:03 PM Julian Hyde  wrote:
> >
> >> I'm still running into this 'Connection refused' error. It's a
> >> show-stopper. I cannot make a release candidate unless this problem is
> >> solved.
> >>
> >> On Mon, Oct 4, 2021 at 10:58 AM Julian Hyde  wrote:
> >>>
> >>> This is the command I used:
> >>>
> >>>docker-compose run -v ~/.gnupg:/.gnupg dry-run
> >>>
> >>> I'm running Docker 18.06.1-ce on Ubuntu 20.04.3 LTS.
> >>>
> >>> By the way, I was thinking of creating a "stage" branch in the Avatica
> >>> git repo. We no longer use a branch per release (which is fine) but
> >>> this means that the commit is not pushed to the git repo until after
> >>> the vote passes. I propose to push the RC commit to the "stage" branch
> >>> (and destructively force push if we need another RC) to give CI a
> >>> chance to run on the RC. After the vote passes I will push a tag.
> >>>
> >>> Julian
> >>>
> >>>
> >>> On Mon, Oct 4, 2021 at 10:48 AM Vladimir Sitnikov
> >>>  wrote:
> >>>>
> >>>> I'm not sure which steps are you following, however, the same sequence
> >>>> seems to work fine:
> >>>>
> >>>> Commands:
> >>>>
> >> https://github.com/vlsi/vlsi-release-plugins/blob/83c85c5faa4c7cd1fe0173b75c1cba5e60c3f209/.github/workflows/release-test.yml#L33-L60
> >>>> Logs:
> >>>>
> >> https://github.com/vlsi/vlsi-release-plugins/runs/3794304055?check_suite_focus=true
> >>>>
> >>>> Vladimir
> >>>>
> >>>>
> >>>> пн, 4 окт. 2021 г. в 20:15, Julian Hyde :
> >>>>
> >>>>> As release manager for the upcoming Avatica 1.19, I just tried to use
> >>>>> the docker-based dry-run. I got the following failure:
> >>>>>
> >>>>> Build calcite-avatica FAILURE reason:
> >>>>>  Execution failed for task ':initializeNexusStagingRepository':
> >>>>>  java.io.UncheckedIOException: java.net.ConnectException:
> >>>>> Failed to connect to /127.0.0.1:8080
> >>>>>  at
> >>>>>
> >> de.marcphilipp.gradle.nexus.internal.NexusClient.findStagingProfileId(NexusClient.kt:84)
> >>>>>  at
> >>>>>
> >> de.marcphilipp.gradle.nexus.InitializeNexusStagingRepository.determineStagingProfileId(InitializeNexusStagingRepository.kt:113)
> >>>>>  at
> >>>>>
> >> de.marcphilipp.gradle.nexus.InitializeNexusStagingRepository.access$determineStagingProfileId(InitializeNexusStagingRepository.kt:38)
> >>>>>
> >>>>> I don't want to spend a morning debugging Docker port-mappings so I'm
> >>>>> moving on.
> >>>>>
> >>>>> Julian
> >>>>>
> >>
> >

Re: Avatica dry-run

[Julian Hyde]

I'm still running into this 'Connection refused' error. It's a
show-stopper. I cannot make a release candidate unless this problem is
solved.

On Mon, Oct 4, 2021 at 10:58 AM Julian Hyde  wrote:
>
> This is the command I used:
>
>   docker-compose run -v ~/.gnupg:/.gnupg dry-run
>
> I'm running Docker 18.06.1-ce on Ubuntu 20.04.3 LTS.
>
> By the way, I was thinking of creating a "stage" branch in the Avatica
> git repo. We no longer use a branch per release (which is fine) but
> this means that the commit is not pushed to the git repo until after
> the vote passes. I propose to push the RC commit to the "stage" branch
> (and destructively force push if we need another RC) to give CI a
> chance to run on the RC. After the vote passes I will push a tag.
>
> Julian
>
>
> On Mon, Oct 4, 2021 at 10:48 AM Vladimir Sitnikov
>  wrote:
> >
> > I'm not sure which steps are you following, however, the same sequence
> > seems to work fine:
> >
> > Commands:
> > https://github.com/vlsi/vlsi-release-plugins/blob/83c85c5faa4c7cd1fe0173b75c1cba5e60c3f209/.github/workflows/release-test.yml#L33-L60
> > Logs:
> > https://github.com/vlsi/vlsi-release-plugins/runs/3794304055?check_suite_focus=true
> >
> > Vladimir
> >
> >
> > пн, 4 окт. 2021 г. в 20:15, Julian Hyde :
> >
> > > As release manager for the upcoming Avatica 1.19, I just tried to use
> > > the docker-based dry-run. I got the following failure:
> > >
> > > Build calcite-avatica FAILURE reason:
> > > Execution failed for task ':initializeNexusStagingRepository':
> > > java.io.UncheckedIOException: java.net.ConnectException:
> > > Failed to connect to /127.0.0.1:8080
> > > at
> > > de.marcphilipp.gradle.nexus.internal.NexusClient.findStagingProfileId(NexusClient.kt:84)
> > > at
> > > de.marcphilipp.gradle.nexus.InitializeNexusStagingRepository.determineStagingProfileId(InitializeNexusStagingRepository.kt:113)
> > > at
> > > de.marcphilipp.gradle.nexus.InitializeNexusStagingRepository.access$determineStagingProfileId(InitializeNexusStagingRepository.kt:38)
> > >
> > > I don't want to spend a morning debugging Docker port-mappings so I'm
> > > moving on.
> > >
> > > Julian
> > >

Re: Avatica dry-run

[Julian Hyde]

This is the command I used:

  docker-compose run -v ~/.gnupg:/.gnupg dry-run

I'm running Docker 18.06.1-ce on Ubuntu 20.04.3 LTS.

By the way, I was thinking of creating a "stage" branch in the Avatica
git repo. We no longer use a branch per release (which is fine) but
this means that the commit is not pushed to the git repo until after
the vote passes. I propose to push the RC commit to the "stage" branch
(and destructively force push if we need another RC) to give CI a
chance to run on the RC. After the vote passes I will push a tag.

Julian


On Mon, Oct 4, 2021 at 10:48 AM Vladimir Sitnikov
 wrote:
>
> I'm not sure which steps are you following, however, the same sequence
> seems to work fine:
>
> Commands:
> https://github.com/vlsi/vlsi-release-plugins/blob/83c85c5faa4c7cd1fe0173b75c1cba5e60c3f209/.github/workflows/release-test.yml#L33-L60
> Logs:
> https://github.com/vlsi/vlsi-release-plugins/runs/3794304055?check_suite_focus=true
>
> Vladimir
>
>
> пн, 4 окт. 2021 г. в 20:15, Julian Hyde :
>
> > As release manager for the upcoming Avatica 1.19, I just tried to use
> > the docker-based dry-run. I got the following failure:
> >
> > Build calcite-avatica FAILURE reason:
> > Execution failed for task ':initializeNexusStagingRepository':
> > java.io.UncheckedIOException: java.net.ConnectException:
> > Failed to connect to /127.0.0.1:8080
> > at
> > de.marcphilipp.gradle.nexus.internal.NexusClient.findStagingProfileId(NexusClient.kt:84)
> > at
> > de.marcphilipp.gradle.nexus.InitializeNexusStagingRepository.determineStagingProfileId(InitializeNexusStagingRepository.kt:113)
> > at
> > de.marcphilipp.gradle.nexus.InitializeNexusStagingRepository.access$determineStagingProfileId(InitializeNexusStagingRepository.kt:38)
> >
> > I don't want to spend a morning debugging Docker port-mappings so I'm
> > moving on.
> >
> > Julian
> >

Avatica dry-run

[Julian Hyde]

As release manager for the upcoming Avatica 1.19, I just tried to use
the docker-based dry-run. I got the following failure:

Build calcite-avatica FAILURE reason:
Execution failed for task ':initializeNexusStagingRepository':
java.io.UncheckedIOException: java.net.ConnectException:
Failed to connect to /127.0.0.1:8080
at 
de.marcphilipp.gradle.nexus.internal.NexusClient.findStagingProfileId(NexusClient.kt:84)
at 
de.marcphilipp.gradle.nexus.InitializeNexusStagingRepository.determineStagingProfileId(InitializeNexusStagingRepository.kt:113)
at 
de.marcphilipp.gradle.nexus.InitializeNexusStagingRepository.access$determineStagingProfileId(InitializeNexusStagingRepository.kt:38)

I don't want to spend a morning debugging Docker port-mappings so I'm moving on.

Julian

Re: [DISCUSS] Add extra check for Avatica to run tests with different timezone

[Julian Hyde]

I don’t think we should do randomized CI.

> On Oct 2, 2021, at 1:32 PM, Vladimir Sitnikov  
> wrote:
> 
> Here's a randomized matrix for Avatica:
> https://github.com/apache/calcite-avatica/pull/156
> https://github.com/apache/calcite-avatica/actions/runs/1298777387
> 
> Vladimir

Re: [DISCUSS] Add extra check for Avatica to run tests with different timezone

[Julian Hyde]

> Do you suggest test all OS, all Guava versions, all timezones in CI?
> That is hard to maintain and it takes CI time

No I don’t suggest that. That seemed to be what you were suggesting!

My suggestion is to test a ‘diagonal’. Suppose we need to test 5 JDK versions, 
5 Guava versions, and 5 operating systems. The full matrix is 5 ^ 3 = 125 
entries. But if we just test the diagonal, {(jdk #0, guava #0, os #0), … (jdk 
#4, guava #4, os #4)} then each parameter gets tested once. That’s not perfect, 
but it’s a huge improvement with a minimal amount of CI resources. We missed 
some bugs because we were testing only one Guava version and only one time zone.

Infrequently - say once a month or once per release - someone can run the whole 
matrix.

Julian


> On Oct 1, 2021, at 10:02 AM, Vladimir Sitnikov  
> wrote:
> 
>> CI should be deterministic
> 
> That is an interesting view.
> One of the approaches that I have not considered earlier is "seed matrix
> randomization from PR index and the branch name".
> That would make the set of jobs consistent for each execution of the same
> PR.
> 
> However, based on my testng/testng experience, it is just fine if the set
> of CI jobs varies.
> The CI job description does clarify what was special (e.g. timezone, JVM,
> OS, etc).
> 
>> And also tests with different Guava versions
> 
> Do you suggest test all OS, all Guava versions, all timezones in CI?
> That is hard to maintain and it takes CI time
> 
>> CI should be deterministic. So that if it breaks, the person to fix it is
> clear
> 
> CI resources are limited, and we can't infinitely scale the build matrix.
> I do not really see why Guava versions are more important than operating
> systems.
> The users should prefer recent Guava versions, and the idea of supporting
> (and testing!) ancient Guava versions is doubtful.
> 
> On the other hand, we might want to verify different cases like
> -XX:hashCode=2 that makes every Object#hashCode() to return 1 always.
> 
> Vladimir

Re: INSERT Query Question

[Julian Hyde]

I’m not sure whether ROW is ‘correct’ SQL, but I agree that it’s not idiomatic 
SQL to use ROW inside a VALUES inside an INSERT.

SQL generation is controlled by the SqlDialect class. Typically we add tests to 
RelToSqlConverterTest. https://issues.apache.org/jira/browse/CALCITE-3344 
 is a good example of this 
kind of change.

I see that RelToSqlConverterTest.testRowValueExpression has a few examples of 
this kind of query. Maybe it needs to be expanded for other dialects.

Can you log a JIRA case for this?

Julian



> On Oct 1, 2021, at 2:07 PM, Charles Givre  wrote:
> 
> Hello Calcite team, 
> I have a quick question.  I'm looking to take an INSERT query, parse it and 
> convert it into the dialect of various databases.  The Inserts will not be 
> complicated.  For instance:
> 
> INSERT INTO mysql_test.data_types 
>   VALUES(1, 2, 3.0, 4.0, '5.0', '2020-12-31', '12:00:00', '2015-12-30 
> 17:55:55')
> 
> Converted into:
> 
> Postgres: INSERT INTO "postgresl_test"."data_types"
> VALUES ROW(1, 2, 3.0, 4.0, '5.0', '2020-12-31', '12:00:00', '2015-12-30 
> 17:55:55')
> MySQL: INSERT INTO `mysql_test`.`data_types`
> VALUES ROW(1, 2, 3.0, 4.0, '5.0', '2020-12-31', '12:00:00', '2015-12-30 
> 17:55:55')
> MSSQL: INSERT INTO [mssql_test].[data_types]
> VALUES ROW(1, 2, 3.0, 4.0, '5.0', '2020-12-31', '12:00:00', '2015-12-30 
> 17:55:55')
> 
> I wrote a function that does this, however as you'll note above, that Calcite 
> is inserting the word ROW before each row to be inserted and this is not 
> correct SQL. 
> 
> public static String cleanQuery(String query, SqlDialect dialect) {
>  SqlParser.Config sqlParserConfig = SqlParser.configBuilder()
>.setParserFactory(SqlDdlParserImpl.FACTORY)
>.setConformance(SqlConformanceEnum.MYSQL_5)
>.setCaseSensitive(true)
>.setLex(Lex.MYSQL)
>.build();
> 
>  try {
>SqlNode node = SqlParser.create(query, sqlParserConfig).parseQuery();
>return node.toSqlString(dialect).getSql();
>  } catch (SqlParseException e) {
>return null;
>  }
> }
> 
> Is there some way to configure Calcite not to insert the word ROW?  Thanks!
> -- C

Re: [DISCUSS] Add extra check for Avatica to run tests with different timezone

[Julian Hyde]

+1 for tests in different time zones. And also tests with different Guava 
versions. In fact I’m working on it: see 
https://github.com/julianhyde/calcite-avatica/commits/stage 
 and 
https://app.travis-ci.com/github/julianhyde/calcite-avatica/builds/238880701 
.

-1 for randomized testing as part of CI. CI should be deterministic. So that if 
it breaks, the person to fix it is clear: it’s generally the last person who 
changed something.

I support randomized testing via scripts. People can run the script for a 
couple of hours and log any bugs it uncovers. We already have several such 
scripts in Calcite (e.g. rex fuzz testing).

Julian


> On Oct 1, 2021, at 3:28 AM, Alessandro Solimando 
>  wrote:
> 
> Thanks Vladimir for the proposal.
> 
> +1 frome me, there are more and more "axes" for testing (OS, locale, JVM,
> timezone, etc.), a randomized approach sounds like a good solution to tame
> the combinatorial explosion.
> 
> Best regards,
> Alessandro
> 
> On Fri, 1 Oct 2021 at 08:32, Vladimir Sitnikov 
> wrote:
> 
>>> Of course it would be fine to just change the timezone for one of more
>>> existing tests which are targeting different JVM versions
>> 
>> Alessandro, team, what do you think of
>> https://github.com/vlsi/github-actions-random-matrix ?
>> 
>> I'm kind of hesitant to introduce more and more /vlsi/ dependencies even
>> though I believe they are the best tools available :)
>> random-matix is a 200 line script that generates build matrix by selecting
>> random values for the axes (e.g. jvm version, os, guava version, etc).
>> 
>> Vladimir
>> 

[jira] [Created] (CALCITE-4815) Avatica's Travis fails with 'LICENSE-like files are missing' error

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4815:


 Summary: Avatica's Travis fails with 'LICENSE-like files are 
missing' error
 Key: CALCITE-4815
 URL: https://issues.apache.org/jira/browse/CALCITE-4815
 Project: Calcite
  Issue Type: Bug
  Components: avatica
Reporter: Julian Hyde


Some configurations of Avatica's Travis CI job fail with the following error. 
{noformat}
> Task :standalone-server:classes
> Task :standalone-server:getLicenses FAILED
> Task :server:checkstyleTest

Build calcite-avatica FAILURE reason:  
  
Execution failed for task 
':standalone-server:getLicenses':
LICENSE-like files are missing
==

MIT
* org.checkerframework:checker-qual:2.11.1

at 
com.github.vlsi.gradle.license.GatherLicenseTask.run(GatherLicenseTask.kt:417)
 at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
at 
org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':standalone-server:getLicenses'.
> LICENSE-like files are missing
  ==
  
  MIT
  * org.checkerframework:checker-qual:2.11.1
 {noformat}
 
@vlsi, Do you have any idea what's going on? Avatica doesn't even use 
checkerframework. You can see the build output (for a few days) at 
[https://app.travis-ci.com/github/julianhyde/calcite-avatica/builds/238877621].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Syntax upgrade about Session Window Table Function

[Julian Hyde]

Thanks for the examples. The PARTITION BY syntax is a clear improvement for the 
SESSION function and I think we should do it, even though it is breaking.

I’ll make further comments against 
https://issues.apache.org/jira/browse/CALCITE-4337 
<https://issues.apache.org/jira/browse/CALCITE-4337>.

> On Sep 29, 2021, at 9:58 PM, JING ZHANG  wrote:
> 
> Hi Julian,
> Thanks for your feedback, the suggestion is very helpful.
> I've added the discussion content the CALCITE-4337
> <https://issues.apache.org/jira/browse/CALCITE-4337> [1]. I would continue
> later discussion in the JIRA case.
> About an example of a query before and after the syntax change. I would use
> the example in session table function document
> <https://calcite.apache.org/docs/reference.html#session> [2].
> Old syntax demo:
> 
>> SELECT * FROM TABLE( SESSION( TABLE orders, DESCRIPTOR(rowtime),
>> DESCRIPTOR(product), INTERVAL '20' MINUTE)); -- or with the named params --
>> note: the DATA param must be the first SELECT * FROM TABLE( SESSION( DATA
>> => TABLE orders, TIMECOL => DESCRIPTOR(rowtime), KEY => DESCRIPTOR(product
>> ), SIZE => INTERVAL '20' MINUTE));
> 
> 
> New syntax demo is as follows, the difference is use PARTITION BY clause to
> replace KEY DESCRIPTOR.
> 
>> SELECT * FROM TABLE( SESSION( TABLE orders PARTITION BY product,
>> DESCRIPTOR(rowtime), INTERVAL '20' MINUTE)); -- or with the named params --
>> note: the DATA param must be the first SELECT * FROM TABLE( SESSION( DATA
>> => TABLE orders PARTITION BY product, TIMECOL => DESCRIPTOR(rowtime), SIZE
>> => INTERVAL '20' MINUTE));
> 
> 
> Best,
> JING ZHANG
> 
> Julian Hyde  于2021年9月30日周四 上午4:55写道：
> 
>> Regarding changes to the syntax of the SESSION table function. I am open
>> to this, even though it would be a breaking change. Can you give an example
>> of a query before and after the syntax change?
>> 
>> I would like to support the new PARTITIONED BY clause for table functions.
>> I encourage you to make the change for table functions in general, before
>> and separately from the change to the SESSION function and window functions.
>> 
>> Please ensure that the discussion gets added to the JIRA case. It might be
>> best if we continue discussion in the JIRA case.
>> 
>> Julian
>> 
>> 
>>> On Sep 28, 2021, at 10:28 PM, JING ZHANG  wrote:
>>> 
>>> Hi community,
>>> I'm now working on CALCITE-4337
>>> <https://issues.apache.org/jira/browse/CALCITE-4337> [1] which aims to
>>> support PARTITION BY clause for table function argument.
>>> I've submitted a pull request
>>> <https://github.com/apache/calcite/pull/2524> [2],
>>> thanks @Danny very much for review.
>>> There are two points left which need more discussion. So I fire this
>>> discussion in order to get more broader suggestions.
>>> 1. SQL standard Polymorphic Table Functions
>>> <
>> https://standards.iso.org/ittf/PubliclyAvailableStandards/c069776_ISO_IEC_TR_19075-7_2017.zip
>>> 
>>> [3]
>>> states:
>>> 
>>>> Input tables have either row semantics or set semantics, as follows:
>>>> a) Row semantics means that the the result of the PTF is decided on a
>>>> row-by-row basis. As an extreme example, the DBMS could atomize the
>> input
>>>> table into individual rows, and send each single row to a different
>> virtual
>>>> processor.
>>>> b) Set semantics means that the outcome of the function depends on how
>> the
>>>> data is partitioned. A partition may not be split across virtual
>>>> processors, nor may a virtual processor handle more than one partition.
>>> 
>>> 
>>> A SESSION window has an input table with set semantics which means it
>>> requires a PARTITION BY clause.
>>> The new syntax is conflict with current session window table function
>>> syntax, please take a look at session table function
>>> <https://calcite.apache.org/docs/reference.html#session> [4].
>>> *Could we replace the old syntax directly, or take compatible into
>>> consideration.*
>>> 2. Based on SQL standard, only input tables with set semantics may be
>>> partitioned while input table with row semantics may not be partitioned.
>>> *Should we have separate branch in Parser.jj for set semantic input table
>>> of table function(Currently, only input table of session window table
>>> function has set semantics)*?
>>> 
>>> Any suggestion is appreciated. Thanks in advanced.
>>> [1] https://issues.apache.org/jira/browse/CALCITE-4337
>>> [2] https://github.com/apache/calcite/pull/2524
>>> [3]
>>> 
>> https://standards.iso.org/ittf/PubliclyAvailableStandards/c069776_ISO_IEC_TR_19075-7_2017.zip
>>> [4] https://calcite.apache.org/docs/reference.html#session
>>> 
>>> Best
>>> JING ZHANG
>> 
>> 

Re: [DISCUSS] Syntax upgrade about Session Window Table Function

[Julian Hyde]

Regarding changes to the syntax of the SESSION table function. I am open to 
this, even though it would be a breaking change. Can you give an example of a 
query before and after the syntax change?

I would like to support the new PARTITIONED BY clause for table functions. I 
encourage you to make the change for table functions in general, before and 
separately from the change to the SESSION function and window functions.

Please ensure that the discussion gets added to the JIRA case. It might be best 
if we continue discussion in the JIRA case.

Julian


> On Sep 28, 2021, at 10:28 PM, JING ZHANG  wrote:
> 
> Hi community,
> I'm now working on CALCITE-4337
>  [1] which aims to
> support PARTITION BY clause for table function argument.
> I've submitted a pull request
>  [2],
> thanks @Danny very much for review.
> There are two points left which need more discussion. So I fire this
> discussion in order to get more broader suggestions.
> 1. SQL standard Polymorphic Table Functions
> 
> [3]
> states:
> 
>> Input tables have either row semantics or set semantics, as follows:
>> a) Row semantics means that the the result of the PTF is decided on a
>> row-by-row basis. As an extreme example, the DBMS could atomize the input
>> table into individual rows, and send each single row to a different virtual
>> processor.
>> b) Set semantics means that the outcome of the function depends on how the
>> data is partitioned. A partition may not be split across virtual
>> processors, nor may a virtual processor handle more than one partition.
> 
> 
> A SESSION window has an input table with set semantics which means it
> requires a PARTITION BY clause.
> The new syntax is conflict with current session window table function
> syntax, please take a look at session table function
>  [4].
> *Could we replace the old syntax directly, or take compatible into
> consideration.*
> 2. Based on SQL standard, only input tables with set semantics may be
> partitioned while input table with row semantics may not be partitioned.
> *Should we have separate branch in Parser.jj for set semantic input table
> of table function(Currently, only input table of session window table
> function has set semantics)*?
> 
> Any suggestion is appreciated. Thanks in advanced.
> [1] https://issues.apache.org/jira/browse/CALCITE-4337
> [2] https://github.com/apache/calcite/pull/2524
> [3]
> https://standards.iso.org/ittf/PubliclyAvailableStandards/c069776_ISO_IEC_TR_19075-7_2017.zip
> [4] https://calcite.apache.org/docs/reference.html#session
> 
> Best
> JING ZHANG

Re: [DISCUSS] Next releases

[Julian Hyde]

Can people please try to review and merge the Avatica PRs this week? Several 
look ready:

https://github.com/apache/calcite-avatica/pulls

Julian



> On Sep 24, 2021, at 6:22 PM, Haisheng Yuan  wrote:
> 
> Sounds good!
> 
> On 2021/09/25 01:13:26, Julian Hyde  wrote: 
>> I propose to start a vote for Avatica 1.19 one week from now, and a
>> vote for Calcite 1.28 two weeks from now. I'll be RM for both. How
>> does that timing sound?
>> 
>> Julian
>> 
>> On Fri, Sep 24, 2021 at 10:12 AM James Starr  wrote:
>>> 
>>> When is calcite planning on going to 2.0?
>>> 
>>> On Fri, Sep 24, 2021 at 3:35 AM Stamatis Zampetakis 
>>> wrote:
>>> 
>>>> From the last discussion [1], the list was the following for Calcite:
>>>> 
>>>> Julian Hyde for 1.28.0
>>>> Rui Wang for 1.29.0
>>>> Andrei Sereda for 1.30.0
>>>> Liya Fan for 1.31.0
>>>> 
>>>> [1]
>>>> 
>>>> https://lists.apache.org/thread.html/r86eead4dc703200c0f7182632bad79198d9c3fab006358c6e7dae8e8%40%3Cdev.calcite.apache.org%3E
>>>> 
>>>> On Fri, Sep 24, 2021 at 6:36 AM Julian Hyde  wrote:
>>>> 
>>>>> We're about due for releases of Avatica and Calcite. Who's next up on
>>>>> the release manager schedule?
>>>>> 
>>>> 
>> 

Re: [DISCUSS] Pronouns

[Julian Hyde]

We should suggest that they add themselves to
https://github.com/apache/calcite/blob/master/site/_data/contributors.yml.
In that file, the instructions should note that the pronouns field is
available if they wish to specify.

On Fri, Sep 24, 2021 at 5:39 AM Stamatis Zampetakis  wrote:
>
> +1
>
> For new committers/PMCs, we could possibly add a sentence in the invitation
> email asking about their preferences and if they want to share them.
>
> Best,
> Stamatis
>
> On Tue, Sep 21, 2021 at 10:07 PM Vladimir Sitnikov <
> sitnikov.vladi...@gmail.com> wrote:
>
> > > possessive pronoun (e.g. his) also included but I don't know enough about
> > the
> > reasons for this to feel strongly either way.
> >
> > There are cases when people prefer specific posessive pronouns. For
> > instance: ze/hir vs ze/zir in http://pronoun.is/
> >
> > Just as an escape hatch, I try using names instead of pronouns.
> >
> > Vladimir
> >

Re: [DISCUSS] Next releases

[Julian Hyde]

I propose to start a vote for Avatica 1.19 one week from now, and a
vote for Calcite 1.28 two weeks from now. I'll be RM for both. How
does that timing sound?

Julian

On Fri, Sep 24, 2021 at 10:12 AM James Starr  wrote:
>
> When is calcite planning on going to 2.0?
>
> On Fri, Sep 24, 2021 at 3:35 AM Stamatis Zampetakis 
> wrote:
>
> > From the last discussion [1], the list was the following for Calcite:
> >
> > Julian Hyde for 1.28.0
> > Rui Wang for 1.29.0
> > Andrei Sereda for 1.30.0
> > Liya Fan for 1.31.0
> >
> > [1]
> >
> > https://lists.apache.org/thread.html/r86eead4dc703200c0f7182632bad79198d9c3fab006358c6e7dae8e8%40%3Cdev.calcite.apache.org%3E
> >
> > On Fri, Sep 24, 2021 at 6:36 AM Julian Hyde  wrote:
> >
> > > We're about due for releases of Avatica and Calcite. Who's next up on
> > > the release manager schedule?
> > >
> >

Re: Error parsing DATE("2021-01-01") for BigQuery, using Calcite

[Julian Hyde]

It’s possible you will also need to use the Babel parser, because DATE is a 
reserved keyword and therefore the parser needs to work in a different mode in 
order to see it as a function name. I think I made the DATE function work for 
Redshift but I’m not sure I did it for BigQuery.


> On Sep 24, 2021, at 1:58 PM, Florent Martineau  
> wrote:
> 
> Dear all,
> 
> Disclaimer: It's the first time I send a message to a mailing list. If it's
> not the right mailing list or if I should use other means (eg.
> Stackoverflow), please do not hesitate to tell me! Also, if you need
> additional pieces of information, I will be glad to provide them!
> 
> My problem is the following: I try to parse SQL queries for Big Query
> dialect, and it seems that it fails whenever I use the DATE keyword.
> 
> I tried to call Bigquery specific functions such as CURRENT_TIMESTAMP,
> DATE_FROM_UNIX or TIMESTAMP_MICROS, and it works. But it fails when trying
> to do things such as DATE("2021-01-01").
> 
> I'm wondering if it's a bug (either from Apache Calcite's implementation of
> Bigquery's dialect, or more likely from my code), or a feature (Bigquery is
> not supposed to support this kind of syntax ==> the query runs well in
> Bigquery's console so I'm doubtful about this hypothesis).
> 
> Also, what's strange is that for the query 'SELECT DATE("2021-01-01")', I
> get an error message telling me that it expects a parenthesis after the
> date, when the parenthesis is actually here. Here is the error message:
> 
> 
> 
> *Error: org.apache.calcite.sql.parser.SqlParseException: Incorrect syntax
> near the keyword 'DATE' at line 1, column 8.Was expecting one of:"ALL"
> ...*
> *[ITEMS OMMITTED]*
> *"(" ...*
> 
> * [ITEMS OMMITTED]*
> 
> 
> 
> Here is the full code to reproduce the error:
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> *package org.apache.calcite;import org.apache.calcite.config.Lex;import
> org.apache.calcite.sql.SqlNode;import
> org.apache.calcite.sql.parser.SqlParseException;import
> org.apache.calcite.sql.parser.SqlParser;import
> org.apache.calcite.sql.parser.impl.SqlParserImpl;import
> org.apache.calcite.sql.validate.SqlConformanceEnum;import
> org.apache.calcite.tools.FrameworkConfig;import
> org.apache.calcite.tools.Frameworks;import
> org.apache.calcite.tools.Planner;public class Demo {  public static void
> main(String[] args) {SqlParser.Config sqlParserConfig =
> SqlParser.config().DEFAULT.withLex(Lex.BIG_QUERY)
> .withConformance(SqlConformanceEnum.BIG_QUERY)
> .withParserFactory(SqlParserImpl.FACTORY);FrameworkConfig
> frameworkConfig = Frameworks.newConfigBuilder()
> .parserConfig(sqlParserConfig).build();String[]
> testSqlFragments = {"1+(2*4) as foo","CURRENT_DATE()",
>  "CURRENT_DATETIME()","DATE_FROM_UNIX(123456)",
> "TIMESTAMP_MICROS(123456)","UNIX_DATE(DATE '2021-01-01')",
> "DATE '2021-01-01'","DATE('2021-01-01')",};for (String
> sqlFragment : testSqlFragments) {  try {String query = "SELECT
> " + sqlFragment;System.out.println("Trying to parse : " + query);
>  Planner planner = Frameworks.getPlanner(frameworkConfig);
> SqlNode node = planner.parse(query);
> System.out.println("Successfully parsed query: " + node);  } catch
> (SqlParseException e) {System.out.println("Error: " + e);  }
> }  }}*
> 
> 
> 
> 
> And here is the output:
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> *Trying to parse : SELECT 1+(2*4) as fooSuccessfully parsed query: SELECT 1
> + 2 * 4 AS `foo`Trying to parse : SELECT CURRENT_DATE()Successfully parsed
> query: SELECT CURRENT_DATE()Trying to parse : SELECT
> CURRENT_DATETIME()Successfully parsed query: SELECT
> `CURRENT_DATETIME`()Trying to parse : SELECT
> DATE_FROM_UNIX(123456)Successfully parsed query: SELECT
> `DATE_FROM_UNIX`(123456)Trying to parse : SELECT
> TIMESTAMP_MICROS(123456)Successfully parsed query: SELECT
> `TIMESTAMP_MICROS`(123456)Trying to parse : SELECT UNIX_DATE(DATE
> '2021-01-01')Error: org.apache.calcite.sql.parser.SqlParseException:
> Incorrect syntax near the keyword 'DATE' at line 1, column 18.Was expecting
> one of:"ALL" ..."CURSOR" ..."DISTINCT" ..."EXISTS" ...
> "NOT" ..."ROW" ..."UNIQUE" ..."WITH" ..."(" ..."+" ... *
> 
> *[CUTTING LONG LIST HERE]*
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> *Trying to parse : SELECT DATE '2021-01-01'Error:
> org.apache.calcite.sql.parser.SqlParseException: Incorrect syntax near the
> keyword 'DATE' at line 1, column 8.Was expecting one of:"ALL" ...
> "CURSOR" ..."DISTINCT" ..."EXISTS" ..."NOT" ..."ROW" ...
> "STREAM" ..."UNIQUE" ..."(" ..."+" ..."-" ..."/*+" ...
>  "INTERVAL" ... ...
>  ... ...
>  ... ...
>  ...*
> 
> *

[DISCUSS] Next releases

[Julian Hyde]

We're about due for releases of Avatica and Calcite. Who's next up on
the release manager schedule?

Re: [DISCUSS] Remove contributors name from commit summary

[Julian Hyde]

A separate but related question: should we give people credit for their 
contributions in the release notes? We could append a contributor’s name to 
each contribution (achieving a similar effect to today). Or we could generate a 
“The following people contributed to this release: …” paragraph per release. Or 
let people’s contributions speak for themselves (if you want to know who did a 
change, browse GitHub).

I lean towards the last of these, but I want to hear what others think. The 
goal, as ever, is to build a strong community.

Julian

> On Sep 23, 2021, at 7:37 AM, Julian Hyde  wrote:
> 
> +1
> 
>> On Sep 23, 2021, at 2:23 AM, Stamatis Zampetakis  wrote:
>> 
>> Hi all,
>> 
>> Currently we are supposed to append the contributors name to the commit
>> summary when they are not committers of the project. The main reason for
>> doing this if I am not mistaken is to give some credit to those people.
>> 
>> I did like this practice in the beginning but I think it adds some small
>> overhead to all parties involved (committers and contributors).
>> 
>> The contributor quite often forgets to include the name in the end so the
>> burden to find and append the name goes to the committer.
>> 
>> In various cases, I've seen PRs ready to merge which were actually missing
>> the name at the end. What usually happens afterwards is one of the
>> following:
>> * the committer merges the PR without amending the name;
>> * the committer rebases the PR, amends the commit, and merges it;
>> * the committer asks the contributor to change the commit message;
>> 
>> I would prefer it if we could avoid this overhead by changing the commit
>> guidelines to not append the contributors name at the end.
>> 
>> GitHub does a great job giving credit to contributors. Moreover in most
>> cases the name appears in the log under the author tag so it is very easy
>> to exploit if we want to extract information and statistics.
>> 
>> What do you think?
>> 
>> Best,
>> Stamatis

Re: [DISCUSS] Remove contributors name from commit summary

[Julian Hyde]

+1

> On Sep 23, 2021, at 2:23 AM, Stamatis Zampetakis  wrote:
> 
> Hi all,
> 
> Currently we are supposed to append the contributors name to the commit
> summary when they are not committers of the project. The main reason for
> doing this if I am not mistaken is to give some credit to those people.
> 
> I did like this practice in the beginning but I think it adds some small
> overhead to all parties involved (committers and contributors).
> 
> The contributor quite often forgets to include the name in the end so the
> burden to find and append the name goes to the committer.
> 
> In various cases, I've seen PRs ready to merge which were actually missing
> the name at the end. What usually happens afterwards is one of the
> following:
> * the committer merges the PR without amending the name;
> * the committer rebases the PR, amends the commit, and merges it;
> * the committer asks the contributor to change the commit message;
> 
> I would prefer it if we could avoid this overhead by changing the commit
> guidelines to not append the contributors name at the end.
> 
> GitHub does a great job giving credit to contributors. Moreover in most
> cases the name appears in the log under the author tag so it is very easy
> to exploit if we want to extract information and statistics.
> 
> What do you think?
> 
> Best,
> Stamatis

[jira] [Created] (CALCITE-4795) In class SqlBasicCall, make the "operands" field private.

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4795:


 Summary: In class SqlBasicCall, make the "operands" field private.
 Key: CALCITE-4795
 URL: https://issues.apache.org/jira/browse/CALCITE-4795
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


In {{class SqlBasicCall}}, the {{operands}} 
[field|https://github.com/apache/calcite/blob/4bc916619fd286b2c0cc4d5c653c96a68801d74e/core/src/main/java/org/apache/calcite/sql/SqlBasicCall.java#L34]
 is a {{public}} array. This seems crazy to me – any client might be writing 
into that field at any time. I propose to make the field private.

This presents some compatibility problems, because people might be depending on 
the field. So I propose a quick deprecation and removal:
 * In release 1.28 (the next release, as I write this) the field and the 
{{public SqlNode[] getOperands()}} method will be marked deprecated. We will 
mirror into another field, {{private final List operandList = 
Arrays.asList(operands);}} We can replace all uses of the {{operands}} field in 
Calcite with uses of the new {{operandList}} field.
 * In release 1.29 (the release after next) the {{operands}} field and the 
{{getOperands()}} method will be removed. People can operate using 
{{List getOperandList()}} and {{setOperand(int, SqlNode)}} methods 
that are inherited from {{SqlCall}}.

After the field is a private list, we could consider making it an immutable 
list. The list would be copied when people call {{setOperand}}, but would not 
need to be cloned when the {{SqlBasicCall}} is created or cloned.

This case completes the work started in CALCITE-147.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: SqlBasicCall.clone shallow copy

[Julian Hyde]

That looks like a bug. Cloning a SqlCall and then calling setOperand on it is 
valid, and should not modify the original SqlCall.

We tend to avoid calling ’set’ methods on SqlNodes - we treat them as immutable 
except that the validator will replace an argument with an equivalent argument 
(e.g. replacing an identifier with a fully-qualified identifier). That’s an 
explanation of why we don’t tend to hit this problem - I’m not claiming it’s 
not a bug.

Can you log a JIRA case please?

Separately, it’s crazy that the “operands" field [1] of SqlBasicCall is public. 
We should deprecate the field for one point release (mirroring it into a 
private mutable list field) and then remove it. Later we can make the private 
list field immutable (copy on write) or something.

Julian

[1] 
https://github.com/apache/calcite/blob/4bc916619fd286b2c0cc4d5c653c96a68801d74e/core/src/main/java/org/apache/calcite/sql/SqlBasicCall.java#L34
 

 




> On Sep 22, 2021, at 3:36 PM, Wen Zhang  wrote:
> 
> Hello,
> 
> I'm trying to make shallow copies of `SqlNode` objects by calling the 
> `clone(SqlParserPos pos) 
> `
>  method. However, the `SqlBasicCall` class overrides this method with an 
> implementation 
> 
>  that might return a new `SqlCall` that shares the operand array with the 
> original. This is due to it calling `SqlOperator.createCall(SqlLiteral 
> functionQualifier, SqlParserPos pos, SqlNode... operands) 
> `,
>  whose default implementation 
> 
>  creates a new `SqlBasicCall` object without copying `operands`.
> 
> Is my understanding correct?
> 
> Thank you!
> 
> Best regards,
> Wen
> 

Re: COALESCE returns maximum of available CHAR capacity.

[Julian Hyde]

The current behavior looks ok to me. The expression has type CHAR(2) because 
the arguments have types CHAR(1) and CHAR(2). So ‘a’ is widened to ‘a ‘.

Julian

> On Sep 22, 2021, at 12:00 AM, stanilovsky evgeny  
> wrote:
> 
> sorry for typo, 'fill the ticket' of course ))
> 
>> hi community !
>> I found that COALESCE('a', 'bb') will return 'a ' <-- whitespace is present, 
>> i found that now it expands for maximum presented CHAR 
>> (RelDataType#inferReturnType logic)
>> sql 92 standard tolds:
>> 
>> '''
>> COALESCE (V1, V2) is equivalent to the following :
>> CASE WHEN V1 IS NOT NULL THEN V1 ELSE V2 END
>> '''
>> 
>> So i suppose that existing behavior is erroneous, if it`s ok i fell the 
>> ticket.
>> 
>> wdyt ?

Re: Guava version

[Julian Hyde]

Thank you! That worked.

I am close to a solution in 
https://github.com/julianhyde/calcite/tree/4789-guava-19 
<https://github.com/julianhyde/calcite/tree/4789-guava-19>. Reviews welcome.


> On Sep 21, 2021, at 8:24 PM, James Starr  wrote:
> 
> I believe you want  ./gradlew -Pguava.version=19.0.
> 
> James
> 
> 
> On Tue, Sep 21, 2021 at 4:09 PM Julian Hyde  wrote:
> 
>> How do I set the Guava version from the Gradle command line?
>> 
>> I’d expect
>> 
>>  ./gradlew -Dguava.version=19.0
>> 
>> to work but it doesn’t.
>> 
>> (We used to test the supported range of Guava versions in CI. Then we
>> stopped because Gradle. And now it turns out that we’re broken on Gradle <
>> 21.0.)
>> 
>> Julian
>> 
>> 

[jira] [Created] (CALCITE-4789) Build is broken on Guava versions < 21

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4789:


 Summary: Build is broken on Guava versions < 21
 Key: CALCITE-4789
 URL: https://issues.apache.org/jira/browse/CALCITE-4789
 Project: Calcite
  Issue Type: Bug
 Environment: The build is currently broken on Guava versions 20 and 
lower. We claim in the release notes to support Guava 19 and higher.

As part of this change, enable testing in CI of the lowest and highest 
supported Guava version. (We used to do this, but we stopped when we moved from 
Maven to Gradle. I still don't know how to set the Guava version from the 
Gradle command line.)
Reporter: Julian Hyde






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Guava version

[Julian Hyde]

How do I set the Guava version from the Gradle command line?

I’d expect

  ./gradlew -Dguava.version=19.0

to work but it doesn’t.

(We used to test the supported range of Guava versions in CI. Then we stopped 
because Gradle. And now it turns out that we’re broken on Gradle < 21.0.)

Julian

Re: ApacheCon 2021 talks about Apache Calcite

[Julian Hyde]

It’s not ApacheCon, but I am giving a talk at StrangeLoop in 2 weeks. The talk 
is about Morel but Calcite gets a prominent mention. 

https://thestrangeloop.com/2021/morel-a-functional-query-language.html 


Julian


> On Sep 20, 2021, at 2:04 PM, Stamatis Zampetakis  wrote:
> 
> Hi all,
> 
> ApacheCon 2021 is starting tomorrow, and I've seen there is a presentation
> by Vladimir Ozerov in the Bigdata track on Wednesday [1].
> 
> Are there any other Calcite related talks?
> 
> Best,
> Stamatis
> 
> [1] https://www.apachecon.com/acah2021/tracks/bigdatasql.html

Re: Tableau dialect specification for Calcite

[Julian Hyde]

Looks very promising, and has the .tdd file that I was looking for. Thanks 
Jacques.

That said, there is no LICENSE file. I will file an issue to clarify the 
license.

Julian



> On Sep 20, 2021, at 1:04 PM, Jacques Nadeau  wrote:
> 
> I wonder if the Dremio connector would be a good place to start since it
> uses Calcite:
> 
> https://github.com/dremio-hub/tableau-connector
> 
> 
> On Fri, Sep 10, 2021 at 12:37 PM Julian Hyde  wrote:
> 
>> Tableau has a text file [1] where you can define your dialect. It lets
>> Tableau know how to generate SQL to extract a year from a date, for
>> example.
>> 
>> Has anyone written that dialect specification for Calcite or a system
>> that uses Calcite's SQL parser (such as Druid, Phoenix)? Since those
>> systems have similar dialects, it would be good to pool resources
>> here.
>> 
>> Julian
>> 
>> [1] https://tableau.github.io/connector-plugin-sdk/docs/dialect
>> 

Tableau dialect specification for Calcite

[Julian Hyde]

Tableau has a text file [1] where you can define your dialect. It lets
Tableau know how to generate SQL to extract a year from a date, for
example.

Has anyone written that dialect specification for Calcite or a system
that uses Calcite's SQL parser (such as Druid, Phoenix)? Since those
systems have similar dialects, it would be good to pool resources
here.

Julian

[1] https://tableau.github.io/connector-plugin-sdk/docs/dialect

[jira] [Created] (CALCITE-4763) EXISTS_AGG, an aggregate function that returns whether count is positive

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4763:


 Summary: EXISTS_AGG, an aggregate function that returns whether 
count is positive
 Key: CALCITE-4763
 URL: https://issues.apache.org/jira/browse/CALCITE-4763
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


Add {{EXISTS_AGG}}, an aggregate function that returns whether count is greater 
than zero.

Thus {{EXISTS_AGG(*)}} is equivalent to {{COUNT(*) > 0}}, and {{EXISTS_AGG(c)}} 
is equivalent to {{COUNT(c) > 0}}.

{{EXISTS_AGG}} would mainly be of use internally. Since it produces a 
{{BOOLEAN}} value, we can use the value directly from an {{Aggregate}} without 
an intervening {{Project}}. It also captures the fact that we don't care how 
many rows were produced.

See also {{TRUE_AGG}}, CALCITE-4334.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-4762) Upgrade to Avatica 1.19

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4762:


 Summary: Upgrade to Avatica 1.19
 Key: CALCITE-4762
 URL: https://issues.apache.org/jira/browse/CALCITE-4762
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


Upgrade Calcite to Avatica version 1.19.

Currently version 1.19 has not been released.

1.19 fixes at least one issue whose test cases are attached as PRs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [jira] [Created] (CALCITE-4760) Should not need to initialize a jdbc:calcite driver to make a RelBuilder

[Julian Hyde]

I am inclined to fix this by creating a connection directly, not via the JDBC 
DriverManager. But that has the potential to break people who are relying on 
the DriverManager. If this would affect you, please speak up by adding comments 
to the JIRA case.

This particular case seems to be related to shading. There was a similar issue 
in Flink [1] whose cause is not clear.

Julian

[1] https://issues.apache.org/jira/browse/FLINK-4581

> On Sep 1, 2021, at 8:35 AM, Steven Talbot (Jira)  wrote:
> 
> Steven Talbot created CALCITE-4760:
> --
> 
> Summary: Should not need to initialize a jdbc:calcite driver to 
> make a RelBuilder
> Key: CALCITE-4760
> URL: https://issues.apache.org/jira/browse/CALCITE-4760
> Project: Calcite
>  Issue Type: Improvement
>Reporter: Steven Talbot
> 
> 
> We just tracked down a nasty issue with Calcite being shaded as a dependency 
> in our code that could cause a stack like
> 
>  
> {noformat}
> No suitable driver found for jdbc:calcite: 
> org/apache/calcite/tools/Frameworks.java:184:in withPrepare 
> org/apache/calcite/tools/RelBuilder.java:225:in create
> {noformat}
> {{}}
> 
> {{}}
> 
> when calling RelBuilder.create.
> 
> Our fault ultimately, but [~julianhyde] pointed out that Calcite really 
> shouldn't be trying to make a JDBC connection here.
> 
> {{}}
> 
> 
> 
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)

Re: Runtime Exception while using SqlParse.

[Julian Hyde]

Yes, the error message is not good enough. Can you please log a bug? The error 
comes from org.apache.calcite.linq4j.tree.BinaryExpression, which I very much 
doubt happens at parse time. If you can add a test case, or a complete error 
stack, that would be appreciated. 

> On Aug 18, 2021, at 9:30 PM, Jariv Narup  wrote:
> 
> Hi Thomas,
>   That is my bad - a carry over from the debugging. However the error
> still persists even with the following query:
> 
> *SELECT Purchase.Purchase_Date, Products.Product_Name FROM Purchase JOIN
> Products ON Purchase.Product_ID = Products.ID WHERE Purchase.Quantity > 10*
> 
> This is what I see:
> 
> *from: org.apache.calcite.sql.SqlNode  =
> {org.apache.calcite.sql.SqlJoin@2039} Method threw
> 'java.lang.RuntimeException' exception. Cannot evaluate
> org.apache.calcite.sql.SqlJoin.toString()* left:
> org.apache.calcite.sql.SqlNode  = {org.apache.calcite.sql.SqlIdentifier@2141}
> "PURCHASE"
> natural: org.apache.calcite.sql.SqlLiteral  =
> {org.apache.calcite.sql.SqlLiteral@2142} "FALSE"
> joinType: org.apache.calcite.sql.SqlLiteral  =
> {org.apache.calcite.sql.SqlLiteral@2143} "INNER"
> right: org.apache.calcite.sql.SqlNode  =
> {org.apache.calcite.sql.SqlIdentifier@2144} "PRODUCTS"
> conditionType: org.apache.calcite.sql.SqlLiteral  =
> {org.apache.calcite.sql.SqlLiteral@2145} "ON"
> condition: org.apache.calcite.sql.SqlNode  =
> {org.apache.calcite.sql.SqlBasicCall@2146} "`PURCHASE`.`PRODUCT_ID` =
> `PRODUCTS`.`ID`"
> pos: org.apache.calcite.sql.parser.SqlParserPos  =
> {org.apache.calcite.sql.parser.SqlParserPos@2147} "line 1, column 68"
> 
> That said, a more directed error text might definitely be more helpful.
> 
> Regards,
> Viraj Purang
> 
> On Tue, Aug 17, 2021 at 1:27 AM Thomas Rebele  wrote:
> 
>> Hi Jariv,
>> 
>> are you sure you want to join two columns of Purchase: *Purchase.Product_ID
>> = Purchase.ID*?
>> If that's the cause of the exception, the error message could be improved.
>> 
>> Cordialement / Best Regards,
>> *Thomas Rebele, PhD* | R Developer | Germany | www.tibco.com
>> 
>> 
>> On Mon, Aug 16, 2021 at 10:58 PM Jariv Narup 
>> wrote:
>> 
>>> Hi Team,
>>>   I am getting the following exception while trying to parse a SQL
>>> statement into its tokens. Is this expected behavior? If not, what would
>>> you suggest to work around this problem. The particulars are given below:
>>> 
>>> *Issue:*
>>> *Method threw 'java.lang.RuntimeException' exception. Cannot evaluate
>>> org.apache.calcite.sql.SqlJoin.toString().*
>>> 
>>> *Where:*
>>> This happens when I use the following API (it is visible on the "from"
>>> value):
>>> 
>>>SqlParser.Config parserConfig = SqlParser.config();
>>>parserConfig
>>>.withCaseSensitive(false)
>>>.withLex(Lex.ORACLE);
>>>SqlParser parser = SqlParser.create(sqlString, parserConfig);
>> [Line with the issue]*SqlNode sqlNode = parser.parseStmt();*
>>> 
>>> 
>>> *Debugger Image:*
>>> [image: image.png]
>>> *Version In Use:*
>>> I am using the following maven GAVs
>>> -org.apache.calcite:calcite-babel:1.27.0, so my assumption is that I am at
>>> the latest.
>>> 
>>> *SQL Statement with the issue:*
>>> 
>>> *SELECT Purchase.Purchase_Date, Products.Product_Name FROM Purchase JOIN
>>> Products ON Purchase.Product_ID = Purchase.ID WHERE Purchase.Quantity > 10*
>>> 
>>> I would appreciate any help on this.
>>> 
>>> Thanks - Jariv
>>> 
>> 

Re: Analyzing SQL queries with default catalog/database names

[Julian Hyde]

A key step in validation is to fully-quality table names. Thus in your example, 
mytable would become the string array [“hive”, “my_database”, “mytable”]. Any 
my_database2.mytable2 would become [“hive”, “my_database2”, “mytable2”]. But 
foo would remain foo, because it is in the query and is not a table.

The validator’s context is a path, which looks something like [[], [“hive”], 
[“hive”, “my_database”]]. Fully-, partially- and non-qualified table names are 
all looked up within that path.

Julian


> On Aug 17, 2021, at 12:02 AM, Zheng Shao  wrote:
> 
> Hi Calcite Dev,
> 
> What is the recommended way to analyze SQL queries with default
> catalog/database names?
> 
> For example, for query like "SELECT * FROM foo" where foo maps to a table
> in hive.my_database.
> 
> In Project Coral
> ,
> the code tries to add catalog/database names right after the parsing stage,
> but that would not work for cases like "WITH foo AS (SELECT * FROM mytable)
> SELECT * from foo" in which case there is no need to prepend.
> 
> -- 
> Zheng

Re: Visualization of Graphviz

[Julian Hyde]

Thomas,

Would you mind reviewing that commit for inclusion in master? I have logged 
https://issues.apache.org/jira/browse/CALCITE-4737 
<https://issues.apache.org/jira/browse/CALCITE-4737> to track.

Julian


> On Aug 13, 2021, at 2:03 AM, Thomas Rebele  wrote:
> 
> Thank you for pointing out the visualizer in the Tempura branch. It looks
> quite promising.
> 
> Cordialement / Best Regards,
> *Thomas Rebele, PhD* | R Developer | Germany | www.tibco.com
> 
> 
> On Thu, Aug 12, 2021 at 7:19 PM Julian Hyde  wrote:
> 
>> I don’t have a strong opinion on this, but I want to point out that there
>> are some GraphViz improvements as part of Tempura (the case is
>> https://issues.apache.org/jira/browse/CALCITE-4568 <
>> https://issues.apache.org/jira/browse/CALCITE-4568>; the specific commit
>> is
>> https://github.com/hbtoo/calcite/commit/c1240ca7bd054830ebb3107c7509e09a998d4b55
>> <
>> https://github.com/hbtoo/calcite/commit/c1240ca7bd054830ebb3107c7509e09a998d4b55
>>> ).
>> 
>> 
>> 
>>> On Aug 12, 2021, at 10:09 AM, Thomas Rebele 
>> wrote:
>>> 
>>> Hello,
>>> 
>>> the dot graphs generated by the planner get quite confusing even for
>> medium
>>> sized plans. I've added line breaks in the labels with a script and
>>> displayed it with the xdot tool, which makes it a bit better, but there
>> is
>>> still a lot of room for improvement. I've looked around for other tools
>>> (Gephi, Cytoscape), but no tool fulfills the requirements:
>>> 
>>> - algorithm(s) to avoid overlap of nodes
>>> - line breaks in node labels
>>> - provide a way to visualize RelSubset
>>> - allow to move nodes, hide them (this should update the visualization of
>>> the subsets)
>>> - search nodes by substrings of the label
>>> - find and highlight paths between nodes
>>> 
>>> Closest was Cytoscape, but I couldn't find a way to visualize the
>> subsets.
>>> Do you have any recommendations? One possibility would be to implement it
>>> with d3js.org, which would allow the integration of the tool in the
>> Calcite
>>> repository. Would there be interest in such a tool?
>>> 
>>> Cordialement / Best Regards,
>>> *Thomas Rebele, PhD* | R Developer | Germany | www.tibco.com
>> 
>> 

[jira] [Created] (CALCITE-4737) Add Volcano visualizer for debugging

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4737:


 Summary: Add Volcano visualizer for debugging
 Key: CALCITE-4737
 URL: https://issues.apache.org/jira/browse/CALCITE-4737
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


Add Volcano visualizer for debugging.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Visualization of Graphviz

[Julian Hyde]

I don’t have a strong opinion on this, but I want to point out that there are 
some GraphViz improvements as part of Tempura (the case is 
https://issues.apache.org/jira/browse/CALCITE-4568 
; the specific commit is 
https://github.com/hbtoo/calcite/commit/c1240ca7bd054830ebb3107c7509e09a998d4b55
 
).



> On Aug 12, 2021, at 10:09 AM, Thomas Rebele  wrote:
> 
> Hello,
> 
> the dot graphs generated by the planner get quite confusing even for medium
> sized plans. I've added line breaks in the labels with a script and
> displayed it with the xdot tool, which makes it a bit better, but there is
> still a lot of room for improvement. I've looked around for other tools
> (Gephi, Cytoscape), but no tool fulfills the requirements:
> 
> - algorithm(s) to avoid overlap of nodes
> - line breaks in node labels
> - provide a way to visualize RelSubset
> - allow to move nodes, hide them (this should update the visualization of
> the subsets)
> - search nodes by substrings of the label
> - find and highlight paths between nodes
> 
> Closest was Cytoscape, but I couldn't find a way to visualize the subsets.
> Do you have any recommendations? One possibility would be to implement it
> with d3js.org, which would allow the integration of the tool in the Calcite
> repository. Would there be interest in such a tool?
> 
> Cordialement / Best Regards,
> *Thomas Rebele, PhD* | R Developer | Germany | www.tibco.com

[jira] [Created] (CALCITE-4728) Parse and validate procedural code (such as SQL/PSM, PL/SQL, PL/pgSQL, T-SQL)

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4728:


 Summary: Parse and validate procedural code (such as SQL/PSM, 
PL/SQL, PL/pgSQL, T-SQL)
 Key: CALCITE-4728
 URL: https://issues.apache.org/jira/browse/CALCITE-4728
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


Parse and validate procedural code (such as the SQL standard's SQL/PSM, 
Oracle's PL/SQL, PostgreSQL's PL/pgSQL, MSSQL's T-SQL).

This would entail:
 * Extensions to the SQL parser. (I'm not sure whether this would be the core 
parser or an extended parser such as Babel.)
 * AST classes (sub-classes of {{SqlNode}}) for functions, procedures, blocks, 
variable and parameter declarations, variable assignment, if-then-else, loop, 
and so forth.
 * Extensions to the validator to validate blocks.
 * Extensions to the validator to validate a SQL statement that is inside a 
block. (Variables and parameters are in scope.)

Optional:
 * Extend Interpreter to execute functions, procedures, blocks.
 * Some means to convert a block into executable Java code.
 * Extend {{SqlDialect}} so that ASTs of functions, procedures and blocks can 
be emitted using the syntax of particular dialects.

The languages are sufficiently similar that we can use the same AST classes for 
all.

I don't think we need to extend {{RelNode}} to represent blocks. (It's not a 
good fit, since a block does not evaluate to a relation.) Possibly we would 
create some data structure to represent a validated block (e.g. the type of 
each variable; each use of a variable points to the variable's definition). 
{{SqlToRelConverter}} would create this data structure in order "freeze" the 
state of the validator.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: QueryableTable can only be implemented as ENUMERABLE convention

[Julian Hyde]

It’s a reasonable ask. Can you please log a JIRA case?

Julian


> On Aug 11, 2021, at 5:09 AM, Martin Jonsson 
>  wrote:
> 
> Hello all.
> It is mentioned in the code (1.27) that QueryableTable can only be with 
> Enumerable Convention. Is there a workaround to have it work with Bindable 
> Convention so not having to reimplement the whole big rule thing as it is 
> done in the Druid adapter?
> My adapter (REST based) is not that complex but the 
> ProjectableFilterableTable is a bit too simple.
> CheersMartin Jonsson

[jira] [Created] (CALCITE-4723) Check whether JDBC adapter generates "GROUP BY ()" against Oracle, DB2, MSSQL

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4723:


 Summary: Check whether JDBC adapter generates "GROUP BY ()" 
against Oracle, DB2, MSSQL
 Key: CALCITE-4723
 URL: https://issues.apache.org/jira/browse/CALCITE-4723
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


Oracle, DB2 and MSSQL have non-standard semantics for "GROUP BY ()". Standard 
behavior is to always return one "grand total" row, but [Oracle, DB2 and MSSQL 
return no rows if the input is 
empty|https://blog.jooq.org/2018/05/25/how-to-group-by-nothing-in-sql/].

Calcite's semantics is that "GROUP BY ()" always returns one row, and the JDBC 
adapter currently assumes that all back ends have the same semantics. On back 
ends that have different semantics, some queries might be giving incorrect 
results.

I suggest the following remedy:
 * Add a {{SqlDialect}} method {{boolean omitGrandTotalOnEmptyInput()}}
 * Run the test suite, and see whether we ever generate "GROUP BY ()" on one of 
the affected dialects. Try to write a test case where we do this.
 * Modify the dialects to generate safe SQL in these cases (possibly "GROUP BY 
()", or possibly something else). As the above article notes, it is 
particularly difficult to find SQL that works for MSSQL, because it bumps into 
the no-constants rule (see CALCITE-4702)




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reviewer/committer needed for [CALCITE-4544] Deprecate Metadata API backed by Java Reflection

[Julian Hyde]

https://issues.apache.org/jira/browse/CALCITE-4544 
 looks like something we 
want, but I am too busy to commit it. It is a major change that requires due 
diligence. Can someone volunteer to review and commit this?

Julian

Re: Integrate the Cosette prover in Calcite's RelNode rewrite tests

[Julian Hyde]

Shuxian,

It would be great to carry on this integration. I downloaded and built your 
project. (I also opened a pull request[1], adding a maven wrapper and a README.)

It’s difficult to get at the RelNodes in RelOptRulesTest if you’re outside 
Calcite. (Because it is a test, we don’t currently include it in a library.) 
Therefore I suggest that you create a fork of Calcite and modify the 
RelOptTestCase.Sql.checkPlanning method[2] to call your checker. Of course 
you’ll have to modify core/build.gradle.kts and add your checker library as a 
testImplementation.

Julian

[1] https://github.com/cosette-solver/cosette-parser/pull/1 

[2] 
https://github.com/apache/calcite/blob/d46137a197a2840ea5ff9f3b38bb86d423c9af11/core/src/test/java/org/apache/calcite/test/RelOptTestBase.java#L87

> On Jul 29, 2021, at 2:26 AM, Shuxian Wang  wrote:
> 
> Hello Calcite developers,
> 
> I am a Berkeley student working on a new version for the [Cosette SQL 
> prover](https://cosette.cs.washington.edu/).
> This [new implementation](https://github.com/cosette-solver/cosette-rs) aims 
> for higher performance and better SQL feature coverage.
> One goal for us is to formally check as much rewrite rules in Calcite as 
> possible, which are those located in 
> https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java.
> 
> The prover initially relies on a custom SQL parser, but we’ve decided to 
> build a [new one](https://github.com/cosette-solver/cosette-parser) depending 
> on Calcite’s parser instead.
> This parser takes in SQL and use Calcite to get a corresponding RelNode.
> Later on the parser translates that RelNode into a JSON format that will be 
> recognized by the prover.
> This makes integrating with the Calcite testing component very feasible, 
> given a way to extract the RelNode before/after each rewrite test and send 
> them to the prover in JSON.
> 
> I writing to ask if there is any general interest in such integration? If so, 
> some help on extracting the RelNode from the Calcite test cases would be 
> appreciated.
> 
> Sincerely,
> 
> Shuxian Wang

[jira] [Created] (CALCITE-4720) Obsolete the Collect relational operator, using Aggregate and ARRAY_AGG (and new aggregate functions MULTISET_AGG and MAP_AGG) instead

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4720:


 Summary: Obsolete the Collect relational operator, using Aggregate 
and ARRAY_AGG (and new aggregate functions MULTISET_AGG and MAP_AGG) instead
 Key: CALCITE-4720
 URL: https://issues.apache.org/jira/browse/CALCITE-4720
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


The {{Collect}} relational operator converts a multi-row relation into a 
relation with a single row and a column whose type is {{MULTISET}}.

But it is difficult to generalize it; we would like to:
*  Generating multiple rows, one for each group key, rather than a single row 
for the whole relation;
* Generate an {{ARRAY}} or {{MAP}} rather than a {{MULTISET};
* Generate a collection of scalars rather than a collection of records if the 
input is a single column (e.g. {{INTEGER MULTISET}} rather than {{ROW(INTEGER 
i) MULTISET}})

And, it is difficult to maintain; it is a minor RelNode that has only 2 
implementations (that I know of) and I'm sure that there are bugs and missing 
support in SqlToRelConverter and the RelOptRule library.

We can achieve the same using the {{Aggregate}} operator and the {{ARRAY_AGG}} 
aggregate function. We would need new aggregate functions (let's call them 
{{MULTISET_AGG}} and {{MAP_AGG}}) for the {{MULTISET}} and {{MAP}} types.

Then we can obsolete {{Collect}}, and make current code paths use {{Aggregate}} 
instead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-4719) Add variants of RexSubQuery that collect sub-queries into MULTISET, ARRAY and MAP collections

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4719:


 Summary: Add variants of RexSubQuery that collect sub-queries into 
MULTISET, ARRAY and MAP collections
 Key: CALCITE-4719
 URL: https://issues.apache.org/jira/browse/CALCITE-4719
 Project: Calcite
  Issue Type: Bug
 Environment: Add variants of {{RexSubQuery}} that collect sub-queries 
into {{MULTISET}}, {{ARRAY}} and {{MAP}} collections. We currently use 
{{RexSubQuery}} for scalar sub-query, {{EXISTS}}, {{IN}} and some others; this 
allows us to defer conversion: convert via rewrites ({{RelOptRule}}) rather 
than in {{SqlToRelConverter}}. The same benefits would apply to {{MULTISET}}, 
{{ARRAY}} and {{MAP}} sub-queries.

Examples:
{code}
SELECT deptno, MULTISET(SELECT * FROM Emp WHERE deptno = d.deptno)
FROM Dept AS d

SELECT deptno, ARRAY(SELECT * FROM Emp WHERE deptno = d.deptno)
FROM Dept AS d

SELECT deptno, MAP(SELECT empno, job FROM Emp WHERE deptno = d.deptno)
FROM Dept AS d
{code}

Reporter: Julian Hyde






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Fwd: [DISCUSS] Developing an "Arrow Compute IR [Intermediate Representation]" to decouple language front ends from Arrow-native compute engines

[Julian Hyde]

On dev@arrow, Wes McKinney just started a discussion of a language for 
“programming” Arrow. One of its explicit goals is for systems like Calcite to 
generate such code as their physical plan.

This would be great for Calcite. Arrow started as an excellent runtime for 
high-performance data processing.

Julian


> Begin forwarded message:
> 
> From: Wes McKinney 
> Subject: [DISCUSS] Developing an "Arrow Compute IR [Intermediate 
> Representation]" to decouple language front ends from Arrow-native compute 
> engines
> Date: August 2, 2021 at 5:16:50 PM PDT
> To: dev 
> Reply-To: d...@arrow.apache.org
> 
> hi folks,
> 
> This idea came up in passing in the past -- given that there are
> multiple independent efforts to develop Arrow-native query engines
> (and surely many more to come), it seems like it would be valuable to
> have a way to enable user languages (like Java, Python, R, or Rust,
> for example) to communicate with backend computing engines (like
> DataFusion, or new computing capabilities being built in the Arrow C++
> library) in a fashion that is "lower-level" than SQL and specialized
> to Arrow's type system. So rather than leaving it to a SQL parser /
> analyzer framework to generate an expression tree of relational
> operators and then translate that to an Arrow-native compute-engine's
> internal grammer, a user framework could provide the desired
> Arrow-native expression tree / data manipulations directly and skip
> the SQL altogether.
> 
> The idea of creating a "serialized intermediate representation (IR)"
> for Arrow compute operations would be to serve use cases large and
> small -- from the most complex TPC-* or time series database query to
> the most simple array predicate/filter sent with an RPC request using
> Arrow Flight. It is deliberately language- and engine-agnostic, with
> the only commonality across usages being the Arrow columnar format
> (schemas and array types). This would be better than leaving it to
> each application to develop its own bespoke expression representations
> for its needs.
> 
> I spent a while thinking about this and wrote up a brain dump RFC
> document [1] and accompanying pull request [2] that makes the possibly
> controversial choice of using Flatbuffers to represent the serialized
> IR. I discuss the rationale for the choice of Flatbuffers in the RFC
> document. This PR is obviously deficient in many regards (incomplete,
> hacky, or unclear in places), and will need some help from others to
> flesh out. I suspect that actually implementing the IR will be
> necessary to work out many of the low-level details.
> 
> Note that this IR is intended to be more of a "superset" project than
> a "lowest common denominator". So there may be things that it includes
> which are only available in some engines (e.g. engines that have
> specialized handling of time series data).
> 
> As some of my personal motivation for the project, concurrent with the
> genesis of Apache Arrow, I started a Python project called Ibis [3]
> (which is similar to R's dplyr project) which serves as a "Python
> analytical query IR builder" that is capable of generating most of the
> SQL standard, targeting many different SQL dialects and other backends
> (like pandas). Microsoft ([4]) and Google ([5]) have used this library
> as a "many-SQL" middleware. As such, I would like to be able to
> translate between the in-memory "logical query" data structures in a
> library like Ibis to a serialized format that can be executed by many
> different Arrow-native query engines. The expression primitives
> available in Ibis should serve as helpful test cases, too.
> 
> I look forward to the community's comments on the RFC document [1] and
> pull request [2] -- I realize that arriving at consensus on a complex
> and ambitious project like this can be challenging so I recommend
> spending time on the "non-goals" section in the RFC and ask questions
> if you are unclear about the scope of what problems this is trying to
> solve. I would be happy to give Edit access on the RFC document to
> others and would consider ideas about how to move forward with
> something that is able to be implemented by different Arrow libraries
> in the reasonably near future.
> 
> Thanks,
> Wes
> 
> [1]: 
> https://docs.google.com/document/d/1C_XVOG7iFkl6cgWWMyzUoIjfKt-X2UxqagPJrla0bAE/edit#
> [2]: https://github.com/apache/arrow/pull/10856
> [3]: https://ibis-project.org/
> [4]: http://cidrdb.org/cidr2021/papers/cidr2021_paper08.pdf
> [5]: 
> https://cloud.google.com/blog/products/databases/automate-data-validation-with-dvt

Re: ClassCastException: FamilyOperandTypeChecker cannot be cast to SqlOperandMetadata

[Julian Hyde]

SqlOperandMetadata only needs to be implemented by user-defined functions. 
(User-defined functions have fixed number of parameters, and the parameters 
have names; built-in functions may have a variable number of arguments, but 
they do not have names.)

So, maybe the cause is a function that isn’t sure whether it’s a UDF or a 
built-in.

My money is on user error rather than Calcite bug.

> On Jul 28, 2021, at 12:19 AM, Yanjing Wang  wrote:
> 
> It may be a bug caused by CALCITE-4427
> , I
> think lookupSubjectRoutines behavior is a bit strange. when found routines
> are less than two, It returns directly whatever the routine SqlKind is.
> otherwise filtered by SqlKind. This behavior may cause empty routine
> results when all candidates are filtered out. @Julian, could you help the
> exception to be fixed?
> 
> wrstrs  于2021年7月27日周二 下午3:33写道：
> 
>> Hi, all
>> 
>> how to solve this problem?
>> 
>> 
>> 
>> 
>> my code:
>> String sql = "select UNIX_TIMESTAMP(concat(substr('2021-07-27
>> 12:12:12',0,11),'09:00:00'))";
>> 
>> SqlOperatorTable opTab =
>>   
>> SqlLibraryOperatorTableFactory.INSTANCE.getOperatorTable(EnumSet.of(
>>SqlLibrary.HIVE,
>> SqlLibrary.SPARK, SqlLibrary.BIG_QUERY, SqlLibrary.ORACLE,
>> SqlLibrary.MYSQL, SqlLibrary.STANDARD));
>> SchemaPlus rootSchema = Frameworks.createRootSchema(true);
>> 
>> FrameworkConfig frameworkConfig = Frameworks.newConfigBuilder()
>>.defaultSchema(rootSchema)
>>.operatorTable(opTab)
>>.build();
>> 
>> Planner planner = Frameworks.getPlanner(frameworkConfig);
>> SqlNode sqlNode = planner.parse(sql);
>> SqlNode validateSqlNode = planner.validate(sqlNode);
>> 
>> 
>> 
>> throw exception:
>> 
>> Exception in thread "main" org.apache.calcite.tools.ValidationException:
>> java.lang.ClassCastException:
>> org.apache.calcite.sql.type.FamilyOperandTypeChecker cannot be cast to
>> org.apache.calcite.sql.type.SqlOperandMetadata
>> 
>>at
>> org.apache.calcite.prepare.PlannerImpl.validate(PlannerImpl.java:228)
>> 
>> Caused by: java.lang.ClassCastException:
>> org.apache.calcite.sql.type.FamilyOperandTypeChecker cannot be cast to
>> org.apache.calcite.sql.type.SqlOperandMetadata
>> 
>>at
>> org.apache.calcite.sql.validate.implicit.TypeCoercionImpl.userDefinedFunctionCoercion(TypeCoercionImpl.java:589)
>> 
>>at
>> org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:303)
>> 
>>at
>> org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:231)

Re: does Calcite have the mathematical equation transformation such a + b = c to a = c - b

[Julian Hyde]

Calcite doesn’t transform

  a + b = c” to “a = c - b

for good reason… someone would also want to transform

  a - b = c” to “a = c + b

and the two transforms would cycle.

To put it another way: Calcite’s expression simplification is not 
smart/powerful enough to do theorem proving. Our rule of thumb is that 
simplification rules must convert to something simpler. If it’s equivalent but 
not (by some measure) simpler, we don’t do it.

That said, I think there’s a way we could handle your case. We could have a 
rule that recognizes

  variable1 - constant1 = constant2

and converts to

  variable1 = constant2 + constant1

(which matches your date_sub case). If someone were to add a rule that 
recognizes

variable1 + constant1 = constant2

and converts to

  variable1 = constant2 + constant1

it wouldn’t form a cycle.

Can you please add a JIRA case for this? You should cover all forms of “+”: 
arithmetic plus, DATETIME_PLUS (e.g. date + interval), and interval + interval.

One thing to watch out for (and test carefully) is arithmetic near the limits 
of data types. If a, b and c are SQL INTEGER values then “a + b = c” is always 
equivalent to “a = c - b”, but I don’t  think that’s the case if they are 
DOUBLE values.

By the way, I am working on making RexSimplifier more pluggable (via a new 
interface RexRule) [1] and this would be a nice implementation of RexRule.

Julian

[1] https://issues.apache.org/jira/browse/CALCITE-4559


> On Jul 29, 2021, at 1:58 AM, Yanjing Wang  wrote:
> 
> Hi, guys
>  I have a scenario to calculate the input ref in an expression, for
> example:
>  date_sub(dt, 1) = '2021-07-29' to dt = date_add('2021-07-29', 1)
>  thus I can conclude that dt =  '2021-07-30'.
> 
>  I start from the most simplest form dt + 1 = 3. but it can't be
> simplified or reduced.
>  I tried dt = 3 - 1 and it can be reduced to dt = 2.
>  I wonder if I can transform dt + 1 = 3 to dt = 3 - 1, the dt will be
> calculated.
> 
>  So I think if we need a transformation rule to implement this kind of
> commute for expression,
>  or we have other existing solution.
> 
>  Thanks for any suggestions.

[jira] [Created] (CALCITE-4711) RexProgramBuilder should not simplify

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4711:


 Summary: RexProgramBuilder should not simplify
 Key: CALCITE-4711
 URL: https://issues.apache.org/jira/browse/CALCITE-4711
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


{{RexProgramBuilder}} currently simplifies expressions even if no 
{{RexSimplifier}} instance is supplied in its constructor: the method 
{{registerInternal}} creates a {{RexSimplifier}}.

This simplification used to be beneficial. For instance, as of 
[4ef9f467|https://github.com/apache/calcite/commits/4ef9f46757528d21c510eb8bd171fa04ba86e36d],
 {{RexProgramBuilder}} can simplify "x AND y AND x" to "x AND y". But now we 
have RexSimplifier, which is applied pretty much any time a RexNode is included 
in a RelNode via a RelBuilder method.

Simplify is now called in too many places, and I suspect that the running time 
is multiplying. Consider: {{RexSimplify.simplifyCast}} calls 
{{RexExecutor.reduce}} to reduce constants. To do this, {{RexExecutor}} uses 
{{RexProgramBuilder}} to generate a program (which will then be executed to 
yield the reduced value), and the RexProgramBuilder calls simplify again. It is 
ridiculous to call simplify at so many levels. It's time that we reduced the 
responsibilities of {{RexProgramBuilder}} to just building programs.

(It is possible that, once a {{RexProgram}} has been created, we can see some 
patterns in the flattened expressions that would not be apparent to 
{{RexSimplify}}. If this happens, we could have a simplify step that works on 
{{Calc}} after its program has been generated.)

After this change, {{RexProgramBuilder}} will simplify only if a 
{{RexSimplify}} is passed in the constructor. {{RexProgramBuilder}} will no 
longer be able to simplify "x AND y AND x". We will add a test to ensure that 
{{RelBuilder.filter}} can do this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Changing the Parser

[Julian Hyde]

There is some documentation: 
https://calcite.apache.org/docs/adapter.html#extending-the-parser 
. 

> On Jul 29, 2021, at 7:04 PM, Vladimir Sitnikov  
> wrote:
> 
> https://issues.apache.org/jira/browse/CALCITE-4701 seems relevant.
> 
> I do not know if "parser extension" build system plugins already exist,
> however, I think it would be nice to have one in Calcite.
> 
> Vladimir

[jira] [Created] (CALCITE-4707) Optimize incremental maintenance of materialized views

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4707:


 Summary: Optimize incremental maintenance of materialized views
 Key: CALCITE-4707
 URL: https://issues.apache.org/jira/browse/CALCITE-4707
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


Optimize incremental maintenance of materialized views, when we know what DML 
has occurred since the last build.

The goal here is to develop an algebraic approach (i.e. new relational 
operators and transformation rules) that will generate an optimal query for 
maintaining the view.

Consider a materialized view
{code}
CREATE TABLE EmpsByDeptno AS
  SELECT deptno, COUNT(*) AS c, SUM(sal) AS ss
  FROM Emps
  GROUP BY deptno;
{code}

We built it at time T1, when the value of {{Emps}} was {{Emps1}}, and it had 
the value {{EmpsByDeptno1}}.

It is now T2, and since then, new employees have been created in the 
{{NewEmps}} table. Thus {{Emps2 = Emps1 UNION ALL NewEmps}}. We could build a 
new MV,
{code}
CREATE TABLE EmpsByDeptno2 AS
  SELECT deptno, COUNT(*) AS c, SUM(sal) AS ss
  FROM (SELECT * FROM Emps1
  UNION ALL
  SELECT * FROM NewEmps)
  GROUP BY deptno;
{code}but we would prefer to generate a DML command to modify {{EmpsByDeptno}} 
in place:{code}
MERGE INTO EmpsByDeptno AS e
  USING (SELECT deptno, COUNT(*) AS c, SUM(sal) AS ss
  FROM NewEmps) AS de
  ON de.deptno = e.deptno
  WHEN MATCHED THEN
UPDATE SET c = c + de.c, ss = ss + de.ss
  WHEN NOT MATCHED THEN
INSERT (deptno, c, ss)
{code}

We propose two new relational operators:
 * {{Diff(S, R)}} computes the difference between two relations. It has a same 
schema as R and S but with an additional column {{delta}}. Rows in S but not in 
R are called insertions, and have delta=1. Rows that are in R but not in S are 
called deletions, and have delta=-1.
 * {{Patch(R, P)}} applies 

The relational operators are named by analogy with the UNIX utilities {{diff}} 
and {{patch}}. (But {{Diff}}'s arguments are reversed compared to {{diff}}.) 
Consider:
{code}
grep r /usr/share/dict/words  > r.txt
grep s /usr/share/dict/words  > s.txt
diff r.txt s.txt > p.patch
patch -p1 r.txt < p.patch
cmp r.txt s.txt.  # r and s are now identical
{code}

The relation {code}R = Patch(S, Diff(R, S)){code} always holds. {{Diff}} 
computes the difference between R and S, and when {{Patch}} applies that 
difference to {{S}} you get {{R}}.

{{Patch}} and {{Diff}} are intended by be generalizations of set operators. If 
there are only additions, i.e. {{R}} is a superset of {{S}}, then you can 
substitute {{Minus}} for {{Diff}}, and {{Union}} for {{Patch}}:
{code}
R = Union(S, Minus(R, S))
{code}

So, how do these relational operators solve our original problem?

An {{INSERT}} statement is assignment to a relation of itself union some new 
rows. ({{INSERT INTO R SELECT * FROM P}} is {{R  R UNION P}}).

A {{MERGE}} statement is similar to {{INSERT}} but allows updates.

The {{MERGE}} query above is represented as {code}EmpsByDeptno  
Patch(EmpsByDeptno, Diff(NewEmpsByDeptno, EmpsByDeptno)){code}

Lastly we need relational transformation rules to optimize the expressions into 
per-key updates.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Support on Avatica driver for Reactive JDBC (R2DBC)

[Julian Hyde]

Avatica does not currently support R2DBC.

I was not previously aware of R2DBC but it looks like a useful extension to 
JDBC for streaming apps. (When I was at SQLstream, we had to make similar 
extensions, e.g. providing a callback method that would be executed each time a 
record was available in a ResultSet. I am glad that this is becoming 
standardized.)

It would be great if someone implemented the R2DBC interface on Avatica’s JDBC 
driver, so that servers such as Druid could use it out of the box.

Julian


> On Jul 19, 2021, at 10:24 PM, Shubham  wrote:
> 
> Hi Apache Team,
> 
> We have adopted Druid in our organisation for OLAP in our Data Engineering
> landscape.
> 
> We are trying to query Druid through Avatica client for jdbc which is the
> popular way of querying Druid.
> 
> We are looking for support for Reactive stack using R2DBC.
> 
> Wanted know if the avatica driver will accept R2DBC or not when connecting
> to Druid?
> 
> Best Regards
> Shubham

On vacation

[Julian Hyde]

I’m on vacation for the next week, pretty much offline. I’d appreciate if 
someone could review PRs, etc. 

Julian

[jira] [Created] (CALCITE-4687) Add LIMIT to WITHIN GROUP clause of aggregate functions

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4687:


 Summary: Add LIMIT to WITHIN GROUP clause of aggregate functions
 Key: CALCITE-4687
 URL: https://issues.apache.org/jira/browse/CALCITE-4687
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


Add LIMIT to WITHIN GROUP clause of aggregate functions. LIMIT is not in the 
SQL standard, but it is useful, and is not hard to implement.

The following query computes the 3 highest paid employees in each department:
{code:java}
SELECT deptno, ARRAY_AGG(sal) WITHIN GROUP (ORDER BY sal DESC LIMIT 3)
FROM Emp
GROUP BY deptno {code}
It can be implemented efficiently (using a merge sort that discards all but the 
top 3 rows in each key, at each pass).

Note that BigQuery does not support the {{WITHIN GROUP}} clause, but in the 
{{ARRAY_AGG}} function, the {{ORDER BY}} and {{LIMIT}} sub-clauses appear 
within the parentheses, like this: {{ARRAY_AGG(sal ORDER BY sal DESC LIMIT 
3)}}. In Calcite, you can use either syntax for {{ARRAY_AGG}}, 
{{ARRAY_CONCAT_AGG}}, {{GROUP_CONCAT}}, {{STRING_AGG}} functions; we should add 
{{LIMIT}} in both.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Support for constraints in jdbc adapter and optional decorrelation step in PlannerImpl.rel?

[Julian Hyde]

Re 1. It seems that the Server parser goes further than the Server 
implementation. You can specify PRIMARY KEY in the DDL, and we can parse it, 
but Server cannot create tables with primary keys. Please log a bug. 
Contributions welcome.

One workaround is for you to parse but not execute. The information you need is 
in the parse tree (SqlCreateTable, SqlKeyConstraint).

Re 2. Can you log a bug? Related to 
https://issues.apache.org/jira/browse/CALCITE-3582 
 and 
https://issues.apache.org/jira/browse/CALCITE-4642 
.

Julian




> On Jul 4, 2021, at 8:14 AM, Arthur Pan  wrote:
> 
> Greetings to dev@calcite:
> 
> I am currently working on extracting RelNodes from DML queries with schemas
> derived from DDL statements, and I have encountered a few issues:
> 
> 1. My current method to derive schemas from DDL statements is
>(a) connect to "jdbc:calcite" with parser factory set to
> ServerDdlExecutor.ParserFactory
>(b) execute the DDL statements
>(c) extract schema with connection.getRootSchem()
>  Currently simple create statements can be correctly executed. However,
> the execution (at
> org.apache.calcite.server.ServerDdlExecutor.execute(ServerDdlExecutor.java:472))
> will fail when constraints (e.g. PRIMARY KEY) are involved. The CREATE
> statement I used is pasted below:
>CREATE TABLE EMP (EMP_ID INTEGER NOT NULL,
> EMP_NAME VARCHAR,
> DEPT_ID INTEGER,
> PRIMARY KEY (DEPT_ID))
>  Is there a way to fix this? Or is there a better way to derive schema
> from DDL statements?
> 
> 2. It seems that currently PlannerImpl.rel(SqlNode) decorrelate (at
> PlannerImpl.java:265) the RelRoot converted from SqlNode, regardless what
> the RelBuilderConfig was passed in to the planner. Is there a way to bypass
> such a transformation to RelNode and fetch the raw RelRoot converted from
> SqlNode by SqlToRelConverter in PlannerImpl?
> 
> I would appreciate any insights into anything related. Thanks in advance!
> 
> Sincerely,
> Arthur Pan

Re: Deduplicate correlate variables question.

[Julian Hyde]

Thanks, yes, that’s better.

> On Jun 30, 2021, at 10:34 PM, stanilovsky evgeny  
> wrote:
> 
> thanks! done, i hope )
> 
>> Thanks! Can you improve the JIRA subject, so that everyone reading the 
>> release notes will understand it? (I had to look up COLLECTION_TABLE.)
>> 
>> 
>>> On Jun 30, 2021, at 5:12 AM, stanilovsky evgeny 
>>>  wrote:
>>> 
>>> Guys, i fill the ticket [1] and PR is ready for check, can anyone take a 
>>> look on it ?
>>> Additional test appended, all tests locally passed.
>>> 
>>> [1] https://issues.apache.org/jira/browse/CALCITE-4673
>>> 
>>> thanks !
>>> 
> The PR needs to fix a bug or implement a feature. So, first you should 
> log a JIRA case describing what doesn’t work. Write tests for what 
> doesn’t work that you want to make work. (Or maybe you can 
> refactor/generalize existing tests.) Then submit a PR, and we will review 
> that PR on the basis of whether it fixes the bug.
> 
> That may sound like a lot of process. But the alternative is people just 
> randomly changing code.
 
 No, this sounds like a proper way )
 Ok, i understand you !
 
> 
>> On Jun 3, 2021, at 7:37 AM, stanilovsky evgeny 
>>  wrote:
>> 
>> Julian, thanks for reply and comments.
>> Can you explain, is it would possible to commit PR containing 
>> deduplication code moved upper this flag ?
>> If so - i will create PR and rerun all existing tests, of course.
>> Thanks !
>> 
>>> Master is a moving target. Apparently you are using a version of the
>>> code from sometime in March. These links work better:
>>> 
>>> [1] 
>>> https://github.com/apache/calcite/blob/796675c9b33e0461bc45a72780162d474a4b098b/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L2881
>>> 
>>> [2] 
>>> https://github.com/apache/calcite/blob/796675c9b33e0461bc45a72780162d474a4b098b/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6209
>>> 
>>> I don't know the exact cause of what you are seeing, but when you set
>>> "expand=false" you are choosing a newer (and better) code path. We do
>>> not have feature parity between the code paths. Nor do we write tests
>>> for both code paths.
>>> 
>>> Some day I'd like to officially deprecate, then obsolete,
>>> "expand=true". It is de facto deprecated because we put more effort
>>> into the "expand=false" path.
>>> 
>>> Julian
>>> 
>>> 
>>> On Fri, May 28, 2021 at 1:19 AM stanilovsky evgeny
>>>  wrote:
 
 Hi, calciters )
 
 I am trying to figure out why DeduplicateCorrelateVariables [1] is 
 called
 only if withExpand [2] flag is true ? Why we don`t need to deduplicate 
 in
 appropriate case ?
 
 [1]
 https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L2881
 
 [2]
 https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6209
 
 Thanks !

Re: Deduplicate correlate variables question.

[Julian Hyde]

Thanks! Can you improve the JIRA subject, so that everyone reading the release 
notes will understand it? (I had to look up COLLECTION_TABLE.)


> On Jun 30, 2021, at 5:12 AM, stanilovsky evgeny  
> wrote:
> 
> Guys, i fill the ticket [1] and PR is ready for check, can anyone take a look 
> on it ?
> Additional test appended, all tests locally passed.
> 
> [1] https://issues.apache.org/jira/browse/CALCITE-4673
> 
> thanks !
> 
>>> The PR needs to fix a bug or implement a feature. So, first you should log 
>>> a JIRA case describing what doesn’t work. Write tests for what doesn’t work 
>>> that you want to make work. (Or maybe you can refactor/generalize existing 
>>> tests.) Then submit a PR, and we will review that PR on the basis of 
>>> whether it fixes the bug.
>>> 
>>> That may sound like a lot of process. But the alternative is people just 
>>> randomly changing code.
>> 
>> No, this sounds like a proper way )
>> Ok, i understand you !
>> 
>>> 
 On Jun 3, 2021, at 7:37 AM, stanilovsky evgeny 
  wrote:
 
 Julian, thanks for reply and comments.
 Can you explain, is it would possible to commit PR containing 
 deduplication code moved upper this flag ?
 If so - i will create PR and rerun all existing tests, of course.
 Thanks !
 
> Master is a moving target. Apparently you are using a version of the
> code from sometime in March. These links work better:
> 
> [1] 
> https://github.com/apache/calcite/blob/796675c9b33e0461bc45a72780162d474a4b098b/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L2881
> 
> [2] 
> https://github.com/apache/calcite/blob/796675c9b33e0461bc45a72780162d474a4b098b/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6209
> 
> I don't know the exact cause of what you are seeing, but when you set
> "expand=false" you are choosing a newer (and better) code path. We do
> not have feature parity between the code paths. Nor do we write tests
> for both code paths.
> 
> Some day I'd like to officially deprecate, then obsolete,
> "expand=true". It is de facto deprecated because we put more effort
> into the "expand=false" path.
> 
> Julian
> 
> 
> On Fri, May 28, 2021 at 1:19 AM stanilovsky evgeny
>  wrote:
>> 
>> Hi, calciters )
>> 
>> I am trying to figure out why DeduplicateCorrelateVariables [1] is called
>> only if withExpand [2] flag is true ? Why we don`t need to deduplicate in
>> appropriate case ?
>> 
>> [1]
>> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L2881
>> 
>> [2]
>> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6209
>> 
>> Thanks !

[jira] [Created] (CALCITE-4671) Allow Avatica connect-string properties in any case (lowerCamel, UPPER_SNAKE, etc.)

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4671:


 Summary: Allow Avatica connect-string properties in any case 
(lowerCamel, UPPER_SNAKE, etc.)
 Key: CALCITE-4671
 URL: https://issues.apache.org/jira/browse/CALCITE-4671
 Project: Calcite
  Issue Type: Bug
  Components: avatica
Reporter: Julian Hyde


Avatica's built-in properties are mostly lower_snake case (e.g. 
{{truststore_password}}; the one exception being {{timeZone}} in lowerCamel; 
see [client 
reference|https://calcite.apache.org/avatica/docs/client_reference.html]); 
Calcite's built-in properties are lowerCamel case (e.g. 
[approximateTopN|https://calcite.apache.org/javadocAggregate/org/apache/calcite/config/CalciteConnectionProperty.html#APPROXIMATE_TOP_N];
 see [Calcite JDBC driver 
reference|https://calcite.apache.org/docs/adapter.html#jdbc-connect-string-parameters%C2%A0]).
 Avatica's properties are inherited by Calcite, so there is bound to be 
confusion.

Avatica's connect string parser (also used in Calcite) should allow any case. 
Thus {{truststore_password}}, {{truststorePassword}}, {{TRUSTSTORE_PASSWORD}}, 
{{TruststorePassword}} are all acceptable. {{trustStorePassword}} is not 
acceptable, because "truststore" is one word.

In Calcite and Avatica doc, add a note that properties can are accepted in any 
case.

Change Avatica's properties to lowerCamel case in both doc and code. (The old 
forms, e.g. "truststore_password", will continue to work.)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New committer: Vladimir Ozerov

[Julian Hyde]

I agree there is a lack of documentation, and it is hurting us.

One explanation is that there are no companies selling Calcite directly. 
(Compare, for example Druid, Hive and Cassandra.) If there were, those 
companies would employ technical writers to contribute documentation.

On a related note. Stamatis and I are going to VLDB in Copenhagen [1] to give a 
tutorial on Calcite to the conference attendees (many of whom are from 
academia). Stamatis has been working hard on a hands-on tutorial that allows 
people to build a useful project using Calcite. This effort should help build 
awareness of Calcite, and provide a concrete way for people to get started 
using Calcite.

Julian

[1] https://boss-workshop.github.io/boss-2021/


> On Jun 25, 2021, at 7:14 AM, Vladimir Ozerov  wrote:
> 
> Thank you, everybody!
> 
> While I enjoy hacking Calcite's codebase, my colleagues and I observe a
> consistent signal from the "field" - lack of documentation blocks many
> attempts to integrate Apache Calcite. Engineers simply get lost with no
> idea on how to move forward. So I more and more believe that (usually
> boring) documentation-related efforts might boost Apache Calcite adoption
> dramatically. Hopefully, we will find time to invest in it.
> 
> Regards,
> Vladimir.
> 
> пт, 25 июн. 2021 г. в 09:52, Fan Liya :
> 
>> Congratulations, Vladimir!
>> Thanks for the good articles.
>> 
>> Best,
>> Liya Fan
>> 
>> On Fri, Jun 25, 2021 at 9:07 AM Julian Hyde 
>> wrote:
>> 
>>> Welcome, Vladimir!
>>> 
>>>> On Jun 24, 2021, at 6:00 PM, Albert  wrote:
>>>> 
>>>> Congrats.
>>>> just found the blog:
>> https://www.querifylabs.com/author/vladimir-ozerov
>>>> 
>>>> On Thu, Jun 24, 2021 at 2:27 PM Alessandro Solimando <
>>>> alessandro.solima...@gmail.com> wrote:
>>>> 
>>>>> Congratulations Vladimir, well deserved, I had the chance to read some
>>>>> of the blog posts and I have appreciated them very much.
>>>>> 
>>>>> Best regards,
>>>>> Alessandro
>>>>> 
>>>>> On Thu, 24 Jun 2021 at 07:58, Viliam Durina 
>>> wrote:
>>>>>> 
>>>>>> Congratulations!
>>>>>> 
>>>>>> Viliam
>>>>>> 
>>>>>> On Thu, 24 Jun 2021 at 06:58, Forward Xu 
>>> wrote:
>>>>>> 
>>>>>>> Congratulations!
>>>>>>> 
>>>>>>> 
>>>>>>> Best，
>>>>>>> 
>>>>>>> Forward
>>>>>>> 
>>>>>>> Danny Chan  于2021年6月24日周四 上午11:51写道：
>>>>>>> 
>>>>>>>> Congrats, Vladimir!
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Danny Chan
>>>>>>>> 
>>>>>>>> Yanjing Wang  于2021年6月24日周四 上午11:41写道：
>>>>>>>> 
>>>>>>>>> Congrats, Vladimir!
>>>>>>>>> 
>>>>>>>>> Roman Kondakov  于2021年6月24日周四
>>>>> 上午11:22写道：
>>>>>>>>> 
>>>>>>>>>> Congratulations, Vladimir!
>>>>>>>>>> 
>>>>>>>>>> Roman Kondakov
>>>>>>>>>> 
>>>>>>>>>> On 24.06.2021 12:23, 段雄 wrote:
>>>>>>>>>>> Congratulations!
>>>>>>>>>>> 
>>>>>>>>>>> XING JIN  于2021年6月24日周四 上午10:21写道：
>>>>>>>>>>> 
>>>>>>>>>>>> Congratulations ~
>>>>>>>>>>>> 
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Jin
>>>>>>>>>>>> 
>>>>>>>>>>>> guangyuan wang  于2021年6月24日周四
>>>>> 上午9:50写道：
>>>>>>>>>>>> 
>>>>>>>>>>>>> Congratulations!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Francis Chuang  于2021年6月24日周四
>>>>>>> 上午6:39写道：
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Congrats, Vladimir!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Francis
>>>>>&g

Re: why do We use inner class for CallCopyingArgHandler within SqlShuttle rather static nested class as ArgHandlerImpl within SqlBasicVisitor.

[Julian Hyde]

> I find the CallCopyingArgHandler is not static, but the doc doesn't specify
> the reason.
> Does someone know why?

If you try making it static you will see that it is unable to use the current 
shuttle to visit the children.

CallCopyingArgHandler exists for a very short duration - visit of one SqlCall 
by the shuttle.

Still, there are multiple CallCopyingArgHandler instances active at the same 
time, so having one CallCopyingArgHandler per Shuttle instance (shared among 
all call visits) is not an option.

> On Jun 25, 2021, at 6:00 AM, Yanjing Wang  wrote:
> 
> In my scenarios, I created some visitors for sql node to rewrite the sql,
> for example, rewriting all literals to generate a sql template.
> 
> But I don't like many new constructors scattered somewhere. I like
> singleton and dependency injection. so I read SqlShuttle source code to
> look for the probability.

I’m not a fan of dependency injection. I can see that it has its place, e.g. 
for gluing together components at the macro level. But it requires mutable 
fields. At the micro level, I am a fan of immutability, final fields assigned 
in the constructor.

I agree that CallCopyingArgHandler is a strange bird. It has the purpose of a 
“strategy pattern”, except that it has some state. It was written a long time 
ago, and I’m not sure I’d do the same today. (I haven’t given it much thought.)

Julian

Re: [ANNOUNCE] New committer: Vladimir Ozerov

[Julian Hyde]

Welcome, Vladimir!

> On Jun 24, 2021, at 6:00 PM, Albert  wrote:
> 
> Congrats.
> just found the blog: https://www.querifylabs.com/author/vladimir-ozerov
> 
> On Thu, Jun 24, 2021 at 2:27 PM Alessandro Solimando <
> alessandro.solima...@gmail.com> wrote:
> 
>> Congratulations Vladimir, well deserved, I had the chance to read some
>> of the blog posts and I have appreciated them very much.
>> 
>> Best regards,
>> Alessandro
>> 
>> On Thu, 24 Jun 2021 at 07:58, Viliam Durina  wrote:
>>> 
>>> Congratulations!
>>> 
>>> Viliam
>>> 
>>> On Thu, 24 Jun 2021 at 06:58, Forward Xu  wrote:
>>> 
 Congratulations!
 
 
 Best，
 
 Forward
 
 Danny Chan  于2021年6月24日周四 上午11:51写道：
 
> Congrats, Vladimir!
> 
> Best,
> Danny Chan
> 
> Yanjing Wang  于2021年6月24日周四 上午11:41写道：
> 
>> Congrats, Vladimir!
>> 
>> Roman Kondakov  于2021年6月24日周四
>> 上午11:22写道：
>> 
>>> Congratulations, Vladimir!
>>> 
>>> Roman Kondakov
>>> 
>>> On 24.06.2021 12:23, 段雄 wrote:
 Congratulations!
 
 XING JIN  于2021年6月24日周四 上午10:21写道：
 
> Congratulations ~
> 
> Best,
> Jin
> 
> guangyuan wang  于2021年6月24日周四
>> 上午9:50写道：
> 
>> Congratulations!
>> 
>> Francis Chuang  于2021年6月24日周四
 上午6:39写道：
>> 
>>> Congrats, Vladimir!
>>> 
>>> Francis
>>> 
>>> On 24/06/2021 7:48 am, Haisheng Yuan wrote:
 Congratulations and thanks for your contributions,
>> Vladimir!
 
 Regards,
 Haisheng
 
 On 2021/06/23 21:34:40, Stamatis Zampetakis <
>> zabe...@gmail.com
> 
> wrote:
> Apache Calcite's Project Management Committee (PMC) has
 invited
>> Vladimir
> Ozerov to
> become a committer, and we are pleased to announce that
>> he has
>> accepted.
> 
> Vladimir is among the few people who know very well the
 internal
>>> workings
> of the
> Calcite optimizer. He started and participated in many
> discussions
>> about
> the core engine and contributed ideas and code for making
>> it
>> better.
> Moreover, Vladimir has blogged and talked about Calcite in
> various
> conferences and meetups giving publicity and showcasing
>> the
>>> capabilities of
> the project.
> 
> Vladimir, welcome, thank you for your contributions, and
>> we
 look
>>> forward to
> your
> further interactions with the community! If you wish,
>> please
> feel
> free
>>> to
> tell
> us more about yourself and what you are working on.
> 
> Stamatis (on behalf of the Apache Calcite PMC)
> 
>>> 
>> 
> 
 
>>> 
>> 
> 
 
>>> 
>>> 
>>> --
>>> Viliam Durina
>>> Jet Developer
>>>  hazelcast®
>>> 
>>>   2 W 5th Ave, Ste 300 | San Mateo, CA
>> 94402 |
>>> USA
>>> +1 (650) 521-5453 | hazelcast.com 
>>> 
>>> --
>>> This message contains confidential information and is intended only for
>> the
>>> individuals named. If you are not the named addressee you should not
>>> disseminate, distribute or copy this e-mail. Please notify the sender
>>> immediately by e-mail if you have received this e-mail by mistake and
>>> delete this e-mail from your system. E-mail transmission cannot be
>>> guaranteed to be secure or error-free as information could be
>> intercepted,
>>> corrupted, lost, destroyed, arrive late or incomplete, or contain
>> viruses.
>>> The sender therefore does not accept liability for any errors or
>> omissions
>>> in the contents of this message, which arise as a result of e-mail
>>> transmission. If verification is required, please request a hard-copy
>>> version. -Hazelcast
>> 
> 
> 
> -- 
> ~~~
> no mistakes
> ~~

Re: Error encountered during build on ppc64le architecture

[Julian Hyde]

It looks as if the Redis adapter test is trying to get an unavailable port. See 
whether the build succeeds if the Redis adapter test is skipped:

./gradlew clean test -x :redis:test

If that passes, it indicates that Calcite is sound. (The Redis adapter is 
probably sound, too, and its test would pass if given a different port.)

Julian


> On Jun 22, 2021, at 9:36 AM, Kishore Kunal Mr  
> wrote:
> 
> 
> 
> Hi Team,
> I encountered an issue while building calcite on ppc64le architecture. For
> the build I followed the steps provided on official site [
> https://calcite.apache.org/docs/howto.html#building-from-git], but failing
> to do so. Hence need your support in fixing the issue.  Logs can be found
> below
> Host details:
> ===
> ubuntu@kishorkunal2:~/newpackages/calcite$ uname -a
> Linux kishorkunal2 4.15.0-115-generic #116-Ubuntu SMP Wed Aug 26 14:34:20
> UTC 2020 ppc64le ppc64le ppc64le GNU/Linux
> ubuntu@kishorkunal2:~/newpackages/calcite$ gradle --version
> 
> Gradle 5.0
> 
> Build time:   2018-11-26 11:48:43 UTC
> Revision: 7fc6e5abf2fc5fe0824aec8a0f5462664dbcd987
> 
> Kotlin DSL:   1.0.4
> Kotlin:   1.3.10
> Groovy:   2.5.4
> Ant:  Apache Ant(TM) version 1.9.13 compiled on July 10 2018
> JVM:  1.8.0_292 (Private Build 25.292-b10)
> OS:   Linux 4.15.0-115-generic ppc64le
> 
> 
> Error log:
> ===
> 
> Gradle Test Executor 1 STANDARD_ERROR
>Jun 22, 2021 4:31:20 PM
> org.junit.jupiter.engine.config.EnumConfigurationParameterConverter get
>INFO: Using parallel execution mode 'CONCURRENT' set via the
> 'junit.jupiter.execution.parallel.mode.default' configuration parameter.
> WARNING   0.1sec,1 completed,   0 failed,   1 skipped,
> org.apache.calcite.adapter.redis.RedisMiniServer
> 
> Gradle Test Executor 1 STANDARD_OUT
>2021-06-22 16:31:21,219 [ForkJoinPool-1-worker-1] INFO  - Loaded
> org.testcontainers.dockerclient.UnixSocketClientProviderStrategy from
> ~/.testcontainers.properties, will try it first
>2021-06-22 16:31:22,516 [ForkJoinPool-1-worker-1] INFO  - Found Docker
> environment with local Unix socket (unix:///var/run/docker.sock)
>2021-06-22 16:31:22,519 [ForkJoinPool-1-worker-1] INFO  - Docker host
> IP address is localhost
>2021-06-22 16:31:22,566 [ForkJoinPool-1-worker-1] INFO  - Connected to
> docker:
>  Server Version: 20.10.2
>  API Version: 1.41
>  Operating System: Ubuntu 18.04.5 LTS
>  Total Memory: 8160 MB
>2021-06-22 16:31:22,571 [ForkJoinPool-1-worker-1] INFO  - Image name
> substitution will be performed by: DefaultImageNameSubstitutor (composite
> of 'ConfigurationFileImageNameSubstitutor' and
> 'PrefixingImageNameSubstitutor')
>2021-06-22 16:31:22,627 [ForkJoinPool-1-worker-1] INFO  - Failure when
> attempting to lookup auth config. Please ignore if you don't have images in
> an authenticated registry. Details: (dockerImageName:
> testcontainers/ryuk:0.3.0, configFile: /home/ubuntu/.docker/config.json.
> Falling back to docker-java default behaviour. Exception
> message: /home/ubuntu/.docker/config.json (No such file or directory)
> 
>> Task :server:test
> 
> Gradle Test Executor 2 STANDARD_ERROR
>Jun 22, 2021 4:31:21 PM
> org.junit.jupiter.engine.config.EnumConfigurationParameterConverter get
>INFO: Using parallel execution mode 'CONCURRENT' set via the
> 'junit.jupiter.execution.parallel.mode.default' configuration parameter.
> 
> ServerTest STANDARD_ERROR
>Jun 22, 2021 4:31:21 PM
> org.junit.jupiter.engine.config.EnumConfigurationParameterConverter get
>INFO: Using parallel execution mode 'CONCURRENT' set via the
> 'junit.jupiter.execution.parallel.mode.default' configuration parameter.
> WARNING   3.8sec,  439 completed,   0 failed,   5 skipped,
> org.apache.calcite.test.ServerParserTest
> WARNING   4.7sec,  439 completed,   0 failed,   6 skipped,
> org.apache.calcite.test.ServerUnParserTest
>  5.2sec, org.apache.calcite.test.ServerTest > testVirtualColumn()
>  6.7sec, org.apache.calcite.test.ServerQuidemTest > test
> (String)[5], [5] sql/table.iq
> WARNING   7.3sec,   15 completed,   0 failed,   1 skipped,
> org.apache.calcite.test.ServerTest
> 
> ServerQuidemTest STANDARD_OUT
>2021-06-22 16:31:30,285 [ForkJoinPool-1-worker-1] INFO  - open start -
> state modified
>2021-06-22 16:31:30,324 [ForkJoinPool-1-worker-1] INFO  - Checkpoint
> start
>2021-06-22 16:31:30,324 [ForkJoinPool-1-worker-1] INFO  - Checkpoint
> end - txts: 25
>  3.5sec, org.apache.calcite.test.ServerQuidemTest > test
> (String)[1], [1] sql/materialized_view.iq
>  8.8sec,6 completed,   0 failed,   0 skipped,
> org.apache.calcite.test.ServerQuidemTest
> WARNING  11.3sec,  899 completed,   0 failed,  12 skipped, Gradle Test
> Run :server:test
> 
>> Task :server:check
>> Task :server:build
> 
>> Task :redis:test FAILED
> FAILURE 

Re: geometry type support problem

[Julian Hyde]

It’s possible that the problem is with the JDBC adapter. Calcite would need to 
connect to PostGIS via its JDBC driver, read the column types, and deduce that 
the column is of GEOMETRY type. JDBC does not have a type-id for GEOMETRY, so 
this process would be a little ad hoc. (Probably using 
ResultSet.getColumnClassName in some way.)

As far as I know, you are the first person to connect to PostGIS via Calcite’s 
JDBC adapter, so it’s not surprising that there are some interoperability 
issues.

Can you please log a JIRA case for this issue?

Julian


> On Jun 22, 2021, at 2:47 AM, tonytao  wrote:
> 
> hi folks,
> 
> I'm using calcite with postgis, and found the support of geometry type is not 
> very good.
> 
> Firstly,  geometry was not added to enum "SqlTypeName" as a jdbc type, 
> geometry type  read from postgresql jdbc will convert to ANY type.
> 
> Since 1.27.0, the hepPlanner will add a cast-to-geometry function to the ANY 
> type  as below:
> 
> table:
> 
> \d cities
>Table "public.cities"
>   Column  | Type  | Collation | Nullable | Default
> --+---+---+--+-
>  id   | integer   |   |  |
>  name | character varying(50) |   |  |
>  the_geom | geometry(Point,4326)  |   |  |
>  x| double precision  |   |  |
>  y| double precision  |   |  |
> 
> query:
> 
> select 
> ST_POINT(ST_x(the_geom),ST_Y(the_geom)),ST_MAKELINE(the_geom,the_geom),ST_AsWKT(the_geom),ST_AsText(\"the_geom\"),ST_x(the_geom),ST_y(the_geom)
>  from pg.cities t
> 
> logical plan of 1.26:
> 
> LogicalProject(EXPR$0=[ST_POINT(ST_X($2), ST_Y($2))], EXPR$1=[ST_MAKELINE($2, 
> $2)], EXPR$2=[ST_ASWKT($2)], EXPR$3=[ST_ASTEXT($2)], EXPR$4=[ST_X($2)], 
> EXPR$5=[ST_Y($2)])
>   JdbcTableScan(table=[[pg, cities]])
> 
> logical plan of 1.27:
> 
> LogicalProject(EXPR$0=[ST_POINT(ST_X(CAST($2):JavaType(interface 
> org.apache.calcite.runtime.Geometries$Geom)), 
> ST_Y(CAST($2):JavaType(interface 
> org.apache.calcite.runtime.Geometries$Geom)))], 
> EXPR$1=[ST_MAKELINE(CAST($2):JavaType(interface 
> org.apache.calcite.runtime.Geometries$Geom), CAST($2):JavaType(interface 
> org.apache.calcite.runtime.Geometries$Geom))], 
> EXPR$2=[ST_ASWKT(CAST($2):JavaType(interface 
> org.apache.calcite.runtime.Geometries$Geom))], 
> EXPR$3=[ST_ASTEXT(CAST($2):JavaType(interface 
> org.apache.calcite.runtime.Geometries$Geom))], 
> EXPR$4=[ST_X(CAST($2):JavaType(interface 
> org.apache.calcite.runtime.Geometries$Geom))], 
> EXPR$5=[ST_Y(CAST($2):JavaType(interface 
> org.apache.calcite.runtime.Geometries$Geom))])
>   JdbcTableScan(table=[[pg, cities]])
> 
> 
> rel2sql couldn't convert logical plan of 1.27 to sql for:
> 
> java.lang.UnsupportedOperationException: Unsupported type when 
> convertTypeToSpec: GEOMETRY
> at 
> org.apache.calcite.sql.type.SqlTypeUtil.convertTypeToSpec(SqlTypeUtil.java:1065)
> at 
> org.apache.calcite.sql.type.SqlTypeUtil.convertTypeToSpec(SqlTypeUtil.java:1087)
> at org.apache.calcite.sql.SqlDialect.getCastSpec(SqlDialect.java:800)
> at 
> org.apache.calcite.sql.dialect.PostgresqlSqlDialect.getCastSpec(PostgresqlSqlDialect.java:92)
> at 
> org.apache.calcite.rel.rel2sql.SqlImplementor$Context.callToSql(SqlImplementor.java:772)
> 
> I think this problem is not only a bug.For the spatial type ,different 
> database has different implements  and definitions.It's hard to write a 
> general solution.In my opinion it could add a hook to support a specific 
> spatial database type convert?
> 
> Many thanks for your work,looking forward for your reply.
> 
> Thanks again!
> 
> Tony Tao
> 
> 

Re: Equals method of AggregateCall

[Julian Hyde]

Not a bug. 

The type is derived from the operator plus argument types, so it is impossible 
for two AggregateCalls to be the same except for type. 

There is no perfect “equals”. The current one seems to serve our needs. If you 
have different needs maybe you need a different “equals” (or use a wrapper 
key). 

Julian

> On Jun 14, 2021, at 08:43, 王腾飞(飞腾)  wrote:
> 
> 
> Hi All,
> 
> I'm working on one query failure of apache drill which is related to calcite. 
> Does anyone know why "RelDataType type" is not included in the equals method 
> of org.apache.calcite.rel.core.AggregateCall? Is this a bug or on purposes?
> 
> 
> Appreciate for your response.

Re: should calcite tolerate some typo mistakes?

[Julian Hyde]

It would have to be a mode that people opt into, but it might be useful. 
Especially for interactive use. 

I think you should write a Jira case with a specification and examples. 

Note that we already detect when people use the right identifier with the wrong 
case, and alert them in the error message. I guess the next step might be to 
“self heal” to find the nearest valid identifier to what they typed.

Julian

> On Jun 16, 2021, at 21:28, Yanjing Wang  wrote:
> 
> for some times, users typo "select `a`" to "select `a `", should calcite
> pass the typo mistake?

[jira] [Created] (CALCITE-4649) Add AggregateExpandGroupingSetsRule, a planner rule that removes GROUPING SETS

[Julian Hyde (Jira)]

Julian Hyde created CALCITE-4649:


 Summary: Add AggregateExpandGroupingSetsRule, a planner rule that 
removes GROUPING SETS
 Key: CALCITE-4649
 URL: https://issues.apache.org/jira/browse/CALCITE-4649
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


Add {{class AggregateExpandGroupingSetsRule}}, a planner rule that removes 
{{GROUPING SETS}}.

Some databases do not support {{GROUPING SETS}}. (For example, BigQuery 
supports {{GROUP BY ROLLUP}} but not general {{GROUPING SETS}}.) But {{GROUPING 
SETS}} is a useful construct, increasingly used in user SQL, and also generated 
by rules (e.g. {{AggregateExpandWithinDistinctRule}}).

The proposed rule would match an {{Aggregate}} whose {{groupSets}} field has 
more than one element, and would convert it to an {{Aggregate}} over a {{Join}} 
whose right-hand side is a list of integers, the group ids active in the query. 
Calls to the {{GROUP_ID}} field would be replaced by references to the group id 
column.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Problem with Spool (de-)serialization

[Julian Hyde]

It would be great to unify this. Can you log a JIRA case?

The changes might be breaking. You can mitigate this by making the
reader more liberal (e.g. accept both upper-case and lower-case enum
names), and also by communicating what you intend to change. People
should speak up if they expect the format to be exactly the same
between versions.

It would be useful if you ensure that everything you intend to change
occurs in at least one test, before you make the change. Then the
before-and-after will be clearly visible in the patch.

Julian

On Wed, Jun 9, 2021 at 7:57 AM Konstantin Orlov  wrote:
>
> Yes, it’s possible.
>
> But my concern is the lack of common approach. Here a few examples:
>
> 1) Correlate and Join both explains joinType as lowerName of enum’s value, 
> hence it could be simply restored by relInput.getEnum("joinType", 
> JoinRelType.class)
>
>   @Override public RelWriter explainTerms(RelWriter pw) {
> return super.explainTerms(pw)
> .item("joinType", joinType.lowerName)
> ...
>   }
>
> 2) MultiJoin uses similar approach, but instead of lowerName uses original 
> enum’s name
>
>   @Override public RelWriter explainTerms(RelWriter pw) {
> List joinTypeNames = new ArrayList<>();
> ...
> for (int i = 0; i < inputs.size(); i++) {
>   joinTypeNames.add(joinTypes.get(i).name());
> ...
> }
>
> return pw.item("joinFilter", joinFilter)
> ...
> .item("joinTypes", joinTypeNames)
> ...
>   }
>
> 3) TableModify uses RelEnumTypes.fromEnum(getOperation())) to explain 
> operation type, but uses input.getEnum("operation", Operation.class) to 
> restore it,
> whereas there is RelEnumTypes#toEnum(String) method
>
> 4) Spool explains both types as enum value, and probably will fail during 
> serialisation to json cause 
> org.apache.calcite.rel.externalize.RelJson#toJson(java.lang.Object)
> doesn’t know how to handle enum
>
>   @Override public RelWriter explainTerms(RelWriter pw) {
> return super.explainTerms(pw)
> .item("readType", readType)
> .item("writeType", writeType);
>   }
>
> Does it make any sense to unify this?
>
>
> --
> Regards,
> Konstantin Orlov
>
>
>
>
> > On 8 Jun 2021, at 22:03, Julian Hyde  wrote:
> >
> > Including class names in serialized JSON makes me nervous. It is a common 
> > attack surface. Could we, say, include the name of the enum (without 
> > package name) and map it to a list of supported Enums?
> >
> >> On Jun 8, 2021, at 9:20 AM, Konstantin Orlov  wrote:
> >>
> >> Hi, folks!
> >>
> >> We have a problem with Spool node deserialisation. Currently it explains 
> >> both readType and writeType fields as Enum,
> >> thus those fields being serialized as map:
> >>
> >> {
> >>"id":"2",
> >>"relOp”:"MyTableSpool",
> >>"readType":{
> >>   "class":"org.apache.calcite.rel.core.Spool$Type",
> >>   "name":"LAZY"
> >>},
> >>"writeType":{
> >>   "class":"org.apache.calcite.rel.core.Spool$Type",
> >>   "name":"EAGER"
> >>}
> >> }
> >>
> >> When deserializing, we use RelInput#getEnum which expects the provided tag 
> >> being a string value representing the enum's value name.
> >>
> >> Should we follow the way used for serialization of JoinRelType within the 
> >> Join node (serialize the enum value as its name in lower case)? Here is 
> >> example:
> >>
> >> {
> >>"id":"4",
> >>"relOp”:"MyJoin",
> >>"condition":{
> >>   "op":{
> >>  "name":">",
> >>  "kind":"SqlKind#GREATER_THAN",
> >>  "syntax":"SqlSyntax#BINARY"
> >>   },
> >>   "operands":[
> >>  {
> >> "input":1,
> >> "name":"$1"
> >>  },
> >>  {
> >> "input":4,
> >> "name":"$4"
> >>  }
> >>   ]
> >>},
> >>"joinType":"inner",
> >>"variablesSet":[0],
> >>"correlationVariables":[0],
> >>"inputs":["0","3"]
> >> }
> >>
> >> Or it's better to get the RelInput being able to deserialize enum 
> >> represented as map as well?
> >>
> >> Personally I prefer the first option cause the type of the enum is known 
> >> for sure, so it's better not to waste the time to (de-)serialize it.
> >>
> >>
> >> --
> >> Regards,
> >> Konstantin Orlov
> >>
> >>
> >>
> >>
> >
>

Re: Optimizer (Hep) NOT scanning RelNodes of Subqqueries

[Julian Hyde]

I presume we’re talking about scalar sub-queries (e.g. sub-query in SELECT) and 
sub-queries with IN and EXISTS, as opposed to sub-queries in the FROM clause. 
The former are represented using RexSubQuery because the queries appear in 
scalar expressions.

There are rules to expand RexSubQuery into relational operators. After those 
rules have been applied, then I’m sure that the Hep engine will be able to find 
them. I think that’s what Haisheng is referring to.

But while those sub-queries are still wrapped in RexSubQuery, it’s possible 
that the Hep engine won’t find them. Because it will have to walk over every 
RexNode inside a Project, Filter or Join and if that RexNode is a RexSubQuery, 
traverse into it to apply the rules to the RelNodes inside. I don’t know 
whether that happens today. Someone should check. It’s a reasonable ask (but 
there might be unforeseen consequences if we enable it by default).

Julian



> On Jun 9, 2021, at 12:02 PM, Haisheng Yuan  wrote:
> 
> Did you include the subquery related rules in the HepPlanner?
> 
> Haisheng
> 
> On 2021/06/09 17:59:44, Krishnakant Agrawal  wrote: 
>> Hi All,
>> 
>> When running a HepOptimizer on top of a RelNode which has a subquery
>> embedded in it, The Optimizer does not take the RelNode representing the
>> subquery up for optimization.
>> 
>> Is this by design,  what's the correct way for the Subquery RelNode to be
>> picked up for Optimization?
>> 
>> If this is a bug, what is the conceptually correct way to go about fixing
>> this?
>> 
>> Thanks,
>> KK
>> 

Re: Problem with Spool (de-)serialization

[Julian Hyde]

Including class names in serialized JSON makes me nervous. It is a common 
attack surface. Could we, say, include the name of the enum (without package 
name) and map it to a list of supported Enums?

> On Jun 8, 2021, at 9:20 AM, Konstantin Orlov  wrote:
> 
> Hi, folks!
> 
> We have a problem with Spool node deserialisation. Currently it explains both 
> readType and writeType fields as Enum,
> thus those fields being serialized as map:
> 
>  {
> "id":"2",
> "relOp”:"MyTableSpool",
> "readType":{
>"class":"org.apache.calcite.rel.core.Spool$Type",
>"name":"LAZY"
> },
> "writeType":{
>"class":"org.apache.calcite.rel.core.Spool$Type",
>"name":"EAGER"
> }
>  }
> 
> When deserializing, we use RelInput#getEnum which expects the provided tag 
> being a string value representing the enum's value name.
> 
> Should we follow the way used for serialization of JoinRelType within the 
> Join node (serialize the enum value as its name in lower case)? Here is 
> example:
> 
>  {
> "id":"4",
> "relOp”:"MyJoin",
> "condition":{
>"op":{
>   "name":">",
>   "kind":"SqlKind#GREATER_THAN",
>   "syntax":"SqlSyntax#BINARY"
>},
>"operands":[
>   {
>  "input":1,
>  "name":"$1"
>   },
>   {
>  "input":4,
>  "name":"$4"
>   }
>]
> },
> "joinType":"inner",
> "variablesSet":[0],
> "correlationVariables":[0],
> "inputs":["0","3"]
>  }
> 
> Or it's better to get the RelInput being able to deserialize enum represented 
> as map as well?
> 
> Personally I prefer the first option cause the type of the enum is known for 
> sure, so it's better not to waste the time to (de-)serialize it.
> 
> 
> -- 
> Regards,
> Konstantin Orlov
> 
> 
> 
> 

Re: [ANNOUNCE] Apache Calcite 1.27.0 released

[Julian Hyde]

Thank you, Stamatis! And the 50 other people who contributed to this release.

On Fri, Jun 4, 2021 at 3:50 PM Haisheng Yuan  wrote:
>
> Thank you Stamatis and all Calciters involved for leading the effort.
>
> Haisheng Yuan
>
> On 2021/06/04 22:16:21, Stamatis Zampetakis  wrote:
> > The Apache Calcite team is pleased to announce the release of Apache
> > Calcite 1.27.0.
> >
> > Calcite is a dynamic data management framework. Its cost-based
> > optimizer converts queries, represented in relational algebra, into
> > executable plans. Calcite supports many front-end languages and
> > back-end data engines, and includes an SQL parser and, as a
> > sub-project, the Avatica JDBC driver.
> >
> > This release comes eight months after 1.26.0. It includes more than
> > 150 resolved issues, comprising a few new features, three minor
> > breaking changes, many bug-fixes and small improvements, as well
> > as code quality enhancements and better test coverage.
> >
> > You can start using it in Maven by simply updating your dependency to:
> >
> >   
> > org.apache.calcite
> > calcite-core
> > 1.27.0
> >   
> >
> > If you'd like to download the source release, you can find it here:
> >
> > https://calcite.apache.org/downloads/
> >
> > You can read more about the release (including release notes) here:
> >
> > https://calcite.apache.org/news/2021/06/03/release-1.27.0/
> >
> > We welcome your help and feedback. For more information on how to
> > report problems, and to get involved, visit the project website at:
> >
> > https://calcite.apache.org/
> >
> > Thanks to everyone involved!
> >
> > Stamatis Zampetakis, on behalf of the Apache Calcite Team
> >

Re: [DISCUSS] Release Managers

[Julian Hyde]

Yes, I am release manager for 1.28. I’ll aim to release first week of August.

Somewhat related to the call for release managers: we need active committers. 
(Stamatis, can you run your stats on PRs and active committers?) On May 21, I 
noted that PR backlog is building up, and appealed for 5 committers to look at 
5 PRs each. But since then only 2 PRs have been committed (thanks Ruben & 
Stamatis). We need to do better.

Julian


> On Jun 3, 2021, at 3:33 PM, Haisheng Yuan  wrote:
> 
> Hi all,
> 
> In the next half year, I hope we can get back to the pace of release for 
> every 2 months. We need more release managers for the next few releases, is 
> there any one interested in volunteering to be release manager?
> Currently Julian is the release manager for 1.28.0, we need 3 more for 1.29.0 
> ~ 1.31.0.
> 
> Thanks,
> Haisheng Yuan

Re: Deduplicate correlate variables question.

[Julian Hyde]

The PR needs to fix a bug or implement a feature. So, first you should log a 
JIRA case describing what doesn’t work. Write tests for what doesn’t work that 
you want to make work. (Or maybe you can refactor/generalize existing tests.) 
Then submit a PR, and we will review that PR on the basis of whether it fixes 
the bug.

That may sound like a lot of process. But the alternative is people just 
randomly changing code.

> On Jun 3, 2021, at 7:37 AM, stanilovsky evgeny  
> wrote:
> 
> Julian, thanks for reply and comments.
> Can you explain, is it would possible to commit PR containing deduplication 
> code moved upper this flag ?
> If so - i will create PR and rerun all existing tests, of course.
> Thanks !
> 
>> Master is a moving target. Apparently you are using a version of the
>> code from sometime in March. These links work better:
>> 
>> [1] 
>> https://github.com/apache/calcite/blob/796675c9b33e0461bc45a72780162d474a4b098b/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L2881
>> 
>> [2] 
>> https://github.com/apache/calcite/blob/796675c9b33e0461bc45a72780162d474a4b098b/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6209
>> 
>> I don't know the exact cause of what you are seeing, but when you set
>> "expand=false" you are choosing a newer (and better) code path. We do
>> not have feature parity between the code paths. Nor do we write tests
>> for both code paths.
>> 
>> Some day I'd like to officially deprecate, then obsolete,
>> "expand=true". It is de facto deprecated because we put more effort
>> into the "expand=false" path.
>> 
>> Julian
>> 
>> 
>> On Fri, May 28, 2021 at 1:19 AM stanilovsky evgeny
>>  wrote:
>>> 
>>> Hi, calciters )
>>> 
>>> I am trying to figure out why DeduplicateCorrelateVariables [1] is called
>>> only if withExpand [2] flag is true ? Why we don`t need to deduplicate in
>>> appropriate case ?
>>> 
>>> [1]
>>> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L2881
>>> 
>>> [2]
>>> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6209
>>> 
>>> Thanks !

Re: RelFieldTrimmer on union distinct

[Julian Hyde]

I ran those queries against Calcite. Calcite gives the correct result. 
RelFieldTrimmer does the right thing.

I was explaining, in simple terms, what would break if you made your proposed 
change to RelFieldTrimmer.

> On Jun 2, 2021, at 3:12 PM, Sean Broeder  wrote:
> 
> Thank you Julian.  
> 
> I am able to reproduce your findings.  
> 
> I am curious if the results highlight a bug in Calcite where we should expect 
> 4 to be returned or is this a query that Calcite should not support?
> 
> Sean
> 
>> On Jun 2, 2021, at 11:37 AM, Julian Hyde  wrote:
>> 
>> Here’s a query that would give the wrong answer if you trim:
>> 
>> select count(*) from (
>>   select deptno from scott.emp
>>   union
>>   select deptno from scott.dept);
>> 
>> returns 4. Note that ‘deptno’ is not used. But when I trim it away,
>> 
>> select count(*) from (
>>   select 'a' from scott.emp
>>   union
>>   select 'a' from scott.dept);
>> 
>> returns 1. (I included ‘a’ because SQL doesn’t allow an empty SELECT clause. 
>> It doesn’t affect the reasoning.)
>> 
>> Julian
>> 
>> 
>> 
>>> On Jun 2, 2021, at 6:03 AM, Sean Broeder  wrote:
>>> 
>>> Currently the RefFieldTrimmer only trims on a UNION ALL operation.  I've
>>> been experimenting to see if it is also possible to trim on UNION
>>> DISTINCT.  Is there a simple query that demonstrates why this is not
>>> possible?
>>> 
>>> Thanks,
>>> Sean
>> 
> 

Re: Minified javascript in source releases

[Julian Hyde]

+0 to remove them entirely, if that is possible.

-1 to keep in git but remove from source releases. We are already in
compliance with Apache policy, best I can tell.


On Wed, Jun 2, 2021 at 12:08 PM Haisheng Yuan  wrote:
>
> +1 for removing them entirely.
>
> Thanks,
> Haisheng Yuan
>
> On 2021/06/02 13:36:05, Alessandro Solimando  
> wrote:
> > Hi all,
> > +1 for removing them as well, all the mentioned sw versions in [5,6]
> > are extremely, extremely old.
> >
> > Best regards,
> > Alessandro
> >
> > On Wed, 2 Jun 2021 at 14:38, Michael Mior  wrote:
> > >
> > > +1 for removing these entirely. I don't believe they are currently 
> > > necessary.
> > > --
> > > Michael Mior
> > > mm...@apache.org
> > >
> > > Le mer. 2 juin 2021 à 06:13, Stamatis Zampetakis  a 
> > > écrit :
> > > >
> > > > Hi all,
> > > >
> > > > In the recent votes for Calcite [1] and Avatica [2] there were some -1
> > > > votes regarding minified javascript files present in the source 
> > > > releases.
> > > >
> > > > The concerns are raised for the following files:
> > > > site/js/html5shiv.min.js
> > > > site/js/respond.min.js
> > > >
> > > > From discussions in other lists it seems that there are Apache source
> > > > releases including minified files so I don't think we are violating any
> > > > policy by including them but there are deferring opinions on this topic
> > > > [3,4].
> > > >
> > > > To avoid bringing back the discussion to if minified javascript should 
> > > > be
> > > > present or not I would like to propose one of the following 
> > > > alternatives:
> > > >
> > > > 1. It seems that both of the aforementioned scripts [5,6] are used for
> > > > compatibility reasons with old browsers. I think most of the people are 
> > > > not
> > > > using these browsers anymore so I would suggest removing them altogether
> > > > from the code.
> > > >
> > > > 2. If for some reason we need to keep them then we could directly use 
> > > > the
> > > > non minified version. The difference of minified vs unminified for both
> > > > files is ~6KB so I doubt it will have any real impact with the internet
> > > > connections that we use nowadays.
> > > >
> > > > What do you think?
> > > >
> > > > Best,
> > > > Stamatis
> > > >
> > > > [1]
> > > > https://lists.apache.org/thread.html/r049cf5b953eb5824ba37df21094dd7c1e92146d659db4735ad121593%40%3Cdev.calcite.apache.org%3E
> > > > [2]
> > > > https://lists.apache.org/thread.html/r074720fc062bde09a7e69dc77f77754c7fd4c22f1357514f6b2196d0%40%3Cdev.calcite.apache.org%3E
> > > > [3]
> > > > https://lists.apache.org/thread.html/rcddc30dd1f0c7f20709e09de7202d4d6885d6235a7fce1c1ab46e4ed%40%3Clegal-discuss.apache.org%3E
> > > > [4]
> > > > http://mail-archives.apache.org/mod_mbox/incubator-general/201805.mbox/%3c89a47c0b-e74e-4571-b282-dbe505291...@classsoftware.com%3e
> > > > [5] https://github.com/scottjehl/Respond
> > > > [6] https://github.com/aFarkas/html5shiv
> >

Re: Deduplicate correlate variables question.

[Julian Hyde]

Master is a moving target. Apparently you are using a version of the
code from sometime in March. These links work better:

[1] 
https://github.com/apache/calcite/blob/796675c9b33e0461bc45a72780162d474a4b098b/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L2881

[2] 
https://github.com/apache/calcite/blob/796675c9b33e0461bc45a72780162d474a4b098b/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6209

I don't know the exact cause of what you are seeing, but when you set
"expand=false" you are choosing a newer (and better) code path. We do
not have feature parity between the code paths. Nor do we write tests
for both code paths.

Some day I'd like to officially deprecate, then obsolete,
"expand=true". It is de facto deprecated because we put more effort
into the "expand=false" path.

Julian


On Fri, May 28, 2021 at 1:19 AM stanilovsky evgeny
 wrote:
>
> Hi, calciters )
>
> I am trying to figure out why DeduplicateCorrelateVariables [1] is called
> only if withExpand [2] flag is true ? Why we don`t need to deduplicate in
> appropriate case ?
>
> [1]
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L2881
>
> [2]
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6209
>
> Thanks !

Re: RelFieldTrimmer on union distinct

[Julian Hyde]

Here’s a query that would give the wrong answer if you trim:

  select count(*) from (
select deptno from scott.emp
union
select deptno from scott.dept);

returns 4. Note that ‘deptno’ is not used. But when I trim it away,

  select count(*) from (
select 'a' from scott.emp
union
select 'a' from scott.dept);

returns 1. (I included ‘a’ because SQL doesn’t allow an empty SELECT clause. It 
doesn’t affect the reasoning.)

Julian



> On Jun 2, 2021, at 6:03 AM, Sean Broeder  wrote:
> 
> Currently the RefFieldTrimmer only trims on a UNION ALL operation.  I've
> been experimenting to see if it is also possible to trim on UNION
> DISTINCT.  Is there a simple query that demonstrates why this is not
> possible?
> 
> Thanks,
> Sean