JavaType with charset

[Zhe Hu]

Hi, community.
Recently, when I try to fix CALCITE-6146, I’ve find something confusing about 
createTypeWithCharsetAndCollation() method in SqlTypeFactoryImpl.
I want to create a new RelDataType with different charset(for deriving type in 
CONVERT function) based on a JavaType like “JavaType(class java.lang.String)”, 
however, the cached type in  DATATYPE_CACHE will be returned, which means I 
cannot get a JavaType with the specific charset.
I'm considering whether adding the charset name for JavaType’s digest is the 
right way to go. Appreciate any suggestions.


Best,
ZheHu

Re: [ANNOUNCE] New committer: Mihai Budiu

[Zhe Hu]

Congrats, Mihai !


Best
ZheHu


 Replied Message 
| From | Bertil Chapuis |
| Date | 10/26/2023 21:09 |
| To |  |
| Subject | Re: [ANNOUNCE] New committer: Mihai Budiu |
Congratulations and welcome!

Bertil

On 26 Oct 2023, at 10:56, Stamatis Zampetakis  wrote:

Apache Calcite's Project Management Committee (PMC) has invited Mihai
Budiu to become a committer, and we are pleased to announce that he
has accepted.

Mihai has addressed numerous compile and runtime issues in SQL
functions and has made several significant improvements to the test
infrastructure to enhance code coverage, quality, and stability. In
addition to code contributions, Mihai is highly active on the mailing
list, where he answers questions, assists others, and shares ideas for
future improvements. Mihai has also been writing blogs and giving
talks about Calcite at various events and conferences, showcasing
Calcite in both simple and more advanced use cases, which can greatly
contribute to the growth of the community.

Mihai, welcome, thank you for your contributions, and we look forward
to your further interactions with the community! If you wish, please
feel free to tell us more about yourself and what you are working on.

As your first commit, please add yourself to the contributors list [1]
and the community page will re-generate [2].

Stamatis (on behalf of the Apache Calcite PMC)

[1] https://github.com/apache/calcite/blob/main/site/_data/contributors.yml
[2] https://calcite.apache.org/community/#project-members

Re:Re: Missing column alia after RelFieldTrimmer

[Zhe Hu]

Benchao, Julian, thanks for your clarification. It makes sense to me.
I also filed CALCITE-6070 to log it.




Best,
ZheHu











At 2023-10-24 08:12:30, "Julian Hyde"  wrote:
>Benchao is correct that we cannot guarantee column names. So, this is a 
>feature request rather than a bug. I still think you should log it.
>
>RelFieldTrimmer works top-down, deducing a bit-set of the columns that each 
>relational operator needs from its input(s). To solve this problem, it would 
>need to pass the desired names of those columns. That might make the trimmer a 
>little bit less efficient, and somewhat more complex.
>
>The trimmer ought to be allowed to ignore the recommended column names; I 
>don’t think we would want to introduce a Project for the sole purpose of 
>renaming columns.
>
>Julian
>
>
>
>> On Oct 23, 2023, at 4:31 AM, Benchao Li  wrote:
>> 
>> AFAIK, usually we cannot guarantee the columns names in the
>> optimization phase. However, I think we can still try our best to
>> preserve it while not introducing too much complexity.
>> 
>> For this case, I'd like to take it as an improvement we can do.
>> 
>> Zhe Hu  于2023年10月23日周一 18:32写道：
>>> 
>>> Hi, community. I encounter a problem when I try RelFieldTrimmer.
>>> I run the following SQL in SqlToRelConverterTest:
>>>select ename as did from emp where ename='a' limit 10
>>> 
>>> 
>>> I get planA without RelFieldTrimmer:
>>> LogicalSort(fetch=[10])
>>>  LogicalProject(DID=[$1])
>>>LogicalFilter(condition=[=($1, 'a')])
>>>  LogicalTableScan(table=[[CATALOG, SALES, EMP]])
>>> 
>>> 
>>> I get planB with the RelFieldTrimmer:
>>> LogicalSort(fetch=[10])
>>>  LogicalFilter(condition=[=($0, 'a')])
>>>LogicalProject(ENAME=[$1])
>>>  LogicalTableScan(table=[[CATALOG, SALES, EMP]])
>>> 
>>> 
>>> However, the column alia DID for ENAME is missing in planB. And I think the 
>>> project should be LogicalProject(DID=[$1]) in planB.
>>> Do I miss some relevant rules in Calcite or is it a issue we need to fix?
>>> 
>>> 
>>> Best,
>>> ZheHu
>> 
>> 
>> 
>> -- 
>> 
>> Best,
>> Benchao Li

Missing column alia after RelFieldTrimmer

[Zhe Hu]

Hi, community. I encounter a problem when I try RelFieldTrimmer.
I run the following SQL in SqlToRelConverterTest:
select ename as did from emp where ename='a' limit 10


I get planA without RelFieldTrimmer:
LogicalSort(fetch=[10])
  LogicalProject(DID=[$1])
LogicalFilter(condition=[=($1, 'a')])
  LogicalTableScan(table=[[CATALOG, SALES, EMP]])


I get planB with the RelFieldTrimmer:
LogicalSort(fetch=[10])
  LogicalFilter(condition=[=($0, 'a')])
LogicalProject(ENAME=[$1])
  LogicalTableScan(table=[[CATALOG, SALES, EMP]])


However, the column alia DID for ENAME is missing in planB. And I think the 
project should be LogicalProject(DID=[$1]) in planB.
Do I miss some relevant rules in Calcite or is it a issue we need to fix?


Best,
ZheHu

Re: Java Doc about RexProgramBuilderBase

[Zhe Hu]

Thank for your clarification! I’ll try to fix it shortly.




 Replied Message 
| From | Julian Hyde |
| Date | 07/8/2023 02:16 |
| To |  |
| Subject | Re: Java Doc about RexProgramBuilderBase |
People refactoring code should remember that their IDE can move and rename 
fields but is not so good at changing documentation. (Maybe in a couple of 
years, with advances in generative AI?!)

And when documentation and code don’t line up, people don’t trust either. (I’m 
as guilty of this as anyone.)

On Jul 7, 2023, at 5:31 AM, Benchao Li  wrote:

This usually happens. When javadoc and the implementation diverges, the
javadoc is mostly possible wrong and need to be improved to match the real
behavior.

For this specific case, I agree with you that the javadoc for
`vParamNotNull` and `vDecimal(int arg)` are not correct, please fix them.
(We can do this kind of trivial work without a Jira ticket)

Zhe Hu  于2023年7月7日周五 14:45写道：

Hi community.
Recently, when I review CALCITE-5769(
https://github.com/apache/calcite/pull/3296), I found something a little
confusing.

First, the java doc in RexProgramBuilderBase.vParamNotNull(), which meant
to create non-nullable variable, but it’s returning description is
“nullable varchar variable”.
Second, we use vDecimal(int arg) to create nullable decimal variable, but
the RelDataType we pass in is “nonNullableDecimal”, which I think should be
“nullableDecimal”. So does the other vXxx() methods.
I’m not sure if I understand right here. If it’s something we can improve,
I’ll file a JIRA case to record and fix it.


Best regards,
Zhe Hu



--

Best,
Benchao Li

Java Doc about RexProgramBuilderBase

[Zhe Hu]

Hi community.
Recently, when I review 
CALCITE-5769(https://github.com/apache/calcite/pull/3296), I found something a 
little
confusing.

First, the java doc in RexProgramBuilderBase.vParamNotNull(), which meant to 
create non-nullable variable, but it’s returning description is “nullable 
varchar variable”.
Second, we use vDecimal(int arg) to create nullable decimal variable, but the 
RelDataType we pass in is “nonNullableDecimal”, which I think should be 
“nullableDecimal”. So does the other vXxx() methods.
I’m not sure if I understand right here. If it’s something we can improve, I’ll 
file a JIRA case to record and fix it.


Best regards,
Zhe Hu

Re: [ANNOUNCE] New committer: TJ Banghart

[Zhe Hu]

Congrats! TJ.
 Replied Message 
| From | Francis Chuang |
| Date | 07/04/2023 15:18 |
| To | dev |
| Subject | Re: [ANNOUNCE] New committer: TJ Banghart |
Congrats, TJ!

On 4/07/2023 5:17 pm, Stamatis Zampetakis wrote:
> Apache Calcite's Project Management Committee (PMC) has invited TJ Banghart to
> become a committer, and we are pleased to announce that they have accepted.
>
> TJ has been contributing to the community for about a year now. They
> introduced many new SQL functions for parsing and formatting dates,
> times, and timestamps extending the capabilities of the BigQuery
> dialect. Furthermore, they pushed various improvements around JSON
> serialization and deserialization, JDBC metadata, and lexical
> policies.
>
> TJ, welcome, thank you for your contributions, and we look forward to your
> further interactions with the community! If you wish, please feel free to tell
> us more about yourself and what you are working on.
>
> As your first commit, please add yourself to the contributors list [1]
> and the community page will re-generate [2].
>
> Stamatis (on behalf of the Apache Calcite PMC)
>
> [1] https://github.com/apache/calcite/blob/main/site/_data/contributors.yml
> [2] https://calcite.apache.org/community/#project-members

Re: [ANNOUNCE] New committer: Dan Zou

[Zhe Hu]

Congrats! Dan.
 Replied Message 
| From | Francis Chuang |
| Date | 07/04/2023 15:18 |
| To | dev |
| Subject | Re: [ANNOUNCE] New committer: Dan Zou |
Congrats, Dan!

On 4/07/2023 5:16 pm, Stamatis Zampetakis wrote:
> Apache Calcite's Project Management Committee (PMC) has invited Dan Zou to
> become a committer, and we are pleased to announce that they have accepted.
>
> Dan has been doing some great work for the project over the past few
> months. They implemented and enabled multiple new SQL functions for
> BigQuery and MSSQL dialects, fixed some optimization rules, and
> improved documentation and test code.
>
> Dan, welcome, thank you for your contributions, and we look forward to your
> further interactions with the community! If you wish, please feel free to tell
> us more about yourself and what you are working on.
>
> As your first commit, please add yourself to the contributors list [1]
> and the community page will re-generate [2].
>
> Stamatis (on behalf of the Apache Calcite PMC)
>
> [1] https://github.com/apache/calcite/blob/main/site/_data/contributors.yml
> [2] https://calcite.apache.org/community/#project-members

Re: [ANNOUNCE] New committer: Zhe Hu

[Zhe Hu]

Thanks a lot you guys! It’s really an honor to be a committer here.


Currently, I’m a BigData Engineer with 3 years of experience in 
Hangzhou(China), and our team mainly focus on processing data analysis from 
multiple data sources. Therefore, Calcite is really the key to us.
For the record, I really like the atmosphere here, and I learned many from our 
members. I hope I can bring more for the community in the future.


Best regards,
Zhe Hu




 Replied Message 
| From | Tanner Clary |
| Date | 06/28/2023 21:29 |
| To |  |
| Subject | Re: [ANNOUNCE] New committer: Zhe Hu |
Congrats, Zhe Hu!!

Best,
Tanner

On Wed, Jun 28, 2023 at 5:52 AM Bertil Chapuis  wrote:

Congratulations, Zhe Hu!

Best regards,

Bertil

On 28 Jun 2023, at 14:26, Dan Zou  wrote:

Congratulations, Zhe Hu!

Best,
Dan Zou





2023年6月28日 20:24，Jiajun Xie  写道：

Congratulations, Zhe Hu!

Welcome!


On Wed, 28 Jun 2023 at 19:39, xiong duan  wrote:

Congratulations, Zhe Hu!

Francis Chuang  于2023年6月28日周三 19:05写道：

Congrats, Zhe Hu!

On 28/06/2023 9:04 pm, Stamatis Zampetakis wrote:
Apache Calcite's Project Management Committee (PMC) has invited Zhe
Hu
to
become a committer, and we are pleased to announce that they have
accepted.

Zhe Hu has been contributing to the project for a while now. They
improved the stability of the ElasticSearch adapter, worked on
supporting new java versions, and landed various enhancements around
CONCAT and CONVERT functions.

Zhe Hu, welcome, thank you for your contributions, and we look
forward
to your
further interactions with the community! If you wish, please feel
free
to tell
us more about yourself and what you are working on.

As your first commit, please add yourself to the contributors list
[1]
and the community page will re-generate [2].

Stamatis (on behalf of the Apache Calcite PMC)

[1]

https://github.com/apache/calcite/blob/main/site/_data/contributors.yml
[2] https://calcite.apache.org/community/#project-members

PR 3258 Review Request

[Zhe Hu]

Hi, community.
I’ve submitted a PR for applying two different NULL semantics for CONCAT 
function, because this function now returns NULL when any of the arguments is 
NULL, however Postgresql and MSSQL have different behavior(they treat NULL as 
empty string).
BTW, CONCAT enabled in Oracle(we call it CONCAT2 in Calcite) has been resolved 
in CALCITE-5745, if anyone needs to refer to.


https://github.com/apache/calcite/pull/3258


Best,
ZheHu

Re: [ANNOUNCE] New committer: Oliver Lee

[Zhe Hu]

Congrats Oliver!
 Replied Message 
| From | xiong duan |
| Date | 06/13/2023 19:17 |
| To | dev |
| Subject | Re: [ANNOUNCE] New committer: Oliver Lee |
Congratulations Oliver!

Dan Zou  于2023年6月13日周二 19:16写道：

> Congratulations Oliver!
>
> Best,
> Dan Zou
>
>
>
>
>
> > 2023年6月13日 18:44，Stamatis Zampetakis  写道：
> >
> > Apache Calcite's Project Management Committee (PMC) has invited Oliver
> > Lee to become a committer, and we are pleased to announce that they
> > have accepted.
> >
> > Oliver started working with us around November 2022 and since then
> > they contributed multiple SQL functions to the Calcite repository
> > bringing lots of improvements to the BigQuery dialect. They improved
> > the extensibility of the SQL validator and pushed various fixes in
> > RelNode serialization to JSON.
> >
> > Oliver, welcome, thank you for your contributions, and we look forward
> > to your further interactions with the community! If you wish, please
> > feel free to tell us more about yourself and what you are working on.
> >
> > As your first commit, please add yourself to the contributors list [1]
> > and the community page will re-generate [2].
> >
> > Stamatis (on behalf of the Apache Calcite PMC)
> >
> > [1]
> https://github.com/apache/calcite/blob/main/site/_data/contributors.yml
> > [2] https://calcite.apache.org/community/#project-members
>
>

Re: subscribe calcite

[Zhe Hu]

Hi Jacky.
The following website will provide more useful information you need.
https://calcite.apache.org/community/#mailing-lists



Best
ZheHu
 Replied Message 
| From | Francis Chuang |
| Date | 05/23/2023 14:59 |
| To | dev,
liuyongvs2 |
| Subject | Re: subscribe calcite |
Hey Jacky,

Please send an email to dev-subscr...@calcite.apache.org to subscribe.

Francis

On 23/05/2023 3:51 pm, Jacky Lau wrote:
> Hi every one:
> I am Jacky Lau, a flink contributor, 7+ years of experience in big data
> development.  I want to subscribe mails. glads to see you guys
> Best Jacky!
>

PR Approval for CI

[Zhe Hu]

Hi, community!
I’ve submitted a PR(https://github.com/apache/calcite/pull/3173) for supporting 
CONVERT/TRANSLATE(value USING transcodingName), which requires approval from a 
maintainer.
Can someone help to approve the CI? Many thanks!


Best,
ZheHu


|


|

Duplicate ConnectionHandle-id when using Avatica

[Zhe Hu]

Hi, fellows,
I have an issue when using Avatica.
In AvaticaConnection, ConnectionHandle is used to make each connection unique, 
whose id is generated by “UUID.randomUUID().toString()”. On the server 
side(like JdbcMeta), client’s connection id is used as key in connectionCache.
Here is what happened in my project. I cached Connection on the client side, 
and the server throws “Connection already exists: …” exception when creating 
new client connection, as a result of duplicate ConnectionHandle.id. Although 
such situation is very rare.
I’m wandering if we need to do some improvement on this, like adding timestamp 
or something else, since ConnectionHandle.id is final and I cannot rewrite it 
in the subclass.


Best regards,
ZheHu

Re:[VOTE] Release Apache Calcite 1.30.0 (release candidate 0)

[Zhe Hu]

Hi, Fan Liya.
Thanks for your great job of releasing Calcite 1.30.0. 
I’m trying to read the release notes in 
https://github.com/apache/calcite/blob/calcite-1.30.0-rc0/site/_docs/history.md.
 However, the latest version of this page is still 1.29.0. I’m not sure if it’s 
my permission issue or not. But I can read it in your 
PR(https://github.com/apache/calcite/pull/2739/files).
By the way, +1 for this release.


Best,
ZheHu
On 03/5/2022 17:56，Fan Liya wrote：
Hi all,

I have created a build for Apache Calcite 1.30.0, release
candidate 0.

Thanks to everyone who has contributed to this release.

You can read the release notes here:
https://github.com/apache/calcite/blob/calcite-1.30.0-rc0/site/_docs/history.md

The commit to be voted upon:
https://gitbox.apache.org/repos/asf?p=calcite.git;a=commit;h=28aacc0bbccfbcd821fece72ae3d92f6c1db8a5e

Its hash is 28aacc0bbccfbcd821fece72ae3d92f6c1db8a5e

Tag:
https://github.com/apache/calcite/tree/calcite-1.30.0-rc0

The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.30.0-rc0
(revision 52873)

The hashes of the artifacts are as follows:
339b279426abfc09a73b0daf94479127aa469b1cd3fa1b1cf0842f57d64daf0e080d4e85ccdd2d32032ed424a39388790743adf2aec869480612e37003eabfc5
*apache-calcite-1.30.0-src.tar.gz

A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachecalcite-1148/org/apache/calcite/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/COMMITTER_ID.asc
https://www.apache.org/dist/calcite/KEYS

To create the jars and test Apache Calcite: "gradle build"
(requires an appropriate Gradle/JDK installation)

Please vote on releasing this package as Apache Calcite 1.30.0.

The vote is open for the next 72 hours and passes if a majority of at
least three +1 PMC votes are cast.

[ ] +1 Release this package as Apache Calcite 1.30.0
[ ]  0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...

Here is my vote:

+1 (binding)

Best,
Liya Fan

Re:Connect Calcite to Sqlite

[Zhe Hu]

Hi, Pan.
It will be much better to go through your issue if you show us how you use 
Calcite to connect Sqlite and what the StackTrace is.


There is one demo in core/src/test/resources/mysql-foodmart-model.json you can 
take a look. By adding a new file(like sqlite.json) which stores connection 
properties for Sqlite, you can create Connection through 
“DriverManager.getConnection(jdbc:calcite:model=sqlite.json)”.
The connection above is what you need if I understand you correctly.


Best,
ZheHu
On 02/14/2022 11:40，Arthur Pan wrote：
Hi dev,

I'm trying to connect Calcite to Sqlite via Jdbc, and extract the existing
schemas from the a real database. I have tried to set up a
CalciteJdbc41Connection, but I have not been successful so far. I'm
wondering if this is the right way to do this, or are there any other
things that I should check first?

Sincerely,
Arthur Pan

Re:List of tables & Columns

[Zhe Hu]

Hi, Sharma. 
As far as I can tell, Calcite doesn’t provide a simple way to get table names 
or column names for a query(or maybe I miss something).
But you can take a look at SqlSelect、SqlOrderBy and SqlWith if you want to do 
such things. Based on my own experience, SqlVisitor seems the only way to get 
table/column names or alias by visiting the SqlNode, at least that’s how I 
extract tables/columns、distinguish SQL type or rewriting SQL in practice.
Hope it’s helpful for you.


Best,
ZheHu




On 02/5/2022 18:39，Yogendra Sharma wrote：
Hi Community,

I had to gather the list of all fully qualified table names and thier column 
names used(referenced) jn a SELECT query.

I simply parsed the query and wrote a visitor to visit the SqlNode recursively.

While its working (may be with few bugs here and there), i am wondering if 
Calcite would already be doing it somewhere ir if there is an easier way to 
achieve this?


Thanks
Y



Get Outlook for Android

Question about creating a SqlBasicCall

[Zhe Hu]

Hi, community.


I have one small question about how to create a SqlBasicCall in Parser.jj 
correctly.
Take TimestampAddFunctionCall as an example, we create it by:
SqlStdOperatorTable.TIMESTAMP_ADD.createCall(s.end(this), args)
As a result, call like “TIMESTAMPADD(arg0, arg1, arg2)” will be formed.


Here is my question:
How does the SqlParser assemble the operands with COMMA correctly?
Since the result is not “TIMESTAMPADD(arg0 arg1, arg2)”, which loses the former 
COMMA; or “TIMESTAMPADD(arg0 arg1, arg2)”、“TIMESTAMPADD(arg0 arg1 arg2)”.


I’ve noticed that SqlParserPos is involved, but I’m not sure if it’s relevant.


Any feedback would be awesome.


Best,
ZheHu

Re: [DISCUSS] New community section for articles/blogs/papers

[Zhe Hu]

It’s definitely a terrific proposal, especially for those new comers and people 
who want to share something useful with the community.


I totally agree with what Jing Zhan and Ruben Q L mentioned, those useful stuff 
has higher requirements for authors and reviewers. What’s more, for each 
release, the RM might need to take a little while to briefly check if those 
materials are still available.


Still, +1 for this.


Best,
ZheHu


On 01/24/2022 17:47，Ruben Q L wrote：
Thanks for the proposal Stamatis.

I agree that such a section could be very helpful, especially for new
comers. As you suggest, perhaps a new bullet "external resources" (or
something like that) under the community page?

I also share the concerns that Jing Zhang expressed. Moreover, there's a
risk that over the time some of the links that we share may become outdated
or obsolete.

Best,
Ruben




On Mon, Jan 24, 2022 at 3:20 AM Jing Zhang  wrote:

Hi Stamatis,

Thanks for driving this discussion.

Big +1 on the idea. They are undoubtedly very helpful for newcomers to
Calcite. I have heard a lot of requests in the dev mailing lists to get
more started documentation.

At the same time, we need take careful to decide what contents to put in
this section. Because the content introduced on the community page means
that the content has been recognized by the community and is more
authoritative.

Best,
Jing Zhang



Stamatis Zampetakis  于2022年1月24日周一 05:19写道：

Hi all,

Taking the recent email of Gavin [1] (sharing his article on GraphQL to
SQL
conversion) as a motive, I wanted to see what people think of creating a
new section in the website for putting this and other useful stuff such
as
Michael's notebooks, Querifylabs blogs [2], etc.

We could have one or more new bullets under the community page [3]. WDYT?

Best,
Stamatis

[1] https://lists.apache.org/thread/3kgwsg18bglszfm51ngy1m58fnh9dd2c
[2] https://www.querifylabs.com/blog
[3] https://calcite.apache.org/community/

Re: [ANNOUNCE] New Calcite PMC chair: Ruben Q L

[Zhe Hu]

Congratulations to Ruben!  Thanks for serving as Chair during the last year, 
Haisheng!


Best regards,
ZheHu






On 01/20/2022 13:09，Jacques Nadeau wrote：
Ditto, thanks Haisheng and congrats Ruben!

On Wed, Jan 19, 2022, 6:35 PM Julian Hyde  wrote:

Congratulations, Ruben, and good luck!

Haisheng, Thank you for serving as Chair.

Julian

On Wed, Jan 19, 2022 at 6:29 PM Haisheng Yuan  wrote:

Calcite community members,

I am pleased to announce that we have a new PMC chair and VP as per our
tradition of rotating the chair once a year. I have resigned, and Ruben
Q L
was duly elected by the PMC and approved unanimously by the Board.

Please join me in congratulating Ruben!

Best,
Haisheng Yuan

Re:[SITE-IMPROVEMENT] Issue need to confirm about the API web

[Zhe Hu]

Hi, I have one small suggestion for the SITE_IMPROVEMENT.
Recently，when I browse howto.html through Chrome Explorer, I’ve found that some 
lines are too long, which causes a little bit format disorder(just based on my 
personal preference). I’ll pick several cases in howto.md below:
(1) Line 73: However, if your IDE does not generate sources…
(2) Line 262: `Exception in thread “main”...
(3) Line 785: “Troubleshooting” character


Best,
ZheHu




On 12/29/2021 06:44，xiong duan wrote：
After checking the regenerate the Web, I noticed a small problem with the
Calcite API URI https://calcite.apache.org/javadocAggregate/ . The doc
title should be "Apache Calcite API", but is "Apache Calcite calcite API".
Same as Avatica, The doc title should be " Apache Calcite Avatica API", but
is "Apache Calcite Avatica calcite-avatica API". I think this is a problem.
Do you think so?

Re: [SITE-IMPROVEMENT] Issue need to confirm about the API web

[Zhe Hu]

+1 for removing the redundant lowercase “calcite”/”calcite-avatica”.


Best,
ZheHu




On 12/29/2021 10:44，Chunwei Lei wrote：
Sounds reasonable to me.


Best,
Chunwei


On Wed, Dec 29, 2021 at 6:44 AM xiong duan  wrote:

After checking the regenerate the Web, I noticed a small problem with the
Calcite API URI https://calcite.apache.org/javadocAggregate/ . The doc
title should be "Apache Calcite API", but is "Apache Calcite calcite API".
Same as Avatica, The doc title should be " Apache Calcite Avatica API", but
is "Apache Calcite Avatica calcite-avatica API". I think this is a problem.
Do you think so?

Re: [QUESTION] Calcite Learning Resources

[Zhe Hu]

Hi, Givre.
You can find some very useful talks at 
http://calcite.apache.org/community/#talks.
Moreover, the website 
https://calcite.apache.org/develop/#download-source-build-and-run-tests is a 
good start for starters, which I find very instructive at least.


Best,
ZheHu

|


|
 Replied Message 
| From | Charles Givre |
| Date | 12/23/2021 23:06 |
| To | dev |
| Subject | [QUESTION] Calcite Learning Resources |
Hello Calcite Team,
I hope everyone is doing well.  I had a question for the team.  We are 
onboarding some new engineers for our product which uses Calcite for a variety 
of purposes.  As the learning curve can be a little steep, I was wondering if 
there are any additional resources other than the docs for a new developer to 
get their heads wrapped around Calcite?  Tutorials, presentations etc.?

Thanks so much and Happy Holidays to all!!
— C

Re: [DISCUSS] Java 16 and 17 support

[Zhe Hu]

Hi, Julian.
I’m happy to do those tasks.
Just one question, where or how can I ensure CI tests for JDK16 and JDK17.


Best,

|
Zhe Hu
|
 Replied Message 
| From | Julian Hyde |
| Date | 12/22/2021 07:37 |
| To | dev@calcite.apache.org |
| Subject | [DISCUSS] Java 16 and 17 support |
Does Calcite now support Java 16 and 17? If think it probably does,
because of these three commits:
* 
https://github.com/apache/calcite/commit/6d51d277b158ff7073f29ada4a96a7a74c0b46fc
* 
https://github.com/apache/calcite/commit/dff775d24ae6f8b2e0f3b68fd727ef16fc9cb428
* 
https://github.com/apache/calcite/commit/a8a6569e6ba75efe9d5725c49338a7f181d3ab5c
/ https://issues.apache.org/jira/browse/CALCITE-4829

However, as of 1.29 RC0 the release notes still say 'This release is
tested on Linux, macOS, Microsoft Windows; using JDK/OpenJDK versions
8 to 15' [1].

The Gradle wrapper uses version 7.3 but the HOWTO still says to use
Gradle 7.2. I don't know whether Vladimir simply forgot to update
howto.md when he made commit 6d51d277 or deliberately did not, because
(in his words [2]) "Manual updates of the version in howto.md is one
of the worst ways to spend time on Calcite."

Anticipating that Vladimir would fail to update howto.md, I added the
instructions "Check that site/_docs/howto.md has the correct Gradle
version" [3] but Rui failed to follow those instructions when making
1.29 RC 0.

If Calcite does indeed support JDK 16 and 17, then the following tasks
need to be done:
* Update the Gradle version in howto.md
* Add JDK 16 and 17 to history.md
* Ensure there are sufficient CI tests for JDK 16 and 17
* Change the subject of
https://issues.apache.org/jira/browse/CALCITE-4547 to "Support Java 16
and 17", mark it fixed in release 1.29, and retrospectively add it to
1.29's release notes.

Any volunteers to do some or all of these tasks?

Julian

[1] 
https://github.com/apache/calcite/blob/calcite-1.29.0-rc0/site/_docs/history.md

[2] 
https://issues.apache.org/jira/browse/CALCITE-4575?focusedCommentId=17340363=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17340363

[3] 
https://github.com/apache/calcite/blob/master/site/_docs/howto.md#making-a-release-candidate

Re:[ANNOUNCE] New committer: Alessandro Solimando

[Zhe Hu]

Congratulations! Alessandro Solimando.


On 12/18/2021 22:08，Stamatis Zampetakis wrote：
Apache Calcite's Project Management Committee (PMC) has invited Alessandro
Solimando to become a committer, and we are pleased to announce that he has
accepted.

Alessandro has been in the community for quite some time now with his first
contribution dating back in 2018. Since then he pushed many fixes in both
Calcite and Avatica touching parts of the core engine,
the Cassandra adapter, and materialized view matching. Apart from code
contributions, Alessandro provides valuable help to the community by doing
reviews, answering questions in the devlist, and testing releases.

Alessandro, welcome, thank you for your contributions, and we look forward
to your further interactions with the community! If you wish, please feel
free to tell us more about yourself and what you are working on.

Stamatis (on behalf of the Apache Calcite PMC)

Re: [Calcite Release] Preparing for 1.29.0 Release

[Zhe Hu]

Hi, Rui.
CALCITE-4860 will be fixed in next release, you can pass it currently.
Thanks for your work!


Best,
ZheHu






On 12/18/2021 06:08，Francesco Gini wrote：
https://issues.apache.org/jira/browse/CALCITE-4907 is in review. I believe
I have addressed all the comments. Let me know if I missed anything.

On Fri, 17 Dec 2021 at 22:42, Jacques Nadeau  wrote:

I've merged CALCITE-4948 to mitigate the concerns around CALCITE-4898 and
maven repositories and category x licenses.

On Fri, Dec 17, 2021 at 1:02 PM Rui Wang  wrote:

For the list of 1.29 fix Jira, several Jira have been marked as resolved
since this Monday. I will do another pass today but probably not all
those
will make the 1.29.0 release.


-Rui

On Fri, Dec 17, 2021 at 11:57 AM Ruben Q L  wrote:

Hello,

FYI https://issues.apache.org/jira/browse/CALCITE-4737 has been
merged.

Regards,
Ruben


On Fri, Dec 17, 2021 at 7:40 PM Julian Hyde 
wrote:

Rui,

Even though https://issues.apache.org/jira/browse/CALCITE-4898 <
https://issues.apache.org/jira/browse/CALCITE-4898> is marked fixed,
there’s some fallout regarding maven repositories and category X.
That
needs to be sorted out before RC0. See [1].

I’m working on getting
https://issues.apache.org/jira/browse/CALCITE-4946
<https://issues.apache.org/jira/browse/CALCITE-4946> done today.
Please
wait for that.

https://issues.apache.org/jira/browse/CALCITE-4885 <
https://issues.apache.org/jira/browse/CALCITE-4885> is a big
refactoring
and, though I’d love to get it in before the release, it isn’t going
to
happen. Don’t wait for it.

I also see the following as open or in progress and fix-in-1.29:
* https://issues.apache.org/jira/browse/CALCITE-4872 <
https://issues.apache.org/jira/browse/CALCITE-4872> (Will Noble)
* https://issues.apache.org/jira/browse/CALCITE-4908 <
https://issues.apache.org/jira/browse/CALCITE-4908> (Woonsan Koo)
* https://issues.apache.org/jira/browse/CALCITE-4704 <
https://issues.apache.org/jira/browse/CALCITE-4704> (Stamatis)
* https://issues.apache.org/jira/browse/CALCITE-4737 <
https://issues.apache.org/jira/browse/CALCITE-4737> (Thomas Rebele)
* https://issues.apache.org/jira/browse/CALCITE-4860 <
https://issues.apache.org/jira/browse/CALCITE-4860> (Zhe Hu / Duan
Xiong)
* https://issues.apache.org/jira/browse/CALCITE-4907 <
https://issues.apache.org/jira/browse/CALCITE-4907> (Francesco Gini)

What’s the status on those?

Julian

[1] https://github.com/apache/calcite/pull/2622 <
https://github.com/apache/calcite/pull/2622>

PS It feels churlish to bring this up, but in time-honored tradition
the
email subject has been ‘[DISCUSS] Towards Calcite x.y.z’.


On Dec 14, 2021, at 10:37 AM, Rui Wang 
wrote:

Hi community,

To prepare for Calcite 1.29.0 release, I will start to check the
JIRA
list
[1] which marks its fix version as 1.29.0 (9 JIRA as of now remains
open/in
progress) and also the pending PRs which looks promising to be
merged.
Please help resolve/review/merge JIRA/PRs if you want some fixes to
be
released.


My current goal is to start freezing the main branch of Calcite by
the
end
of this week.

Please let me know if you have any questions.

[1]:




https://issues.apache.org/jira/browse/CALCITE-4941?jql=project%20%3D%20CALCITE%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22)%20AND%20fixVersion%20%3D%201.29.0


-Rui







--
Francesco Gini

Re:[DISCUSS] JIRA vs GitHub Issues

[Zhe Hu]

+0.5
As you mentioned, "co-locating issues and PRs would make external
contributions easier”, I do agree with it. What’s more, when people browse this 
Project, they could have a basic picture about the latest progress more 
intuitive.
But I also have some doubts about this migration. How do we process the 
existing issues in JIRA? Moving all of them to Github or just the open issues?


Best,
Zhe Hu







On 11/29/2021 00:43，Vladimir Sitnikov wrote：
Hi,

What do you think of moving to GitHub Issues for Calcite?

Currently, my (Calcite) development workflow focuses on pull requests.
That is all contributions I see come via pull requests.

At the same time, issues are hosted in JIRA, which creates friction: PR
merging requires changes to both GitHub and JIRA.
I guess it would be easier if issue management was in GitHub as well.

Then, I believe, co-locating issues and PRs would make external
contributions easier.
The links between Issues and PRs would be easier to navigate.

There's a possibility to enable GitHub discussions as well (see
https://github.com/apache/airflow/discussions ).
Co-locating Issues, and Discussions look promising.

I think it is not that painful, however, GitHub has a limitation of 20
collaborators (non-committers who want to assign, edit, close issues, and
pull requests):
https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-AssigningexternalcollaboratorswiththetriageroleonGitHub

I have no strong opinion, however, I just realized having issues in GitHub
would ease friction.

Any thoughts?

Just to gather opinion:
-1..+1 keep using ASF JIRA for issue management
-1..+1 migrate to GitHub Issues

Here's my vote:
+0.5 keep using ASF JIRA for issue management. It more-or-less works,
however, merging PRs and navigating from PR to issue is far from perfect.
+1 migrate to GitHub Issues. It would simplify navigation between issues
and PRs, and I believe it would reduce friction for external contributors.
In theory, GitHub Issues can be automated with Actions (e.g. labels). I'm
not sure if we need that, however, it might be useful.

Vladimir

Re: [DISCUSS] Upgrading Elasticsearch in Calcite from 7.0.1 to 7.15

[Zhe Hu]

Hi, Julian, Andrei.
I’ve upgraded the embedded Elasticsearch version to 7.15.2, so does the 
RestClient, and all the existing tests could run through. But when I try new ES 
features in unit test(such as multi_terms、top_metrics), results show that they 
are not available(cannot be parsed).
I’ve also installed ES 7.15.2 locally, and I can get correct results with the 
above new features by running client codes, which are the same with the codes 
in calcite.
So I’m wandering if I missed something necessary. It will be very helpful if 
you guys have some suggestions or directions.


Best,
Zhe Hu






On 11/23/2021 09:08，Zhe Hu wrote：
That’s very helpful, thanks for your reminder.
I will look into whether ES rest client is backward compatible.






On 11/23/2021 07:34，Andrei Sereda wrote：
+1

Also take a look at vendor's official support schedule:

https://www.elastic.co/support/eol

I think it is safe to upgrade  ES rest client to currently supported
version.




On Mon, Nov 22, 2021, 19:00 Julian Hyde  wrote:

+1

Sounds like a good idea.

On Nov 22, 2021, at 6:40 AM, Zhe Hu  wrote:

Hi, community.


Recently, when I tried to fix bugs in Elasticsearch Adapter, I’ve found
that some new features in released Elasticsearch cannot be applied in
Calcite, since the Elasticsearch version in Calcite is 7.0.1.


For instance, as CALCITE-4868 mentioned, I want to sort aggregation
results by multi_terms(supported after ES 7.1.11) as it’s more suitable
under such circumstances than terms aggregation, however the DSL is not
supported in Calcite currently.


Moreover, I've browsed Elasticsearch's release notes roughly, and there
are many new SQL features were added from 7.0.1 to 7.15.2, and lots of bugs
were fixed meanwhile(if needed, I will spend some time to collect such info
that benefits Calcite users).


Since we use RestClient in Elasticsearch Adapter, there are still some
cons about upgrading ES version, like new DSL couldn’t be used in old
version Elasticsearch cluster, current use case might need to be adjusted.


Nevertheless, I’m still wondering if we can upgrade the Elasticsearch
version to 7.15, which could enrich ES Adapter's ablility as much as
possible.


Best regards,
Zhe Hu

Re:Calcite to build a query

[Zhe Hu]

Hi, Sandeep
If I'm understanding you correctly, it seems that you wanna find a way to 
transfer a SqlNode or RelNode to SQL statement.
As far as I know, you can take a look at SqlNode, where the SqlNode can be 
transferred to SQL expression through toString() method. And, 
RelToSqlConverterTest illustrates how to assemble a SQL by RelNode.


Hope it’s helpful for you.


Best,
Zhe Hu






On 11/24/2021 18:38，Sandeep Nayak wrote：
Hello,

I have been looking for a library which will allow me to use an object tree
to generate a SQL. Calcite came up in my search but after taking a look at
the codebase my conclusion is that Calcite provides the ability to parse
SQL statements into an object tree represented as SQLNode (and derived
classes) instances which subsequently can be used to perform optimizations
by generating  relational algebra for the logical query.

Calcite however does not provide classes which can be used to assemble a
tree and generate SQL statements. From the contracts at least SQLNode takes
in a Parser position and there are no other contracts which indicate such
an option. Can someone confirm if this is inaccurate?

If not Calcite, are there other open source libraries out there which allow
this?

Thanks in advance.

-Sandeep

Re: [DISCUSS] Upgrading Elasticsearch in Calcite from 7.0.1 to 7.15

[Zhe Hu]

That’s very helpful, thanks for your reminder.
I will look into whether ES rest client is backward compatible.






On 11/23/2021 07:34，Andrei Sereda wrote：
+1

Also take a look at vendor's official support schedule:

https://www.elastic.co/support/eol

I think it is safe to upgrade  ES rest client to currently supported
version.




On Mon, Nov 22, 2021, 19:00 Julian Hyde  wrote:

+1

Sounds like a good idea.

On Nov 22, 2021, at 6:40 AM, Zhe Hu  wrote:

Hi, community.


Recently, when I tried to fix bugs in Elasticsearch Adapter, I’ve found
that some new features in released Elasticsearch cannot be applied in
Calcite, since the Elasticsearch version in Calcite is 7.0.1.


For instance, as CALCITE-4868 mentioned, I want to sort aggregation
results by multi_terms(supported after ES 7.1.11) as it’s more suitable
under such circumstances than terms aggregation, however the DSL is not
supported in Calcite currently.


Moreover, I've browsed Elasticsearch's release notes roughly, and there
are many new SQL features were added from 7.0.1 to 7.15.2, and lots of bugs
were fixed meanwhile(if needed, I will spend some time to collect such info
that benefits Calcite users).


Since we use RestClient in Elasticsearch Adapter, there are still some
cons about upgrading ES version, like new DSL couldn’t be used in old
version Elasticsearch cluster, current use case might need to be adjusted.


Nevertheless, I’m still wondering if we can upgrade the Elasticsearch
version to 7.15, which could enrich ES Adapter's ablility as much as
possible.


Best regards,
Zhe Hu

Re: [DISCUSS] Apache Calcite Online Meetup January 2022

[Zhe Hu]

Got it. Thanks！
|


|
 Replied Message 
| From | xiong duan |
| Date | 11/22/2021 23:09 |
| To | dev |
| Subject | Re: [DISCUSS] Apache Calcite Online Meetup January 2022 |
Yes. You can find the relative talks in
http://calcite.apache.org/community/#talks .

Zhe Hu  于2021年11月22日周一 下午10:50写道：

> +1 for online meetup.
> And I’m curious about if there is any playback for this meetup, because
> somebody might couldn't attend due to timezone variety or any other reasons.
>
>
>
>
>
>
> On 11/22/2021 22:26，xiong duan wrote：
> Great idea. +1 for an online meetup.
>
> Justin Swanhart  于2021年11月20日周六 上午12:42写道：
>
> +1 for an online meetup
>
> On Fri, Nov 19, 2021, 11:32 AM Jacques Nadeau  wrote:
>
> WRT to talk structure, I propose we do 5 or 10 minute lightning talks as
> opposed to something longer. It ensures a wider variety of topics, better
> attendance and you never have to sit that long through something you're
> not
> interested in...
>
> On Thu, Nov 18, 2021 at 11:47 PM Vladimir Sitnikov <
> sitnikov.vladi...@gmail.com> wrote:
>
> Are the people willing to give a talk around Calcite?
>
> I may present a talk regarding Gradle, and the upcoming migration to
> the
> idiomatic structure:
> https://issues.apache.org/jira/browse/CALCITE-4832
> https://github.com/apache/calcite/pull/2566
>
> https://docs.gradle.org/7.3/userguide/structuring_software_products.html
>
> What do people think in terms of timing?
>
> I think the new build structure is more like 30-45 min talk
>
> Vladimir
>
>
>
>

Re: [DISCUSS] Apache Calcite Online Meetup January 2022

[Zhe Hu]

+1 for online meetup.
And I’m curious about if there is any playback for this meetup, because 
somebody might couldn't attend due to timezone variety or any other reasons.






On 11/22/2021 22:26，xiong duan wrote：
Great idea. +1 for an online meetup.

Justin Swanhart  于2021年11月20日周六 上午12:42写道：

+1 for an online meetup

On Fri, Nov 19, 2021, 11:32 AM Jacques Nadeau  wrote:

WRT to talk structure, I propose we do 5 or 10 minute lightning talks as
opposed to something longer. It ensures a wider variety of topics, better
attendance and you never have to sit that long through something you're
not
interested in...

On Thu, Nov 18, 2021 at 11:47 PM Vladimir Sitnikov <
sitnikov.vladi...@gmail.com> wrote:

Are the people willing to give a talk around Calcite?

I may present a talk regarding Gradle, and the upcoming migration to
the
idiomatic structure:
https://issues.apache.org/jira/browse/CALCITE-4832
https://github.com/apache/calcite/pull/2566

https://docs.gradle.org/7.3/userguide/structuring_software_products.html

What do people think in terms of timing?

I think the new build structure is more like 30-45 min talk

Vladimir

[DISCUSS] Upgrading Elasticsearch in Calcite from 7.0.1 to 7.15

[Zhe Hu]

Hi, community.


Recently, when I tried to fix bugs in Elasticsearch Adapter, I’ve found that 
some new features in released Elasticsearch cannot be applied in Calcite, since 
the Elasticsearch version in Calcite is 7.0.1.


For instance, as CALCITE-4868 mentioned, I want to sort aggregation results by 
multi_terms(supported after ES 7.1.11) as it’s more suitable under such 
circumstances than terms aggregation, however the DSL is not supported in 
Calcite currently.


Moreover, I've browsed Elasticsearch's release notes roughly, and there are 
many new SQL features were added from 7.0.1 to 7.15.2, and lots of bugs were 
fixed meanwhile(if needed, I will spend some time to collect such info that 
benefits Calcite users).


Since we use RestClient in Elasticsearch Adapter, there are still some cons 
about upgrading ES version, like new DSL couldn’t be used in old version 
Elasticsearch cluster, current use case might need to be adjusted.


Nevertheless, I’m still wondering if we can upgrade the Elasticsearch version 
to 7.15, which could enrich ES Adapter's ablility as much as possible.


Best regards,
Zhe Hu

Re: [jira] [Created] (CALCITE-4896) GROUP BY might get wrong results under certain circumstances in Elasticsearch Adapter

[Zhe Hu]

I do agree it’s reasonable in Calcite(however not 100% correct).
And as far as I know, in Elasticsearch, reserved value for NULL is often used 
when people know specific data range.
As for this issue, someone else may find a better solution. Are you interested? 
Swanhart.




On 11/21/2021 21:30，Justin Swanhart wrote：
Thanks for explaining because I did not get that at all, but it makes
sense.  Some column stores use a reserved value for NULL too which reduces
the overall range of values available.

On Sun, Nov 21, 2021 at 7:09 AM Zhe Hu  wrote:

Yes, when you group an Integer field in Elasticsearch Adapter, NULL and
Integer.MIN_VALUE will be in the same group, which is not correct for SQL
users.


On 11/21/2021 19:56，Justin Swanhart wrote：
If I am understanding correctly, the field2 value is NULL.

Count(0) should be 2 and count(field2)==2 because of the way aggregates
treat NULL values.

On Fri, Nov 19, 2021, 8:21 PM ZheHu (Jira)  wrote:

ZheHu created CALCITE-4896:
--

Summary: GROUP BY might get wrong results under certain
circumstances in Elasticsearch Adapter
Key: CALCITE-4896
URL: https://issues.apache.org/jira/browse/CALCITE-4896
Project: Calcite
Issue Type: Bug
Components: elasticsearch-adapter
Affects Versions: 1.28.0
Reporter: ZheHu


In Elasticsearch Adapter, if one field is grouped by along with missing
value, SQL user might get incorrect results. Take the following case as an
example:
{code:java}
ES mappings: {"int_field1" : integer, "int_field2" : integer}

doc1 = {"int_field1":1, "int_field2": -2147483648}
doc1 = {"int_field1":2}
{code}

When I try "select count(1) as CNT from view group by int_field2", the
result is 2.

For Integer type, the missing value is replaced by Integer.MIN_VALUE, so
doc1 and doc2 will be divided in the same group.

Any other data types like short、long、float、double, they also have such
problem.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [jira] [Created] (CALCITE-4896) GROUP BY might get wrong results under certain circumstances in Elasticsearch Adapter

[Zhe Hu]

Yes, when you group an Integer field in Elasticsearch Adapter, NULL and 
Integer.MIN_VALUE will be in the same group, which is not correct for SQL users.


On 11/21/2021 19:56，Justin Swanhart wrote：
If I am understanding correctly, the field2 value is NULL.

Count(0) should be 2 and count(field2)==2 because of the way aggregates
treat NULL values.

On Fri, Nov 19, 2021, 8:21 PM ZheHu (Jira)  wrote:

ZheHu created CALCITE-4896:
--

Summary: GROUP BY might get wrong results under certain
circumstances in Elasticsearch Adapter
Key: CALCITE-4896
URL: https://issues.apache.org/jira/browse/CALCITE-4896
Project: Calcite
Issue Type: Bug
Components: elasticsearch-adapter
Affects Versions: 1.28.0
Reporter: ZheHu


In Elasticsearch Adapter, if one field is grouped by along with missing
value, SQL user might get incorrect results. Take the following case as an
example:
{code:java}
ES mappings: {"int_field1" : integer, "int_field2" : integer}

doc1 = {"int_field1":1, "int_field2": -2147483648}
doc1 = {"int_field1":2}
{code}

When I try "select count(1) as CNT from view group by int_field2", the
result is 2.

For Integer type, the missing value is replaced by Integer.MIN_VALUE, so
doc1 and doc2 will be divided in the same group.

Any other data types like short、long、float、double, they also have such
problem.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

PR review request(CALCITE-4868)

[Zhe Hu]

Hi team, 
This PR is about supporting sorting aggregation results in Elasticsearch 
Adapter, please help review it in your free time. Thanks a lot!


What’s more, the git workflow shows “First-time contributors need a maintainer 
to approve running workflows”, awaiting approval?


PR:https://github.com/apache/calcite/pull/2599
ISSUE:https://issues.apache.org/jira/projects/CALCITE/issues/CALCITE-4868


Best regards.