Re: Flakey test on Linux (JDK 11), Avatica master

[xu]

Hi Haisheng,



I spend some time debugging and find the possible root cause.



In BasicSqlType, the value of `toString` result and instance member
`digest` might be different. For TIMESTAMP without precision (precision =
-1), toString returns TIMESTAMP and digest is TIMESTAMP(0), which conflicts
with real TIMESTAMP(0) with 0 as precision.


`Interner DATATYPE_CACHE = Interners.newWeakInterner()` makes
sure SqlType is singleton between invocations, TIMESTAMP(0) might return
SqlType - TIMESTAMP without precision literally which is wrong. Because the
global cache uses weak reference, so the cache would usually invalidate
after Java GC, which mitigates the impact of the hidden bug. As for the
specific environment Linux JDK 11 situation, I could not give a reasonable
explanation yet but I suppose the cause is clear.



I find `org.apache.calcite.rel.rules.DateRangeRulesTest` could create
TIMESTAMP without precision, just right before the errors occurred, and the
error always starts from org.apache.calcite.test.SqlLimitsTest >
testPrintLimits() according to the two failure builds per your mail.



If the analysis above makes sense, I can create an issue in JIRA and fix
the problem to make digest & toString return the same value for TIMESTAMP.


Best,

Xu



Haisheng Yuan  于2020年4月14日周二 上午10:00写道：

> Hi,
>
> There are frequently flakey test on Linux(JDK 11) related with
> TIMESTAMP(0) vs TIMESTAMP.
>
> Below is failure test from [1].
> FAILURE 0.0sec, org.apache.calcite.test.SqlToRelConverterExtendedTest >
> testTableValuedFunctionTumbleWithSubQueryParam()
> 1663 org.opentest4j.AssertionFailedError: plan ==> expected: <
> 1664 LogicalProject(ORDERID=[$0], ROWTIME=[$1], window_start=[$2],
> window_end=[$3])
> 1665 LogicalTableFunctionScan(invocation=[TUMBLE($1, DESCRIPTOR($1),
> 6:INTERVAL MINUTE)], rowType=[RecordType(INTEGER ORDERID, TIMESTAMP(0)
> ROWTIME, TIMESTAMP(0) window_start, TIMESTAMP(0) window_end)])
> 1666 LogicalProject(ORDERID=[$0], ROWTIME=[$1])
> 1667 LogicalTableScan(table=[[CATALOG, SALES, SHIPMENTS]])
> 1668 > but was: <
> 1669 LogicalProject(ORDERID=[$0], ROWTIME=[$1], window_start=[$2],
> window_end=[$3])
> 1670 LogicalTableFunctionScan(invocation=[TUMBLE($1, DESCRIPTOR($1),
> 6:INTERVAL MINUTE)], rowType=[RecordType(INTEGER ORDERID, TIMESTAMP
> ROWTIME, TIMESTAMP window_start, TIMESTAMP window_end)])
> 1671 LogicalProject(ORDERID=[$0], ROWTIME=[$1])
> 1672 LogicalTableScan(table=[[CATALOG, SALES, SHIPMENTS]])
> 1673 >
>
> Below is failure test from [2]:
> testPrintLimits
> The only diff is TIMESTAMP(0) vs TIMESTAMP
>
> Any clue?
>
> [1] https://github.com/apache/calcite/runs/576436249
> [2] https://github.com/apache/calcite/runs/584340845
>
>
> - Haisheng
>
>

-- 

Best regards,

Xu Zhang (张旭)

Re: Re: Flakey test on Linux (JDK 11), Avatica master

[xu]

Yes, will do, working on this now. BTW, neoremind is my github id 😃

Haisheng Yuan  于2020年4月15日周三 上午11:30写道：

> Hi Xu,
>
> Thanks for your quick response and looking into this issue.
>
> Looks like neoremind just opened a pull request based on your analysis
> (thanks neoremind!), can you help review?
> https://github.com/apache/calcite/pull/1916
>
> - Haisheng
>
> --
> 发件人：xu
> 日 期：2020年04月14日 20:13:15
> 收件人：
> 主 题：Re: Flakey test on Linux (JDK 11), Avatica master
>
> Hi Haisheng,
>
>
>
> I spend some time debugging and find the possible root cause.
>
>
>
> In BasicSqlType, the value of `toString` result and instance member
> `digest` might be different. For TIMESTAMP without precision (precision =
> -1), toString returns TIMESTAMP and digest is TIMESTAMP(0), which conflicts
> with real TIMESTAMP(0) with 0 as precision.
>
>
> `Interner DATATYPE_CACHE = Interners.newWeakInterner()` makes
> sure SqlType is singleton between invocations, TIMESTAMP(0) might return
> SqlType - TIMESTAMP without precision literally which is wrong. Because the
> global cache uses weak reference, so the cache would usually invalidate
> after Java GC, which mitigates the impact of the hidden bug. As for the
> specific environment Linux JDK 11 situation, I could not give a reasonable
> explanation yet but I suppose the cause is clear.
>
>
>
> I find `org.apache.calcite.rel.rules.DateRangeRulesTest` could create
> TIMESTAMP without precision, just right before the errors occurred, and the
> error always starts from org.apache.calcite.test.SqlLimitsTest >
> testPrintLimits() according to the two failure builds per your mail.
>
>
>
> If the analysis above makes sense, I can create an issue in JIRA and fix
> the problem to make digest & toString return the same value for TIMESTAMP.
>
>
> Best,
>
> Xu
>
>
>
> Haisheng Yuan  于2020年4月14日周二 上午10:00写道：
>
> > Hi,
> >
> > There are frequently flakey test on Linux(JDK 11) related with
> > TIMESTAMP(0) vs TIMESTAMP.
> >
> > Below is failure test from [1].
> > FAILURE 0.0sec, org.apache.calcite.test.SqlToRelConverterExtendedTest >
> > testTableValuedFunctionTumbleWithSubQueryParam()
> > 1663 org.opentest4j.AssertionFailedError: plan ==> expected: <
> > 1664 LogicalProject(ORDERID=[$0], ROWTIME=[$1], window_start=[$2],
> > window_end=[$3])
> > 1665 LogicalTableFunctionScan(invocation=[TUMBLE($1, DESCRIPTOR($1),
>
> > 6:INTERVAL MINUTE)], rowType=[RecordType(INTEGER ORDERID, TIMESTAMP(0)
> > ROWTIME, TIMESTAMP(0) window_start, TIMESTAMP(0) window_end)])
> > 1666 LogicalProject(ORDERID=[$0], ROWTIME=[$1])
> > 1667 LogicalTableScan(table=[[CATALOG, SALES, SHIPMENTS]])
> > 1668 > but was: <
> > 1669 LogicalProject(ORDERID=[$0], ROWTIME=[$1], window_start=[$2],
> > window_end=[$3])
> > 1670 LogicalTableFunctionScan(invocation=[TUMBLE($1, DESCRIPTOR($1),
> > 6:INTERVAL MINUTE)], rowType=[RecordType(INTEGER ORDERID, TIMESTAMP
> > ROWTIME, TIMESTAMP window_start, TIMESTAMP window_end)])
> > 1671 LogicalProject(ORDERID=[$0], ROWTIME=[$1])
> > 1672 LogicalTableScan(table=[[CATALOG, SALES, SHIPMENTS]])
> > 1673 >
> >
> > Below is failure test from [2]:
> > testPrintLimits
> > The only diff is TIMESTAMP(0) vs TIMESTAMP
> >
> > Any clue?
> >
> > [1] https://github.com/apache/calcite/runs/576436249
> > [2] https://github.com/apache/calcite/runs/584340845
> >
> >
> > - Haisheng
> >
> >
>
> --
>
> Best regards,
>
> Xu Zhang (张旭)
>
>

-- 

Best regards,

Xu

Re: [ANNOUNCE] New committer: Vineet Garg

[xu]

Congrats, Vineet!

Danny Chan  于2020年4月26日周日 下午4:52写道：

> Congrats, Vineet!
>
> Best,
> Danny Chan
> 在 2020年4月26日 +0800 PM1:55，dev@calcite.apache.org，写道：
> >
> > Congrats, Vineet!
>


-- 

Best regards,

Xu

Re: Using indexes rather than table scans with Calcite

[xu]

Hi Tim,

I am working on MySQL InnoDB adapter and trying to introduce this to
Calcite, currently it is only in early stage, and not approved/reviewed by
committers yet. Anyway, we are facing the same problem like what index to
use, how to push down order by operation, etc. I have developed a simple
rule based adapter to be "index aware" and being able to leverage a MySQL
InnoDB storage engine written in Java. Hope this will help you to explore
more options.

https://issues.apache.org/jira/browse/CALCITE-4034

Thanks,
Xu

Haisheng Yuan  于2020年5月31日周日 下午10:06写道：

> Hi Roman,
>
> Thank you for sharing your thoughts.
>
> > It can be very tricky because the rule should consider not
> > only filters, but also collations. This leads to increasing the
> > complexity of such rules.
>
> Logical transformation rules like FilterTableScan2IndexTableScanRule
> should not consider physical properties, like collation, distribution. You
> forgot that we just reached consensus in CALCITE-3972. :)
>
> Regarding the 2nd option that uses index b, it is indeed not that easy for
> Calcite 1.22.0. In latest version, it now becomes possible. After rule
> transformation, during top-down trait request, the collation is passed
> through Filter, down to physical TableScan, which accepts the trait request
> with collation on b, find there is an index on b, and return a new RelNode
> IndexScan. This process can be done in
> EnumerableTableScan#passThrough(required).
>
> > I hope, when the Cascades-style optimizer become a part of Calcite, the
> search space
> > pollution will not be a serious issue anymore.
>
> I hope so too. Top-down style can help alleviate space pollution by space
> pruning. But the space pollution caused by LogicalProject operator and its
> related rules still can't be avoided. :)
>
> Again, thanks for sharing your experience with us.
>
> Haisheng
>
> On 2020/05/31 09:58:36, Roman Kondakov 
> wrote:
> > Hi Haisheng,
> >
> > The basic rationale behind the using materialized views for secondary
> > index representation instead of special rules like mentioned
> > FilterTableScan2IndexTableScanRule is the simplicity of implementation.
> >
> > You are absolutely right that materialized views approach has an obvious
> > drawback that it should register all indexes as materialized views in
> > the optimizer's search space. But we expect our users will not overuse
> > indexes because in general having too many indexes is a bad practice: on
> > each table update we also should update it's indexes and it can cause
> > some performance degradation of the system as a whole, not only
> > optimizer. So we expect the number of indexes will be relatively small.
> >
> > Ignite uses indexes not only for index lookups, but also for exploiting
> > it's sortedness. In this case materialized view's approach can show some
> > advantages. Let's consider the example:
> >
> > SELECT * FROM foo WHERE a > 100 ORDER BY b;
> >
> > where both fields 'a' and 'b' are indexed. In this case we will have two
> > alternatives of the query execution:
> >
> > 1. Use index 'a' for index conditioned scan and then sort rows by 'b'
> > 2. Use index scan over 'b' and then apply filter over 'a' - here we can
> > avoid sorting, because index over 'b' is already sorted.
> >
> > If I understand the approach with FilterTableScan2IndexTableScanRule
> > correctly, at this step the rule should make a decision about which
> > index to use. It can be very tricky because the rule should consider not
> > only filters, but also collations. This leads to increasing the
> > complexity of such rules. With materialized views approach we just
> > register all indexes with their caollation and let the cost model do its
> > job. Our rules are very simple. The complexity is encapsulated in the
> > index scan cost estimation.
> >
> > As for your example with disjunctive predicate:
> >
> > > SELECT * FROM foo WHERE a > 100 or b < 1000;
> >
> > I think with materialized views approach we can do a complex logic as
> > well. For example, we may implement a special rule for such cases:
> > BitmapOrRule. Or, even better, we will rewrite such predicates to a
> > UNION ALL clause. A very good explanation and a comparison of both
> > approaches I've found in this Oracle's blog post [1].
> >
> > As a conclusion: choosing between materialized views approach and
> > IndexRule-based approach is a trade-off between complexity of
> > implementation and search 

Re: [ANNOUNCE] New Calcite PMC chair: Haisheng Yuan

[xu]

Congratulations, Haisheng! Thank you for your work, Stamatis!

Ruben Q L  于2020年12月18日周五 下午5:08写道：

> Congratulations Haisheng!
> Thanks for your work Stamatis!
>
>
> On Fri, Dec 18, 2020 at 8:17 AM Alessandro Solimando <
> alessandro.solima...@gmail.com> wrote:
>
> > Thanks Stamatis for your hard work and dedication, and congratulations to
> > Haisheng for this appointment!
> >
> > Best regards,
> > Alessandro
> >
> > On Fri, 18 Dec 2020 at 07:31, Xin Wang  wrote:
> >
> > > Congrats Haisheng! Thanks for your work, Stamatis!
> > >
> > >
> > > Fan Liya  于2020年12月18日周五 下午12:08写道：
> > >
> > > > Congratulations, Haisheng!
> > > > Looking forward to your great work in the coming year!
> > > >
> > > > Stamatis, thanks for your great work in the past year!
> > > >
> > > > Best,
> > > > Liya Fan
> > > >
> > > >
> > > > On Fri, Dec 18, 2020 at 11:11 AM Feng Zhu 
> > wrote:
> > > >
> > > > > Thanks for your work and effort, Stamatis!
> > > > > Congratulations, Haisheng!
> > > > >
> > > > > Stamatis Zampetakis  于2020年12月17日周四 下午9:49写道：
> > > > >
> > > > > > Calcite community members,
> > > > > >
> > > > > > I am pleased to announce that we have a new PMC chair and VP as
> per
> > > our
> > > > > > tradition of rotating the chair once a year. I have resigned, and
> > > > > > Haisheng was duly elected by the PMC and approved unanimously by
> > the
> > > > > Board.
> > > > > >
> > > > > > Please join me in congratulating Haisheng!
> > > > > >
> > > > > > Best,
> > > > > > Stamatis
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Thanks,
> > > Xin
> > >
> >
>


-- 

Best regards,

Xu

Re: [ANNOUNCE] New committer: Liya Fan

[xu]

Congrats Liya!

Forward Xu  于2021年2月22日周一 上午11:28写道：

> Congratulations Liya!
>
>
> Best,
>
> Forward
>
> Chunwei Lei  于2021年2月22日周一 上午11:13写道：
>
> > Congratulations Liya!
> >
> > Best,
> > Chunwei
> >
> >
> > On Fri, Feb 12, 2021 at 5:28 PM Fan Liya  wrote:
> >
> > > Dear all,
> > >
> > > I want to express my sincere gratitude to you for all your kind help.
> > > I feel honored to have the opportunity of working with you in this
> > awesome
> > > community.
> > >
> > > In the past years, I have been working on some different components of
> > the
> > > SQL engine, now in Alibaba, and previously in Microsoft.
> > > Through this experience, I find the SQL optimizer especially
> interesting,
> > > because it involves so many interesting problems.
> > >
> > > In the future, I want to make greater contributions to the community
> and
> > > help our users build better SQL engines.
> > >
> > > Best,
> > > Liya Fan
> > >
> > >
> > > On Thu, Feb 11, 2021 at 1:54 AM Rui Wang  wrote:
> > >
> > > > Congratulations Liya!
> > > >
> > > >
> > > > -Rui
> > > >
> > > > On Wed, Feb 10, 2021 at 6:39 AM Michael Mior 
> wrote:
> > > >
> > > > > Congratulations and welcome Liya!
> > > > > --
> > > > > Michael Mior
> > > > > mm...@apache.org
> > > > >
> > > > > Le mer. 10 févr. 2021 à 05:23, Francis Chuang
> > > > >  a écrit :
> > > > > >
> > > > > > Congrats Liya and well deserved!
> > > > > >
> > > > > > Francis
> > > > > >
> > > > > > On 10/02/2021 8:53 pm, Stamatis Zampetakis wrote:
> > > > > > > Apache Calcite's Project Management Committee (PMC) has invited
> > > Liya
> > > > > Fan to
> > > > > > > become a committer, and we are pleased to announce that they
> have
> > > > > accepted.
> > > > > > >
> > > > > > > Liya has made a lot of contributions to the project, enhancing
> > > > > optimizer
> > > > > > > rules,
> > > > > > > improving the efficiency of graph traversal algorithms, as well
> > as
> > > > > fixing
> > > > > > > various
> > > > > > > bugs in the project.
> > > > > > >
> > > > > > > Liya, welcome, thank you for your contributions, and we look
> > > forward
> > > > > to your
> > > > > > > further interactions with the community! If you wish, please
> feel
> > > > free
> > > > > to
> > > > > > > tell
> > > > > > > us more about yourself and what you are working on.
> > > > > > >
> > > > > > > Stamatis (on behalf of the Apache Calcite PMC)
> > > > > > >
> > > > >
> > > >
> > >
> >
>


-- 

Best regards,

Xu

Questions regarding replacing table with SQL query using calcite

[Leon Xu]

Hi Calcite devs,

We are modeling a usecase using a DAG. Each node in the DAG will be some
select query. We would like to use calcite to convert all these queries
into one consolidated query.
e.g.
In node1, we have query: `select a from node2;`
In node2, we have query: `select a, b from foo;`

should be converted to `select a from (select a, b from foo);`

If we were to use calcite to achieve this, what would be the best
implementation? Should I extend a substituionVisitor and replace table name
with the correspoinding query?

Thanks

Re: Questions regarding replacing table with SQL query using calcite

[Leon Xu]

Thanks Julian, another follow-up question, is there a way to ask Calcite to
ignore validating a table if that table is not found?
In the example I gave: In node2, we have query: `select a, b from foo;`
I might not have information about table `foo`, is there a way to ask
calcite to just recognize it as a raw table and proceed with validation?
By default setting it will errors out with object 'foo' not found, if I am
using parser to validate the sql.

Thanks
Leon

On Sat, Aug 17, 2019 at 12:29 AM Julian Hyde  wrote:

> That is standard view expansion. We support this.
>
> Julian
>
> > On Aug 16, 2019, at 06:12, Leon Xu  wrote:
> >
> > Hi Calcite devs,
> >
> > We are modeling a usecase using a DAG. Each node in the DAG will be some
> > select query. We would like to use calcite to convert all these queries
> > into one consolidated query.
> > e.g.
> > In node1, we have query: `select a from node2;`
> > In node2, we have query: `select a, b from foo;`
> >
> > should be converted to `select a from (select a, b from foo);`
> >
> > If we were to use calcite to achieve this, what would be the best
> > implementation? Should I extend a substituionVisitor and replace table
> name
> > with the correspoinding query?
> >
> > Thanks
>

Apply contribution permission

[Leonard Xu]

Hi,calcite community.

I’d like to contribute to community, could you kindly give me the contribution 
permission.
My JIRA ID is:Leonard Xu

Best，
Leonard

Re: [ANNOUNCE] Danny Chan joins Calcite PMC

[Leonard Xu]

Congratulation！ Danny


> On 2019年10月31日, at 上午9:06, Andrei Sereda  wrote:
> 
> Congrats, Danny!
> 
> On Wed, Oct 30, 2019 at 8:08 PM Julian Hyde  wrote:
> 
>> Congratulations, Danny! Thank you for all of your hard work on code,
>> reviewing others' work, answering questions, and generally making our
>> community welcoming to all!
>> 
>> Julian
>> 
>> On Wed, Oct 30, 2019 at 4:32 PM Michael Mior  wrote:
>>> 
>>> Welcome on board Danny! Glad to have you.
>>> --
>>> Michael Mior
>>> mm...@apache.org
>>> 
>>> Le mer. 30 oct. 2019 à 17:22, Francis Chuang
>>>  a écrit :
 
 I'm pleased to announce that Danny has accepted an invitation to
 join the Calcite PMC. Danny has been a consistent and helpful
 figure in the Calcite community for which we are very grateful. We
 look forward to the continued contributions and support.
 
 Please join me in congratulating Danny!
 
 - Francis (on behalf of the Calcite PMC)
>> 

Re: [ANNOUNCE] Haisheng Yuan joins Calcite PMC

[Leonard Xu]

Congratulations! HaiSheng

Best,
Leonard Xu
> On 2019年11月11日, at 下午1:22, Danny Chan  wrote:
> 
> Congratulations HaiSheng!

[discuss] Enhanced MATH Function

[Forward Xu]

Hi everybody,



I'd like to kick off a discussion on Enhanced MATH functions in calcite.



I recently checked java's math function and found that some functions in
calcite are not supported. So I made some comparisons with some commonly
used databases to discuss whether these functions need to be supported. The
comparison details are in [2]. And built an umbrella jira for this [1].



Would love to hear your thoughts.



Best,

Forward


[1] https://issues.apache.org/jira/browse/CALCITE-3683

[2]
https://docs.google.com/spreadsheets/d/1Q6Mj1ub-etyE0dbwvsaLquy6bz_fKuFX0vU3mGn8QKA/edit?usp=sharing

Re: Gradle documentation update

[Forward Xu]

OK, I can try to fix this.

Feng Zhu  于2020年1月7日周二 下午8:05写道：

> I also noticed the same problem several days ago when a colleague
> complained he failed to build Calcite according to the documents on
> website.
>
>
> E.g.,
>
> https://calcite.apache.org/docs/howto.html#building-from-a-source-distribution
>
>
>
>
>
>
>
>
> *$ git clone git://github.com/apache/calcite.git
> $ cd calcite$ ./mvnw install $ mvn
> clean$ mvn package... hacks ...$ mvn package -DskipGenerate*
>
> https://calcite.apache.org/develop/#download-source-build-and-run-tests
>
>
> *$ git clone git://github.com/apache/calcite.git
> $ cd calcite$ mvn install*
>
> Vladimir Sitnikov  于2020年1月7日周二 下午7:44写道：
>
> > Can you clarify an url which has stale information?
> >
> > Vladimir
> >
>

[discuss] Implement Presto adapter

[Forward Xu]

Hi everybody,

I'd like to kick off a discussion on Implement Presto adapter in calcite.

We know that presto is also a powerful SQL query tool, which is widely used
with the olap of big data. So I want to add an adapter for presto to calite.

Would love to hear your thoughts.

Best,
Forward

Re: [discuss] Implement Presto adapter

[Forward Xu]

Hi, Stamatis

Because we have a number of internal clusters already using and installing
presto. And we want use calcite to optimize the parser and push it down to
presto, spark and other calculation engines or olaps. Utilize the sql
optimization function of calite in combination with distributed computing
engines or distributed olaps such as presto. In this way, each performance
can be exerted.

Best,
Forward

Stamatis Zampetakis  于2020年1月9日周四 下午8:51写道：

> Hi,
>
> Presto seems to have many similarities with Calcite itself.
> As you mentioned, Presto is not really a database but a distributed SQL
> framework to query heterogeneous data sources.
> Presto connectors are in some sense similar to Calcite adapters and if the
> Enumerable convention was capable of performing distributed joins etc.,
> then the similarities would be even bigger.
>
> To the best of my knowledge, the majority of our users adopt Calcite either
> to query multiple data sources in a unified way or to provide SQL query
> processing capabilities to data sources that do not already have. Presto
> seems to have similar goals, thus, I am not sure who will benefit from this
> new adapter.
>
> I never used Presto myself so I may have missed some important points. Can
> you share some use-cases about who may use this adapter and how?
>
> Best,
> Stamatis
>
>
> On Wed, Jan 8, 2020 at 2:44 PM Forward Xu  wrote:
>
> > Hi everybody,
> >
> > I'd like to kick off a discussion on Implement Presto adapter in calcite.
> >
> > We know that presto is also a powerful SQL query tool, which is widely
> used
> > with the olap of big data. So I want to add an adapter for presto to
> > calite.
> >
> > Would love to hear your thoughts.
> >
> > Best,
> > Forward
> >
>

Re: [discuss] Implement Presto adapter

[Forward Xu]

Hi Julian,

The current idea is through com.facebook.presto.spi.
1. Can be achieved through plugs provided by presto. For example [1].
2. You can construct Connector directly through
com.facebook.presto.spi.ConnectorFactory [2].

Best,
Forward

[1]
https://www.codota.com/code/java/methods/com.facebook.presto.spi.Plugin/getConnectorFactories
[2]
https://www.programcreek.com/java-api-examples/?code=y-lan/presto/presto-master/presto-raptor/src/main/java/com/facebook/presto/raptor/RaptorPlugin.java

Julian Hyde  于2020年1月10日周五 上午2:03写道：

> Sounds like you are planning to target Presto’s internal engine rather
> than Presto the database. To clarify, can you give an example of the code
> that you would generate and send to Presto for a simple SQL query?
>
> Julian
>
>
> > On Jan 9, 2020, at 5:07 AM, Forward Xu  wrote:
> >
> > Hi, Stamatis
> >
> > Because we have a number of internal clusters already using and
> installing
> > presto. And we want use calcite to optimize the parser and push it down
> to
> > presto, spark and other calculation engines or olaps. Utilize the sql
> > optimization function of calite in combination with distributed computing
> > engines or distributed olaps such as presto. In this way, each
> performance
> > can be exerted.
> >
> > Best,
> > Forward
> >
> > Stamatis Zampetakis  于2020年1月9日周四 下午8:51写道：
> >
> >> Hi,
> >>
> >> Presto seems to have many similarities with Calcite itself.
> >> As you mentioned, Presto is not really a database but a distributed SQL
> >> framework to query heterogeneous data sources.
> >> Presto connectors are in some sense similar to Calcite adapters and if
> the
> >> Enumerable convention was capable of performing distributed joins etc.,
> >> then the similarities would be even bigger.
> >>
> >> To the best of my knowledge, the majority of our users adopt Calcite
> either
> >> to query multiple data sources in a unified way or to provide SQL query
> >> processing capabilities to data sources that do not already have. Presto
> >> seems to have similar goals, thus, I am not sure who will benefit from
> this
> >> new adapter.
> >>
> >> I never used Presto myself so I may have missed some important points.
> Can
> >> you share some use-cases about who may use this adapter and how?
> >>
> >> Best,
> >> Stamatis
> >>
> >>
> >> On Wed, Jan 8, 2020 at 2:44 PM Forward Xu 
> wrote:
> >>
> >>> Hi everybody,
> >>>
> >>> I'd like to kick off a discussion on Implement Presto adapter in
> calcite.
> >>>
> >>> We know that presto is also a powerful SQL query tool, which is widely
> >> used
> >>> with the olap of big data. So I want to add an adapter for presto to
> >>> calite.
> >>>
> >>> Would love to hear your thoughts.
> >>>
> >>> Best,
> >>> Forward
> >>>
> >>
>
>

Re: [discuss] Implement Presto adapter

[Forward Xu]

Hi Julian,

In addition, we can also implement a PrestoSqlDialect.


Best,

Forward

Forward Xu  于2020年1月10日周五 下午2:23写道：

> Hi Julian,
>
> The current idea is through com.facebook.presto.spi.
> 1. Can be achieved through plugs provided by presto. For example [1].
> 2. You can construct Connector directly through
> com.facebook.presto.spi.ConnectorFactory [2].
>
> Best,
> Forward
>
> [1]
> https://www.codota.com/code/java/methods/com.facebook.presto.spi.Plugin/getConnectorFactories
> [2]
> https://www.programcreek.com/java-api-examples/?code=y-lan/presto/presto-master/presto-raptor/src/main/java/com/facebook/presto/raptor/RaptorPlugin.java
>
> Julian Hyde  于2020年1月10日周五 上午2:03写道：
>
>> Sounds like you are planning to target Presto’s internal engine rather
>> than Presto the database. To clarify, can you give an example of the code
>> that you would generate and send to Presto for a simple SQL query?
>>
>> Julian
>>
>>
>> > On Jan 9, 2020, at 5:07 AM, Forward Xu  wrote:
>> >
>> > Hi, Stamatis
>> >
>> > Because we have a number of internal clusters already using and
>> installing
>> > presto. And we want use calcite to optimize the parser and push it down
>> to
>> > presto, spark and other calculation engines or olaps. Utilize the sql
>> > optimization function of calite in combination with distributed
>> computing
>> > engines or distributed olaps such as presto. In this way, each
>> performance
>> > can be exerted.
>> >
>> > Best,
>> > Forward
>> >
>> > Stamatis Zampetakis  于2020年1月9日周四 下午8:51写道：
>> >
>> >> Hi,
>> >>
>> >> Presto seems to have many similarities with Calcite itself.
>> >> As you mentioned, Presto is not really a database but a distributed SQL
>> >> framework to query heterogeneous data sources.
>> >> Presto connectors are in some sense similar to Calcite adapters and if
>> the
>> >> Enumerable convention was capable of performing distributed joins etc.,
>> >> then the similarities would be even bigger.
>> >>
>> >> To the best of my knowledge, the majority of our users adopt Calcite
>> either
>> >> to query multiple data sources in a unified way or to provide SQL query
>> >> processing capabilities to data sources that do not already have.
>> Presto
>> >> seems to have similar goals, thus, I am not sure who will benefit from
>> this
>> >> new adapter.
>> >>
>> >> I never used Presto myself so I may have missed some important points.
>> Can
>> >> you share some use-cases about who may use this adapter and how?
>> >>
>> >> Best,
>> >> Stamatis
>> >>
>> >>
>> >> On Wed, Jan 8, 2020 at 2:44 PM Forward Xu 
>> wrote:
>> >>
>> >>> Hi everybody,
>> >>>
>> >>> I'd like to kick off a discussion on Implement Presto adapter in
>> calcite.
>> >>>
>> >>> We know that presto is also a powerful SQL query tool, which is widely
>> >> used
>> >>> with the olap of big data. So I want to add an adapter for presto to
>> >>> calite.
>> >>>
>> >>> Would love to hear your thoughts.
>> >>>
>> >>> Best,
>> >>> Forward
>> >>>
>> >>
>>
>>

Re: Contributors of 2019

[Forward Xu]

Hi Stamatis

Thank you very much.

Best,
Forward

Stamatis Zampetakis  于2020年1月12日周日 下午10:47写道：

> Hi all,
>
> Looking back into 2019, I wanted to take a moment and thank all these
> people who helped the project in one way or another. I know that stats do
> not always tell the truth but it is a concrete measure that shows the
> extraordinary effort of some people.
>
> Doing some post-processing on the commit logs (of Calcite & Avatica master)
> between Jan 2019 and Jan 2020, here what the numbers say:
>
> *Reviewers of the year*
> Committer mail COUNTA of Review SUM of Lines modified
> jh...@apache.org 60 20652
> h.y...@alibaba-inc.com 79 8531
> yuzhao@gmail.com 56 7004
> mailto...@126.com 8 3736
> zabe...@gmail.com 25 3465
> michael.m...@gmail.com 16 3431
> 25229979+asereda...@users.noreply.github.com 25 3353
> rube...@gmail.com 9 2293
> hon...@apache.org 13 2026
> jcama...@apache.org 13 1293
> sitnikov.vladi...@gmail.com 20 1049
> vvo...@gmail.com 2 899
> lauren...@users.noreply.github.com 6 867
> mm...@cs.rit.edu 4 675
> chun...@apache.org 10 674
> laur...@dremio.com 3 668
> els...@apache.org 3 610
> francischu...@apache.org 6 324
> kris...@apache.org 4 318
> k...@rxd.hu 2 101
> An vital part of every project is reviews. The people above have dedicated
> a big part of their time reviewing and merging work of others. Looking into
> our backlog of PRs this is an area that we are a bit behind and we should
> try to do better next year. Of course there are a few individuals (Julian,
> Haisheng, Danny) who took on their shoulders a big burden and whom we
> should try to imitate within our capacity.
>
> I would like also to mention that there are many people (committers & non
> committers) who are helping a lot the project by reviewing PRs in GitHub.
> Unfortunately, I didn't find an easy way to extract such stats from GitHub
> to include them above.
>
> *Committers of the year*
> Author mail SUM of Lines modified
> COUNTA of Commit
> sitnikov.vladi...@gmail.com 83780 97
> jh...@apache.org 36662 63
> yuzhao@gmail.com 22391 55
> kht...@linkedin.com 7273 1
> zabe...@gmail.com 7122 34
> hon...@apache.org 5691 20
> rube...@gmail.com 5475 30
> chun...@apache.org 4341 27
> x1q...@163.com 4210 15
> jinxing.co...@gmail.com 3227 24
> h.y...@alibaba-inc.com 3026 28
> 1989yanlinw...@163.com 2562 15
> j.feina...@pragmaticminds.de 2426 2
> wellfeng...@gmail.com 1932 16
> jcama...@apache.org 1711 12
> vg...@apache.org 1630 11
> ihor.huzenko@gmail.com 1545 5
> 25229979+asereda...@users.noreply.github.com 1388 21
> francischu...@apache.org 1355 35
> yanzhi@antfin.com 1298 14
> mouhoubi.kha...@gmail.com 1295 2
> 953396...@qq.com 1061 10
> shuo...@alibaba-inc.com 1051 4
> ming...@ebay.com 1047 1
> laur...@apache.org 1009 10
> k...@rxd.hu 989 15
> j...@dremio.com 909 5
> wuchong...@alibaba-inc.com 818 1
> vvo...@gmail.com 769 5
> riteshkapoor.opensou...@gmail.com 747 4
> hhlai1...@gmail.com 727 1
> siddha...@dremio.com 705 3
> absolute...@qq.com 685 1
> prav...@dremio.com 650 1
> wenhui_t...@yeah.net 646 8
> m.gelb...@gmail.com 577 3
> kris...@apache.org 558 13
> mm...@cs.rit.edu 545 17
> st...@stoty.hu 532 1
> ming.moria...@gmail.com 488 4
> chunwei...@163.com 406 2
> bohdan.kazy...@gmail.com 404 2
> cyazb...@murex.com 347 1
> xndai@live.com 345 9
> godfre...@163.com 321 3
> shlok7...@github.com 320 1
> ryan...@looker.com 320 3
> lame...@users.noreply.github.com 316 4
> chiboch...@gmail.com 299 1
> zuoz...@gmail.com 279 1
> yuzhao@alibaba-inc.com 246 1
> amaliu...@163.com 227 6
> wmy7...@gmail.com 201 3
> krishnakant.agra...@datametica.com 199 2
> 785294...@qq.com 193 1
> dam6...@gmail.com 156 2
> bot...@apache.org 154 2
> snuyan...@gmail.com 139 3
> nishant.mon...@gmail.com 129 1
> chenghe...@gmail.com 124 1
> weidong3...@126.com 120 2
> wuchienc...@qq.com 118 2
> b.hano...@criteo.com 116 1
> justin.sw...@looker.com 104 1
> whlwanghail...@didichuxing.com 103 1
> asi...@maprtech.com 96 1
> pengzhiwei2...@icloud.com 95 3
> ste...@looker.com 94 2
> stuti.gu...@datametica.com 92 1
> mel...@murex.com 90 1
> 1269223...@qq.com 90 2
> walter_...@hotmail.com 88 1
> justin.szel...@gmail.com 88 3
> hpe...@uber.com 88 1
> mail.mondal.s...@gmail.com 85 1
> doc...@gmail.com 79 2
> beyond1...@126.com 77 1
> 490081...@qq.com 76 2
> wellfeng...@tencent.com 73 1
> zhaindr...@hortonworks.com 72 1
> fudian...@alibaba-inc.com 62 1
> bake@alibaba-inc.com 59 1
> shikha.som...@servicesource.com 57 1
> laur...@dremio.com 53 1
> seancx...@gmail.com 48 2
> sahi...@spotify.com 47 1
> 9326...@gmail.com 37 1
> –fengfeng_...@163.com 32 1
> renhe@antfin.com 31 1
> els...@apache.org 31 6
> zhuan...@dragonsoft.com.cn 29 1
> divyansh...@datametica.com 29 1
> wuyunfen...@baidu.com 28 1
> hello...@gmail.com 28 1
> pressenna.sockalingas...@omnisci.com 26 1
> liusiyu...@360.cn 25 1
> li.x...@kyligence.io 25 1
> j...@apache.org 25 1
> 44845836+chad...@users.noreply.github.com 23 1
> zj48...@ly.com 19 1
> dijksp...@outlook.com 17 1
>

Re: Contributors of 2019

[Forward Xu]

Hi Stamatis

Thank you very much.   I will also participate in the review later.

Best,
Forward

Forward Xu  于2020年1月12日周日 下午10:59写道：

> Hi Stamatis
>
> Thank you very much.
>
> Best,
> Forward
>
> Stamatis Zampetakis  于2020年1月12日周日 下午10:47写道：
>
>> Hi all,
>>
>> Looking back into 2019, I wanted to take a moment and thank all these
>> people who helped the project in one way or another. I know that stats do
>> not always tell the truth but it is a concrete measure that shows the
>> extraordinary effort of some people.
>>
>> Doing some post-processing on the commit logs (of Calcite & Avatica
>> master)
>> between Jan 2019 and Jan 2020, here what the numbers say:
>>
>> *Reviewers of the year*
>> Committer mail COUNTA of Review SUM of Lines modified
>> jh...@apache.org 60 20652
>> h.y...@alibaba-inc.com 79 8531
>> yuzhao@gmail.com 56 7004
>> mailto...@126.com 8 3736
>> zabe...@gmail.com 25 3465
>> michael.m...@gmail.com 16 3431
>> 25229979+asereda...@users.noreply.github.com 25 3353
>> rube...@gmail.com 9 2293
>> hon...@apache.org 13 2026
>> jcama...@apache.org 13 1293
>> sitnikov.vladi...@gmail.com 20 1049
>> vvo...@gmail.com 2 899
>> lauren...@users.noreply.github.com 6 867
>> mm...@cs.rit.edu 4 675
>> chun...@apache.org 10 674
>> laur...@dremio.com 3 668
>> els...@apache.org 3 610
>> francischu...@apache.org 6 324
>> kris...@apache.org 4 318
>> k...@rxd.hu 2 101
>> An vital part of every project is reviews. The people above have dedicated
>> a big part of their time reviewing and merging work of others. Looking
>> into
>> our backlog of PRs this is an area that we are a bit behind and we should
>> try to do better next year. Of course there are a few individuals (Julian,
>> Haisheng, Danny) who took on their shoulders a big burden and whom we
>> should try to imitate within our capacity.
>>
>> I would like also to mention that there are many people (committers & non
>> committers) who are helping a lot the project by reviewing PRs in GitHub.
>> Unfortunately, I didn't find an easy way to extract such stats from GitHub
>> to include them above.
>>
>> *Committers of the year*
>> Author mail SUM of Lines modified
>> COUNTA of Commit
>> sitnikov.vladi...@gmail.com 83780 97
>> jh...@apache.org 36662 63
>> yuzhao@gmail.com 22391 55
>> kht...@linkedin.com 7273 1
>> zabe...@gmail.com 7122 34
>> hon...@apache.org 5691 20
>> rube...@gmail.com 5475 30
>> chun...@apache.org 4341 27
>> x1q...@163.com 4210 15
>> jinxing.co...@gmail.com 3227 24
>> h.y...@alibaba-inc.com 3026 28
>> 1989yanlinw...@163.com 2562 15
>> j.feina...@pragmaticminds.de 2426 2
>> wellfeng...@gmail.com 1932 16
>> jcama...@apache.org 1711 12
>> vg...@apache.org 1630 11
>> ihor.huzenko@gmail.com 1545 5
>> 25229979+asereda...@users.noreply.github.com 1388 21
>> francischu...@apache.org 1355 35
>> yanzhi@antfin.com 1298 14
>> mouhoubi.kha...@gmail.com 1295 2
>> 953396...@qq.com 1061 10
>> shuo...@alibaba-inc.com 1051 4
>> ming...@ebay.com 1047 1
>> laur...@apache.org 1009 10
>> k...@rxd.hu 989 15
>> j...@dremio.com 909 5
>> wuchong...@alibaba-inc.com 818 1
>> vvo...@gmail.com 769 5
>> riteshkapoor.opensou...@gmail.com 747 4
>> hhlai1...@gmail.com 727 1
>> siddha...@dremio.com 705 3
>> absolute...@qq.com 685 1
>> prav...@dremio.com 650 1
>> wenhui_t...@yeah.net 646 8
>> m.gelb...@gmail.com 577 3
>> kris...@apache.org 558 13
>> mm...@cs.rit.edu 545 17
>> st...@stoty.hu 532 1
>> ming.moria...@gmail.com 488 4
>> chunwei...@163.com 406 2
>> bohdan.kazy...@gmail.com 404 2
>> cyazb...@murex.com 347 1
>> xndai@live.com 345 9
>> godfre...@163.com 321 3
>> shlok7...@github.com 320 1
>> ryan...@looker.com 320 3
>> lame...@users.noreply.github.com 316 4
>> chiboch...@gmail.com 299 1
>> zuoz...@gmail.com 279 1
>> yuzhao@alibaba-inc.com 246 1
>> amaliu...@163.com 227 6
>> wmy7...@gmail.com 201 3
>> krishnakant.agra...@datametica.com 199 2
>> 785294...@qq.com 193 1
>> dam6...@gmail.com 156 2
>> bot...@apache.org 154 2
>> snuyan...@gmail.com 139 3
>> nishant.mon...@gmail.com 129 1
>> chenghe...@gmail.com 124 1
>> weidong3...@126.com 120 2
>> wuchienc...@qq.com 118 2
>> b.hano...@criteo.com 116 1
>> justin.sw...@looker.com 104 1
>> whlwanghail...@didichuxing.com 103 1
>> asi...@maprtech.com 96 1
>> pengzhiwei2...@icloud

Re: [DISCUSS] Towards Calcite 1.22.0

[Forward Xu]

Thanks, Danny.

+1


Best,

Forwardxu

Chunwei Lei  于2020年2月21日周五 上午8:51写道：

> Thanks, Danny.
>
> +1 to Danny’s timetable (RC on 2/24, release by 2/29).
>
>
> Best,
> Chunwei
>
>
> On Fri, Feb 21, 2020 at 2:37 AM Julian Hyde  wrote:
>
> > +1 to Danny’s timetable (RC on 2/24, release by 2/29).
> >
> > We can have a quick 1.23 (say 1 month after 1.22) if there are important
> > PRs that we don’t have time for this release. Of course we still have to
> > find reviewers for those PRs.
> >
> > Julian
> >
> >
> > > On Feb 19, 2020, at 11:18 PM, Danny Chan  wrote:
> > >
> > > Hi all,
> > >
> > > Approximately 5 months have passed from the previous release
> > > (Calcite 1.21.0) and I was thinking that it is the time to have the
> next
> > > release by the end of February. To do this I think we should try to
> have
> > an
> > > RC around the 24 of February(the next Monday).
> > >
> > > The progress towards the next release can be seen in [1].
> > > I have made some pre-edit of the issues, we do not have many issues
> > marked
> > > for fixing in version 1.22.0.
> > >
> > > Because it is a long-overdue, I would suggest we only concentrate on
> > > solving these issues in the following days,
> > > I’m not saying we are ignoring the existing PRs, but we did need a
> > release
> > > now that should happen 2 months ago.
> > >
> > > If you really think the issue/PR should be into release-1.22.0, please
> > > comment in this thread. I’m planning to pull the RC0 on
> > > 24 of February, thanks ~
> > >
> > > Don't hesitate to reply to this thread if the plan above is not
> > convenient
> > > for you!
> > >
> > > [1]
> > >
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12333950
> > >
> > > Best,
> > > Danny Chan
> >
> >
>

Re: [ANNOUNCE] New committer: Feng Zhu

[Forward Xu]

Congratulations！


Best，

Forward

Julian Hyde  于2020年3月1日周日 上午7:33写道：

> PS This is the “powered by” page:
> https://calcite.apache.org/docs/powered_by.html <
> https://calcite.apache.org/docs/powered_by.html>
>
> > On Feb 29, 2020, at 3:32 PM, Julian Hyde  wrote:
> >
> > Congratulations, thanks, and welcome!
> >
> > Feng, I wasn’t aware of Tencent’s SuperSQL project. It would be a great
> addition to the “powered by” page. If you would like to add it, please
> create a PR to add a paragraph to that page (no need for a JIRA case), and
> provide a link to a logo for Tencent that I can add to the picture.
> >
> > Julian
> >
> >
> >
> >> On Feb 29, 2020, at 3:15 PM, Rui Wang  wrote:
> >>
> >> Congratulations Feng! Well deserved!
> >>
> >>
> >>
> >> -Rui
> >>
> >> On Sat, Feb 29, 2020 at 2:10 AM Feng Zhu  wrote:
> >>
> >>> Thank you everyone for your warm welcome!
> >>> Currently I am working at SuperSQL team of Tencent in Shenzhen,
> Guangdong,
> >>> China.
> >>> The project relies heavily on Calcite to analyze data residing in
> thousands
> >>> of heterogeneous data sources.
> >>>
> >>> It is my greate honor to be part of the community. Calcite is an
> excellent
> >>> and promising project!
> >>> I will do my best to contribute to the project.
> >>>
> >>> Best regards,
> >>> Feng
> >>>
> >>> Chunwei Lei  于2020年2月29日周六 下午4:46写道：
> >>>
>  Congratulations！
> 
> 
>  Best,
>  Chunwei
> 
> 
>  On Sat, Feb 29, 2020 at 4:41 PM Danny Chan 
> wrote:
> 
> > Congratulations!
> >
> > Francis Chuang 于2020年2月29日 周六下午4:35写道：
> >
> >> Congrats, well-deserved!
> >>
> >> On 29/02/2020 6:26 pm, Stamatis Zampetakis wrote:
> >>> Apache Calcite's Project Management Committee (PMC) has invited
> >>> Feng
> >>> Zhu to become a committer, and we are pleased to announce that he
> >>> has accepted.
> >>>
> >>> Feng is an active contributor and has contributed a lot of code to
>  some
> >>> fairly complex parts of Calcite. He has been very helpful in
> > discussions
> >>> and reviews, especially around code generation, and his work is
>  always
> >>> of high quality.
> >>>
> >>> Feng, welcome, thank you for your contributions, and we look
> >>> forward
> >>> your further interactions with the community! If you wish, please
>  feel
> >>> free to tell us more about yourself and what you are working on.
> >>>
> >>> Stamatis (on behalf of the Apache Calcite PMC)
> >>>
> >>
> >
> 
> >>>
> >
>
>

[discuss] Add the default mode to the path in the Json functions.

[Forward Xu]

hi everyone, I recently found some discussable optimizations when I
contributed the flip-90 [1] Json functions:
The current json functions in calcite: JSON_EXISTS, JSON_VALUE, JSON_QUERY,
JSON_OBJECT, JSON_OBJECTAGG, JSON_ARRAY, JSON_ARRAYAGG and IS JSON
predication functions. These functions are implemented based on the SQL
2016-2017 standard [2]. According to this standard, the path of the json
function must be used in one of strict or lax mode. such as:
json_exists ('{"foo": "bar"}', 'lax $ .foo') or
json_exists ('{"foo": "bar"}', 'strict $ .foo')
Can we give a default mode to simplify the use of lax and strict. For
example, we default to lax mode. In this way, the use of our json function
can be simplified to:
json_exists ('{"foo": "bar"}', '$ .foo')
Implementation idea improvement JsonFunctions jsonApiCommonSyntax path
judgment to increase the default lax mode logic.
Of course, these changes are not described in SQL2016-2017.
I want to hear your opinion here.

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=141724550
[2]
https://standards.iso.org/ittf/PubliclyAvailableStandards/c067367_ISO_IEC_TR_19075-6_2017.zip

Re: [discuss] Add the default mode to the path in the Json functions.

[Forward Xu]

Hi Julian,
Thank you for bringing such a good shared link, there are some areas in the
JSON function that need to be improved and implemented. I will continue to
improve it.

Forward

Julian Hyde  于2020年4月21日周二 上午1:28写道：

> Speaking of JSON functions, JOOQ creator Lukas Eder has been giving JSON
> functions in MySQL/MariaDB a good workout over the last few days. It’s
> amusing to read what he has discovered: https://twitter.com/lukaseder <
> https://twitter.com/lukaseder>
>
> Julian
>
>
> > On Apr 20, 2020, at 8:27 AM, Stamatis Zampetakis 
> wrote:
> >
> > Hi,
> >
> > I know that Oracle uses lax by default [1] but I don't remember what
> other
> > DBMS do.
> >
> > In any case adopting a default mode sounds like a reasonable thing to do.
> >
> > Best,
> > Stamatis
> >
> > [1] https://docs.oracle.com/database/121/ADXDB/json.htm#ADXDB6259
> >
> >
> > On Mon, Apr 20, 2020, 7:28 AM Chunwei Lei 
> wrote:
> >
> >> Thank you for proposing this, Forward.
> >>
> >> I am wondering whether it is really useful for users who want to use
> these
> >> functions
> >> since they probably know what the syntax is. But personally I would
> like to
> >> have the
> >> default mode since it is more convenient.
> >>
> >>
> >> Best,
> >> Chunwei
> >>
> >>
> >> On Sun, Apr 19, 2020 at 9:51 AM Forward Xu 
> wrote:
> >>
> >>> hi everyone, I recently found some discussable optimizations when I
> >>> contributed the flip-90 [1] Json functions:
> >>> The current json functions in calcite: JSON_EXISTS, JSON_VALUE,
> >> JSON_QUERY,
> >>> JSON_OBJECT, JSON_OBJECTAGG, JSON_ARRAY, JSON_ARRAYAGG and IS JSON
> >>> predication functions. These functions are implemented based on the SQL
> >>> 2016-2017 standard [2]. According to this standard, the path of the
> json
> >>> function must be used in one of strict or lax mode. such as:
> >>> json_exists ('{"foo": "bar"}', 'lax $ .foo') or
> >>> json_exists ('{"foo": "bar"}', 'strict $ .foo')
> >>> Can we give a default mode to simplify the use of lax and strict. For
> >>> example, we default to lax mode. In this way, the use of our json
> >> function
> >>> can be simplified to:
> >>> json_exists ('{"foo": "bar"}', '$ .foo')
> >>> Implementation idea improvement JsonFunctions jsonApiCommonSyntax path
> >>> judgment to increase the default lax mode logic.
> >>> Of course, these changes are not described in SQL2016-2017.
> >>> I want to hear your opinion here.
> >>>
> >>> [1]
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=141724550
> >>> [2]
> >>>
> >>>
> >>
> https://standards.iso.org/ittf/PubliclyAvailableStandards/c067367_ISO_IEC_TR_19075-6_2017.zip
> >>>
> >>
>
>

Re: [ANNOUNCE] New committer: Vineet Garg

[Forward Xu]

Congratulations

best,
Forward

Francis Chuang  于2020年4月26日周日 上午6:04写道：

> Congrats, Vineet!
>
> On 26/04/2020 7:52 am, Stamatis Zampetakis wrote:
> > Apache Calcite's Project Management Committee (PMC) has invited Vineet
> > Garg to become a committer, and we are pleased to announce that he
> > has accepted.
> >
> > With the first code contribution in Calcite back in 2017, Vineet is
> > definitely
> > not new to the project. Since then he has contributed many patches,
> > fixing and improving various modules of Calcite, notably things around
> > subqueries.
> >
> > Vineet, welcome, thank you for your contributions, and we look forward
> > your further interactions with the community! If you wish, please feel
> > free to tell us more about yourself and what you are working on.
> >
> > Stamatis (on behalf of the Apache Calcite PMC)
> >
>

Re: [ANNOUNCE] New committer: Vineet Garg

[Leonard Xu]

Congratulations, Vineet!

Best,
Leonard Xu
> 在 2020年4月26日，18:07，xu  写道：
> 
> Congrats, Vineet!
> 
> Danny Chan  于2020年4月26日周日 下午4:52写道：
> 
>> Congrats, Vineet!
>> 
>> Best,
>> Danny Chan
>> 在 2020年4月26日 +0800 PM1:55，dev@calcite.apache.org，写道：
>>> 
>>> Congrats, Vineet!
>> 
> 
> 
> -- 
> 
> Best regards,
> 
> Xu

Re: [ANNOUNCE] New committer: Wang Yanlin

[Forward Xu]

Congrats


Best,

Forward

953396112 <953396...@qq.com> 于2020年4月29日周三 上午8:26写道：

> Congrats, Wang Yanlin!
>
>
>
>
> ---Original---
> From: "Stamatis Zampetakis" Date: Wed, Apr 29, 2020 05:51 AM
> To: "dev" Subject: [ANNOUNCE] New committer: Wang Yanlin
>
>
> Apache Calcite's Project Management Committee (PMC) has invited Wang Yanlin
> to
> become a committer, and we are pleased to announce that he has accepted.
>
> Wang has pushed numerous fixes and improvements to the project, landing in
> total
> the impressive number of 30 commits to the master. Among other things, he
> contributed some important features in the Interpreter.
>
> Wang, welcome, thank you for your contributions, and we look forward your
> further interactions with the community! If you wish, please feel free to
> tell
> us more about yourself and what you are working on.
>
> Stamatis (on behalf of the Apache Calcite PMC)

Re: [ANNOUNCE] New committer: Jin Xing

[Forward Xu]

Congrats


best,

Forward

953396112 <953396...@qq.com> 于2020年4月29日周三 上午8:21写道：

> Congrats, Jin Xing!
>
>
> ---Original---
> From: "Stamatis Zampetakis" Date: Wed, Apr 29, 2020 05:47 AM
> To: "dev" Subject: [ANNOUNCE] New committer: Jin Xing
>
>
> Apache Calcite's Project Management Committee (PMC) has invited Jin Xing to
> become a committer, and we are pleased to announce that he has accepted.
>
> Jin has contributed a lot of code in the project and many
> recent improvements in
> materialized view matching have his signature on them. Apart from code
> contributions, Jin provides valuable help to the community by doing reviews
> and
> answering questions in the devlist.
>
> Jin, welcome, thank you for your contributions, and we look forward to your
> further interactions with the community! If you wish, please feel free to
> tell
> us more about yourself and what you are working on.
>
> Stamatis (on behalf of the Apache Calcite PMC)

Re: [ANNOUNCE] New committer: Forward Xu

[Forward Xu]

Thank you everyone for your warm welcome!
I'm working in the TBDS team of Tencent in Shenzhen. TBDS (Tencent Big Data
Suite) is similar to Alibaba's EMR, TBDS is a big data ecosystem. I am
responsible for Oceanus(flink streaming jobs) and Tdbank (Tencent real-time
data collection system). I‘m very happy to become calcite committer and
looking forward to make more contributions.

Best,
Forward

Zoltan Haindrich  于2020年4月29日周三 下午1:58写道：

> Congratulations!
>
> On 4/29/20 7:31 AM, Enrico Olivelli wrote:
> > Congrats!
> >
> > Enrico
> >
> > Il Mer 29 Apr 2020, 04:52 Feng Zhu  ha scritto:
> >
> >>   Congrations! Forward!
> >>
> >> best,
> >> Feng
> >>
> >> Chunwei Lei  于2020年4月29日周三 上午10:17写道：
> >>
> >>> Congrats, Forward!
> >>>
> >>>
> >>>
> >>> Best,
> >>> Chunwei
> >>>
> >>>
> >>> On Wed, Apr 29, 2020 at 6:46 AM Rui Wang  wrote:
> >>>
> >>>> Congrats!
> >>>>
> >>>>
> >>>> -Rui
> >>>>
> >>>> On Tue, Apr 28, 2020 at 3:04 PM Francis Chuang <
> >> francischu...@apache.org
> >>>>
> >>>> wrote:
> >>>>
> >>>>> Congrats, Forward!
> >>>>>
> >>>>> Francis
> >>>>>
> >>>>> On 29/04/2020 7:53 am, Stamatis Zampetakis wrote:
> >>>>>> Apache Calcite's Project Management Committee (PMC) has invited
> >>> Forward
> >>>>> Xu
> >>>>>> to
> >>>>>> become a committer, and we are pleased to announce that he has
> >>>> accepted.
> >>>>>>
> >>>>>> Forward has been helping the project for some time now. He added
> >> many
> >>>> new
> >>>>>> SQL
> >>>>>> functions to the project and is one of our JSON experts. On top of
> >>>> that,
> >>>>> and
> >>>>>> other fixes, he is the one who added the Redis adapter to the
> >>> project.
> >>>>>>
> >>>>>> Forward, welcome, thank you for your contributions, and we look
> >>> forward
> >>>>> to
> >>>>>> your
> >>>>>> further interactions with the community! If you wish, please feel
> >>> free
> >>>> to
> >>>>>> tell
> >>>>>> us more about yourself and what you are working on.
> >>>>>>
> >>>>>> Stamatis (on behalf of the Apache Calcite PMC)
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>

[discuss] Implement Elasticsearch jdbc driver

[Forward Xu]

Hi everybody,

I'd like to kick off a discussion on Implement Elasticsearch jdbc driver in
calcite.

Calcite has already implemented the elasticsearch adapter. I want to add
standard jdbc driver support based on the adapter. The advantage is that it
can be more easily used for visualization through the jdbc driver-based bi
system. It can also enhance the DDL and DML of the current elasticsearch
adapter. Implement es jdbc driver similar to aws open source [1]

Would love to hear your thoughts.

Best,
Forward

[1]https://github.com/opendistro-for-elasticsearch/sql

Re: [VOTE] Release apache-calcite-1.23.0 (release candidate 1)

[Forward Xu]

Thanks, Haisheng!


Environment: Mac-OS Mojave (10.14.6)

JDK version: 1.8.0_211


Release notes: OK

Build with tests: OK

Checksums and signatures: OK


Vote:

+1 (non-binding)


Best,

Forward

XING JIN  于2020年5月19日周二 上午10:53写道：

> Thanks, Haisheng ~
>
> Local Calcite build with tests enabled on Linux: *OK*
> Calcite-based system test suite: *OK*
>
> Vote:
> +1 (non-binding)
>
> Francis Chuang  于2020年5月18日周一 下午2:00写道：
>
> > Thanks for making this release available for voting, Haisheng!
> >
> > Verified GPG signature - OK
> > Verified SHA512 - OK
> > Ran tests per HOWTO (./gradlew check) - OK
> > Quickly skimmed release notes - OK
> > Spotted checked a few JARs in the Maven repository - OK
> >
> > Environment (OpenJDK:latest docker container):
> > Gradle 6.3 (via gradlew)
> > Debian GNU/Linux 8
> > openjdk version "1.8.0_111"
> > OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2~bpo8+1-b14)
> > OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
> >
> > My vote is: +1 (binding)
> >
> > Francis
> >
> > On 16/05/2020 2:02 pm, Haisheng Yuan wrote:
> > > Hi all,
> > >
> > > I have created a build for Apache Calcite 1.23.0, release
> > > candidate 1.
> > >
> > > Thanks to everyone who has contributed to this release.
> > >
> > > You can read the release notes here:
> > >
> >
> https://github.com/apache/calcite/blob/calcite-1.23.0-rc1/site/_docs/history.md
> > >
> > > The commit to be voted upon:
> > >
> >
> https://gitbox.apache.org/repos/asf?p=calcite.git;a=commit;h=b708fdc46d4c5fd4c5a6c7a398823318a7b4dce3
> > >
> > > Its hash is b708fdc46d4c5fd4c5a6c7a398823318a7b4dce3
> > >
> > > Tag:
> > > https://github.com/apache/calcite/tree/calcite-1.23.0-rc1
> > >
> > > The artifacts to be voted on are located here:
> > >
> https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.23.0-rc1
> > > (revision 39622)
> > >
> > > The hashes of the artifacts are as follows:
> > >
> >
> 961c4f13199e199c669a6168ba655a9492bdd80d644da375a684b732c0b628b8a2ffacea5da97c82e8702a8e3bf7a1f58784baa49509fb3c48ef593259e11f46
> > > *apache-calcite-1.23.0-src.tar.gz
> > >
> > > A staged Maven repository is available for review at:
> > >
> >
> https://repository.apache.org/content/repositories/orgapachecalcite-1089/org/apache/calcite/
> > >
> > > Release artifacts are signed with the following key:
> > > https://dist.apache.org/repos/dist/release/calcite/KEYS
> > >
> > > N.B.
> > > To create the jars and test Apache Calcite: "./gradlew build".
> > >
> > > If you do not have a Java environment available, you can run the tests
> > > using docker. To do so, install docker and docker-compose, then run
> > > "docker-compose run test" from the root of the directory.
> > >
> > > Please vote on releasing this package as Apache Calcite 1.23.0.
> > >
> > > The vote is open for the next 72 hours and passes if a majority of at
> > > least three +1 PMC votes are cast.
> > >
> > > [ ] +1 Release this package as Apache Calcite 1.23.0
> > > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> > > [ ] -1 Do not release this package because...
> > >
> > >
> > > Here is my vote:
> > >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Haisheng Yuan
> > >
> >
>

Re: [ANNOUNCE] Apache Calcite 1.23.0 released

[Forward Xu]

Nice job, Haisheng! Thanks.


Best,

ForwardXu

Enrico Olivelli  于2020年5月28日周四 下午1:48写道：

> Great
>
>
> Enrico
>
> Il Gio 28 Mag 2020, 05:12 Chunwei Lei  ha scritto:
>
> > Nice job, Haisheng!
> >
> >
> > Best,
> > Chunwei
> >
> >
> > On Thu, May 28, 2020 at 10:42 AM XING JIN 
> wrote:
> >
> > > Thanks a lot for driving this, Haisheng!
> > >
> > > Best,
> > > Jin
> > >
> > > Stamatis Zampetakis  于2020年5月28日周四 上午7:14写道：
> > >
> > > > Many thanks again Haisheng, the release has your signature on it and
> it
> > > is
> > > > in many places :)
> > > >
> > > > On Wed, May 27, 2020 at 9:34 PM Haisheng Yuan 
> > wrote:
> > > >
> > > > > The Apache Calcite team is pleased to announce the release of
> Apache
> > > > > Calcite 1.23.0.
> > > > >
> > > > > Calcite is a dynamic data management framework. Its cost-based
> > > > > optimizer converts queries, represented in relational algebra, into
> > > > > executable plans. Calcite supports many front-end languages and
> > > > > back-end data engines, and includes an SQL parser and, as a
> > > > > sub-project, the Avatica JDBC driver.
> > > > >
> > > > > This release comes two months after 1.22.0. It includes more than
> > > > > 100 resolved issues, comprising of a few new features as well as
> > > > > general improvements and bug-fixes. It includes support for top
> > > > > down trait request and trait enforcement without abstract
> converter,
> > > > > ClickHouse dialect, SESSION and HOP Table functions, and many
> > > > > more bug fixes and improvements.
> > > > >
> > > > > You can start using it in Maven by simply updating your dependency
> > to:
> > > > >
> > > > >   
> > > > > org.apache.calcite
> > > > > calcite-core
> > > > > 1.23.0
> > > > >   
> > > > >
> > > > > If you'd like to download the source release, you can find it here:
> > > > >
> > > > >   https://calcite.apache.org/downloads/
> > > > >
> > > > > You can read more about the release (including release notes) here:
> > > > >
> > > > >   https://calcite.apache.org/news/2020/05/23/release-1.23.0/
> > > > >
> > > > > We welcome your help and feedback. For more information on how to
> > > > > report problems, and to get involved, visit the project website at:
> > > > >
> > > > >   https://calcite.apache.org/
> > > > >
> > > > > Thanks to everyone involved!
> > > > >
> > > >
> > >
> >
>

Re: Draft board report for July 2020

[Forward Xu]

Looks good!

Thanks Stamatis！


Best,

Forward



Chunwei Lei  于2020年7月3日周五 上午10:47写道：

> Looks great! Thanks Stamatis~~
>
> Best,
> Chunwei
>
>
> On Fri, Jul 3, 2020 at 9:08 AM Julian Hyde  wrote:
>
> > Looks good. Thanks!
> >
> > I regard the report as for the quarter (April 1st - June 30th), so the
> > Avatica-go release should rightly go into the Summer report.
> >
> > Julian
> >
> > > On Jul 2, 2020, at 4:31 AM, Stamatis Zampetakis 
> > wrote:
> > >
> > > Attached below is a draft of this month's board report. I plan to
> > submit it
> > > on July 7. Please let me know if you have any additions or corrections.
> > >
> > > ## Description:
> > > Apache Calcite is a highly customizable framework for parsing and
> > planning
> > > queries on data in a wide variety of formats. It allows database-like
> > > access,
> > > and in particular a SQL interface and advanced query optimization, for
> > data
> > > not
> > > residing in a traditional database.
> > >
> > > Avatica is a sub-project within Calcite and provides a framework for
> > > building
> > > local and remote JDBC and ODBC database drivers. Avatica has an
> > independent
> > > release schedule and its own repository.
> > >
> > > ## Issues:
> > > There are no issues requiring board attention.
> > >
> > > ## Membership Data:
> > > Apache Calcite was founded 2015-10-22 (4 years ago)
> > > There are currently 50 committers and 22 PMC members in this project.
> > > The Committer-to-PMC ratio is roughly 7:3.
> > >
> > > Community changes, past quarter:
> > > - No new PMC members. Last addition was Haisheng Yuan on 2019-11-11.
> > > - Forward Xu was added as committer on 2020-04-28
> > > - Xing Jin was added as committer on 2020-04-28
> > > - Vineet Garg was added as committer on 2020-04-24
> > > - Yanlin Wang was added as committer on 2020-04-28
> > >
> > > ## Project Activity:
> > > Avatica 1.17.0 was released on 2020-06-22. It is a small routine
> release
> > > that includes a
> > > few dependency updates and bug fixes.
> > >
> > > [Include Avatica Go if it is released on time]
> > > Avatica Go 5.0.0 was released on 2020-07-XX. It is a major release of
> > > Avatica Go with a number of improvements and a breaking change
> affecting
> > > connection metadata. Worth mentioning the support for batching query
> > string
> > > parameters in the DSN allowing updates to the server to be executed
> once
> > > Close() is called on the prepared statement.
> > >
> > > Calcite 1.23.0 was released on 2020-05-23, including more than 100
> > resolved
> > > issues. It includes many new futures such as an SQL dialect for
> > ClickHouse
> > > and SESSION/HOP table functions (particularly useful for queries on
> > > streams), along with significant improvements to the optimizer reducing
> > > optimization latency up to a factor of 50x for some complex queries.
> > >
> > > [Do we have any talks or new adopters to put in the report?]
> > >
> > > ## Community Health:
> > >
> > > The overall activity on the mailing lists has increased (-7% dev@,
> +27%
> > > issues@) with many interesting topics some very important for the
> > future of
> > > the project such as the redesign of the optimizer, modelling indexes,
> > and a
> > > big refactoring of the ruleset.
> > >
> > > Regarding opened/resolved issues the things are rather balanced. We
> had a
> > > slight decrease in new issues (-6% JIRA, -13% GitHub) counterbalanced
> by
> > a
> > > small increase in closed issues (+14% JIRA, +4% GitHub). Since we have
> a
> > > large
> > > backlog of PRs, the increased ratio of closed PRs/issues is a good
> sign.
> > >
> > > Although the number of commits has slightly decreased in the past
> quarter
> > > (-16%) we had more code contributors (+17%) among them some new faces
> > which
> > > is
> > > very promising for the future of the project.
> >
>

Re: [DISCUSS] Binary files for testing InnoDB adapter

[Forward Xu]

+1


If we can add the function(Example: innodb-java-writer) of generating ibd
files in the next version, I think it will be better. In this way, InnoDB
adapter initially has the capability of a database, and can further improve
the current unit test.


best

Forward

Stamatis Zampetakis  于2020年7月17日周五 上午6:25写道：

> Hello,
>
> It's good to avoid non-readable files when possible but I don't think we
> have to worry too much about it.
>
> The ASF foundation does not require all files in a release to be
> protected by copyright law and lists a few exceptions [1].
>
> The file is a database dump so I don't see much creativity involved so it
> could be classified under the following case.
>
> "A file without any degree of creativity in either its literal elements or
> its structure is not protected by copyright law; therefore, such a file
> does not require a license header."
>
> Based on my understanding of the policy, I don't see a problem leaving this
> file as is even after 1.24 but if there are doubts I can ping legal@.
>
> Best,
> Stamatis
>
> [1] https://www.apache.org/legal/src-headers.html#faq-exceptions
>
>
> On Thu, Jul 16, 2020 at 10:54 PM Enrico Olivelli 
> wrote:
>
> > Il Gio 16 Lug 2020, 21:28 Julian Hyde  ha scritto:
> >
> > > Michael,
> > >
> > > I think that sentence covers the situation where, say, we ship a
> > > .class file without shipping the .java it was generated from.
> > >
> > > In our case, we are shipping the equivalent of the .class file and the
> > > .java file (namely EMP.ibd and the scott.sql that MySQL generated it
> > > from). We have to ship the .class file equivalent (.ibd) because we
> > > don't require that people running the tests have the javac equivalent
> > > (MySQL).
> > >
> > > The concern about auditability remains.
> > >
> >
> > You can wrap the files in a text based format with a readable license
> > header and the test case can unpack the files and use them. It is tricky
> > but it can work
> >
> >
> > Enrico
> >
> >
> > > Julian
> > >
> > > On Thu, Jul 16, 2020 at 8:34 AM Michael Mior  wrote:
> > > >
> > > > I don't personally have a problem with this, but it seems as though
> it
> > > > might violate the release policy. Specifically the statement "It is
> > > > also necessary for the PMC to ensure that the source package is
> > > > sufficient to build any binary artifacts associated with the
> release."
> > > >
> > > > https://www.apache.org/legal/release-policy.html#what
> > > >
> > > > --
> > > > Michael Mior
> > > > mm...@apache.org
> > > >
> > > > Le mer. 15 juil. 2020 à 20:02, Julian Hyde  a
> écrit
> > :
> > > > >
> > > > > TL;DR: PMC members, would you vote for a release 1.24 if it
> includes
> > > > > binary files necessary for testing?
> > > > >
> > > > > I would like to include the InnoDB adapter [1] in release 1.24. It
> is
> > > > > well written, well documented, and it is ready.
> > > > >
> > > > > There is one problem: there are some binary files (in InnoDB
> format)
> > > > > [2] that are included for testing. As a general rule, Apache does
> not
> > > > > release binary files as part of the source release because they are
> > > > > difficult to audit for provenance.
> > > > >
> > > > > I think we should make an exception, for just release 1.24, because
> > > > > the files are small (just EMP and DEPT tables) and generated by
> hand.
> > > > >
> > > > > I have asked the author to do a follow-up task to remove the files
> > > > > before next release.
> > > > >
> > > > > PMC members, please reply to this email and indicate whether this
> > > > > would cause you to vote -1 on the upcoming release.
> > > > >
> > > > > Julian
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/CALCITE-4034
> > > > >
> > > > > [2]
> > >
> >
> https://github.com/apache/calcite/tree/b5e1622e7a43a3468a880c374f9161eee3ffa1ea/innodb/src/test/resources/data
> > >
> >
>

Re: [ANNOUNCE] Apache Calcite 1.24.0 released

[Forward Xu]

Thanks! Chunwei! Good job.

The link address of the Presto dialect implementation in
https://calcite.apache.org/news/2020/07/24/release-1.24.0/ seems to be
wrong.

Best,
Forward

Chunwei Lei  于2020年7月26日周日 上午10:30写道：

> > I assume that master is open for commits?
> Yes. The master is open for commits now.
>
>
> Best,
> Chunwei
>
>
> On Sun, Jul 26, 2020 at 5:43 AM Julian Hyde 
> wrote:
>
> > Thanks for being release manager, Chunwei! Great job.
> >
> > I assume that master is open for commits? I have quite a few stacked up:
> > 3923, 4079, 4134, 4128, 4080, 2569, 2160.
> >
> > I’d like Ruben to review 3923/4079 (class loader deadlock), Rui & Michael
> > to review 2569/2160 (rework table functions, and spatial table functions)
> > and would like volunteers to review 4134 (interval expressions), 4080
> > (character literals as column aliases), 4128 (refactor CSV/file
> adapters).
> >
> > Julian
> >
> >
> > > On Jul 25, 2020, at 9:19 AM, Stamatis Zampetakis 
> > wrote:
> > >
> > > Thanks Chunwei for leading this and of course all of our contributors
> for
> > > their high quality work.
> > >
> > > Best,
> > > Stamatis
> > >
> > > On Sat, Jul 25, 2020, 10:27 AM Chunwei Lei 
> > wrote:
> > >
> > >> The Apache Calcite team is pleased to announce the release of Apache
> > >> Calcite 1.24.0.
> > >>
> > >> Calcite is a dynamic data management framework. Its cost-based
> > >> optimizer converts queries, represented in relational algebra, into
> > >> executable plans. Calcite supports many front-end languages and
> > >> back-end data engines, and includes an SQL parser and, as a
> > >> sub-project, the Avatica JDBC driver.
> > >>
> > >> This release comes two months after 1.23.0. It includes more than
> > >> 80 resolved issues, comprising of a few new features as well as
> > >> general improvements and bug-fixes. It includes top-down rule
> > >> apply and upper bound space pruning, Presto dialect and many
> > >> more bug fixes and improvements.
> > >>
> > >> You can start using it in Maven by simply updating your dependency to:
> > >>
> > >>  
> > >>org.apache.calcite
> > >>calcite-core
> > >>1.24.0
> > >>  
> > >>
> > >> If you'd like to download the source release, you can find it here:
> > >>
> > >>  https://calcite.apache.org/downloads/
> > >>
> > >> You can read more about the release (including release notes) here:
> > >>
> > >>  https://calcite.apache.org/news/2020/07/24/release-1.24.0/
> > >>
> > >> We welcome your help and feedback. For more information on how to
> > >> report problems, and to get involved, visit the project website at:
> > >>
> > >>  https://calcite.apache.org/
> > >>
> > >> Thanks to everyone involved!
> > >>
> > >>
> > >> Best,
> > >> Chunwei
> > >>
> >
> >
>

Re: [ANNOUNCE] Ruben Quesada Lopez joins Calcite PMC

[Forward Xu]

Congrats, Ruben!


Best

Forward

XING JIN  于2020年8月12日周三 上午8:58写道：

> Congrats, Ruben!
>
> 953396112 <953396...@qq.com> 于2020年8月12日周三 上午7:47写道：
>
> > Congratulations, Ruben!
> >
> >
> > xzh
> > -- 原始邮件 --
> > 发件人:
> >   "dev"
> > <
> > zabe...@gmail.com>;
> > 发送时间: 2020年8月12日(星期三) 凌晨5:53
> > 收件人: "dev" >
> > 主题: [ANNOUNCE] Ruben Quesada Lopez joins Calcite PMC
> >
> >
> >
> > I'm pleased to announce that Ruben has accepted an invitation to
> > join the Calcite PMC. Ruben has been a consistent and helpful
> > figure in the Calcite community for which we are very grateful. We
> > look forward to the continued contributions and support.
> >
> > Please join me in congratulating Ruben!
> >
> > - Stamatis (on behalf of the Calcite PMC)
>

Re: [ANNOUNCE] New committer: Rui Wang

[Forward Xu]

Congratulations Rui!


Best,

Forward

Fan Liya  于2020年9月10日周四 上午10:07写道：

> Congratulations, Rui!
>
> Best,
> Liya Fan
>
> On Thu, Sep 10, 2020 at 9:45 AM Danny Chan  wrote:
>
> > Congrats Rui!
> >
> > Best,
> > Danny Chan
> >
> >
> >
> > > 在 2020年9月10日，上午6:28，Francis Chuang  写道：
> > >
> > > Congrats Rui!
> >
> >
>

Re: [DISCUSS] State of the project 2020

[Forward Xu]

; Thanks for your great work!
> > > >
> > > >
> > > > Calcite is very good at semantic transformation of relational
> algebra.
> > In
> > > > the process of the project, users can optimize relational algebra by
> > > > implementing RelOptRule or RelShuttle and so on, so as to bring
> > > relational
> > > > algebra into physical query engine. Another point is the construction
> > of
> > > > materialized view recognition framework, which realizes the ability
> of
> > > > materialized view recognition of common relational algebra. The
> > > expression
> > > > of relational algebra is very flexible, and it can do better in
> > > identifying
> > > > various materialized views.
> > > >
> > > >
> > > > +1 for voting Haisheng.
> > > >
> > > >
> > > > Best,
> > > > Zhaohui Xu
> > > >
> > > >
> > > > -- 原始邮件 --
> > > > 发件人:
> > > >   "dev"
> > > > <
> > > > zabe...@gmail.com>;
> > > > 发送时间: 2020年11月5日(星期四) 上午6:26
> > > > 收件人: "dev" > > >
> > > > 主题: [DISCUSS] State of the project 2020
> > > >
> > > >
> > > >
> > > > Hi Calcite community members,
> > > >
> > > > A bit more than five years ago (October 22, 2015) Calcite graduated
> to
> > a
> > > > top-level Apache project[1]. At that time, the community decided to
> > have
> > > an
> > > > annual “state of the project” discussion and to vote for a new PMC
> > > > chair/VP[2]. So, I’m kicking off both of those discussions.
> > > >
> > > > I think it was an excellent year so far in many aspects.
> > > >
> > > > We were lucky to have many high quality contributions including:
> > notable
> > > > improvements in the Volcano planner (for speed, plan quality,
> > > > extensibility) bringing it a bit closer to Cascades and Columbia [6,
> 7,
> > > 8,
> > > > 9]; easier and more extensible parameterization of rules [3]; new
> > > dialects
> > > > such as ClickHouse [4], and Presto [5]; support for SQL hints [10];
> new
> > > > adapters for querying Redis [11] and InnoDB [12] through Calcite;
> > various
> > > > enhancements in streaming SQL. The previous list is by no means
> > > exhaustive.
> > > >
> > > > Apart from the new features, certainly worth mentioning is the
> > > > modernization of the build and test infrastructure (for both Calcite
> > and
> > > > Avatica), with the migration from maven to gradle, JUnit4 to JUnit5,
> > and
> > > > the introduction of GitHub actions as part of the CI.
> > > >
> > > > In terms of CI, I am happy to see a few more integration tests (IT)
> > > running
> > > > on a regular basis on GitHub. Eventually, it will be nice to have
> even
> > > more
> > > > IT tests to help us catch regressions early on and improve the
> quality
> > of
> > > > our releases.
> > > >
> > > > We wouldn’t have so many great contributions, if we didn’t also have
> > > > prolific contributors.
> > > > Our community has grown with Danny, Haisheng, Ruben, joining the PMC,
> > > > Forward, Xing, Vineet, Yanlin, Feng, Rui, joining as committers, and
> > many
> > > > more people chiming in discussions, reviews, and submitting pull
> > > requests,
> > > > who are not yet committers but I’m sure some of whom will become in
> the
> > > > near future.
> > > >
> > > > We have had five Calcite releases (1.22.0 to 1.26.0), one Avatica
> > release
> > > > (1.17.0), and one Avatica Go (5.0.0) so far in 2020, and I think that
> > is
> > > a
> > > > great tempo that we should strive to maintain in the years to come.
> One
> > > > thing to improve is the poor implication of other people than Francis
> > on
> > > > the Avatica side; the rest of us, putting myself first, should try to
> > be
> > > > more involved by reviewing PRs, preparing releases, voting etc.
> > > >
> > > > It was nice to see our community members giving talks to confere

Re: [ANNOUNCE] New committer: Liya Fan

[Forward Xu]

Congratulations Liya!


Best,

Forward

Chunwei Lei  于2021年2月22日周一 上午11:13写道：

> Congratulations Liya!
>
> Best,
> Chunwei
>
>
> On Fri, Feb 12, 2021 at 5:28 PM Fan Liya  wrote:
>
> > Dear all,
> >
> > I want to express my sincere gratitude to you for all your kind help.
> > I feel honored to have the opportunity of working with you in this
> awesome
> > community.
> >
> > In the past years, I have been working on some different components of
> the
> > SQL engine, now in Alibaba, and previously in Microsoft.
> > Through this experience, I find the SQL optimizer especially interesting,
> > because it involves so many interesting problems.
> >
> > In the future, I want to make greater contributions to the community and
> > help our users build better SQL engines.
> >
> > Best,
> > Liya Fan
> >
> >
> > On Thu, Feb 11, 2021 at 1:54 AM Rui Wang  wrote:
> >
> > > Congratulations Liya!
> > >
> > >
> > > -Rui
> > >
> > > On Wed, Feb 10, 2021 at 6:39 AM Michael Mior  wrote:
> > >
> > > > Congratulations and welcome Liya!
> > > > --
> > > > Michael Mior
> > > > mm...@apache.org
> > > >
> > > > Le mer. 10 févr. 2021 à 05:23, Francis Chuang
> > > >  a écrit :
> > > > >
> > > > > Congrats Liya and well deserved!
> > > > >
> > > > > Francis
> > > > >
> > > > > On 10/02/2021 8:53 pm, Stamatis Zampetakis wrote:
> > > > > > Apache Calcite's Project Management Committee (PMC) has invited
> > Liya
> > > > Fan to
> > > > > > become a committer, and we are pleased to announce that they have
> > > > accepted.
> > > > > >
> > > > > > Liya has made a lot of contributions to the project, enhancing
> > > > optimizer
> > > > > > rules,
> > > > > > improving the efficiency of graph traversal algorithms, as well
> as
> > > > fixing
> > > > > > various
> > > > > > bugs in the project.
> > > > > >
> > > > > > Liya, welcome, thank you for your contributions, and we look
> > forward
> > > > to your
> > > > > > further interactions with the community! If you wish, please feel
> > > free
> > > > to
> > > > > > tell
> > > > > > us more about yourself and what you are working on.
> > > > > >
> > > > > > Stamatis (on behalf of the Apache Calcite PMC)
> > > > > >
> > > >
> > >
> >
>

Re: Reviewers

[Alex Xu]

Agree ！

ptr.bo...@gmail.com 于2018年5月22日 周二上午4:26写道：

> Hello,
>
> I've just fired at you Julian a pull request, just like you said. Sorry for
> that.
>
> Currently, each bug at JIRA is assigned to you automatically - this can be
> misleading, when someone is logging a bug and deliver a fix just after.
> This is my case. Maybe it would be better to leave a JIRA issue unassigned?
>
>
> On Mon, May 21, 2018 at 6:58 PM Julian Hyde  wrote:
>
> > Calcite contributors,
> >
> > If you are submitting a PR, please stop tagging me personally in JIRA
> > cases to review your PR. I do review a lot of PRs in this project but I
> am
> > short of time and. If you tag me, other committers will be discouraged
> from
> > doing it.
> >
> > Of course we are delighted that you are contributing to the project, but
> > we need a process that can scale, and directing everything to me does not
> > help.
> >
> > When you make a PR, add a comment to the JIRA. If a committer does not
> > engage within 24 hours, feel free to send a message to the dev list
> asking
> > for review.
> >
> > Committers,
> >
> > As I have said several times, I need other committers to step up and
> help.
> > I am getting close to burning out.
> >
> > Julian
>
>
>
> --
> Piotr Bojko
> http://about.me/ptr.bojko
>

'Spool' Node support

[Ted Xu]

Hi folks,

I'm not sure if there is a recommended way to represent diverged (multiple
parents) plan in Calcite. It’s true that RelNode data structure is
compatible with multiple parents, but it is not working in optimizer.

For example, if we have query as follows,

FROM (SELECT c1, random() as c2, c3 FROM src)
INSERT OVERWRITE TABLE src1 SELECT c1, c2
INSERT OVERWRITE TABLE src2 SELECT c3, c2

TableSink1(on columns c1, c2)
Project(c1, random() as c2, c3)
TableScan
TableSink2(on columns c3, c2)
Project(c1, random() as c2, c3)
TableScan

Planners will recognize Projects and TableScans share the common digests
thus merged together, but Project Transpose Rules splits them, which breaks
the random assumption.

My solution is to add a Spool node to prevent any rule to further split a
sub-plan, but it generates sub-optimal result. I've noticed there is a
really old JIRA ticket https://jira.apache.org/jira/browse/CALCITE-481 but
it was somehow suspended.

I'd like to move on on this feature, but there are still something to do
first:

1. Let RelOptRuleCall to aware parents, currently only HepRelOptRuleCall
passes parents in certain cases.
2. Let RelOptRuleOperand to define multiple parent patterns

Please correct me if I'm something wrong, any suggestion will be much
appreciated.

Re: 'Spool' Node support

[Ted Xu]

Thanks Julian for the comment.

To some extent it is "split" in VolcanoPlanner. Let's revisit the example I
mentioned above:

TableSink1(on columns c1, c2)
Project(c1, random() as c2, c3)
TableScan
TableSink2(on columns c3, c2)
Project(c1, random() as c2, c3)
TableScan

The 2 projects share the common digests, thus it is recognized in
VolcanoPlanner, the plan in memo looks like:

TableSink1 TableSink2
 \ /
 Project (c1, random() as c2, c3)
 |
TableScan

However, if we "add" a project here, the project will further match other
rule pattern (e.g., PrjectMergeRule), the random project will no longer
share the same digest, new plan looks like:

TableSink1  TableSink2
  |   |
Project (c1, random())   Project(c3, random())
  |   |
TableScan(c1) TableScan(c3)

Notice that the random function call is executed twice, which breaks the
assumption of original sql query.

I think Spool node should prevent such kind of transformation.

On Tue, Oct 23, 2018 at 11:30 AM Julian Hyde  wrote:

> I assume you’re talking about HepPlanner? VolcanoPlanner doesn’t “split”
> anything, it only adds new things.
>
> As you’ve noticed Spool isn’t finished, but the idea would be to use
> VolcanoPlanner, because it can truly handle plans that are DAGs, then use
> some kind of costing trick to ensure that nodes that are shared are only
> counted in the overall cost once.
>
> > On Oct 22, 2018, at 8:26 PM, Ted Xu  wrote:
> >
> > Hi folks,
> >
> > I'm not sure if there is a recommended way to represent diverged
> (multiple
> > parents) plan in Calcite. It’s true that RelNode data structure is
> > compatible with multiple parents, but it is not working in optimizer.
> >
> > For example, if we have query as follows,
> >
> > FROM (SELECT c1, random() as c2, c3 FROM src)
> > INSERT OVERWRITE TABLE src1 SELECT c1, c2
> > INSERT OVERWRITE TABLE src2 SELECT c3, c2
> >
> > TableSink1(on columns c1, c2)
> >Project(c1, random() as c2, c3)
> >TableScan
> > TableSink2(on columns c3, c2)
> >Project(c1, random() as c2, c3)
> >TableScan
> >
> > Planners will recognize Projects and TableScans share the common digests
> > thus merged together, but Project Transpose Rules splits them, which
> breaks
> > the random assumption.
> >
> > My solution is to add a Spool node to prevent any rule to further split a
> > sub-plan, but it generates sub-optimal result. I've noticed there is a
> > really old JIRA ticket https://jira.apache.org/jira/browse/CALCITE-481
> but
> > it was somehow suspended.
> >
> > I'd like to move on on this feature, but there are still something to do
> > first:
> >
> > 1. Let RelOptRuleCall to aware parents, currently only HepRelOptRuleCall
> > passes parents in certain cases.
> > 2. Let RelOptRuleOperand to define multiple parent patterns
> >
> > Please correct me if I'm something wrong, any suggestion will be much
> > appreciated.
>
>

Re: Looking for help on query optimization statistics

[Ted Xu]

Hi Tian,

To inject statistics, you need to implement your own RelMetadataProvider
and register it into your runtime environment.

I'm not able to find a valid example in Calcite, but there is a good
example you can find in Drill
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java#L405
.


On Mon, Nov 5, 2018 at 11:54 AM Tian Ye  wrote:

> Hi,
>
> I'm working on comparing the query plan generated by Calcite with that
> from a DBMS I'm developing.
>
> In particular, I can generate some TPC-H data and a SQL query, and I want
> to let Calcite generate as good query plan as it can. So far, I haven't
> passed any statistics to Calcite, so the query plan it generates is just a
> baseline.
>
> Now I wonder how I could pass statistics to Calcite. What I don't
> understand is that since Calcite does not load the data until query plan is
> actually executed, how could I pass statistics to it?
> I fail to find any document talking about how to do that in detail. The
> last paragraphs of  http://calcite.apache.org/docs/adapter.html do
> mention it but are not so clear.
>
>
>
>
> Would you please help me?
>
> Thanks a lot.
>
>
>
>
> 
> Tian Ye
> y...@pku.edu.cn

Re: Does calcite support chinese?

[Ted Xu]

Hi,

You could use optional charset literal here. E.g.,

> upsert into t1(c1,c2,c3) values(_UTF16'中文',2,2);

For some reason Calcite currently only support UTF-16 for Unicode. Here
CALCITE-2619 is an on going work to support more charsets like UTF-8,
GB2312 and GBK.


On Wed, Nov 7, 2018 at 5:30 PM 朱力(梧空)  wrote:

>
> dear guys,
>
> does calcite support chinese? in my case, what should i do ?
> 0: jdbc:lindorm:100.81.140.116:30024:test> upsert into t1(c1,c2,c3)
> values('中文',2,2); Error: Error while executing SQL "upsert into
> t1(c1,c2,c3) values('中文',2,2)": Failed to encode '中文' in character set
> 'ISO-8859-1' (state=,code=0)
>
> thanks

Re: does calcite support chinese?

[Ted Xu]

+1 for full charset support.

There may be multiple changes to make, not only related to literals but
also to type system, casts, (jdbc) binary protocols etc.

I'd like to propose some design ideas since I'm from a CJK country and
we've already tackled most of Calcite's charset issues.

On Thu, Nov 8, 2018 at 3:22 AM Julian Hyde  wrote:

> > I can't remember another database that allows just ISO-8859-1 in simple
> > string literals.
> > That makes it very surprising.
>
> Try SQL Server on rextester.com . You need to
> prefix literals with ’N’, like this: N’привет'
>
> > What is the reason for Calcite to enforce ISO-8859-1 by default?
> >
> > In other words, I can't imagine a project that would pick ISO-8859-1 as a
> > default string literal encoding if they HAVE to make an explicit choice
> > (e.g. no default within Calcite).
>
> I really don’t know.
>
> I would very much appreciate if someone took ownership of this issue, took
> the time to understand what Calcite does today, document it, understand
> what the SQL standard says, and make improvements.
>
> Julian
>
>
> > On Nov 7, 2018, at 11:08 AM, Vladimir Sitnikov <
> sitnikov.vladi...@gmail.com> wrote:
> >
> >> The issue is not the encoding of our Java code. The issue is the
> encoding
> > of the SQL we process. That SQL may or may not come from Java source
> files.
> >
> > I can't remember another database that allows just ISO-8859-1 in simple
> > string literals.
> > That makes it very surprising.
> >
> > For instance, https://rextester.com/l/postgresql_online_compiler allows
> > <>
> >
> > What is the reason for Calcite to enforce ISO-8859-1 by default?
> >
> > In other words, I can't imagine a project that would pick ISO-8859-1 as a
> > default string literal encoding if they HAVE to make an explicit choice
> > (e.g. no default within Calcite).
> >
> > Vladimir
>
>

Charset support proposal

[Ted Xu]

Hi folks,

I created a design doc
https://docs.google.com/document/d/1wo5byn_6K_YOKiPdXNav1zgzt9IBC3SbPvpPnIShtXk/edit?usp=sharing
for supporting charset in calcite, per previous discussions on this topic.

One thing I'm not sure is runtime (Codegen on Enumerable and RelExecutor
etc) change. Since I/O is decoupled by pluggable points like
Schemas#enumerable, that part looks good to me already.

I'm sure there are a lot misunderstandings and missing pieces in that doc
above, please feel free to leave comments.

Re: Charset support proposal

[Ted Xu]

Thanks for the comments, please find my replies inline.

On Fri, Nov 16, 2018 at 2:06 AM Julian Hyde  wrote:

> Looks great - thank you for writing this. I have some questions. If they
> are already answered in your document, forgive me, and just say “That’s
> answered in the document."
>
> I very much like the idea of adding default charset and collation to
> RelDataTypeSystem. This will help to carry them to all points in the code
> where they are needed.
>
> I also like the idea of adding charset and collation as table options. It
> seems that this feature is non-essential, and could be done in phase 2, if
> necessary. Also, it mainly applies to SQL DDL, i.e. the “server” module. I
> don’t we need to add default charset and collation to the Table or
> RelOptTable interfaces, just SqlCreateTable.
>

Agreed.


>
> Regarding the column options. Could charset and collection not be
> specified as part of the column’s data type?
>

Yes, if not specified, column charset is deduced from table default, or
session default, or system default charset.


>
> When we are parsing a SQL character literal, the characters of that
> literal are in the same encoding as the SQL string itself. The parser (see
> the line ‘UNICODE_INPUT = true;' generated Parser.jj file) seems to assume
> that input is unicode. That seems fine to me — do you agree?
>

Agreed. By fixing 'core charset' be UTF-16, we have better performance and
lower coding effort.


>
> Unqualified character literals (e.g. ‘hello’, vs qualified _UTF8’hello’)
> are always UTF16. Is that correct? Should we provide a way to change that
> default? Do any major databases provide a way to change that default?
>
>
IMO unqualified characters should have default charset, instead of treating
'hello' as _UTF16'hello', it is more convenient to treat it as
_${DEFAULT_CHARSET}'hello', where DEFAULT_CHARSET is defined by
session/system configuration (connection/startup configuration in mysql
https://goo.gl/67hOXK , or SqlSetOption in Calcite) or type system.

In a scenario where different columns have different charsets/collations, I
> assume that there will be a lot of implicit conversion going on. (Not to
> mention explicit conversion, using CONVERT.) Are there concerns about this?
> Are the rules well-defined if, say we compare a UTF8 with a UTF16 string,
> or concatenate a UTF8 with a UTF16 string?
>

There may be concerns. I've already found 2 points,

1. Sql Function return type inference.
2. RelDataFactory#leastRestrictive

>From which there may impacts on rules like ReduceExpressionRules.


>
> I saw that MySQL has problems with 3-byte utf8 (aka utf8mb3) and 4-byte
> utf8mb4[1]. Are we going to avoid those problems?
>

I'm not sure but Java UTF-8 encoder/decoder look good.


>
> Julian
>
> [1] https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html
>
>
>
> > On Nov 15, 2018, at 4:13 AM, Ted Xu  wrote:
> >
> > Hi folks,
> >
> > I created a design doc
> >
> https://docs.google.com/document/d/1wo5byn_6K_YOKiPdXNav1zgzt9IBC3SbPvpPnIShtXk/edit?usp=sharing
> > for supporting charset in calcite, per previous discussions on this
> topic.
> >
> > One thing I'm not sure is runtime (Codegen on Enumerable and RelExecutor
> > etc) change. Since I/O is decoupled by pluggable points like
> > Schemas#enumerable, that part looks good to me already.
> >
> > I'm sure there are a lot misunderstandings and missing pieces in that doc
> > above, please feel free to leave comments.
>
>

Re: Charset support proposal

[Ted Xu]

Thanks Stamatis for the comment and advice!

You've made a good point on RelDataType. I agree RelDataTypeSystem change
is not a necessity.

I also agree with the work prioritization, the DDL changes are not urgent
compare to core correctness. I'd like to create JIRA tickets on those
topics which we've already have common agreements like types and rules.

On Fri, Nov 16, 2018 at 6:29 AM Stamatis Zampetakis 
wrote:

> Thanks for the initiative Ted!
>
> The document mentions various interesting features like enriching DDL
> operations with charset and collation but I think it is also worth
> mentioning a few points in the existing API that are related with these
> notions.
>
> The RelDataType (column data type) includes already the notions of charset
> (RelDataType#getCharset) and collation (RelDataType#getCollation). Moreover
> the RelDataTypeFactory allows creating types with a specific charset and
> collation through the method
> RelDataTypeFactory#createTypeWithCharsetAndCollation and it also provides a
> method to obtain the default charset
> (RelDataTypeFactory#getDefaultCharset).
>
> Basically with the above it is already possible for users to pass their own
> charset and collation as far as it concerns tables and columns. Also by
> providing a custom RelDataTypeFactory they can also control what is the
> default charset that is going to used.
>
> Regarding the operations performed on columns with charsets/collations the
> SQL standard has specific rules on what should be done or not along with
> implicit/explicit conversions [1]. I don't know to what extend these rules
> existing in Calcite but by looking in various places they don't seem to be
> present.
>
> Apart from the DDL that can be extended relatively easy I think the biggest
> challenge is to incorporate the Standard rules in the necessary places.
> From my point of view, it seems more urgent to improve the runtime to
> behave correctly in the presence of charset and collations. Overall, I find
> the subject very interesting and at the same time very important so thanks
> a lot for working on it.
>
> Best,
> Stamatis
>
> [1] 4.2.3 Rules determining collating sequence usage
> <http://web.cecs.pdx.edu/~len/sql1999.pdf>
>
> Στις Πέμ, 15 Νοε 2018 στις 7:06 μ.μ., ο/η Julian Hyde 
> έγραψε:
>
> > Looks great - thank you for writing this. I have some questions. If they
> > are already answered in your document, forgive me, and just say “That’s
> > answered in the document."
> >
> > I very much like the idea of adding default charset and collation to
> > RelDataTypeSystem. This will help to carry them to all points in the code
> > where they are needed.
> >
> > I also like the idea of adding charset and collation as table options. It
> > seems that this feature is non-essential, and could be done in phase 2,
> if
> > necessary. Also, it mainly applies to SQL DDL, i.e. the “server” module.
> I
> > don’t we need to add default charset and collation to the Table or
> > RelOptTable interfaces, just SqlCreateTable.
> >
> > Regarding the column options. Could charset and collection not be
> > specified as part of the column’s data type?
> >
> > When we are parsing a SQL character literal, the characters of that
> > literal are in the same encoding as the SQL string itself. The parser
> (see
> > the line ‘UNICODE_INPUT = true;' generated Parser.jj file) seems to
> assume
> > that input is unicode. That seems fine to me — do you agree?
> >
> > Unqualified character literals (e.g. ‘hello’, vs qualified _UTF8’hello’)
> > are always UTF16. Is that correct? Should we provide a way to change that
> > default? Do any major databases provide a way to change that default?
> >
> > In a scenario where different columns have different charsets/collations,
> > I assume that there will be a lot of implicit conversion going on. (Not
> to
> > mention explicit conversion, using CONVERT.) Are there concerns about
> this?
> > Are the rules well-defined if, say we compare a UTF8 with a UTF16 string,
> > or concatenate a UTF8 with a UTF16 string?
> >
> > I saw that MySQL has problems with 3-byte utf8 (aka utf8mb3) and 4-byte
> > utf8mb4[1]. Are we going to avoid those problems?
> >
> > Julian
> >
> > [1] https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html
> >
> >
> >
> > > On Nov 15, 2018, at 4:13 AM, Ted Xu  wrote:
> > >
> > > Hi folks,
> > >
> > > I created a design doc
> > >
> >
> https://docs.google.com/document/d/1wo5byn_6K_YOKiPdXNav1zgzt9IBC3SbPvpPnIShtXk/edit?usp=sharing
> > > for supporting charset in calcite, per previous discussions on this
> > topic.
> > >
> > > One thing I'm not sure is runtime (Codegen on Enumerable and
> RelExecutor
> > > etc) change. Since I/O is decoupled by pluggable points like
> > > Schemas#enumerable, that part looks good to me already.
> > >
> > > I'm sure there are a lot misunderstandings and missing pieces in that
> doc
> > > above, please feel free to leave comments.
> >
> >
>

Re: Can’t seem to parse DDL staements

[Ted Xu]

Hi Dilip,

DDL related code including statement parse and data structures are located
in calcite-server sub-project.

The unit test 'ServerParserTest' may be a good starting point to look into.
You can find it here
https://github.com/apache/calcite/blob/master/server/src/test/java/org/apache/calcite/test/ServerParserTest.java
 .

On Mon, Dec 17, 2018 at 1:34 AM Dilip Raj Baral 
wrote:

> Hi, team.
>
> I have been using Apache Calcite for about two weeks now for one of the
> projects at my company. I have found it fun and pretty useful so far. I
> have been able to parse DMLs very smoothly. However, when I tried to parse
> DDLs like CREATE TABLE, ALTER TABLE, etc., the SqlParser.parseStmt() or
> SqlParser.parseQuery() failed.
>
> I have detailed the specifics in the following SO question.
>
>
> https://stackoverflow.com/questions/53801005/apache-calcite-cant-seem-to-parse-ddl-statements
>
> Looking forward to a suggestion.
>
> Best regards,
> Dilip Raj Baral
> https://diliprajbaral.com
>
>
>

a new adapter for Apache Kafka

[Mingmin Xu]

Hi all,

I add a new adapter for Apache Kafka, to allow users to query Kafka topics
with Calcite STREAM SQL. Can someone take a look at the PR[1]?

[1]. https://github.com/apache/calcite/pull/1127

Thanks!
Mingmin

Re: a new adapter for Apache Kafka

[Mingmin Xu]

Thank you @Andrei @Michael. Currently it's a straightforward implementation
of stream ScannableTable, a site page is added for usage. Kindly let me
know if something important is not covered.

Mingmin

On Wed, Apr 24, 2019 at 1:24 PM Michael Mior  wrote:

> I think we'd generally be willing to accept any adapter which is
> well-written, well-tested, and likely to be generally useful. I think
> Kafka meets the final criterion and with some work the current code
> could meet the first two. That said, even if the adapter isn't
> accepted into the main Calcite codebase, I would encourage you to
> publish it as a separate package anyway.
> --
> Michael Mior
> mm...@apache.org
>
> Le mer. 24 avr. 2019 à 08:07, Andrei Sereda  a écrit :
> >
> > Hi Mingmin,
> >
> > I'm happy to review it.
> >
> > Before would like to confirm with this list that they're OK adding new
> > adapter (kafka) to calcite codebase ?
> >
> > Regards,
> > Andrei.
> >
> > On Tue, Apr 23, 2019 at 10:43 PM Mingmin Xu  wrote:
> >
> > > Hi all,
> > >
> > > I add a new adapter for Apache Kafka, to allow users to query Kafka
> topics
> > > with Calcite STREAM SQL. Can someone take a look at the PR[1]?
> > >
> > > [1]. https://github.com/apache/calcite/pull/1127
> > >
> > > Thanks!
> > > Mingmin
> > >
>


-- 

Mingmin

Re: a new adapter for Apache Kafka

[Mingmin Xu]

Quick ping ~~~
Can someone take a look at this PR(1)? I suppose a Kafka adapter will be
very helpful to accelerate the adoption of Calcite Stream SQL.

[1]. https://github.com/apache/calcite/pull/1127

Mingmin

On Wed, Apr 24, 2019 at 11:31 PM Mingmin Xu  wrote:

> Thank you @Andrei @Michael. Currently it's a straightforward
> implementation of stream ScannableTable, a site page is added for usage.
> Kindly let me know if something important is not covered.
>
> Mingmin
>
> On Wed, Apr 24, 2019 at 1:24 PM Michael Mior  wrote:
>
>> I think we'd generally be willing to accept any adapter which is
>> well-written, well-tested, and likely to be generally useful. I think
>> Kafka meets the final criterion and with some work the current code
>> could meet the first two. That said, even if the adapter isn't
>> accepted into the main Calcite codebase, I would encourage you to
>> publish it as a separate package anyway.
>> --
>> Michael Mior
>> mm...@apache.org
>>
>> Le mer. 24 avr. 2019 à 08:07, Andrei Sereda  a écrit :
>> >
>> > Hi Mingmin,
>> >
>> > I'm happy to review it.
>> >
>> > Before would like to confirm with this list that they're OK adding new
>> > adapter (kafka) to calcite codebase ?
>> >
>> > Regards,
>> > Andrei.
>> >
>> > On Tue, Apr 23, 2019 at 10:43 PM Mingmin Xu  wrote:
>> >
>> > > Hi all,
>> > >
>> > > I add a new adapter for Apache Kafka, to allow users to query Kafka
>> topics
>> > > with Calcite STREAM SQL. Can someone take a look at the PR[1]?
>> > >
>> > > [1]. https://github.com/apache/calcite/pull/1127
>> > >
>> > > Thanks!
>> > > Mingmin
>> > >
>>
>
>
> --
> 
> Mingmin
>


-- 

Mingmin

Re: a new adapter for Apache Kafka

[Mingmin Xu]

Thank you Andrei, appreciated for your time.

On Thu, May 9, 2019 at 11:29 AM Andrei Sereda  wrote:

> Mingmin, I'll be reviewing the PR this week.
> Sorry was busy last couple of weeks.
>
> On Thu, May 9, 2019 at 12:54 PM Mingmin Xu  wrote:
>
> > Quick ping ~~~
> > Can someone take a look at this PR(1)? I suppose a Kafka adapter will be
> > very helpful to accelerate the adoption of Calcite Stream SQL.
> >
> > [1]. https://github.com/apache/calcite/pull/1127
> >
> > Mingmin
> >
> > On Wed, Apr 24, 2019 at 11:31 PM Mingmin Xu  wrote:
> >
> > > Thank you @Andrei @Michael. Currently it's a straightforward
> > > implementation of stream ScannableTable, a site page is added for
> usage.
> > > Kindly let me know if something important is not covered.
> > >
> > > Mingmin
> > >
> > > On Wed, Apr 24, 2019 at 1:24 PM Michael Mior  wrote:
> > >
> > >> I think we'd generally be willing to accept any adapter which is
> > >> well-written, well-tested, and likely to be generally useful. I think
> > >> Kafka meets the final criterion and with some work the current code
> > >> could meet the first two. That said, even if the adapter isn't
> > >> accepted into the main Calcite codebase, I would encourage you to
> > >> publish it as a separate package anyway.
> > >> --
> > >> Michael Mior
> > >> mm...@apache.org
> > >>
> > >> Le mer. 24 avr. 2019 à 08:07, Andrei Sereda  a
> écrit
> > :
> > >> >
> > >> > Hi Mingmin,
> > >> >
> > >> > I'm happy to review it.
> > >> >
> > >> > Before would like to confirm with this list that they're OK adding
> new
> > >> > adapter (kafka) to calcite codebase ?
> > >> >
> > >> > Regards,
> > >> > Andrei.
> > >> >
> > >> > On Tue, Apr 23, 2019 at 10:43 PM Mingmin Xu 
> > wrote:
> > >> >
> > >> > > Hi all,
> > >> > >
> > >> > > I add a new adapter for Apache Kafka, to allow users to query
> Kafka
> > >> topics
> > >> > > with Calcite STREAM SQL. Can someone take a look at the PR[1]?
> > >> > >
> > >> > > [1]. https://github.com/apache/calcite/pull/1127
> > >> > >
> > >> > > Thanks!
> > >> > > Mingmin
> > >> > >
> > >>
> > >
> > >
> > > --
> > > 
> > > Mingmin
> > >
> >
> >
> > --
> > 
> > Mingmin
> >
>


-- 

Mingmin

RelMetadataQuery is not pluggable?

[Ted Xu]

Hi all,

After [CALCITE-604 <https://issues.apache.org/jira/browse/CALCITE-604>], I
found the methods of RelMetadataQuery is not pluggable. By 'pluggable', I
mean RelMetadataQuery's behaviour can be re-defined by registering a
RelMetadataProvider in RelOptCluster.

For example, if I defined a RelMetadataProvider as follows and register it
using RelOptCluster#setMetadataProvider,

public class MyRelMetadataProvider extends ChainedRelMetadataProvider {
  public MyRelMetadataProvider() {
super(
ImmutableList.of(
MyRelMdRowCount.SOURCE));
  }
}

I expect RelMetadataQuery#getRowCount will invoke MyRelMdRowCount.SOURCE.
However I noticed that after CALCITE-604 it always use
DefaultRelMetadataProvider.

I do noticed an example of overriding RelMetadataQuery in RelMetadataTest,
but I guess by now I can only add metadata support rather than override
existing metadata?

I'm not sure if I'm using it in the right way, please correct me if I'm
wrong. Thanks!

Regards,
Ted Xu.

Re: RelMetadataQuery is not pluggable?

[Ted Xu]

Thanks Julian!

I managed to inject my code by setting a ThreadLocal attribute in
RelMetadataQuery, as follows,

RelMetadataQuery.THREAD_PROVIDERS.set(
 JaninoRelMetadataProvider.of(new MyRelMetadataProvider()));

Now it works.

However, does that mean RelOptCluster#setMetadataProvider is deprecated?

On Tue, Mar 29, 2016 at 12:26 PM Julian Hyde  wrote:

> Did you take a look at https://issues.apache.org/jira/browse/CALCITE-1147
> <https://issues.apache.org/jira/browse/CALCITE-1147>? It helps make
> RelMetadataQuery pluggable.
>
> > On Mar 28, 2016, at 9:21 PM, Ted Xu  wrote:
> >
> > Hi all,
> >
> > After [CALCITE-604 <https://issues.apache.org/jira/browse/CALCITE-604>],
> I
> > found the methods of RelMetadataQuery is not pluggable. By 'pluggable', I
> > mean RelMetadataQuery's behaviour can be re-defined by registering a
> > RelMetadataProvider in RelOptCluster.
> >
> > For example, if I defined a RelMetadataProvider as follows and register
> it
> > using RelOptCluster#setMetadataProvider,
> >
> > public class MyRelMetadataProvider extends ChainedRelMetadataProvider {
> >  public MyRelMetadataProvider() {
> >super(
> >ImmutableList.of(
> >MyRelMdRowCount.SOURCE));
> >  }
> > }
> >
> > I expect RelMetadataQuery#getRowCount will invoke MyRelMdRowCount.SOURCE.
> > However I noticed that after CALCITE-604 it always use
> > DefaultRelMetadataProvider.
> >
> > I do noticed an example of overriding RelMetadataQuery in
> RelMetadataTest,
> > but I guess by now I can only add metadata support rather than override
> > existing metadata?
> >
> > I'm not sure if I'm using it in the right way, please correct me if I'm
> > wrong. Thanks!
> >
> > Regards,
> > Ted Xu.
>
>

how to declare a column monotonic in a StreamableTable

[Mingmin Xu]

Hello all,

I'm writing some examples to understand the execution plan(more exactly
RelNode) of Calcite Streaming, and meet the error 'Streaming aggregation
requires at least one monotonic expression in GROUP BY clause' with
GROUP-BY.  Can anybody share an Java example on how to declare a column
monotonic in a StreamableTable? And, can I disable this check rule?

Thank you!
-- 

Mingmin

Re: how to declare a column monotonic in a StreamableTable

[Mingmin Xu]

Thanks, that can help to generate the RelNode. However it brings a new
issue, not sure it's related.

Briefly I'm testing to replace RelNode with my own RelNode, to build a
pipeline based on the RelNode tree. When running planner.transform(), an
error message is thrown as
https://gist.github.com/XuMingmin/491ff85011d2312acf086c2c0b7343c8.


The converter can work without the monotonic column with output:

SQL>:
select stream * from SOJ_EVENT
SQLPlan>
LogicalDelta
  LogicalProject(EVENTTIMESTAMP=[$0], ITEMID=[$1], SITEID=[$2],
PAGEID=[$3], PAGENAME=[$4])
LogicalTableScan(table=[[SOJ_EVENT]])

beamPlan>
BeamProjectRel(EVENTTIMESTAMP=[$0], ITEMID=[$1], SITEID=[$2], PAGEID=[$3],
PAGENAME=[$4])
  BeamStreamScanRel(table=[[SOJ_EVENT, (STREAM)]])


Also paste my DeltaRule at
https://gist.github.com/XuMingmin/402cb9b3969763ee6b1efc80016c9a3f, Table
schema class here
https://gist.github.com/XuMingmin/0f0efa51be7e5567a7b2d93c206f2275.

Mingmin




On Tue, Feb 28, 2017 at 7:26 PM, 陈江枫  wrote:

> implement
>
> Statistic getStatistic()  in table interface should work.
>
>
> @Override
> public Statistic getStatistic() {
> return Statistics.of(100d,
> ImmutableList.of(),
> RelCollations.createSingleton(0));
> }
>
> is a simple implementation declaring the first field to be ascending
>
>
> 2017-03-01 9:56 GMT+08:00 Mingmin Xu :
>
> > Hello all,
> >
> > I'm writing some examples to understand the execution plan(more exactly
> > RelNode) of Calcite Streaming, and meet the error 'Streaming aggregation
> > requires at least one monotonic expression in GROUP BY clause' with
> > GROUP-BY.  Can anybody share an Java example on how to declare a column
> > monotonic in a StreamableTable? And, can I disable this check rule?
> >
> > Thank you!
> > --
> > 
> > Mingmin
> >
>



-- 

Mingmin

Re: how to declare a column monotonic in a StreamableTable

[Mingmin Xu]

btw, same case when I remove BeamDeltaRule from ruleset, the error message
is a little different as
https://gist.github.com/XuMingmin/b4e4eb40f3d8169cf9d11e10a6da92a8

On Tue, Feb 28, 2017 at 9:08 PM, Mingmin Xu  wrote:

> Thanks, that can help to generate the RelNode. However it brings a new
> issue, not sure it's related.
>
> Briefly I'm testing to replace RelNode with my own RelNode, to build a
> pipeline based on the RelNode tree. When running planner.transform(), an
> error message is thrown as https://gist.github.com/XuMingmin/
> 491ff85011d2312acf086c2c0b7343c8.
>
> 
> The converter can work without the monotonic column with output:
>
> SQL>:
> select stream * from SOJ_EVENT
> SQLPlan>
> LogicalDelta
>   LogicalProject(EVENTTIMESTAMP=[$0], ITEMID=[$1], SITEID=[$2],
> PAGEID=[$3], PAGENAME=[$4])
> LogicalTableScan(table=[[SOJ_EVENT]])
>
> beamPlan>
> BeamProjectRel(EVENTTIMESTAMP=[$0], ITEMID=[$1], SITEID=[$2],
> PAGEID=[$3], PAGENAME=[$4])
>   BeamStreamScanRel(table=[[SOJ_EVENT, (STREAM)]])
>
>
> Also paste my DeltaRule at https://gist.github.com/XuMingmin/
> 402cb9b3969763ee6b1efc80016c9a3f, Table schema class here
> https://gist.github.com/XuMingmin/0f0efa51be7e5567a7b2d93c206f2275.
>
> Mingmin
>
>
>
>
> On Tue, Feb 28, 2017 at 7:26 PM, 陈江枫  wrote:
>
>> implement
>>
>> Statistic getStatistic()  in table interface should work.
>>
>>
>> @Override
>> public Statistic getStatistic() {
>> return Statistics.of(100d,
>> ImmutableList.of(),
>> RelCollations.createSingleton(0));
>> }
>>
>> is a simple implementation declaring the first field to be ascending
>>
>>
>> 2017-03-01 9:56 GMT+08:00 Mingmin Xu :
>>
>> > Hello all,
>> >
>> > I'm writing some examples to understand the execution plan(more exactly
>> > RelNode) of Calcite Streaming, and meet the error 'Streaming aggregation
>> > requires at least one monotonic expression in GROUP BY clause' with
>> > GROUP-BY.  Can anybody share an Java example on how to declare a column
>> > monotonic in a StreamableTable? And, can I disable this check rule?
>> >
>> > Thank you!
>> > --
>> > 
>> > Mingmin
>> >
>>
>
>
>
> --
> 
> Mingmin
>



-- 

Mingmin

Re: Calcite to Beam

[Mingmin Xu]

We have some work already, will create a feature branch later soon.



On Wed, Apr 5, 2017 at 3:37 PM, Khai Tran 
wrote:

> Thanks, Julian. That's what I need.
>
> On Wed, Apr 5, 2017 at 3:26 PM, Julian Hyde  wrote:
>
> > + dev@beam (please moderate through!)
> >
> > There’s only been talk so far.
> >
> > I found https://issues.apache.org/jira/browse/BEAM-301 <
> > https://issues.apache.org/jira/browse/BEAM-301>. Does that match what
> you
> > were thinking? If so, let’s continue the conversation there.
> >
> > Julian
> >
> > > On Apr 5, 2017, at 3:13 PM, Khai Tran 
> > wrote:
> > >
> > > Hi,
> > > Just want to check if there is any ongoing effort to convert Calcite
> > > logical plan into Apache Beam APIs for stream processing?
> > >
> > > Thanks,
> > > Khai
> >
> >
>



-- 

Mingmin

Re: Help with planner optimize performance issue

[Ted Xu]

Hi JD,

If you're using Volcano planner, it is common sense that applying rules
will expand search space. However, if you put your rules carefully, the
space expansion is linear to rule apply and rels. According to your test
result, it's better if you can check your rules to see if there is
exponential expansion (e.g., unlimited join ordering).

There are still 2 ways to narrow down your search space,

1. Set RelNode importantce = 0 during optimzation, which tells the Volcano
planner to cease further exploration (rule apply) on the very RelNode. You
should use it carefully because it may cause CannotPlanException.
2. Use HepPlanner instead of VolcanoPlanner. Instead of retaining all
optional plan, HepPlanner will heuristically choose one route at each rule
apply. The search space is very limited but you may get sub-optimal plan.

Regards,
Ted Xu

On Wed, Aug 9, 2017 at 9:53 AM JD Zheng  wrote:

> Hi,
>
> We use RelBuilder to build logical relational algebra trees for our domain
> specific language. It is possible that the user might construct a
> relational algebra tree with several “where” clause. In this case, the
> RelBuilder will build a relnode tree with multiple filters one followed by
> another back to back. We noticed that the optimize time for such tree grows
> exponentially. For example, with 1 where clause, normally it took a few ms.
> However, with 7 where clauses, the optimize time will take almost 1 minute.
>
> Before I dig into the planner, does anyone encounter similar performance
> issue or have any suggestion as what to look into? Thanks,
>
> -JD

Re: Usage of RelTrait and RelTraitDef

[Ted Xu]

Hi Renjie,

RelTraitDef is the definition or template of RelTrait, while the RelTrait
concept corresponds to the concept of 'physical property' in *The Volcano
Optimizer Generator: Extensibility and Efficient Search*, by G Graefe.

Regards,
Ted Xu

On Wed, Aug 9, 2017 at 5:02 PM Renjie Liu  wrote:

> Hi, all:
> I'm reading calcite's source code and confused with these two classes. Why
> these two classes exists and what's their role in query optimizer process?
> --
> Liu, Renjie
> Software Engineer, MVAD
>

Re: Proposal to extend Calcite into a incremental query optimizer

[Forward Xu]

Thanks for sharing +1


forward

Albert  于2021年5月14日周五 下午2:02写道：

> Thanks for sharing
>
>
> 在 2021年5月14日星期五，Julian Hyde  写道：
>
> > During the meeting we agreed to start progressing this contribution in
> the
> > usual Apache Way, with conversations on the dev list and in the
> > https://issues.apache.org/jira/browse/CALCITE-4568 <
> > https://issues.apache.org/jira/browse/CALCITE-4568> JIRA case. So, it
> > should be easy for you to participate.
> >
> > Botong said he would share the slides. (He might be unwilling to make
> them
> > public, because they are his presentation for a conference that has not
> > happened yet. Reach out to him one-to-one.)
> >
> > Next step is for someone on the Alibaba side to create a PR that is
> > rebased on the latest Calcite master, and add a comment to the JIRA case.
> > Then we can discuss what needs to be done for that PR. Code quality,
> adding
> > comments, breaking up into smaller commits, additional tests, renaming
> > packages/classes, restructuring into plugins are all possibilities.
> >
> > Our side of the bargain, as committers, is that we should review in a
> > timely manner, and not move the goal posts — if the contributors make the
> > changes we request then we will land this code in master in a reasonable
> > amount of time.
> >
> > We also discussed incremental view maintenance (IVM). Tempura solves a
> > more general problem (finding the optimal K steps to maintain a
> > materialized view as data arrives in K points in time) but if we set K=2,
> > we can generate a plan for how to update a materialized view given a
> delta
> > table. The plan will be different based on cost - e.g. whether the delta
> > table is small or large. This is a problem that many of our users would
> > like to solve. It will exercise much of Tempura’s code base, and
> encourage
> > contributions.
> >
> > In my opinion, we should do IVM at launch. It should be the main example
> > we use in conference talks, blog posts, etc. When people understand that
> > case, we can explain how we generalize from K=2 to arbitrary K.
> >
> > Julian
> >
> >
> > > On May 13, 2021, at 9:51 AM, Rui Wang  wrote:
> > >
> > > I apologize that I had a wrong impression on the meeting time (I
> thought
> > it
> > > should be on Thursday but it is Wednesday). I can follow up your
> meeting
> > > records if you have any.
> > >
> > >
> > > -Rui
> > >
> > > On Tue, May 11, 2021 at 8:17 PM Botong Huang  wrote:
> > >
> > >> Hi all,
> > >>
> > >> This is a reminder that we are going to have our second discussion
> > meeting
> > >> tomorrow at 10-11pm PST. Please find the link below, everyone is
> > welcome to
> > >> join!
> > >>
> > >> Join Zoom Meeting
> > >> https://uci.zoom.us/j/91986206610
> > >> <
> > >> https://www.google.com/url?q=https%3A%2F%2Fuci.zoom.us%2Fj%
> > 2F91986206610&sa=D&source=calendar&usd=2&usg=AOvVaw24sxPtI6hbukCSo3nlQsbn
> > >>>
> > >>
> > >> Meeting ID: 919 8620 6610
> > >> One tap mobile
> > >> +16699006833 <(669)%20900-6833>,,91986206610# US (San Jose)
> > >> +12532158782 <(253)%20215-8782>,,91986206610# US (Tacoma)
> > >>
> > >> Dial by your location
> > >>+1 669 900 6833 <(669)%20900-6833> US (San Jose)
> > >>+1 253 215 8782 <(253)%20215-8782> US (Tacoma)
> > >>+1 346 248 7799 <(346)%20248-7799> US (Houston)
> > >>+1 301 715 8592 <(301)%20715-8592> US (Washington DC)
> > >>+1 312 626 6799 <(312)%20626-6799> US (Chicago)
> > >>+1 646 558 8656 <(646)%20558-8656> US (New York)
> > >> Meeting ID: 919 8620 6610
> > >> Find your local number: https://uci.zoom.us/u/acyXcc43Cd
> > >> <
> > >> https://www.google.com/url?q=https%3A%2F%2Fuci.zoom.us%2Fu%
> > 2FacyXcc43Cd&sa=D&source=calendar&usd=2&usg=AOvVaw2W08kj_8hEx44dryeZlXb6
> > >>>
> > >>
> > >> Join by Skype for Business
> > >> https://uci.zoom.us/skype/91986206610
> > >> <
> > >> https://www.google.com/url?q=https%3A%2F%2Fuci.zoom.us%
> > 2Fskype%2F91986206610&sa=D&source=calendar&usd=2&usg=
> > AOvVaw3w0M0YYbcjPyBXzNpyyk0Z
> > >>>
> > >>
> > >> Thanks,
> > >> Botong
> > >>
> > >> On Wed, May 5, 2021 at 9:55 AM Botong Huang  wrote:
> > >>
> > >>> Hi Stamatis and all,
> > >>>
> > >>> Thanks for the interest! Let's tentatively schedule the next meeting
> > next
> > >>> Wednesday at May 12, 10pm-11pm PST then. Please let us know if
> there's
> > >> new
> > >>> needs showing up.
> > >>>
> > >>> Best,
> > >>> Botong
> > >>>
> > >>> On Sun, May 2, 2021 at 2:59 PM Stamatis Zampetakis <
> zabe...@gmail.com>
> > >>> wrote:
> > >>>
> >  Hello,
> > 
> >  I really regret missing the first meeting, sorry about that. I added
> > my
> >  preferences in the document.
> >  I will make sure to attend the next one and help as much as I can.
> > 
> >  I didn't have the chance yet to go over the paper but will try to do
> > it
> >  before the next meeting.
> > 
> >  For me the following dates are more convenient than others so it
> would
> > >> be
> >  nice if we cou

[HELP] Calcite interpreting SQL to Druid DSL Question

[Xu Chi]

Hi team,

I’m trying to integrate Druid with Calcite by using calcite-druid adapter.
After following the tutorial, we could connect Calcite with Druid and fetching 
metadata back.
But when trying to execute query with below SQL, the DSL generated by Calcite 
is translating the __time where clause by using the combination of full time 
period intervals + filters to specify the specified time range.  But this kind 
of interpreting results in a bad query performance.

Is there any way that we could achieve that translating the __time where clause 
directly into interval field as shown below in the “Desired DSL for interval 
field” section?

Thank you.

SQL:
SELECT * FROM xxx WHERE __time > '2021-06-06T00:00:15+08:00' AND __time < 
'2021-06-07T00:00:15+08:00' LIMIT 10

DSL Generated by Calcite:
{
"queryType": "scan",
"dataSource": "xxx",
"intervals": [
"1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z"
],
"filter": {
"type": "and",
"fields": [
{
"type": "expression",
"expression": "(\"__time\" > 
timestamp_parse('2021-06-06T00:00:15\\u002B08:00','','Asia/Shanghai'))"
},
{
"type": "expression",
"expression": "(\"__time\" < 
timestamp_parse('2021-06-07T00:00:15\\u002B08:00','','Asia/Shanghai'))"
}
]
},
"columns": [
"__time",
...
],
"resultFormat": "compactedList",
"limit": 10
}

Desired DSL for the interval field:
"intervals": [ "2021-06-07T00:00:15+08:00/2021-06-08T00:00:15+08:00" ]


Regards,
Chi

Re: [HELP] Calcite interpreting SQL to Druid DSL Question

[Xu Chi]

Hi Ben,

Thanks for your prompt response.
I checked the data schema in Druid and found below definition which specify the 
timestamp column.
If I query with “__time”, it will be the situation which mentioned in previous 
email. And I tried to query with the column name “at”, it shows there’s no such 
column which I think it has been filled into the “__time” column.
Did I missed any steps when setting up the model or define Calcite’s 
DruidSchema which resulting this issue? 
Please advise. Thanks.

{
...
"timestampSpec": {
"column": "at",
"format": "posix"
}
...
}

Regards,
Chi
> 
> On Jun 9, 2021, at 12:01 PM, Ben Krug  wrote:
> 
> I'm coming from the druid side, not a calcite person, but I wanted to ask.
> What tutorial do you mean?  Did you specify a model, and did it specify
> timestampColumn?  (If so, was it __time?)
> 
> I notice that in the druid console, if you use your SQL, it translate
> correctly, as in your desired DSL, so I imagine it's something in the
> configuration, maybe the model configuration?
> 
>> On Tue, Jun 8, 2021 at 8:37 PM Xu Chi  wrote:
>> 
>> Hi team,
>> 
>> I’m trying to integrate Druid with Calcite by using calcite-druid adapter.
>> After following the tutorial, we could connect Calcite with Druid and
>> fetching metadata back.
>> But when trying to execute query with below SQL, the DSL generated by
>> Calcite is translating the __time where clause by using the combination of
>> full time period intervals + filters to specify the specified time range.
>> But this kind of interpreting results in a bad query performance.
>> 
>> Is there any way that we could achieve that translating the __time where
>> clause directly into interval field as shown below in the “Desired DSL for
>> interval field” section?
>> 
>> Thank you.
>> 
>> SQL:
>> SELECT * FROM xxx WHERE __time > '2021-06-06T00:00:15+08:00' AND __time <
>> '2021-06-07T00:00:15+08:00' LIMIT 10
>> 
>> DSL Generated by Calcite:
>> {
>>"queryType": "scan",
>>"dataSource": "xxx",
>>"intervals": [
>>"1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z"
>>],
>>"filter": {
>>"type": "and",
>>"fields": [
>>{
>>"type": "expression",
>>"expression": "(\"__time\" >
>> timestamp_parse('2021-06-06T00:00:15\\u002B08:00','','Asia/Shanghai'))"
>>},
>>{
>>"type": "expression",
>>"expression": "(\"__time\" <
>> timestamp_parse('2021-06-07T00:00:15\\u002B08:00','','Asia/Shanghai'))"
>>}
>>]
>>},
>>"columns": [
>>"__time",
>>...
>>],
>>"resultFormat": "compactedList",
>>"limit": 10
>> }
>> 
>> Desired DSL for the interval field:
>> "intervals": [ "2021-06-07T00:00:15+08:00/2021-06-08T00:00:15+08:00" ]
>> 
>> 
>> Regards,
>> Chi

Re: [ANNOUNCE] New committer: Vladimir Ozerov

[Forward Xu]

Congratulations!


Best，

Forward

Danny Chan  于2021年6月24日周四 上午11:51写道：

> Congrats, Vladimir!
>
> Best,
> Danny Chan
>
> Yanjing Wang  于2021年6月24日周四 上午11:41写道：
>
> > Congrats, Vladimir!
> >
> > Roman Kondakov  于2021年6月24日周四 上午11:22写道：
> >
> > > Congratulations, Vladimir!
> > >
> > > Roman Kondakov
> > >
> > > On 24.06.2021 12:23, 段雄 wrote:
> > > > Congratulations!
> > > >
> > > > XING JIN  于2021年6月24日周四 上午10:21写道：
> > > >
> > > >> Congratulations ~
> > > >>
> > > >> Best,
> > > >> Jin
> > > >>
> > > >> guangyuan wang  于2021年6月24日周四 上午9:50写道：
> > > >>
> > > >>> Congratulations!
> > > >>>
> > > >>> Francis Chuang  于2021年6月24日周四 上午6:39写道：
> > > >>>
> > >  Congrats, Vladimir!
> > > 
> > >  Francis
> > > 
> > >  On 24/06/2021 7:48 am, Haisheng Yuan wrote:
> > > > Congratulations and thanks for your contributions, Vladimir!
> > > >
> > > > Regards,
> > > > Haisheng
> > > >
> > > > On 2021/06/23 21:34:40, Stamatis Zampetakis 
> > > >> wrote:
> > > >> Apache Calcite's Project Management Committee (PMC) has invited
> > > >>> Vladimir
> > > >> Ozerov to
> > > >> become a committer, and we are pleased to announce that he has
> > > >>> accepted.
> > > >>
> > > >> Vladimir is among the few people who know very well the internal
> > >  workings
> > > >> of the
> > > >> Calcite optimizer. He started and participated in many
> discussions
> > > >>> about
> > > >> the core engine and contributed ideas and code for making it
> > better.
> > > >> Moreover, Vladimir has blogged and talked about Calcite in
> various
> > > >> conferences and meetups giving publicity and showcasing the
> > >  capabilities of
> > > >> the project.
> > > >>
> > > >> Vladimir, welcome, thank you for your contributions, and we look
> > >  forward to
> > > >> your
> > > >> further interactions with the community! If you wish, please
> feel
> > > >> free
> > >  to
> > > >> tell
> > > >> us more about yourself and what you are working on.
> > > >>
> > > >> Stamatis (on behalf of the Apache Calcite PMC)
> > > >>
> > > 
> > > >>>
> > > >>
> > > >
> > >
> >
>

Re: [HELP] Calcite interpreting SQL to Druid DSL Question

[Xu Chi]

Hi Ben,

Sorry for the late reply. The Druid is not owned by our team and I’m trying to 
figure out whether it’s resulting from some wrong configuration on the Druid 
side. 
Regarding the tutorial, I’m able to run the tutorial and the problem happened 
when I’m trying to use code querying from some real Druid. There’s no error in 
the tutorial, sorry about the confusion.
Will come back to this thread if we have any new findings. Thanks.

Regards,
Chi

发自我的iPhone

> 在 2021年6月9日，下午11:31，Ben Krug  写道：
> 
> Hi Xu -
> 
> I think it should work the way you expect.  It would help to know the steps
> that you followed to set it up.
> When you say a tutorial, which one?  Do you mean this -
> https://calcite.apache.org/docs/druid_adapter.html ?
> What steps did you take to set it up and run the query?
> 
> Thank you,
> 
> Ben
> 
>> On Tue, Jun 8, 2021 at 10:26 PM Xu Chi  wrote:
>> 
>> Hi Ben,
>> 
>> Thanks for your prompt response.
>> I checked the data schema in Druid and found below definition which
>> specify the timestamp column.
>> If I query with “__time”, it will be the situation which mentioned in
>> previous email. And I tried to query with the column name “at”, it shows
>> there’s no such column which I think it has been filled into the “__time”
>> column.
>> Did I missed any steps when setting up the model or define Calcite’s
>> DruidSchema which resulting this issue?
>> Please advise. Thanks.
>> 
>> {
>>...
>>"timestampSpec": {
>>"column": "at",
>>"format": "posix"
>>}
>>...
>> }
>> 
>> Regards,
>> Chi
>>> 
>>>> On Jun 9, 2021, at 12:01 PM, Ben Krug  wrote:
>>> 
>>> I'm coming from the druid side, not a calcite person, but I wanted to
>> ask.
>>> What tutorial do you mean?  Did you specify a model, and did it specify
>>> timestampColumn?  (If so, was it __time?)
>>> 
>>> I notice that in the druid console, if you use your SQL, it translate
>>> correctly, as in your desired DSL, so I imagine it's something in the
>>> configuration, maybe the model configuration?
>>> 
>>>> On Tue, Jun 8, 2021 at 8:37 PM Xu Chi  wrote:
>>>> 
>>>> Hi team,
>>>> 
>>>> I’m trying to integrate Druid with Calcite by using calcite-druid
>> adapter.
>>>> After following the tutorial, we could connect Calcite with Druid and
>>>> fetching metadata back.
>>>> But when trying to execute query with below SQL, the DSL generated by
>>>> Calcite is translating the __time where clause by using the combination
>> of
>>>> full time period intervals + filters to specify the specified time
>> range.
>>>> But this kind of interpreting results in a bad query performance.
>>>> 
>>>> Is there any way that we could achieve that translating the __time where
>>>> clause directly into interval field as shown below in the “Desired DSL
>> for
>>>> interval field” section?
>>>> 
>>>> Thank you.
>>>> 
>>>> SQL:
>>>> SELECT * FROM xxx WHERE __time > '2021-06-06T00:00:15+08:00' AND __time
>> <
>>>> '2021-06-07T00:00:15+08:00' LIMIT 10
>>>> 
>>>> DSL Generated by Calcite:
>>>> {
>>>>   "queryType": "scan",
>>>>   "dataSource": "xxx",
>>>>   "intervals": [
>>>>   "1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z"
>>>>   ],
>>>>   "filter": {
>>>>   "type": "and",
>>>>   "fields": [
>>>>   {
>>>>   "type": "expression",
>>>>   "expression": "(\"__time\" >
>>>> timestamp_parse('2021-06-06T00:00:15\\u002B08:00','','Asia/Shanghai'))"
>>>>   },
>>>>   {
>>>>   "type": "expression",
>>>>   "expression": "(\"__time\" <
>>>> timestamp_parse('2021-06-07T00:00:15\\u002B08:00','','Asia/Shanghai'))"
>>>>   }
>>>>   ]
>>>>   },
>>>>   "columns": [
>>>>   "__time",
>>>>   ...
>>>>   ],
>>>>   "resultFormat": "compactedList",
>>>>   "limit": 10
>>>> }
>>>> 
>>>> Desired DSL for the interval field:
>>>> "intervals": [ "2021-06-07T00:00:15+08:00/2021-06-08T00:00:15+08:00" ]
>>>> 
>>>> 
>>>> Regards,
>>>> Chi
>> 
>> 
> 

Re: [ANNOUNCE] New committer: Zhaohui Xu

[Forward Xu]

Congratulations, Zhaohui


Forward

JiaTao Tao  于2021年10月12日周二 下午3:18写道：

> Congratulations
>
> Regards!
>
> Aron Tao
>
>
> Alessandro Solimando  于2021年10月7日周四
> 下午3:44写道：
>
> > Congratulations Zhaohui, I have followed some of the contributions on the
> > view-based rewritings, thanks for all the improvements!
> >
> > Best regards,
> > Alessandro
> >
> > On Thu, 7 Oct 2021 at 07:27, Haisheng Yuan  wrote:
> >
> > > Congratulations, Zhaohui, well deserved!
> > >
> > > Haisheng
> > >
> > > On 2021/10/06 21:14:00, Francis Chuang 
> wrote:
> > > > Congratulations!
> > > >
> > > > On 7/10/2021 7:47 am, Stamatis Zampetakis wrote:
> > > > > Apache Calcite's Project Management Committee (PMC) has invited
> > > Zhaohui Xu
> > > > > to
> > > > > become a committer, and we are pleased to announce that they have
> > > accepted.
> > > > >
> > > > > Numbers speak for themselves and Zhaohui has over 30 commits
> already
> > in
> > > > > master
> > > > > and more than 20 open pull requests waiting to get in. Great record
> > so
> > > far
> > > > > including
> > > > > (but not limited to) improvements and fixes in the view based
> > > > > rewriting modules,
> > > > > JSON serialization, metadata, and field trimming.
> > > > >
> > > > > Zhaohui, welcome, thank you for your contributions, and we look
> > > forward to
> > > > > your
> > > > > further interactions with the community! If you wish, please feel
> > free
> > > to
> > > > > tell
> > > > > us more about yourself and what you are working on.
> > > > >
> > > > > Stamatis (on behalf of the Apache Calcite PMC)
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] State of the project 2021

[Forward Xu]

Thank you very much Ruben for his work.

Ruben PMC chair +1


Forwardxu

Haisheng Yuan  于2021年12月11日周六 09:03写道：

> Hi Calcite community members,
>
> It has been 6 years since Calcite graduated to a top level Apache project.
> I am so excited to witness how vivid the community has become and how far
> we have come.
>
> We have seen 2 releases so far for Calcite this year (with another release
> v1.29.0 ongoing), with each release containing a large amount of changes.
> Specifically, there are many new features introduced: the InnoDB Adaptor,
> three-value logic for SEARCH operator, Enumerable MergeUnion operator, the
> UNIQUE sub-query predicate... as well as the improvements to materialized
> view recognition. We won't see Calcite's continuous improvements without
> the community members' collaboration and dedication.
>
> In terms of Calcite Avatica, we had 2 releases for Avatica and 0 releases
> for Avatica Go.  Avatica consistently has fewer contributors and pull
> requests than Calcite, maybe because it has fewer users and the code hasn't
> changed aggressively. But more contributors and code reviewers are highly
> welcome and appreciated.
>
> I am glad to see that we had several technical meetups, not to mention at
> ApacheCon and StangeLoop, to talk about the use cases and improvements that
> we have done to Calcite. Many thanks to those who presented talks to give
> more people opportunities to know and use Calcite, and more importantly, to
> enrich the Calcite community.
>
> It's been a great privilege to be able to serve as Calcite's PMC chair in
> 2021. It's also been a big learning experience for me and I am grateful to
> those who trusted me and helped me in the journey. I am also extremely
> proud of having this opportunity to work with Calcite contributors and
> committers.
>
> Last but not least, we need to discuss who should be the new PMC chair of
> Calcite after I step down in December. I would like to nominate Ruben
> Quesada Lopez as the first candidate in the vote. Ruben has been a great
> contributor to the project. He has shown a lot of care for the Calcite
> project and the community, which I feel would make him a great PMC chair.
>
> To repeat questions from previous years:
> 1) What else are we doing well in the project?
> 2) What areas do we need to do better?
>
> Please take some time to share your thoughts!
>
> Thanks,
> Haisheng
>

Re: [ANNOUNCE] Apache Calcite 1.29.0 released

[Forward Xu]

Thanks very much for being RM for this release, Rui!


Forward


Julian Hyde  于2021年12月27日周一 15:37写道：

> Thanks for being RM, Rui!
>
> I made a few fix-ups:
>  * I made a force push to ensure that
> https://github.com/apache/calcite/commit/cbfe0609edcc4a843d71497f159e3687a834119e
> <
> https://github.com/apache/calcite/commit/cbfe0609edcc4a843d71497f159e3687a834119e>
> is still on the master branch.
>  * I amended
> https://github.com/apache/calcite/commit/fb2e28ae17d28657f86cd78503edc402464a8da7
> <
> https://github.com/apache/calcite/commit/fb2e28ae17d28657f86cd78503edc402464a8da7>
> to fix JdbcTest.java (it asserted that minorVersion < 30, which is no
> longer true)
>  * I revised the release notes, adding "[CALCITE-4547] Support Java 16 and
> 17” to the list of bugs fixed
>  * In the news item, I changed the commit SHA from
> dec167ac18272c0cd8be477d6b162d7a31a62114 to
> cbfe0609edcc4a843d71497f159e3687a834119e. This is important because the
> downloads page is generated from it.
>
> Can someone update the site, and re-generate javadoc?
>
> Julian
>
>
> > On Dec 26, 2021, at 9:17 PM, Francis Chuang 
> wrote:
> >
> > Thanks so much for being RM for this release, Rui!
> >
> > Francis
> >
> > On 27/12/2021 4:16 pm, Rui Wang wrote:
> >> Hi Community,
> >> I am happy to announce that 1.29.0 is released and the Calcite main
> branch
> >> is unlocked now.
> >> -Rui
> >> -- Forwarded message -
> >> From: Rui Wang 
> >> Date: Sun, Dec 26, 2021 at 9:15 PM
> >> Subject: [ANNOUNCE] Apache Calcite 1.29.0 released
> >> To: 
> >> The Apache Calcite team is pleased to announce the release of Apache
> >> Calcite 1.29.0.
> >> Calcite is a dynamic data management framework. Its cost-based
> >> optimizer converts queries, represented in relational algebra, into
> >> executable plans. Calcite supports many front-end languages and
> >> back-end data engines, and includes an SQL parser and, as a
> >> sub-project, the Avatica JDBC driver.
> >> This release comes two months after 1.28.0. It contains contributions
> >> from 23 authors, and resolves 47 issues, comprising of general
> improvements
> >> and bug-fixes. It upgrades log4j2 to 2.17.0 to fix security
> vulnerabilities
> >> like CVE-2021-45105.
> >> You can start using it in Maven by simply updating your dependency to:
> >>   
> >> org.apache.calcite
> >> calcite-core
> >> 1.29.0
> >>   
> >> If you'd like to download the source release, you can find it here:
> >>   https://calcite.apache.org/downloads/
> >> You can read more about the release (including release notes) here:
> >>   https://calcite.apache.org/docs/history.html#v1-29-0
> >> We welcome your help and feedback. For more information on how to
> >> report problems, and to get involved, visit the project website at:
> >>https://calcite.apache.org/
> >> Thanks to everyone involved!
> >> Rui Wang, on behalf of the Apache Calcite Team
>
>

Re: [ANNOUNCE] New Calcite PMC chair: Ruben Q L

[Forward Xu]

Congratulations to Ruben!  Thanks for serving as Chair during the last
year, Haisheng!


forward

Ruben Q L  于2022年1月20日周四 19:38写道：

> Thanks everyone!
> And thank you Haisheng for being our PMC Chair during last year!
>
>
>
> On Thu, Jan 20, 2022 at 8:52 AM Alessandro Solimando <
> alessandro.solima...@gmail.com> wrote:
>
> > Congratulations to Ruben and thanks a lot to Haisheng!
> >
> > On Thu, 20 Jan 2022 at 08:13, 953396112 <13282155...@qq.com.invalid>
> > wrote:
> >
> > > Congratulations to Ruben!  
> > > Thanks for serving as Chair, Haisheng!
> > >
> > >
> > > Best regards,
> > > Zhaohui Xu
> > >
> > >
> > >
> > >
> > > -- 原始邮件 --
> > > 发件人:
> > >   "dev"
> > > <
> > > hy...@apache.org>;
> > > 发送时间: 2022年1月20日(星期四) 上午10:29
> > > 收件人: "dev" > >
> > > 主题: [ANNOUNCE] New Calcite PMC chair: Ruben Q L
> > >
> > >
> > >
> > > Calcite community members,
> > >
> > > I am pleased to announce that we have a new PMC chair and VP as per our
> > > tradition of rotating the chair once a year. I have resigned, and Ruben
> > Q L
> > > was duly elected by the PMC and approved unanimously by the Board.
> > >
> > > Please join me in congratulating Ruben!
> > >
> > > Best,
> > > Haisheng Yuan
> >
>

Re: [ANNOUNCE] Chunwei Lei joins Calcite PMC

[Forward Xu]

Congratulations, Chunwei!


Best,

Forwardxu

Stamatis Zampetakis  于2022年5月25日周三 14:46写道：

> Congratulations Chunwei. You have been doing a lot of work for the project
> and this is very much appreciated!
> Thanks for all your efforts.
>
> Best,
> Stamatis
>
> On Wed, May 25, 2022 at 8:32 AM Haisheng Yuan  wrote:
>
> > Congratulations, Chunwei!
> >
> > On 2022/05/25 06:26:35 Michael Mior wrote:
> > > Congratulations and thank you Chunwei!
> > >
> > > --
> > > Michael Mior
> > > mm...@apache.org
> > >
> > >
> > > Le mar. 24 mai 2022 à 16:47, Ruben Q L  a écrit :
> > >
> > > > I am pleased to announce that Chunwei has accepted an invitation to
> > join
> > > > the
> > > >  Calcite PMC. Chunwei has been a consistent and helpful figure in the
> > > > Calcite community for which we are very grateful. We look forward to
> > the
> > > > continued contributions and support.
> > > >
> > > > Please join me in congratulating Chunwei!
> > > >
> > > > - Ruben (on behalf of the Calcite PMC)
> > > >
> > >
> >
>

Re: [ANNOUNCE] Vladimir Ozerov joins Calcite PMC

[Forward Xu]

Congratulations Vladimir!


Best,

ForwardXu

Stamatis Zampetakis  于2022年5月25日周三 15:05写道：

> Congratulations Vladimir. You have shown that you care for the project in
> many different ways and it's only natural to see you in the PMC.
>
> Best,
> Stamatis
>
> On Wed, May 25, 2022 at 8:33 AM Haisheng Yuan  wrote:
>
> > Congratulations Vladimir!
> >
> > On 2022/05/25 06:27:16 Michael Mior wrote:
> > > Congratulations Vladimir!
> > >
> > > --
> > > Michael Mior
> > > mm...@apache.org
> > >
> > >
> > > Le mar. 24 mai 2022 à 16:47, Ruben Q L  a écrit :
> > >
> > > > I am pleased to announce that Vladimir has accepted an invitation to
> > join
> > > > the Calcite PMC. Vladimir has been a consistent and helpful figure in
> > the
> > > > Calcite community for which we are very grateful. We look forward to
> > the
> > > > continued contributions and support.
> > > >
> > > > Please join me in congratulating Vladimir!
> > > >
> > > > - Ruben (on behalf of the Calcite PMC)
> > > >
> > >
> >
>

Re: [ANNOUNCE] New committer: Jing Zhang

[Forward Xu]

Congratulations!


Best,

ForwardXu

Benchao Li  于2022年7月4日周一 13:11写道：

> Congratulations Jing!
>
> Francis Chuang  于2022年7月4日周一 10:43写道：
>
> > Congrats!
> >
> > On 4/07/2022 12:42 pm, Haisheng Yuan wrote:
> > > Apache Calcite's Project Management Committee (PMC) has invited Jing
> > Zhang
> > > to
> > > become a committer, and we are pleased to announce that she has
> accepted.
> > >
> > > Since Dec 2017, Jing has been an active and continuous contributor to
> the
> > > Apache
> > > Calcite project. She has pushed high quality patches, fixing and
> > improving
> > > code
> > > around streaming plan.
> > >
> > > Jing, welcome, thank you for your contributions, and we look forward to
> > your
> > > further interactions with the community! If you wish, please feel free
> to
> > > tell
> > > us more about yourself and what you are working on.
> > >
> > > Haisheng (on behalf of the Apache Calcite PMC)
> > >
> >
>
>
> --
>
> Best,
> Benchao Li
>

Re: [ANNOUNCE] New committer: Benchao Li

[Forward Xu]

Congratulations!


Best,

ForwardXu

Francis Chuang  于2022年7月4日周一 10:21写道：

> Congrats!
>
> On 4/07/2022 12:13 pm, Jiajun Xie wrote:
> > Congrat Benchao Li,  Well deserved!
> >
> > On Mon, 4 Jul 2022 at 09:21, Haisheng Yuan  wrote:
> >
> >> Apache Calcite's Project Management Committee (PMC) has invited Benchao
> Li
> >> to
> >> become a committer, and we are pleased to announce that he has accepted.
> >>
> >> Benchao has pushed a lot of high quality patches, fixing and improving
> code
> >> around plan simplification and rules. Apart from code contributions, he
> has
> >> been regularly involved in the discussion and helping out others in the
> >> mailing
> >> list.
> >>
> >> Benchao, welcome, thank you for your contributions, and we look forward
> to
> >> your
> >> further interactions with the community! If you wish, please feel free
> to
> >> tell
> >> us more about yourself and what you are working on.
> >>
> >> Haisheng (on behalf of the Apache Calcite PMC)
> >>
> >
>

Re: [DISCUSS] How we choose a PMC chair

[Forward Xu]

+1

Haisheng Yuan  于2022年7月4日周一 08:32写道：

> +1 to the improvement.
>
> On 2022/07/04 00:13:58 Francis Chuang wrote:
> > +1 I think this is a good idea. We have a lot of capable PMC members and
> > it would be of great benefit to the project if all of them were
> > considered during the PMC chair selection process.
> >
> > On 4/07/2022 9:46 am, Julian Hyde wrote:
> > > As you know, Calcite has a tradition of choosing a new PMC chair (VP)
> > > each year, around the anniversary of the project's graduation[1][2]. I
> > > think this is a great tradition, but I'd like to discuss an
> > > improvement to that process.
> > >
> > > (I'm starting the conversation now - several months after the previous
> > > vote, and several months before the next - so that it's clear that I
> > > am not criticizing the process or the outcome or previous votes.)
> > >
> > > I've noticed that the outgoing chair sends an email on dev@ saying
> > > words to the following effect:
> > >
> > >I think Xyz would be a great person to succeed me.
> > >What do you all think?
> > >
> > > (I fear that I may have started this tradition when, at the end of my
> > > tenure as first chair, I approached Jesus and asked him whether he'd
> > > be prepared to do the job[3]. Mea culpa.)
> > >
> > > After such an outright endorsement, especially on a public list, it
> > > would be churlish for someone to reply "Actually, I think Abc would be
> > > better." As a result, it's rather difficult to have an open debate,
> > > and the candidate selected by the outgoing chair tends to win
> > > unopposed.
> > >
> > > I suggest that the outgoing chair says something like
> > >
> > >It's time to change the PMC chair.
> > >Please send nominations to private@ and the PMC will discuss and
> vote.
> > >
> > > That would allow for several nominations, allow people to give reasons
> > > why they prefer a candidate (without disparaging other candidates),
> > > and lead to a more informed outcome.
> > >
> > > What do you think? Are there any other aspects of the election process
> > > we should change?
> > >
> > > Julian
> > >
> > > [1] https://lists.apache.org/thread/rmj9qm9wlol3nb7z4phddoljbgvypkrt
> > > [2] https://lists.apache.org/thread/5tzb8w655pj2vo9omz20th5jnbn9zww7
> > > [3] https://lists.apache.org/thread/y4wjdj5h1y3sypnlmhpoz9r6bkk3cv6o
> >
>

Re: Apache Calcite add materialized view recommend module

[Forward Xu]

Hi feiheng,


+1 for this.


Forwardxu

Benchao Li  于2022年11月28日周一 20:58写道：

> Hi feiheng,
>
> It's great to hear that you want to contribute this to the community, I'm
> very interested in this.
>
> Is there any design doc or code branch or something that we can see now?
>
> feiheng ye  于2022年11月28日周一 13:47写道：
>
> > -- Forwarded message -
> > 发件人： Julian Hyde 
> > Date: 2022年11月22日周二 03:03
> > Subject: Re: Apache Calcite add materialized view recommend module
> > To: feiheng ye 
> >
> >
> > One-to-one emails are not very efficient. Can you please send to
> > dev@calcite
> > ?
> >
> >
> > > On Nov 17, 2022, at 5:57 PM, feiheng ye 
> > wrote:
> > >
> > > Hi jhyde.
> > >
> > > Nice to meet you.
> > >
> > > I get the fork from github and secondly develop the apache calcite. I
> > found the materialize view function is imperfect and I want to add
> > materialized view auto recommend module.
> > >
> > >
> > > 1. The module input is source RelNode and output is List.
> > > 2. The source RelNode maybe converted by user source sql.
> > > 3. The output  List is recommend materialized views which user
> > can use them to accelerate the source sql.
> > >
> > >
> > > This function is already complete, does the members of Apache Calcite
> > want to add it?
> > >
> > >
> > > Thanks Regards.
> >
>
>
> --
>
> Best,
> Benchao Li
>

Re: Feature: Support gremlin adapter

[Forward Xu]

hi

This is a great feature to extend calcite from regular data queries to
graph queries (calculations),
+1 for looking forward to it.

forwardxu

柳尘  于2023年12月24日周日 11:20写道：

> Motivation
>
> Hi, community. Currently, more and more users are using some graph
> databases, such as JanusGraph, HugeGraph, etc.
> To do some relationship representation of personnel social networks,
> It is used to count the activity of each user in the social network. Most
> graph databases are in the graph building and graph traversal stage.
> All will be implemented using Gremlin syntax.
> However, this is very unfriendly to users who are not familiar with gremlin
> syntax. While calcite exists as a query framework,
> It also provides an adapter interface to adapt to different database
> dialects, such as parsing, relational algebra conversion, and query plan
> binding.
> Our company has solved the problem of adapting various graph databases.
> This is my warehouse: https://github.com/kaori-seasons/calcite-gremlin-sql
>
>
> Background
>
> Calcite itself supports the database language expansion of the adapter,
> which enables users to understand the meaning of the grammar.
> Complete the simplification of the dialect. For example, expand SqlNode to
> implement syntax analysis, and expand RelNode to implement logical plan
> mapping.
>
> thinkerpop is an adaptation framework for various graph databases. In this
> framework, gremlin syntax is mentioned for the first time.
> It has now become a universal query layer for graph databases. While
> expanding query statements through calcite’s adapter interface,
> We will also use thinkerpop's universal graph database API to provide
> dialect compatibility for different graph databases.
>
> Give a simple example:
> From
>
> SELECT "key" FROM inttype
>
>  maps to
>
>
> g.V().hasLabel("inttype").group().unfold().select(Column.values).order().by(_.unfold().id()).project("inttype").
>
> by(.project("key").by(.unfold().choose(.has("key"),.values("key"),_.constant("\$%#NULL#
> %\$"
>
>
>
>
>
> The design architecture is divided into three layers.
>
> Analytical syntax layer, relational algebra transformation layer, logical
> plan binding layer.
>
> Parsing syntax layer: In the parsing query statement phase, fields and
> equivalent conditions are split and converted into points and edges.
>
> Relational algebra layer: Split it into a collection of points and edges,
> and convert it into a TableScan during the aggregation/sorting/query stage
> where calcite abstracts it.
> It is convenient to generate query plans based on conditions and field
> information.
> Connection scanning/single table filtering and other methods can be used to
> traverse from any edge/any starting point in the graph
>
> Logical plan binding layer: Bind behaviors such as connection
> scanning/single table filtering/projection to calcite’s planner to generate
> query plans.
>
> The process of generating Gremlin logical plan using select statement:
>
> 1. First of all, all graph databases start from a source point to build the
> graph. We will use the GraphTraversalSource provided by thinkerpop.
> As the origin, extract the syntax of the incoming point and side
> information. This step will be implemented in SqlSchemaGrabber
> 2. Secondly, for select/where/having/order by/group by our plan in the
> parsing phase is as follows:
>
>   - group by: for a point. There are out-degree and in-degree
> attributes in the graph. From the perspective of the data table, it is
> equivalent to converting the table data into IN or OUT.
> These two dimensions are aggregated. This behavior also corresponds to
> the table traversal graph operation. During the graph traversal
> process, fold/unfold tags will be generated, representing the
> direction of graph traversal.
>   - select: For the select operation, the operation of scanning the
> entire table can be regarded as projecting all columns into point
> attributes. The value changes of each column are mapped to the gremlin
> operation of adding points.
>   - where: The filter operation is used in graph computing semantic
> scenarios. It can be regarded as the edges connected by the out-degree
> and in-degree of the filter point, so it does not involve the
> conversion of relational algebra.
>   Instead, it is pushed directly to the logical plan.
>   - order by: In the process of graph traversal, we mentioned that
> fold/unflod will be generated on the path to represent the
> forward/backward direction.
>   If a field is encountered and there is no value that can be sorted,
> we will fall back to the origin of GraphTraversalSource and end the
> sorting operation.
>   If there are values that can be sorted, they will be unified in
> SqlTraversalEngine, in-degree and out-degree will be counted
> separately for aggregation, and then used with group by according to
> label (IN/OUT)
>   - having: The meaning is the same as group by, but the label is
> different (in addition to the IN/OUT co

Re: [ANNOUNCE] New Calcite PMC chair: Benchao Li

[Forward Xu]

Congratulations!

Best,
Forward Xu

Hongyu Guo  于2023年12月22日周五 16:59写道：

> Congratulations Benchao!
> Thanks Stamatis for his great work!
>
>
> On Fri, Dec 22, 2023 at 10:58 AM Ran Tao  wrote:
>
> > Congratulations Benchao!
> > and thanks Stamatis for his great job!
> >
> > Best Regards,
> > Ran Tao
> >
> >
> > Benchao Li  于2023年12月22日周五 10:41写道：
> >
> > > Thank you everyone! And thanks Stamatis for being a great chair, twice!
> > >
> > > I'm very honored to be elected as the chair of Calcite for next year.
> > > This is my first time to take this role, so it means that I need to
> > > learn a lot of new things, I will try to follow the great examples
> > > that the previous chairs have set so far and try my best to fulfil my
> > > duties for next year.
> > >
> > > Jiajun Xie  于2023年12月22日周五 09:33写道：
> > > >
> > > > Congratulations Benchao!
> > > > Thanks Stamatis for his great job!
> > > >
> > > > On Fri, 22 Dec 2023 at 05:01, Julian Hyde  wrote:
> > > >
> > > > > Congratulations, Benchao. And thank you - for your work over many
> > > > > years making our community kinder and healthier. I know you will be
> > an
> > > > > excellent PMC chair.
> > > > >
> > > > > Thank you, Stamatis, for serving as chair not once but twice.
> > > > >
> > > > > Julian
> > > > >
> > > > > On Thu, Dec 21, 2023 at 6:40 AM Enrico Olivelli <
> eolive...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Congratulations !
> > > > > >
> > > > > > Enrico
> > > > > >
> > > > > > Il giorno gio 21 dic 2023 alle ore 15:07 Tanner Clary
> > > > > >  ha scritto:
> > > > > > >
> > > > > > > Congrats, Benchao :)
> > > > > > >
> > > > > > > -Tanner
> > > > > > >
> > > > > > > On Thu, Dec 21, 2023 at 2:33 AM Alessandro Solimando <
> > > > > > > alessandro.solima...@gmail.com> wrote:
> > > > > > >
> > > > > > > > Congratulations Benchao, and thanks again to Stamatis for his
> > > great
> > > > > job!
> > > > > > > >
> > > > > > > > On Thu, 21 Dec 2023 at 11:18, Ruben Q L 
> > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations Benchao!!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Dec 21, 2023 at 10:05 AM Sergey Nuyanzin <
> > > > > snuyan...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congratulations, Benchao!
> > > > > > > > > >
> > > > > > > > > > On Thu, Dec 21, 2023 at 10:47 AM Francis Chuang <
> > > > > > > > > francischu...@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Congratulations, Benchao!
> > > > > > > > > > >
> > > > > > > > > > > On 21/12/2023 8:24 pm, Guangdong Liu wrote:
> > > > > > > > > > > > Congratulations!
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > >
> > > > > > > > > > > > Best Regards
> > > > > > > > > > > >
> > > > > > > > > > > > 
> > > > > > > > > > > >
> > > > > > > > > > > > Liugddx
> > > > > > > > > > > > liug...@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Stamatis Zampetakis 
> > 于2023年12月21日周四
> > > > > 17:18写道：
> > > > > > > > > > > >
> > > > > > > > > > > >> Calcite community members,
> > > > > > > > > > > >>
> > > > > > > > > > > >> I am pleased to announce that we have a new PMC
> chair
> > > and
> > > > > VP as
> > > > > > > > per
> > > > > > > > > > > >> our tradition of rotating the chair once a year. I
> > have
> > > > > resigned,
> > > > > > > > > and
> > > > > > > > > > > >> Benchao Li was duly elected by the PMC and approved
> > > > > unanimously by
> > > > > > > > > the
> > > > > > > > > > > >> Board.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Please join me in congratulating Benchao!
> > > > > > > > > > > >>
> > > > > > > > > > > >> Best regards,
> > > > > > > > > > > >> Stamatis
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Best regards,
> > > > > > > > > > Sergey
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > >
> > >
> > >
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> > >
> >
>

Re: Multi-query optimization

[Forward Xu]

Thanks to the community for the related work that has been done, This is a
good discussion,  and I think it can be used in, for example, better
multi-query optimization algorithms, more effective query result caching
strategies, smarter materialized view selection and maintenance methods,
etc. These directions require further research and exploration.

Finally here is a related paper MULTI-QUERY OPTIMIZATION AND APPLICATIONS]

.

Best，
ForwardXu

Julian Hyde  于2024年1月4日周四 04:09写道：

> Multi-query optimization is one of the big challenges of our field.
> Examples of multi-queries:
>  * an INSERT statement that writes into a table but also updates an index,
>  * a DAG that represents an ETL/ELT job;
>  * a query that produces several data sets (say a list of invoices for
> orders and a list of products that need to be restocked);
>  * a query that uses intermediate results more than once.
>
> Common features of multi-queries are multiple output data sets, and
> re-used intermediate results. In other words, the dataflow graph is a
> DAG rather than just a tree.
>
> I think it would be useful to frame the problem by extending SQL so
> that such multi-queries can be represented as a single unit. From that
> would follow extensions to relational algebra, and improvements to
> planner algorithms and cost models.
>
> I would like to hear people's thoughts before I log a Jira case with a
> sketch of the problem.
>
> Related work:
>  * https://issues.apache.org/jira/browse/CALCITE-481 Spool operator
>  * https://issues.apache.org/jira/browse/CALCITE-1440 Multiple RelNodes
>  * https://issues.apache.org/jira/browse/CALCITE-4568 Incremental
> query optimization
>  * https://issues.apache.org/jira/browse/CALCITE-129 Recursive queries
>
> Julian
>

Re: [MINOR]

[Forward Xu]

If the fix is very small and there is no related work order or issue, we
can briefly describe the problem and explain the reason for the fix when
submitting the PR. If we need to standardize this type of PR submission, we
can form some PR templates or documentation.

Best,
ForwardXu

Julian Hyde  于2024年1月4日周四 04:21写道：

> It's so difficult to agree on a vocabulary that everyone will agree
> on, I don't know whether we should even try.
>
> I follow the universal rule: "When in Rome, do as the Romans do".
> Which is to say, follow the existing standard (even if it's
> undocumented, which it probably is) and don't invent your own. If I'm
> about to make a commit that fixes a typo, or fixes lint errors, or
> upgrades a component, I browse the commit log to see what commit
> messages people have used for similar changes.
>
> Julian
>
> On Wed, Jan 3, 2024 at 4:11 AM Benchao Li  wrote:
> >
> > The word [MINOR] is very coarse-grained to me, as Stamatis mentions
> > above, it may contains small improvements to javadoc, site, test,
> > error message, method/variable name, etc. And it is subjective to the
> > author/committer whether a PR should be minor.
> >
> > Looking at the git history, I found it useful when a commit without
> > Jira id starts with "Site:", "Refactor:", "Code style:", "Lint".
> > Maybe we can add more useful prefixes for common small changes, such
> > as "Mailmap: ", "Javadoc: ", "Typo:". For others which are not very
> > common small changes, just a clear/concise commit message without any
> > prefix is good enough.
> >
> > Anyway, we'd better reach an agreement when we can omit Jira id and
> > consider a PR is small.
> >
> > Stamatis Zampetakis  于2024年1月3日周三 18:27写道：
> > >
> > > The presence of the "minor" keyword in the commit summary is a bit
> > > redundant. I would argue that if the message is precise enough the
> > > person reading it can infer it is minor or not.
> > >
> > > Moreover, the minor classification is subjective. Some people consider
> > > minor things that do not change code at all. Others consider minor
> > > things that don't touch production code. The addition of tests and
> > > fixing a broken build is also considered minor by few people. All
> > > these examples are taken from the current commits which landed in
> > > Calcite.
> > >
> > > For the reasons above, I feel that we don't really need to include
> > > "minor" in the commit summary but don't feel very strongly about it
> > > anyways.
> > >
> > > In the past we agreed that if a change is trivial/minor we don't need
> > > a JIRA id. Although, this is convenient for getting this committed
> > > faster without too much hassle it has some drawbacks.
> > >
> > > If at some point in the future we want to attach more information to a
> > > particular commit, this is not possible since we cannot modify the git
> > > history. When there is a JIRA we can always add new comments and
> > > clarifications there even if the ticket is resolved.
> > >
> > > Having a JIRA id is also a convenient way and readable way for
> > > referring to particular commits. The commit hash can be used to
> > > identify a commit in Apache main branch but if the same commit is
> > > backported to other forks/branches using the hash becomes more
> > > cumbersome. This is especially useful in downstream projects and forks
> > > for tracking that a specific change landed in various branches.
> > >
> > > Maybe in the future we should reconsider the optionality of the JIRA
> > > id in the commit summary. If this happens then [CALCITE-X][MINOR]
> > > would start getting overly long so this may be another argument
> > > against including "minor" in the message.
> > >
> > > Best,
> > > Stamatis
> > >
> > > On Wed, Jan 3, 2024 at 10:25 AM Ran Tao  wrote:
> > > >
> > > > This format most likely comes from other open source projects.
> > > > If calcite has its own specifications, such as how to set the title
> for PRs
> > > > that do not require a jira name,
> > > > IMHO, it can be introduced in the contribution doc.
> > > > Commiters can also review PRs according to this specification.
> > > >
> > > > Best Regards,
> > > > Ran Tao
> > > >
> > > >
> > > > Istvan Toth  于2024年1月3日周三 16:11写道：
> > > >
> > > > > Perhaps the square bracket convention ?
> > > > > If the ticket starts with CALICITE-\d+ , then make sure that the
> JIRA
> > > > > ticket id is between brackets.
> > > > >
> > > > > Also check for Gerrit Change IDs which are often added
> automatically, and a
> > > > > paint to remove.
> > > > >
> > > > > Istvan
> > > > >
> > > > > On Tue, Jan 2, 2024 at 10:50 PM Tanner Clary <
> tannercl...@google.com
> > > > > .invalid>
> > > > > wrote:
> > > > >
> > > > > > I like the [MINOR] prefix because it makes it easy to identify
> simple
> > > > > > commits (via grep or ctrl+f), the same way [CALCITE-1234] makes
> it easy
> > > > > to
> > > > > > find commits related to [CALCITE-1234]. I also like that it
> maintains the
> > > > > > "[...]" styling at the be

About SqlFunctions.java slimming in calcite

[Forward Xu]

hi,
I noticed that the code file of the SqlFunctions.java class is getting
larger and larger and the functions are increasing. I think some functions
can be broken down into different function classes to slim down
SqlFunctions. For example: json functions use JsonFunctions, and only
SqlFunctions retain Function name reference.

What do you think?

Best,
ForwardXu

Re: About SqlFunctions.java slimming in calcite

[Forward Xu]

hi Tanner,
Thank you very much for your reply.
In the ISO/IEC SQL standard, scalar functions can be classified into the
following types:
1. String functions: Functions for manipulating string data, such as
CONCAT(), SUBSTRING().
2. Numeric functions: Functions for working with numeric data, such as
ABS(), ROUND().
3. Date and time functions: Functions for manipulating date and time data,
such as DATEADD(), DATEDIFF().
4. System functions: Functions providing information about the database
environment and other system-related details, such as CURRENT_DATE,
CURRENT_USER.

Best,
ForwardXu

Tanner Clary  于2024年1月30日周二 12:21写道：

> I don't think it's a bad idea but I'm also not sure whether it's worth the
> time and effort of deciding what functions should be grouped together, etc.
> Seeing as how methods in that class are mostly (always?) called via
> reflection, I think having them in one place is helpful in ensuring
> consistent code style/patterns. Test classes in Calcite, for instance, vary
> in thoroughness, documentation, formatting, etc, but individual tests in
> the same class rarely vary from one another. That's just from a
> maintenance/consistency perspective, if there's some performance issues or
> otherwise that I'm neglecting then maybe it is worth the attention.
>
> Tanner
>
> On Mon, Jan 29, 2024 at 8:02 PM Forward Xu  wrote:
>
> > hi,
> > I noticed that the code file of the SqlFunctions.java class is getting
> > larger and larger and the functions are increasing. I think some
> functions
> > can be broken down into different function classes to slim down
> > SqlFunctions. For example: json functions use JsonFunctions, and only
> > SqlFunctions retain Function name reference.
> >
> > What do you think?
> >
> > Best,
> > ForwardXu
> >
>

[jira] [Created] (CALCITE-3329) Implement osquery command for OS adapter

[Forward Xu (Jira)]

Forward Xu created CALCITE-3329:
---

 Summary: Implement osquery command for OS adapter
 Key: CALCITE-3329
 URL: https://issues.apache.org/jira/browse/CALCITE-3329
 Project: Calcite
  Issue Type: New Feature
Reporter: Forward Xu


Implement osquery command for OS adapter



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify

[Leonard Xu (Jira)]

Leonard Xu created CALCITE-3368:
---

 Summary: Nonequivalent simplification happens in RexSimplify
 Key: CALCITE-3368
 URL: https://issues.apache.org/jira/browse/CALCITE-3368
 Project: Calcite
  Issue Type: Bug
  Components: core
Affects Versions: 1.21.0
Reporter: Leonard Xu
 Fix For: 1.22.0


When I  do some overflow test ，I found that Calcite's simplification logic of 
isNull expression will convert  from

*"f(operand0, operand1) IS NULL"* to 

*"operand0 IS NULL OR operand1 IS NULL"*  if the Policy of  RexNode‘s SqlKind 
is ANY 。

This simplification will not calculate  the real vale of  *f(operand0, 
operand1)* (eg..'f0 + 'f1 ）which maybe overflow. 
{code:java}
//org.apache.calcite.rex.RexSimplify.java

private RexNode simplifyIsNull(RexNode a) {
 // Simplify the argument first,
 // call ourselves recursively to see whether we can make more progress.
 // For example, given
 // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
 // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
 a = simplify(a, UNKNOWN);
 if (!a.getType().isNullable() && isSafeExpression(a)) {
 return rexBuilder.makeLiteral(false);
 }
 if (RexUtil.isNull(a)) {
 return rexBuilder.makeLiteral(true);
 }
 if (a.getKind() == SqlKind.CAST) {
 return null;
 }
 switch (Strong.policy(a.getKind())) {
 case NOT_NULL:
 return rexBuilder.makeLiteral(false);
 case ANY:
 // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
 // to "operand0 IS NULL OR operand1 IS NULL"
 final List operands = new ArrayList<>();
 for (RexNode operand : ((RexCall) a).getOperands()) {
 final RexNode simplified = simplifyIsNull(operand);
 if (simplified == null) {
 operands.add(
 rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
 } else {
 operands.add(simplified);
 }
 }
 return RexUtil.composeDisjunction(rexBuilder, operands, false);
 case AS_IS:
 default:
 return null;
 }
}{code}
And most of calculating SqlKinds are assigned *Policy.ANY*  at present. 
{code:java}
//org.apache.calcite.plan.Strong.java
public static Policy policy(SqlKind kind) {
  return MAP.getOrDefault(kind, Policy.AS_IS);
}



map.put(SqlKind.PLUS, Policy.ANY);
map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
map.put(SqlKind.MINUS, Policy.ANY);
map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
map.put(SqlKind.TIMES, Policy.ANY);
map.put(SqlKind.DIVIDE, Policy.ANY);

 * that operator evaluates to null. */
public enum Policy {
  /** This kind of expression is never null. No need to look at its arguments,
   * if it has any. */
  NOT_NULL,

  /** This kind of expression has its own particular rules about whether it
   * is null. */
  CUSTOM,

  /** This kind of expression is null if and only if at least one of its
   * arguments is null. */
  ANY,

  /** This kind of expression may be null. There is no way to rewrite. */
  AS_IS,
}{code}
 

It should be an obvious nonequivalent simplification.

And this issue come from Flink 
（[FLINK-14030|https://issues.apache.org/jira/browse/FLINK-14030]）.

 

[~danny0405] could you have a look at this?

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3510) Implement sqlquery for Redis adapter

[Forward Xu (Jira)]

Forward Xu created CALCITE-3510:
---

 Summary: Implement sqlquery for Redis adapter
 Key: CALCITE-3510
 URL: https://issues.apache.org/jira/browse/CALCITE-3510
 Project: Calcite
  Issue Type: New Feature
Reporter: Forward Xu


sqlline> select * from "csv_02";
++---+
| DEPTNO | NAME |
++---+
| 20 | Sales |
| 10 | Sales |
++---+
2 rows selected (0.014 seconds)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3601) Update cassandra, elasticsearch, geode, linq4j, mongodb test upgrade from junit4 to junit5

[Forward Xu (Jira)]

Forward Xu created CALCITE-3601:
---

 Summary: Update cassandra, elasticsearch, geode, linq4j, mongodb 
test upgrade from junit4 to junit5
 Key: CALCITE-3601
 URL: https://issues.apache.org/jira/browse/CALCITE-3601
 Project: Calcite
  Issue Type: Test
Reporter: Forward Xu


Update `cassandra, elasticsearch, geode, linq4j, and mongodb` tests upgrade 
from junit4 to junit5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3622) Update geode tests upgrade from junit4 to junit5

[Forward Xu (Jira)]

Forward Xu created CALCITE-3622:
---

 Summary: Update geode tests upgrade from junit4 to junit5
 Key: CALCITE-3622
 URL: https://issues.apache.org/jira/browse/CALCITE-3622
 Project: Calcite
  Issue Type: Improvement
Reporter: Forward Xu


Update `geode` tests upgrade from junit4 to junit5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3625) Update linq4j tests upgrade from junit4 to junit5

[Forward Xu (Jira)]

Forward Xu created CALCITE-3625:
---

 Summary: Update linq4j tests upgrade from junit4 to junit5
 Key: CALCITE-3625
 URL: https://issues.apache.org/jira/browse/CALCITE-3625
 Project: Calcite
  Issue Type: Improvement
Reporter: Forward Xu


Update `linq4j` tests upgrade from junit4 to junit5



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3637) Update linq4j tests upgrade from junit4 to junit5

[Forward Xu (Jira)]

Forward Xu created CALCITE-3637:
---

 Summary: Update linq4j tests upgrade from junit4 to junit5
 Key: CALCITE-3637
 URL: https://issues.apache.org/jira/browse/CALCITE-3637
 Project: Calcite
  Issue Type: Improvement
Reporter: Forward Xu


Update `linq4j` tests upgrade from junit4 to junit5



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3642) Update cassandra tests upgrade from junit4 to junit5

[Forward Xu (Jira)]

Forward Xu created CALCITE-3642:
---

 Summary: Update cassandra tests upgrade from junit4 to junit5
 Key: CALCITE-3642
 URL: https://issues.apache.org/jira/browse/CALCITE-3642
 Project: Calcite
  Issue Type: Improvement
Reporter: Forward Xu


Update `Cassandra` tests upgrade from junit4 to junit5



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3683) Enhanced MATH Function

[Forward Xu (Jira)]

Forward Xu created CALCITE-3683:
---

 Summary: Enhanced MATH Function
 Key: CALCITE-3683
 URL: https://issues.apache.org/jira/browse/CALCITE-3683
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: Forward Xu






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3684) IMPLEMENT CBRT FUNCTION

[Forward Xu (Jira)]

Forward Xu created CALCITE-3684:
---

 Summary: IMPLEMENT CBRT FUNCTION
 Key: CALCITE-3684
 URL: https://issues.apache.org/jira/browse/CALCITE-3684
 Project: Calcite
  Issue Type: Improvement
Reporter: Forward Xu
Assignee: Forward Xu


IMPLEMENT CBRT FUNCTION.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3685) IMPLEMENT RINT FUNCTION

[Forward Xu (Jira)]

Forward Xu created CALCITE-3685:
---

 Summary: IMPLEMENT RINT FUNCTION
 Key: CALCITE-3685
 URL: https://issues.apache.org/jira/browse/CALCITE-3685
 Project: Calcite
  Issue Type: Improvement
Reporter: Forward Xu
Assignee: Forward Xu


IMPLEMENT RINT FUNCTION



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3686) IMPLEMENT ADDEXACT FUNCTION

[Forward Xu (Jira)]

Forward Xu created CALCITE-3686:
---

 Summary: IMPLEMENT ADDEXACT FUNCTION
 Key: CALCITE-3686
 URL: https://issues.apache.org/jira/browse/CALCITE-3686
 Project: Calcite
  Issue Type: Improvement
Reporter: Forward Xu
Assignee: Forward Xu


IMPLEMENT ADDEXACT FUNCTION



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3687) IMPLEMENT SUBTRACTEXACT FUNCTION

[Forward Xu (Jira)]

Forward Xu created CALCITE-3687:
---

 Summary: IMPLEMENT SUBTRACTEXACT FUNCTION
 Key: CALCITE-3687
 URL: https://issues.apache.org/jira/browse/CALCITE-3687
 Project: Calcite
  Issue Type: Improvement
Reporter: Forward Xu
Assignee: Forward Xu


IMPLEMENT SUBTRACTEXACT FUNCTION



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3689) IMPLEMENT INCREMENTEXACT FUNCTION

[Forward Xu (Jira)]

Forward Xu created CALCITE-3689:
---

 Summary: IMPLEMENT INCREMENTEXACT FUNCTION
 Key: CALCITE-3689
 URL: https://issues.apache.org/jira/browse/CALCITE-3689
 Project: Calcite
  Issue Type: Improvement
Reporter: Forward Xu
Assignee: Forward Xu


IMPLEMENT INCREMENTEXACT FUNCTION



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3688) IMPLEMENT MULTIPLYEXACT FUNCTION

[Forward Xu (Jira)]

Forward Xu created CALCITE-3688:
---

 Summary: IMPLEMENT MULTIPLYEXACT FUNCTION
 Key: CALCITE-3688
 URL: https://issues.apache.org/jira/browse/CALCITE-3688
 Project: Calcite
  Issue Type: Improvement
Reporter: Forward Xu
Assignee: Forward Xu


IMPLEMENT MULTIPLYEXACT FUNCTION



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3690) IMPLEMENT DECREMENTEXACT FUNCTION

[Forward Xu (Jira)]

Forward Xu created CALCITE-3690:
---

 Summary: IMPLEMENT DECREMENTEXACT FUNCTION
 Key: CALCITE-3690
 URL: https://issues.apache.org/jira/browse/CALCITE-3690
 Project: Calcite
  Issue Type: Improvement
Reporter: Forward Xu
Assignee: Forward Xu


IMPLEMENT DECREMENTEXACT FUNCTION



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3691) IMPLEMENT NEGATEEXACT FUNCTION

[Forward Xu (Jira)]

Forward Xu created CALCITE-3691:
---

 Summary: IMPLEMENT NEGATEEXACT FUNCTION
 Key: CALCITE-3691
 URL: https://issues.apache.org/jira/browse/CALCITE-3691
 Project: Calcite
  Issue Type: Improvement
Reporter: Forward Xu
Assignee: Forward Xu


IMPLEMENT NEGATEEXACT FUNCTION



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3692) IMPLEMENT TOINTEXACT FUNCTION

[Forward Xu (Jira)]

Forward Xu created CALCITE-3692:
---

 Summary: IMPLEMENT TOINTEXACT FUNCTION
 Key: CALCITE-3692
 URL: https://issues.apache.org/jira/browse/CALCITE-3692
 Project: Calcite
  Issue Type: Improvement
Reporter: Forward Xu
Assignee: Forward Xu


IMPLEMENT TOINTEXACT FUNCTION



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3693) IMPLEMENT ULP FUNCTION

[Forward Xu (Jira)]

Forward Xu created CALCITE-3693:
---

 Summary: IMPLEMENT ULP FUNCTION
 Key: CALCITE-3693
 URL: https://issues.apache.org/jira/browse/CALCITE-3693
 Project: Calcite
  Issue Type: Improvement
Reporter: Forward Xu
Assignee: Forward Xu


IMPLEMENT ULP FUNCTION



--
This message was sent by Atlassian Jira
(v8.3.4#803005)