Re: [DISCUSS] Draft board report for July 2019

[Michael Mior]

+1 Thanks Francis!
--
Michael Mior
mm...@apache.org

Le mar. 9 juil. 2019 à 00:25, Francis Chuang
 a écrit :
>
> Thanks, Michael! I'll add that into the report. If there are no
> objections, I plan to submit this to the board tonight.
>
> Updated report as follows:
>
> ## Description:
>
> Apache Calcite is a highly customizable framework for parsing and
> planning queries on data in a wide variety of formats. It allows
> database-like access, and in particular a SQL interface and advanced
> query optimization, for data not residing in a traditional database.
>
> Avatica is a sub-project within Calcite and provides a framework for
> building local and remote JDBC and ODBC database drivers. Avatica has an
> independent release schedule and its own repository.
>
> ## Issues:
> There are no issues requiring board attention at this time
>
> ## Activity:
> Development and mailing list activity is steady for both Calcite and its
> Avatica sub-project.
>
> Since the last report, we have added 4 new committers and 1 new PMC
> member. In terms of pull requests and mailing list participation, we are
> also seeing a lot new contributors. In our last report, we also noted an
> improvement in the review of pull requests by new committers as well as
> contributors without committer status. We are seeing a continual
> improvement in this area with most PRs receiving an initial ping from a
> committer or a contributor within the first couple of days after it's
> been opened.
>
> A paper illustrating the use of Calcite for streaming-SQL applications
> was presented by members of the Apache Calcite team in conjunction with
> members of Apache Flink and Apache Beam at SIGMOD 2019.
>
> In terms of releases, Calcite 1.20.0 was released in late June,
> containing more than 130 fixes and enhancements. This makes it one of
> our largest releases ever. We also released Avatica 1.14.0 in late
> April, which was then followed by 1.15.0 in mid-May, due to a regression
> that broke existing behavior expected by Avatica clients. A release of
> Avatica-Go, 4.0.0, was also released in the middle of May.
>
> We are seeing continual momentum in the project and are very happy to
> see new faces, both on the mailing list as well as contributing code via
> pull requests. We believe this is a testament to our welcoming community
> as well as the commercial applicability of the project.
>
> ## Health report:
> Activity levels on mailing lists, git and JIRA are normal for both
> Calcite and Avatica.
>
> ## PMC changes:
>
>   - Currently 20 PMC members.
>   - Stamatis Zampetakis was added to the PMC on Sat Apr 13 2019
>
> ## Committer base changes:
>
>   - Currently 43 committers.
>   - New commmitters:
>  - Chunwei Lei was added as a committer on Sat Apr 27 2019
>  - Danny Chen was added as a committer on Tue May 14 2019
>  - Ruben Q L was added as a committer on Sat Apr 27 2019
>  - Zhiwei Peng was added as a committer on Sat Apr 27 2019
>
> ## Releases:
>
>   - 1.20.0 was released on Mon Jun 24 2019
>   - avatica-1.14.0 was released on Mon Apr 29 2019
>   - avatica-1.15.0 was released on Mon May 13 2019
>   - calcite-avatica-go-4.0.0 was released on Thu May 16 2019
>
> ## JIRA activity:
>
>   - 195 JIRA tickets created in the last 3 months
>   - 161 JIRA tickets closed/resolved in the last 3 months
>
>
> On 8/07/2019 9:40 pm, Michael Mior wrote:
> > Looks good to me! I would suggest also including the SIGMOD paper
> > Julian et al. contributed to.
> >
> > https://dl.acm.org/citation.cfm?id=3314040
> > --
> > Michael Mior
> > mm...@apache.org
> >
> > Le lun. 8 juil. 2019 à 12:09, Volodymyr Vysotskyi
> >  a écrit :
> >>
> >> +1
> >>
> >> Kind regards,
> >> Volodymyr Vysotskyi
> >>
> >>
> >> On Mon, Jul 8, 2019 at 12:57 PM Danny Chan  wrote:
> >>
> >>> Thanks for the report, +999 !
> >>>
> >>> Best,
> >>> Danny Chan
> >>> 在 2019年7月8日 +0800 AM6:38，Francis Chuang ，写道：
>  Attached below is a draft of this month's board report. Please let me
>  know if you have any additions or corrections.
> 
>  ## Description:
> 
>  Apache Calcite is a highly customizable framework for parsing and
>  planning queries on data in a wide variety of formats. It allows
>  database-like access, and in particular a SQL interface and advanced
>  query optimization, for data not residing in a traditional database.
> 
>  Avatica is a sub-project within Calcite and provides a framework for
>  building local and remote JDBC and ODBC database drivers. Avatica has an
>  independent release schedule and its own repository.
> 
>  ## Issues:
>  There are no issues requiring board attention at this time
> 
>  ## Activity:
>  Development and mailing list activity is steady for both Calcite and its
>  Avatica sub-project.
> 
>  Since the last report, we have added 4 new committers and 1 new PMC
>  member. In terms of pull requests and mailing list participation, we are
>  als

Re: Sharing the word at SIGMOD 2019

[Michael Mior]

Thanks for sharing Stamatis. I was the only one there from RIT, so I
suppose in that specific case it was me. However, Fabian, Slim, and
Edmon were also there sharing (and perhaps others I missed as well),
so thanks to all! It was great to see another Calcite-related paper :)
--
Michael Mior
mm...@apache.org

Le mar. 9 juil. 2019 à 00:42, Stamatis Zampetakis  a écrit :
>
> Hello,
>
> I just happened to discuss with a few people who attended SIGMOD this year
> in Amsterdam and I was very happy to learn that they had a brief
> introduction to Calcite by some people from RIT.
>
> I don't know if it was Michael or somebody else from there who helped
> sharing the word but it certainly gives a great boost for expanding the
> community.
>
> Keep it up,
> Stamatis

Re: [DISCUSS] Draft board report for July 2019

[Chunwei Lei]

Great report! +1.

Best,
Chunwei


On Tue, Jul 9, 2019 at 6:45 AM Julian Hyde  wrote:

> > Francis wrote:
> >
> > We are seeing continual momentum in the project and are very happy to
> see new
> > faces, both on the mailing list as well as contributing code via pull
> requests. We
> > believe this is a testament to our welcoming community as well as the
> commercial
> > applicability of the project.
>
> +1
>
> Thanks for writing the report!
>
> Julian
>
>
> > On Jul 8, 2019, at 4:40 AM, Michael Mior  wrote:
> >
> > Looks good to me! I would suggest also including the SIGMOD paper
> > Julian et al. contributed to.
> >
> > https://dl.acm.org/citation.cfm?id=3314040
> > --
> > Michael Mior
> > mm...@apache.org
> >
> > Le lun. 8 juil. 2019 à 12:09, Volodymyr Vysotskyi
> >  a écrit :
> >>
> >> +1
> >>
> >> Kind regards,
> >> Volodymyr Vysotskyi
> >>
> >>
> >> On Mon, Jul 8, 2019 at 12:57 PM Danny Chan 
> wrote:
> >>
> >>> Thanks for the report, +999 !
> >>>
> >>> Best,
> >>> Danny Chan
> >>> 在 2019年7月8日 +0800 AM6:38，Francis Chuang ，写道：
>  Attached below is a draft of this month's board report. Please let me
>  know if you have any additions or corrections.
> 
>  ## Description:
> 
>  Apache Calcite is a highly customizable framework for parsing and
>  planning queries on data in a wide variety of formats. It allows
>  database-like access, and in particular a SQL interface and advanced
>  query optimization, for data not residing in a traditional database.
> 
>  Avatica is a sub-project within Calcite and provides a framework for
>  building local and remote JDBC and ODBC database drivers. Avatica has
> an
>  independent release schedule and its own repository.
> 
>  ## Issues:
>  There are no issues requiring board attention at this time
> 
>  ## Activity:
>  Development and mailing list activity is steady for both Calcite and
> its
>  Avatica sub-project.
> 
>  Since the last report, we have added 4 new committers and 1 new PMC
>  member. In terms of pull requests and mailing list participation, we
> are
>  also seeing a lot new contributors. In our last report, we also noted
> an
>  improvement in the review of pull requests by new committers as well
> as
>  contributors without committer status. We are seeing a continual
>  improvement in this area with most PRs receiving an initial ping from
> a
>  committer or a contributor within the first couple of days after it's
>  been opened.
> 
>  In terms of releases, Calcite 1.20.0 was released in late June,
>  containing more than 130 fixes and enhancements. This makes it one of
>  our largest releases ever. We also released Avatica 1.14.0 in late
>  April, which was then followed by 1.15.0 in mid-May, due to a
> regression
>  that broke existing behavior expected by Avatica clients. A release of
>  Avatica-Go, 4.0.0, was also released in the middle of May.
> 
>  We are seeing continual momentum in the project and are very happy to
>  see new faces, both on the mailing list as well as contributing code
> via
>  pull requests. We believe this is a testament to our welcoming
> community
>  as well as the commercial applicability of the project.
> 
>  ## Health report:
>  Activity levels on mailing lists, git and JIRA are normal for both
>  Calcite and Avatica.
> 
>  ## PMC changes:
> 
>  - Currently 20 PMC members.
>  - Stamatis Zampetakis was added to the PMC on Sat Apr 13 2019
> 
>  ## Committer base changes:
> 
>  - Currently 43 committers.
>  - New commmitters:
>  - Chunwei Lei was added as a committer on Sat Apr 27 2019
>  - Danny Chen was added as a committer on Tue May 14 2019
>  - Ruben Q L was added as a committer on Sat Apr 27 2019
>  - Zhiwei Peng was added as a committer on Sat Apr 27 2019
> 
>  ## Releases:
> 
>  - 1.20.0 was released on Mon Jun 24 2019
>  - avatica-1.14.0 was released on Mon Apr 29 2019
>  - avatica-1.15.0 was released on Mon May 13 2019
>  - calcite-avatica-go-4.0.0 was released on Thu May 16 2019
> 
>  ## JIRA activity:
> 
>  - 195 JIRA tickets created in the last 3 months
>  - 161 JIRA tickets closed/resolved in the last 3 months
> 
> >>>
>
>

[jira] [Created] (CALCITE-3184) Add the new logo to the website

[Stamatis Zampetakis (JIRA)]

Stamatis Zampetakis created CALCITE-3184:


 Summary: Add the new logo to the website
 Key: CALCITE-3184
 URL: https://issues.apache.org/jira/browse/CALCITE-3184
 Project: Calcite
  Issue Type: Task
  Components: site
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis
 Fix For: 1.21.0


Finalize and add the new logo to the website as discussed in the dev list 
[[1|https://lists.apache.org/thread.html/582b4f14dcce117594abba7f36e46725afaa811194def510c246c909@%3Cdev.calcite.apache.org%3E],
 
[2|https://lists.apache.org/thread.html/d6138f938fad7402376718ade286aae50d12d04b63d8ead78685830b@%3Cdev.calcite.apache.org%3E]].




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Streams Query Optimization Using Rate and Window Size

[Stamatis Zampetakis]

Hi Alireza,

Cost models for streams is a very cool topic but I don't have much
knowledge in the domain.

Regarding the implementation details if you have custom physical operators
then it makes sense to implement computeSelfCost() function as you see fit.

Another option is to plug in your custom RelMetadataProvider [1]; you can
find a few examples in RelMetadataTest [2].
That way you can also change the cost function of existing operators
(logical or not) without changing the operators themselves.

As far as it concerns the cost of logical operators the behavior of the
planner can be customized [3].
The most common configuration is to ignore the cost of logical operators so
leaving it as infinite.

Best,
Stamatis

[1]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/metadata/RelMetadataProvider.java
[2]
https://github.com/apache/calcite/blob/e8d598a434e8dbadaf756f8c57c748f4d7e16fdf/core/src/test/java/org/apache/calcite/test/RelMetadataTest.java#L1005

On Tue, Jul 2, 2019 at 11:23 PM Alireza Samadian
 wrote:

> Dear Members of Calcite Community,
>
> I'm working on Apache Beam SQL and we use Calcite for query optimization.
> We represent both tables and streams as a subclass of
> AbstractQueryableTable. In calcite implementation of cost model and
> statistics, one of the key elements is row count. Also all the nodes can
> present a rowcount estimation based on their inputs. For instance, in the
> case of joins, the rowcount is estimated by:
> left.rowCount*right.rowCount*selectivity_estimate.
>
> My first question is, what is the correct way of representing streams in
> Calcite's Optimizer? Does calcite still uses row_count for streams? If so,
> what does row count represent in case of streams?
>
> In [1] they are suggesting to use both window (size of window in terms of
> tuples) and rate to represent output of all nodes in stream processing
> systems, and for every node these two values are estimated. For instance,
> they suggest to estimate window and rate of the joins using:
> join_rate = (left_rate*right_window + right_rate*left_window)*selectivitiy
> join_window = (left_window*right_window)*selectivitiy
>
> We were thinking about using this approach for Beam SQL; however, I am
> wondering where would be the point of extension? I was thinking to
> implement computeSelfCost() using a different cost model (rate,
> window_size) for our physical Rel Nodes, in which we don't call
> estimate_row_count and instead we use inputs' non cumulative cost to
> estimate the node's cost. However, I am not sure if this is a good approach
> and whether this can potentially cause problems in the optimization
> (because there will still be logical nodes that are implemented in calcite
> and may use row count estimation). Does calcite uses cost estimation for
> logical nodes such as logical join? or it only calculates the cost when the
> nodes are physical?
>
> I will appreciate if someone can help me. I will also appreciate if someone
> has other suggestions for streams query optimization.
>
> Best,
> Alireza Samadian
>
> [1] Ayad, Ahmed M., and Jeffrey F. Naughton. "Static optimization of
> conjunctive queries with sliding windows over infinite streams."
> *Proceedings
> of the 2004 ACM SIGMOD international conference on Management of data*.
> ACM, 2004.
>

Re: [DISCUSS] Draft board report for July 2019

[Julian Hyde]

> Francis wrote:
>
> We are seeing continual momentum in the project and are very happy to see new
> faces, both on the mailing list as well as contributing code via pull 
> requests. We
> believe this is a testament to our welcoming community as well as the 
> commercial
> applicability of the project.

+1

Thanks for writing the report!

Julian


> On Jul 8, 2019, at 4:40 AM, Michael Mior  wrote:
> 
> Looks good to me! I would suggest also including the SIGMOD paper
> Julian et al. contributed to.
> 
> https://dl.acm.org/citation.cfm?id=3314040
> --
> Michael Mior
> mm...@apache.org
> 
> Le lun. 8 juil. 2019 à 12:09, Volodymyr Vysotskyi
>  a écrit :
>> 
>> +1
>> 
>> Kind regards,
>> Volodymyr Vysotskyi
>> 
>> 
>> On Mon, Jul 8, 2019 at 12:57 PM Danny Chan  wrote:
>> 
>>> Thanks for the report, +999 !
>>> 
>>> Best,
>>> Danny Chan
>>> 在 2019年7月8日 +0800 AM6:38，Francis Chuang ，写道：
 Attached below is a draft of this month's board report. Please let me
 know if you have any additions or corrections.
 
 ## Description:
 
 Apache Calcite is a highly customizable framework for parsing and
 planning queries on data in a wide variety of formats. It allows
 database-like access, and in particular a SQL interface and advanced
 query optimization, for data not residing in a traditional database.
 
 Avatica is a sub-project within Calcite and provides a framework for
 building local and remote JDBC and ODBC database drivers. Avatica has an
 independent release schedule and its own repository.
 
 ## Issues:
 There are no issues requiring board attention at this time
 
 ## Activity:
 Development and mailing list activity is steady for both Calcite and its
 Avatica sub-project.
 
 Since the last report, we have added 4 new committers and 1 new PMC
 member. In terms of pull requests and mailing list participation, we are
 also seeing a lot new contributors. In our last report, we also noted an
 improvement in the review of pull requests by new committers as well as
 contributors without committer status. We are seeing a continual
 improvement in this area with most PRs receiving an initial ping from a
 committer or a contributor within the first couple of days after it's
 been opened.
 
 In terms of releases, Calcite 1.20.0 was released in late June,
 containing more than 130 fixes and enhancements. This makes it one of
 our largest releases ever. We also released Avatica 1.14.0 in late
 April, which was then followed by 1.15.0 in mid-May, due to a regression
 that broke existing behavior expected by Avatica clients. A release of
 Avatica-Go, 4.0.0, was also released in the middle of May.
 
 We are seeing continual momentum in the project and are very happy to
 see new faces, both on the mailing list as well as contributing code via
 pull requests. We believe this is a testament to our welcoming community
 as well as the commercial applicability of the project.
 
 ## Health report:
 Activity levels on mailing lists, git and JIRA are normal for both
 Calcite and Avatica.
 
 ## PMC changes:
 
 - Currently 20 PMC members.
 - Stamatis Zampetakis was added to the PMC on Sat Apr 13 2019
 
 ## Committer base changes:
 
 - Currently 43 committers.
 - New commmitters:
 - Chunwei Lei was added as a committer on Sat Apr 27 2019
 - Danny Chen was added as a committer on Tue May 14 2019
 - Ruben Q L was added as a committer on Sat Apr 27 2019
 - Zhiwei Peng was added as a committer on Sat Apr 27 2019
 
 ## Releases:
 
 - 1.20.0 was released on Mon Jun 24 2019
 - avatica-1.14.0 was released on Mon Apr 29 2019
 - avatica-1.15.0 was released on Mon May 13 2019
 - calcite-avatica-go-4.0.0 was released on Thu May 16 2019
 
 ## JIRA activity:
 
 - 195 JIRA tickets created in the last 3 months
 - 161 JIRA tickets closed/resolved in the last 3 months
 
>>> 

Sharing the word at SIGMOD 2019

[Stamatis Zampetakis]

Hello,

I just happened to discuss with a few people who attended SIGMOD this year
in Amsterdam and I was very happy to learn that they had a brief
introduction to Calcite by some people from RIT.

I don't know if it was Michael or somebody else from there who helped
sharing the word but it certainly gives a great boost for expanding the
community.

Keep it up,
Stamatis

Re: [DISCUSS] Draft board report for July 2019

[Francis Chuang]

Thanks, Michael! I'll add that into the report. If there are no 
objections, I plan to submit this to the board tonight.


Updated report as follows:

## Description:

Apache Calcite is a highly customizable framework for parsing and 
planning queries on data in a wide variety of formats. It allows 
database-like access, and in particular a SQL interface and advanced 
query optimization, for data not residing in a traditional database.


Avatica is a sub-project within Calcite and provides a framework for 
building local and remote JDBC and ODBC database drivers. Avatica has an 
independent release schedule and its own repository.


## Issues:
There are no issues requiring board attention at this time

## Activity:
Development and mailing list activity is steady for both Calcite and its 
Avatica sub-project.


Since the last report, we have added 4 new committers and 1 new PMC 
member. In terms of pull requests and mailing list participation, we are 
also seeing a lot new contributors. In our last report, we also noted an 
improvement in the review of pull requests by new committers as well as 
contributors without committer status. We are seeing a continual 
improvement in this area with most PRs receiving an initial ping from a 
committer or a contributor within the first couple of days after it's 
been opened.


A paper illustrating the use of Calcite for streaming-SQL applications 
was presented by members of the Apache Calcite team in conjunction with 
members of Apache Flink and Apache Beam at SIGMOD 2019.


In terms of releases, Calcite 1.20.0 was released in late June, 
containing more than 130 fixes and enhancements. This makes it one of 
our largest releases ever. We also released Avatica 1.14.0 in late 
April, which was then followed by 1.15.0 in mid-May, due to a regression 
that broke existing behavior expected by Avatica clients. A release of 
Avatica-Go, 4.0.0, was also released in the middle of May.


We are seeing continual momentum in the project and are very happy to 
see new faces, both on the mailing list as well as contributing code via 
pull requests. We believe this is a testament to our welcoming community 
as well as the commercial applicability of the project.


## Health report:
Activity levels on mailing lists, git and JIRA are normal for both 
Calcite and Avatica.


## PMC changes:

 - Currently 20 PMC members.
 - Stamatis Zampetakis was added to the PMC on Sat Apr 13 2019

## Committer base changes:

 - Currently 43 committers.
 - New commmitters:
- Chunwei Lei was added as a committer on Sat Apr 27 2019
- Danny Chen was added as a committer on Tue May 14 2019
- Ruben Q L was added as a committer on Sat Apr 27 2019
- Zhiwei Peng was added as a committer on Sat Apr 27 2019

## Releases:

 - 1.20.0 was released on Mon Jun 24 2019
 - avatica-1.14.0 was released on Mon Apr 29 2019
 - avatica-1.15.0 was released on Mon May 13 2019
 - calcite-avatica-go-4.0.0 was released on Thu May 16 2019

## JIRA activity:

 - 195 JIRA tickets created in the last 3 months
 - 161 JIRA tickets closed/resolved in the last 3 months


On 8/07/2019 9:40 pm, Michael Mior wrote:

Looks good to me! I would suggest also including the SIGMOD paper
Julian et al. contributed to.

https://dl.acm.org/citation.cfm?id=3314040
--
Michael Mior
mm...@apache.org

Le lun. 8 juil. 2019 à 12:09, Volodymyr Vysotskyi
 a écrit :


+1

Kind regards,
Volodymyr Vysotskyi


On Mon, Jul 8, 2019 at 12:57 PM Danny Chan  wrote:


Thanks for the report, +999 !

Best,
Danny Chan
在 2019年7月8日 +0800 AM6:38，Francis Chuang ，写道：

Attached below is a draft of this month's board report. Please let me
know if you have any additions or corrections.

## Description:

Apache Calcite is a highly customizable framework for parsing and
planning queries on data in a wide variety of formats. It allows
database-like access, and in particular a SQL interface and advanced
query optimization, for data not residing in a traditional database.

Avatica is a sub-project within Calcite and provides a framework for
building local and remote JDBC and ODBC database drivers. Avatica has an
independent release schedule and its own repository.

## Issues:
There are no issues requiring board attention at this time

## Activity:
Development and mailing list activity is steady for both Calcite and its
Avatica sub-project.

Since the last report, we have added 4 new committers and 1 new PMC
member. In terms of pull requests and mailing list participation, we are
also seeing a lot new contributors. In our last report, we also noted an
improvement in the review of pull requests by new committers as well as
contributors without committer status. We are seeing a continual
improvement in this area with most PRs receiving an initial ping from a
committer or a contributor within the first couple of days after it's
been opened.

In terms of releases, Calcite 1.20.0 was released in late June,
containing more than 130 fixes and enhancements. This makes it one o

Calcite-Master - Build # 1247 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Calcite-Master (build #1247)

Status: Still Failing

Check console output at https://builds.apache.org/job/Calcite-Master/1247/ to 
view the results.

[jira] [Created] (CALCITE-3183) trimming method for Filter rel uses wrong traitSet

[Juhwan Kim (JIRA)]

Juhwan Kim created CALCITE-3183:
---

 Summary: trimming method for Filter rel uses wrong traitSet
 Key: CALCITE-3183
 URL: https://issues.apache.org/jira/browse/CALCITE-3183
 Project: Calcite
  Issue Type: Bug
Reporter: Juhwan Kim
Assignee: Juhwan Kim


It seems like there is a bug here: 
https://github.com/apache/calcite/blob/e8d598a434e8dbadaf756f8c57c748f4d7e16fdf/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java#L487.

Unlike other trimming methods, filter trim function copies the current filter 
rel and directly pushes it to the builder instead of calling factory method for 
filter rel. The problem with the current code is that it uses the same traitSet 
even though it would no longer be valid after trimming its input. For example, 
fields in collation might have been updated after trimming. We should reflect 
this change when creating a new rel.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Add Logical filter on a query plan from Flink Table API

[Stamatis Zampetakis]

Hi Felipe,

If you want to build a plan (add a logical filter in the query) then
instead of using rules the most straightforward way would be to use the
RelBuilder [1] API of Calcite or the Table [2] API of Flink as you
mentioned.
On the other hand if you want to do things where you need information about
the whole plan (e.g., the plan must not contain a filter) then the visitor
approach seems more adequate [3, 4].
Creating rules which match anything (i.e., the whole plan) are not that
frequent.

Best,
Stamatis

[1]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java
[2]
https://github.com/apache/flink/blob/master/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java
[3]
https://github.com/apache/calcite/blob/e8d598a434e8dbadaf756f8c57c748f4d7e16fdf/core/src/main/java/org/apache/calcite/rel/RelNode.java#L433
[4]
https://github.com/apache/calcite/blob/e8d598a434e8dbadaf756f8c57c748f4d7e16fdf/core/src/main/java/org/apache/calcite/rel/RelNode.java#L247
On Mon, Jul 8, 2019 at 3:27 PM Felipe Gutierrez <
felipe.o.gutier...@gmail.com> wrote:

> Hi,
>
> I am a newbie in Apache Calcite. I am trying to use it with Apache Flink.
> To start I am trying to create a HelloWorld which just add a logical filter
> on my query.
> 1 - I have my Flink app using Table API [1].
> 2 - I have created my Calcite filter rule which is applied to my FLink
> query if I use CalciteConfig cc = new
>
> CalciteConfigBuilder().addLogicalOptRuleSet(RuleSets.ofList(MyFilterRule.INSTANCE)).build()
> [2];
> 3 - The debug thread only goes to my rule if there is a filter on my query.
>
> I would like to create a logical filter if there is no filter set on the
> logical query. How should I implement it?
> I see my LogicalFilter been created when I call "tableEnv.explain()"
> method. I suppose that I can add some logical filters on the plan.
>
> == Abstract Syntax Tree ==
> LogicalFilter(condition=[>=($6, 50)])
>   LogicalTableScan(table=[[TicketsStation01Plat01]])
>
> == Optimized Logical Plan ==
> DataStreamCalc(select=[sensorId, sensorType, platformId, platformType,
> stationId, timestamp, value, trip, eventTime], where=[>=(value, 50)])
>   StreamTableSourceScan(table=[[TicketsStation01Plat01]], fields=[sensorId,
> sensorType, platformId, platformType, stationId, timestamp, value, trip,
> eventTime], source=[SensorTuples])
>
> == Physical Execution Plan ==
> 
>
> Thanks,
> Felipe
>
> [1]
>
> https://github.com/felipegutierrez/explore-flink/blob/master/src/main/java/org/sense/flink/examples/stream/table/HelloWorldCalcitePlanTableAPI.java#L62
> [2]
>
> https://github.com/felipegutierrez/explore-flink/blob/master/src/main/java/org/sense/calcite/rules/MyFilterRule.java#L14
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez*
> *--* *https://felipeogutierrez.blogspot.com
> *
>

Calcite-Avatica-Master - Build # 166 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Calcite-Avatica-Master (build #166)

Status: Still Failing

Check console output at 
https://builds.apache.org/job/Calcite-Avatica-Master/166/ to view the results.

Calcite-Master - Build # 1246 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Calcite-Master (build #1246)

Status: Still Failing

Check console output at https://builds.apache.org/job/Calcite-Master/1246/ to 
view the results.

Add Logical filter on a query plan from Flink Table API

[Felipe Gutierrez]

Hi,

I am a newbie in Apache Calcite. I am trying to use it with Apache Flink.
To start I am trying to create a HelloWorld which just add a logical filter
on my query.
1 - I have my Flink app using Table API [1].
2 - I have created my Calcite filter rule which is applied to my FLink
query if I use CalciteConfig cc = new
CalciteConfigBuilder().addLogicalOptRuleSet(RuleSets.ofList(MyFilterRule.INSTANCE)).build()
[2];
3 - The debug thread only goes to my rule if there is a filter on my query.

I would like to create a logical filter if there is no filter set on the
logical query. How should I implement it?
I see my LogicalFilter been created when I call "tableEnv.explain()"
method. I suppose that I can add some logical filters on the plan.

== Abstract Syntax Tree ==
LogicalFilter(condition=[>=($6, 50)])
  LogicalTableScan(table=[[TicketsStation01Plat01]])

== Optimized Logical Plan ==
DataStreamCalc(select=[sensorId, sensorType, platformId, platformType,
stationId, timestamp, value, trip, eventTime], where=[>=(value, 50)])
  StreamTableSourceScan(table=[[TicketsStation01Plat01]], fields=[sensorId,
sensorType, platformId, platformType, stationId, timestamp, value, trip,
eventTime], source=[SensorTuples])

== Physical Execution Plan ==


Thanks,
Felipe

[1]
https://github.com/felipegutierrez/explore-flink/blob/master/src/main/java/org/sense/flink/examples/stream/table/HelloWorldCalcitePlanTableAPI.java#L62
[2]
https://github.com/felipegutierrez/explore-flink/blob/master/src/main/java/org/sense/calcite/rules/MyFilterRule.java#L14
*--*
*-- Felipe Gutierrez*

*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
*

[jira] [Created] (CALCITE-3182) Trim unused fields for plan of materialized-view before matching.

[jin xing (JIRA)]

jin xing created CALCITE-3182:
-

 Summary: Trim unused fields for plan of materialized-view before 
matching.
 Key: CALCITE-3182
 URL: https://issues.apache.org/jira/browse/CALCITE-3182
 Project: Calcite
  Issue Type: Improvement
Reporter: jin xing


In current code, before matching query with materialized-view, unused fields of 
query is trimed but materialized-view is not. Thus below simple SQL fails to be 
matched though query and materialized-
view are exactly the same
```
 @Test public void testMaterializationAfterTrimingOfUnusedFields() {
 String sql =
 "select \"y\".\"deptno\", \"y\".\"name\", \"x\".\"sum_salary\"\n" +
 "from\n" +
 " (select \"deptno\", sum(\"salary\") \"sum_salary\" from \"emps\" group by 
\"deptno\") \"x\"\n" +
 " join\n" +
 " \"depts\" \"y\"\n" +
 " on \"x\".\"deptno\"=\"y\".\"deptno\"\n";
 checkMaterialize(sql, sql);
 }
```
Checking \{{CalciteMaterializer}} 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java#L83
 , I think the code intends to do the trimming, but didn't call the method.
Since query is trimed for unused fields in anyway 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L995
 , thus I think there's no n
ecessity to keep the materialized-view un-trimmed



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Draft board report for July 2019

[Michael Mior]

Looks good to me! I would suggest also including the SIGMOD paper
Julian et al. contributed to.

https://dl.acm.org/citation.cfm?id=3314040
--
Michael Mior
mm...@apache.org

Le lun. 8 juil. 2019 à 12:09, Volodymyr Vysotskyi
 a écrit :
>
> +1
>
> Kind regards,
> Volodymyr Vysotskyi
>
>
> On Mon, Jul 8, 2019 at 12:57 PM Danny Chan  wrote:
>
> > Thanks for the report, +999 !
> >
> > Best,
> > Danny Chan
> > 在 2019年7月8日 +0800 AM6:38，Francis Chuang ，写道：
> > > Attached below is a draft of this month's board report. Please let me
> > > know if you have any additions or corrections.
> > >
> > > ## Description:
> > >
> > > Apache Calcite is a highly customizable framework for parsing and
> > > planning queries on data in a wide variety of formats. It allows
> > > database-like access, and in particular a SQL interface and advanced
> > > query optimization, for data not residing in a traditional database.
> > >
> > > Avatica is a sub-project within Calcite and provides a framework for
> > > building local and remote JDBC and ODBC database drivers. Avatica has an
> > > independent release schedule and its own repository.
> > >
> > > ## Issues:
> > > There are no issues requiring board attention at this time
> > >
> > > ## Activity:
> > > Development and mailing list activity is steady for both Calcite and its
> > > Avatica sub-project.
> > >
> > > Since the last report, we have added 4 new committers and 1 new PMC
> > > member. In terms of pull requests and mailing list participation, we are
> > > also seeing a lot new contributors. In our last report, we also noted an
> > > improvement in the review of pull requests by new committers as well as
> > > contributors without committer status. We are seeing a continual
> > > improvement in this area with most PRs receiving an initial ping from a
> > > committer or a contributor within the first couple of days after it's
> > > been opened.
> > >
> > > In terms of releases, Calcite 1.20.0 was released in late June,
> > > containing more than 130 fixes and enhancements. This makes it one of
> > > our largest releases ever. We also released Avatica 1.14.0 in late
> > > April, which was then followed by 1.15.0 in mid-May, due to a regression
> > > that broke existing behavior expected by Avatica clients. A release of
> > > Avatica-Go, 4.0.0, was also released in the middle of May.
> > >
> > > We are seeing continual momentum in the project and are very happy to
> > > see new faces, both on the mailing list as well as contributing code via
> > > pull requests. We believe this is a testament to our welcoming community
> > > as well as the commercial applicability of the project.
> > >
> > > ## Health report:
> > > Activity levels on mailing lists, git and JIRA are normal for both
> > > Calcite and Avatica.
> > >
> > > ## PMC changes:
> > >
> > > - Currently 20 PMC members.
> > > - Stamatis Zampetakis was added to the PMC on Sat Apr 13 2019
> > >
> > > ## Committer base changes:
> > >
> > > - Currently 43 committers.
> > > - New commmitters:
> > > - Chunwei Lei was added as a committer on Sat Apr 27 2019
> > > - Danny Chen was added as a committer on Tue May 14 2019
> > > - Ruben Q L was added as a committer on Sat Apr 27 2019
> > > - Zhiwei Peng was added as a committer on Sat Apr 27 2019
> > >
> > > ## Releases:
> > >
> > > - 1.20.0 was released on Mon Jun 24 2019
> > > - avatica-1.14.0 was released on Mon Apr 29 2019
> > > - avatica-1.15.0 was released on Mon May 13 2019
> > > - calcite-avatica-go-4.0.0 was released on Thu May 16 2019
> > >
> > > ## JIRA activity:
> > >
> > > - 195 JIRA tickets created in the last 3 months
> > > - 161 JIRA tickets closed/resolved in the last 3 months
> > >
> >

Re: [DISCUSS] Draft board report for July 2019

[Volodymyr Vysotskyi]

+1

Kind regards,
Volodymyr Vysotskyi


On Mon, Jul 8, 2019 at 12:57 PM Danny Chan  wrote:

> Thanks for the report, +999 !
>
> Best,
> Danny Chan
> 在 2019年7月8日 +0800 AM6:38，Francis Chuang ，写道：
> > Attached below is a draft of this month's board report. Please let me
> > know if you have any additions or corrections.
> >
> > ## Description:
> >
> > Apache Calcite is a highly customizable framework for parsing and
> > planning queries on data in a wide variety of formats. It allows
> > database-like access, and in particular a SQL interface and advanced
> > query optimization, for data not residing in a traditional database.
> >
> > Avatica is a sub-project within Calcite and provides a framework for
> > building local and remote JDBC and ODBC database drivers. Avatica has an
> > independent release schedule and its own repository.
> >
> > ## Issues:
> > There are no issues requiring board attention at this time
> >
> > ## Activity:
> > Development and mailing list activity is steady for both Calcite and its
> > Avatica sub-project.
> >
> > Since the last report, we have added 4 new committers and 1 new PMC
> > member. In terms of pull requests and mailing list participation, we are
> > also seeing a lot new contributors. In our last report, we also noted an
> > improvement in the review of pull requests by new committers as well as
> > contributors without committer status. We are seeing a continual
> > improvement in this area with most PRs receiving an initial ping from a
> > committer or a contributor within the first couple of days after it's
> > been opened.
> >
> > In terms of releases, Calcite 1.20.0 was released in late June,
> > containing more than 130 fixes and enhancements. This makes it one of
> > our largest releases ever. We also released Avatica 1.14.0 in late
> > April, which was then followed by 1.15.0 in mid-May, due to a regression
> > that broke existing behavior expected by Avatica clients. A release of
> > Avatica-Go, 4.0.0, was also released in the middle of May.
> >
> > We are seeing continual momentum in the project and are very happy to
> > see new faces, both on the mailing list as well as contributing code via
> > pull requests. We believe this is a testament to our welcoming community
> > as well as the commercial applicability of the project.
> >
> > ## Health report:
> > Activity levels on mailing lists, git and JIRA are normal for both
> > Calcite and Avatica.
> >
> > ## PMC changes:
> >
> > - Currently 20 PMC members.
> > - Stamatis Zampetakis was added to the PMC on Sat Apr 13 2019
> >
> > ## Committer base changes:
> >
> > - Currently 43 committers.
> > - New commmitters:
> > - Chunwei Lei was added as a committer on Sat Apr 27 2019
> > - Danny Chen was added as a committer on Tue May 14 2019
> > - Ruben Q L was added as a committer on Sat Apr 27 2019
> > - Zhiwei Peng was added as a committer on Sat Apr 27 2019
> >
> > ## Releases:
> >
> > - 1.20.0 was released on Mon Jun 24 2019
> > - avatica-1.14.0 was released on Mon Apr 29 2019
> > - avatica-1.15.0 was released on Mon May 13 2019
> > - calcite-avatica-go-4.0.0 was released on Thu May 16 2019
> >
> > ## JIRA activity:
> >
> > - 195 JIRA tickets created in the last 3 months
> > - 161 JIRA tickets closed/resolved in the last 3 months
> >
>

Re: [DISCUSS] Draft board report for July 2019

[Danny Chan]

Thanks for the report, +999 !

Best,
Danny Chan
在 2019年7月8日 +0800 AM6:38，Francis Chuang ，写道：
> Attached below is a draft of this month's board report. Please let me
> know if you have any additions or corrections.
>
> ## Description:
>
> Apache Calcite is a highly customizable framework for parsing and
> planning queries on data in a wide variety of formats. It allows
> database-like access, and in particular a SQL interface and advanced
> query optimization, for data not residing in a traditional database.
>
> Avatica is a sub-project within Calcite and provides a framework for
> building local and remote JDBC and ODBC database drivers. Avatica has an
> independent release schedule and its own repository.
>
> ## Issues:
> There are no issues requiring board attention at this time
>
> ## Activity:
> Development and mailing list activity is steady for both Calcite and its
> Avatica sub-project.
>
> Since the last report, we have added 4 new committers and 1 new PMC
> member. In terms of pull requests and mailing list participation, we are
> also seeing a lot new contributors. In our last report, we also noted an
> improvement in the review of pull requests by new committers as well as
> contributors without committer status. We are seeing a continual
> improvement in this area with most PRs receiving an initial ping from a
> committer or a contributor within the first couple of days after it's
> been opened.
>
> In terms of releases, Calcite 1.20.0 was released in late June,
> containing more than 130 fixes and enhancements. This makes it one of
> our largest releases ever. We also released Avatica 1.14.0 in late
> April, which was then followed by 1.15.0 in mid-May, due to a regression
> that broke existing behavior expected by Avatica clients. A release of
> Avatica-Go, 4.0.0, was also released in the middle of May.
>
> We are seeing continual momentum in the project and are very happy to
> see new faces, both on the mailing list as well as contributing code via
> pull requests. We believe this is a testament to our welcoming community
> as well as the commercial applicability of the project.
>
> ## Health report:
> Activity levels on mailing lists, git and JIRA are normal for both
> Calcite and Avatica.
>
> ## PMC changes:
>
> - Currently 20 PMC members.
> - Stamatis Zampetakis was added to the PMC on Sat Apr 13 2019
>
> ## Committer base changes:
>
> - Currently 43 committers.
> - New commmitters:
> - Chunwei Lei was added as a committer on Sat Apr 27 2019
> - Danny Chen was added as a committer on Tue May 14 2019
> - Ruben Q L was added as a committer on Sat Apr 27 2019
> - Zhiwei Peng was added as a committer on Sat Apr 27 2019
>
> ## Releases:
>
> - 1.20.0 was released on Mon Jun 24 2019
> - avatica-1.14.0 was released on Mon Apr 29 2019
> - avatica-1.15.0 was released on Mon May 13 2019
> - calcite-avatica-go-4.0.0 was released on Thu May 16 2019
>
> ## JIRA activity:
>
> - 195 JIRA tickets created in the last 3 months
> - 161 JIRA tickets closed/resolved in the last 3 months
>

[jira] [Created] (CALCITE-3181) Support limit per group in Window

[Haisheng Yuan (JIRA)]

Haisheng Yuan created CALCITE-3181:
--

 Summary: Support limit per group in Window
 Key: CALCITE-3181
 URL: https://issues.apache.org/jira/browse/CALCITE-3181
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: Haisheng Yuan


We have a lot of queries like the following to retrieve top N tuples per group:

{code:java}
SELECT x, y FROM
 (SELECT x, y, ROW_NUMBER() OVER (PARTITION BY x ORDER BY y) 
 AS rn FROM t1) t2 WHERE rn <= 3;
{code}

The performance is not good if each group has a lot more tuples than wanted, 
because we will retrieve and sort all the tuples, instead of just doing a top-N 
heap sort.

In order to do optimization for this kind of query, we need to extend window to 
support limit, if and only if there is only 1 window function, and it is 
{{row_number()}}. We also need a substitute rule to push the limit into window. 
Of course, we also need to modify executor to support this optimization (can be 
later).

Thoughts? Objections?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Calcite-Master - Build # 1245 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Calcite-Master (build #1245)

Status: Still Failing

Check console output at https://builds.apache.org/job/Calcite-Master/1245/ to 
view the results.

Re: [DISCUSS] Draft board report for July 2019

[Stamatis Zampetakis]

Great report Francis! +1

On Mon, Jul 8, 2019 at 12:38 AM Francis Chuang 
wrote:

> Attached below is a draft of this month's board report. Please let me
> know if you have any additions or corrections.
>
> ## Description:
>
> Apache Calcite is a highly customizable framework for parsing and
> planning queries on data in a wide variety of formats. It allows
> database-like access, and in particular a SQL interface and advanced
> query optimization, for data not residing in a traditional database.
>
> Avatica is a sub-project within Calcite and provides a framework for
> building local and remote JDBC and ODBC database drivers. Avatica has an
> independent release schedule and its own repository.
>
> ## Issues:
> There are no issues requiring board attention at this time
>
> ## Activity:
> Development and mailing list activity is steady for both Calcite and its
> Avatica sub-project.
>
> Since the last report, we have added 4 new committers and 1 new PMC
> member. In terms of pull requests and mailing list participation, we are
> also seeing a lot new contributors. In our last report, we also noted an
> improvement in the review of pull requests by new committers as well as
> contributors without committer status. We are seeing a continual
> improvement in this area with most PRs receiving an initial ping from a
> committer or a contributor within the first couple of days after it's
> been opened.
>
> In terms of releases, Calcite 1.20.0 was released in late June,
> containing more than 130 fixes and enhancements. This makes it one of
> our largest releases ever. We also released Avatica 1.14.0 in late
> April, which was then followed by 1.15.0 in mid-May, due to a regression
> that broke existing behavior expected by Avatica clients. A release of
> Avatica-Go, 4.0.0, was also released in the middle of May.
>
> We are seeing continual momentum in the project and are very happy to
> see new faces, both on the mailing list as well as contributing code via
> pull requests. We believe this is a testament to our welcoming community
> as well as the commercial applicability of the project.
>
> ## Health report:
> Activity levels on mailing lists, git and JIRA are normal for both
> Calcite and Avatica.
>
> ## PMC changes:
>
>   - Currently 20 PMC members.
>   - Stamatis Zampetakis was added to the PMC on Sat Apr 13 2019
>
> ## Committer base changes:
>
>   - Currently 43 committers.
>   - New commmitters:
>  - Chunwei Lei was added as a committer on Sat Apr 27 2019
>  - Danny Chen was added as a committer on Tue May 14 2019
>  - Ruben Q L was added as a committer on Sat Apr 27 2019
>  - Zhiwei Peng was added as a committer on Sat Apr 27 2019
>
> ## Releases:
>
>   - 1.20.0 was released on Mon Jun 24 2019
>   - avatica-1.14.0 was released on Mon Apr 29 2019
>   - avatica-1.15.0 was released on Mon May 13 2019
>   - calcite-avatica-go-4.0.0 was released on Thu May 16 2019
>
> ## JIRA activity:
>
>   - 195 JIRA tickets created in the last 3 months
>   - 161 JIRA tickets closed/resolved in the last 3 months
>
>

[jira] [Created] (CALCITE-3180) Bump httpclient from 4.5.6 to 4.5.9

[Fokko Driesprong (JIRA)]

Fokko Driesprong created CALCITE-3180:
-

 Summary: Bump httpclient from 4.5.6 to 4.5.9
 Key: CALCITE-3180
 URL: https://issues.apache.org/jira/browse/CALCITE-3180
 Project: Calcite
  Issue Type: Improvement
  Components: avatica
Reporter: Fokko Driesprong


Bump httpclient from 4.5.6 to 4.5.9



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (CALCITE-3179) Bump Jackson from 2.9.8 to 2.9.9

[Fokko Driesprong (JIRA)]

Fokko Driesprong created CALCITE-3179:
-

 Summary: Bump Jackson from 2.9.8 to 2.9.9
 Key: CALCITE-3179
 URL: https://issues.apache.org/jira/browse/CALCITE-3179
 Project: Calcite
  Issue Type: Improvement
Reporter: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)