Re: [ANNOUNCE] New committers: Chunwei Lei

[Michael Mior]

Congratulations and welcome Chunwei!
--
Michael Mior
mm...@apache.org

Le ven. 26 avr. 2019 à 22:34, Francis Chuang
 a écrit :
>
> Apache Calcite's Project Management Committee (PMC) has invited Chunwei
> Lei to become a committer, and we are pleased to announce that he has
> accepted.
>
> Over the past few months, Chunwei has proposed numerous pull requests,
> reviewed PRs and opened JIRA issues for the project.
>
> Chunwei, welcome, thank you for your contributions, and we look forward
> your further interactions with the community! If you wish, please feel
> free to tell us more about yourself and what you are working on.
>
> Francis (on behalf of the Apache Calcite PMC)

Re: [VOTE] Release apache-calcite-avatica-1.14.0 (release candidate 0)

[Michael Mior]

+1 Checked hashes and signatures and compiled and ran tests. Thanks Francis!
--
Michael Mior
mm...@apache.org

Le jeu. 25 avr. 2019 à 18:18, Francis Chuang
 a écrit :
>
> Hi all,
>
> I have created a build for Apache Calcite Avatica 1.14.0, release
> candidate 0.
>
> Thanks to everyone who has contributed to this release.
>
> You can read the release notes here:
> https://github.com/apache/calcite-avatica/blob/branch-avatica-1.14/site/_docs/history.md
>
> The commit to be voted upon:
> https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=commit;h=4fe0f9b8c7df2aa061caaff12e5ff02ceb8c02c0
>
> Its hash is 4fe0f9b8c7df2aa061caaff12e5ff02ceb8c02c0.
>
> The artifacts to be voted on are located here:
> https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-1.14.0-rc0/
>
> The hashes of the artifacts are as follows:
> src.tar.gz.sha512
> 58b264957da88c43ab1aa650a47b3728f9c94403a29ddf34d57d1fc24e8d4c005b58dd9f81fd6a0e7ce0dddbe6123a98f94b19c8cd760935890df4e78b5476c9
>
> A staged Maven repository is available for review at:
> https://repository.apache.org/content/repositories/orgapachecalcite-1058
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/francischuang.asc
>
> If you do not have a Java environment available, you can run the tests
> using docker. To do so, install docker and docker-compose, then run
> docker-compose run test" from the root of the directory.
>
> Please vote on releasing this package as Apache Calcite Avatica 1.14.0.
>
> The vote is open for the next 72 hours and passes if a majority of
> at least three +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Calcite 1.14.0
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
>
> Here is my vote:
>
> +1 (binding)
>
> Francis

Re: Calcite fuzz testing

[Michael Mior]

It would be interesting if the Tracehash author had a source of bug
reports identified as duplicates along with stack traces to see how
well it works in practice. At this point, it seems like it's just a
heuristic based on an opinion of what's important.
--
Michael Mior
mm...@apache.org

Le ven. 26 avr. 2019 à 15:10, Michael Mior  a écrit :
>
> I could see some might dismiss this as noise, but I really like the
> idea of tracehash and it would be nice to see that catch on. (I think
> it would be interesting if it could be structured something like a
> geohash so truncation would reduce specificity, but it's less obvious
> how to do this here.) Since it takes up minimal space, I would be open
> to considering including it in stack traces.
>
> --
> Michael Mior
> mm...@apache.org
>
> Le ven. 26 avr. 2019 à 14:36, Vladimir Sitnikov
>  a écrit :
> >
> > Let me post a couple of links I've came across today (it comes out of this
> > Twitter thread: https://twitter.com/backendsecret/status/1121290210464034816
> > ):
> >
> > https://github.com/alexknvl/fuzzball -- it is a machine learning driven
> > fuzzer for Scala which identifies quite a few bugs in Scala compiler.
> >
> > The beauty of ML is we don't need to somehow declare the grammar, but it
> > can just learn from lots of samples.
> > I've no idea if that would play well for SQL (we need to declare metadata
> > somehow), however it might still work somehow.
> >
> > Then there's https://github.com/cretz/javan-warty-pig a fuzzer + bytecode
> > agent to trace execution (it remembers the taken paths, so it distinguishes
> > "different" executions.
> >
> > https://github.com/alexknvl/tracehash -- a library that produces short
> > summaries for exception stacktraces.
> > Those signatures might be a good aid for "stackoverflow-guided-development"
> > (== we might want to print stacktrace signatures by default for Calcite
> > exceptions).
> >
> > Vladimir

Re: Calcite fuzz testing

[Michael Mior]

I could see some might dismiss this as noise, but I really like the
idea of tracehash and it would be nice to see that catch on. (I think
it would be interesting if it could be structured something like a
geohash so truncation would reduce specificity, but it's less obvious
how to do this here.) Since it takes up minimal space, I would be open
to considering including it in stack traces.

--
Michael Mior
mm...@apache.org

Le ven. 26 avr. 2019 à 14:36, Vladimir Sitnikov
 a écrit :
>
> Let me post a couple of links I've came across today (it comes out of this
> Twitter thread: https://twitter.com/backendsecret/status/1121290210464034816
> ):
>
> https://github.com/alexknvl/fuzzball -- it is a machine learning driven
> fuzzer for Scala which identifies quite a few bugs in Scala compiler.
>
> The beauty of ML is we don't need to somehow declare the grammar, but it
> can just learn from lots of samples.
> I've no idea if that would play well for SQL (we need to declare metadata
> somehow), however it might still work somehow.
>
> Then there's https://github.com/cretz/javan-warty-pig a fuzzer + bytecode
> agent to trace execution (it remembers the taken paths, so it distinguishes
> "different" executions.
>
> https://github.com/alexknvl/tracehash -- a library that produces short
> summaries for exception stacktraces.
> Those signatures might be a good aid for "stackoverflow-guided-development"
> (== we might want to print stacktrace signatures by default for Calcite
> exceptions).
>
> Vladimir

Re: Function sets (aka flavor and dialect)

[Michael Mior]

I think "function set" sounds like a reasonable name. My current
interpretation of dialect is that it's more related to the SQL syntax
accepted by each system. I'm not really sure what the intended
difference is between dialect and conformance is, but it seems like
perhaps these two concepts could be merged.
--
Michael Mior
mm...@apache.org

Le ven. 26 avr. 2019 à 14:14, Julian Hyde  a écrit :
>
> There’s a discussion in https://issues.apache.org/jira/browse/CALCITE-2846 
> <https://issues.apache.org/jira/browse/CALCITE-2846> about reorganizing the 
> Sql operator table. The idea is for people to be able to start a connection 
> with, say, the standard set of SQL functions, plus functions to emulate 
> MySQL, plus spatial functions. And for us to reorganize the code so that if a 
> function is in both MySQL and Oracle but not in the SQL Standard we only 
> define that function in one place.
>
> We need a word for a "set of functions". (Standard, MySQL, Oracle and Spatial 
> are examples of sets of functions in the above paragraph.) It is tempting to 
> call this a dialect, but that word has an existing meaning that we do not 
> want to change. “Conformance” is another existing concept that we need to 
> work with. I suggested “flavor” in the JIRA case, but now I’m thinking it is 
> an arbitrary word that gives very little clue as to its purpose. The concept 
> is already exposed via the connect-string parameter “fun" (e.g. 
> “jdbc:calcite:fun=spatial,oracle”).
>
> Any ideas for a better word, or a better way of organizing the dialect / 
> conformance / function set concepts.
>
> Julian
>

Re: Google BigQuery - zetasql parser/analyzer

[Michael Mior]

This is basically the point of the Babel parser, to be as liberal as
possible in what SQL is accepted and configurable where things
necessarily conflict between different implementations.

Andrew, could you clarify what you're hoping to contribute to Babel? I
think support for any dialect of SQL will generally be welcomed :)

--
Michael Mior
mm...@apache.org

Le jeu. 25 avr. 2019 à 23:39, Lai Zhou  a écrit :
>
> Good news.
> I found Julian created a new issue about various sql parsing.
>  https://issues.apache.org/jira/browse/CALCITE-3022
> I think it may be helpful to support various sql parsing in a unified
> manner.
> Calcite did use the SqlConformance to enable various SQL compatibility
> modes,
> but it's not enough to solve all the problems.
> Except sql parsing, different sql engines also have differences on
> type inferring,
> type checking and  implicit casting.
>
>
>
>
>
>
>
> Andrew Pilloud  于2019年4月26日周五 上午7:28写道：
>
> > I was intending to send an email about this, thanks for starting the
> > discussion. I'm on the team at Google that is open sourcing ZetaSQL.
> > It is a C++ SQL parser used internally for the BigQuery standard sql
> > parser, among other things.
> >
> > We've open source the Java frontend and Rui currently working on a
> > adapter between ZetaSQL and Calcite for Apache Beam, but we'd probably
> > be interested in contributing it directly to Calcite Bable at some
> > point. Any thoughts on that?
> >
> > Andrew
> >
> > On Thu, Apr 25, 2019 at 4:08 PM Kevin Risden  wrote:
> > >
> > > https://github.com/google/zetasql
> > >
> > > Saw this come by on twitter not too long ago and figured share here since
> > > it definitely overlaps with Calcite.
> > >
> > > Kevin Risden
> >

Re: a new adapter for Apache Kafka

[Michael Mior]

I think we'd generally be willing to accept any adapter which is
well-written, well-tested, and likely to be generally useful. I think
Kafka meets the final criterion and with some work the current code
could meet the first two. That said, even if the adapter isn't
accepted into the main Calcite codebase, I would encourage you to
publish it as a separate package anyway.
--
Michael Mior
mm...@apache.org

Le mer. 24 avr. 2019 à 08:07, Andrei Sereda  a écrit :
>
> Hi Mingmin,
>
> I'm happy to review it.
>
> Before would like to confirm with this list that they're OK adding new
> adapter (kafka) to calcite codebase ?
>
> Regards,
> Andrei.
>
> On Tue, Apr 23, 2019 at 10:43 PM Mingmin Xu  wrote:
>
> > Hi all,
> >
> > I add a new adapter for Apache Kafka, to allow users to query Kafka topics
> > with Calcite STREAM SQL. Can someone take a look at the PR[1]?
> >
> > [1]. https://github.com/apache/calcite/pull/1127
> >
> > Thanks!
> > Mingmin
> >

Re: Connecting Calcite with Apache Phoenix

[Michael Mior]

I took a closer look at this myself and it's more complicated than I
was hoping. You might be able to work with the shaded JAR instead.

https://mvnrepository.com/artifact/org.apache.calcite.avatica/avatica

--
Michael Mior
mm...@apache.org

Le mer. 17 avr. 2019 à 17:50, Rakesh Nair <1994hsekar1...@gmail.com> a écrit :
>
> Hello Michael,
> Would you mind elaborating a bit on how to accomplish this?
>
> Regards,
> Rakesh
>
> On Wed, Apr 17, 2019 at 11:04 PM Michael Mior  wrote:
>
> > This dependency is due to files which were automatically generated
> > with version 3 of the protobuf. You should hopefully be able to
> > regenerate these files with version 2 to remove the dependency.
> > --
> > Michael Mior
> > mm...@apache.org
> >
> > Le mer. 17 avr. 2019 à 03:56, Rakesh Nair <1994hsekar1...@gmail.com> a
> > écrit :
> > >
> > > Hi Chunwei,
> > > I do not believe that is gonna work because, to my understanding, Calcite
> > > has internal dependency on *com/google/protobuf/GeneratedMessageV3 *which
> > > is only available in Protobuf version 3+ onwards. Error trace:-
> > >
> > > *Exception in thread "main" java.lang.NoClassDefFoundError:
> > > com/google/protobuf/GeneratedMessageV3*
> > > * at java.lang.ClassLoader.defineClass1(Native Method)*
> > > * at java.lang.ClassLoader.defineClass(ClassLoader.java:763)*
> > > * at
> > > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)*
> > > * at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)*
> > > * at java.net.URLClassLoader.access$100(URLClassLoader.java:73)*
> > > * at java.net.URLClassLoader$1.run(URLClassLoader.java:368)*
> > > * at java.net.URLClassLoader$1.run(URLClassLoader.java:362)*
> > > * at java.security.AccessController.doPrivileged(Native Method)*
> > > * at java.net.URLClassLoader.findClass(URLClassLoader.java:361)*
> > > * at java.lang.ClassLoader.loadClass(ClassLoader.java:424)*
> > > * at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)*
> > > * at java.lang.ClassLoader.loadClass(ClassLoader.java:357)*
> > > * at
> > >
> > org.apache.calcite.avatica.ConnectionPropertiesImpl.(ConnectionPropertiesImpl.java:38)*
> > > * at org.apache.calcite.avatica.MetaImpl.(MetaImpl.java:72)*
> > > * at
> > > org.apache.calcite.jdbc.CalciteMetaImpl.(CalciteMetaImpl.java:85)*
> > > * at org.apache.calcite.jdbc.Driver.createMeta(Driver.java:169)*
> > > * at
> > >
> > org.apache.calcite.avatica.AvaticaConnection.(AvaticaConnection.java:121)*
> > > *at
> > >
> > org.apache.calcite.jdbc.CalciteConnectionImpl.(CalciteConnectionImpl.java:118)*
> > > * at
> > >
> > org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection.(CalciteJdbc41Factory.java:115)*
> > > * at
> > >
> > org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:59)*
> > > * at
> > >
> > org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:1)*
> > > * at
> > >
> > org.apache.calcite.jdbc.CalciteFactory.newConnection(CalciteFactory.java:53)*
> > > * at
> > >
> > org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)*
> > > * at java.sql.DriverManager.getConnection(DriverManager.java:664)*
> > > * at java.sql.DriverManager.getConnection(DriverManager.java:208)*
> > > *at
> > > org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:153)*
> > > * at org.apache.calcite.tools.RelBuilder.create(RelBuilder.java:209)*
> > >
> > > Is there any other alternate way I can use making this connection?
> > >
> > > Regards,
> > > Rakesh
> > >
> > > On Wed, Apr 17, 2019 at 12:59 PM Chunwei Lei 
> > wrote:
> > >
> > > > Hi, Rakesh
> > > >
> > > > I find that it says Protobuf doesn't provide source/binary
> > > > compatibility across different versions after searching the error
> > > > message using google[1].
> > > >
> > > > Maybe you can try to compile source code of Calcite which uses the
> > > > same version of Protobuf in Phoenix. But I am not sure that it can
> > > > work.
> > > >
> > > >
> > > > [1] https://groups.google.com/forum/#!topic/protobuf/N8rznIMcwWg
> > > >
> > > >
> > > > Bests,
> > > > Chunwei
> > &g

Re: Connecting Calcite with Apache Phoenix

[Michael Mior]

This dependency is due to files which were automatically generated
with version 3 of the protobuf. You should hopefully be able to
regenerate these files with version 2 to remove the dependency.
--
Michael Mior
mm...@apache.org

Le mer. 17 avr. 2019 à 03:56, Rakesh Nair <1994hsekar1...@gmail.com> a écrit :
>
> Hi Chunwei,
> I do not believe that is gonna work because, to my understanding, Calcite
> has internal dependency on *com/google/protobuf/GeneratedMessageV3 *which
> is only available in Protobuf version 3+ onwards. Error trace:-
>
> *Exception in thread "main" java.lang.NoClassDefFoundError:
> com/google/protobuf/GeneratedMessageV3*
> * at java.lang.ClassLoader.defineClass1(Native Method)*
> * at java.lang.ClassLoader.defineClass(ClassLoader.java:763)*
> * at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)*
> * at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)*
> * at java.net.URLClassLoader.access$100(URLClassLoader.java:73)*
> * at java.net.URLClassLoader$1.run(URLClassLoader.java:368)*
> * at java.net.URLClassLoader$1.run(URLClassLoader.java:362)*
> * at java.security.AccessController.doPrivileged(Native Method)*
> * at java.net.URLClassLoader.findClass(URLClassLoader.java:361)*
> * at java.lang.ClassLoader.loadClass(ClassLoader.java:424)*
> * at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)*
> * at java.lang.ClassLoader.loadClass(ClassLoader.java:357)*
> * at
> org.apache.calcite.avatica.ConnectionPropertiesImpl.(ConnectionPropertiesImpl.java:38)*
> * at org.apache.calcite.avatica.MetaImpl.(MetaImpl.java:72)*
> * at
> org.apache.calcite.jdbc.CalciteMetaImpl.(CalciteMetaImpl.java:85)*
> * at org.apache.calcite.jdbc.Driver.createMeta(Driver.java:169)*
> * at
> org.apache.calcite.avatica.AvaticaConnection.(AvaticaConnection.java:121)*
> *at
> org.apache.calcite.jdbc.CalciteConnectionImpl.(CalciteConnectionImpl.java:118)*
> * at
> org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection.(CalciteJdbc41Factory.java:115)*
> * at
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:59)*
> * at
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:1)*
> * at
> org.apache.calcite.jdbc.CalciteFactory.newConnection(CalciteFactory.java:53)*
> * at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)*
> * at java.sql.DriverManager.getConnection(DriverManager.java:664)*
> * at java.sql.DriverManager.getConnection(DriverManager.java:208)*
> *at
> org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:153)*
> * at org.apache.calcite.tools.RelBuilder.create(RelBuilder.java:209)*
>
> Is there any other alternate way I can use making this connection?
>
> Regards,
> Rakesh
>
> On Wed, Apr 17, 2019 at 12:59 PM Chunwei Lei  wrote:
>
> > Hi, Rakesh
> >
> > I find that it says Protobuf doesn't provide source/binary
> > compatibility across different versions after searching the error
> > message using google[1].
> >
> > Maybe you can try to compile source code of Calcite which uses the
> > same version of Protobuf in Phoenix. But I am not sure that it can
> > work.
> >
> >
> > [1] https://groups.google.com/forum/#!topic/protobuf/N8rznIMcwWg
> >
> >
> > Bests,
> > Chunwei
> >
> > Rakesh Nair <1994hsekar1...@gmail.com> 于2019年4月17日周三 下午3:03写道：
> > >
> > > Hi,
> > > Could someone kindly help me with the problem I'm facing in my previous
> > > email? I've been sitting on this for some time now and would really like
> > to
> > > solve it.
> > > For what it's worth, these are the dependencies I'm using,
> > > **
> > > * *
> > > * *
> > > * org.apache.calcite*
> > > * calcite-core*
> > > * 1.19.0*
> > > * *
> > > * *
> > > * *
> > > * org.apache.phoenix*
> > > * phoenix-core*
> > > * 5.0.0.3.0.1.0-187*
> > > * *
> > > * *
> > > * *
> > > * *
> > > * Hortonworks Repository*
> > > * http://repo.hortonworks.com/content/repositories/releases/
> > > <http://repo.hortonworks.com/content/repositories/releases/>*
> > > * *
> > > * *
> > >
> > > Waiting for you response.
> > > Regards,
> > > Rakesh.
> > >
> > > On Tue, Apr 16, 2019 at 7:03 PM Rakesh Nair <1994hsekar1...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > > So I've trying to connect Calcite with Apache Phoenix

Re: [DISCUSS] Draft board report for April 2019

[Michael Mior]

Looks good Francis! Doesn't need to be in the report, but I do like
that I can now easily rebase and merge PRs right from the GitHub web
interface after seeing passing Travis tests. For smallish PRs, I find
that means I don't even need to check out a copy of the code.

--
Michael Mior
mm...@apache.org

Le lun. 1 avr. 2019 à 19:29, Francis Chuang  a écrit :
>
> Good catch, Stamatis! The bit about PRs on Github does seem silly.
>
> Updated report as attached:
>
> ## Description:
>
> Apache Calcite is a highly customizable framework for parsing and
> planning queries on data in a wide variety of formats. It allows
> database-like access, and in particular a SQL interface and advanced
> query optimization, for data not residing in a traditional database.
>
> Avatica is a sub-project within Calcite and provides a framework for
> building local and remote JDBC and ODBC database drivers. Avatica has an
> independent release schedule and its own repository.
>
> ## Issues:
> There are no issues requiring board attention at this time
>
> ## Activity:
> Development and mailing list activity is steady for both Calcite and its
> Avatica sub-project.
>
> We have been seeing a lot of new contributors open pull requests and
> participating in our mailing lists to answer questions and partake in
> discussions. This is most likely due to Calcite being increasingly
> adopted as part of commercial offerings offered by cloud service
> providers as well as being used to build internal tools. The increased
> participation by new contributors is reflected in the 4 new committers
> we have added to the project in the last 3 months.
>
> In the last report, we mentioned that there were some issues with pull
> requests not being reviewed in a timely manner. Over the last few
> months, we have observed new contributors and non-committers reviewing
> open pull requests in our repositories in conjunction with our existing
> committers. This is a great improvement and we hope this will put us on
> the path to stop PRs from getting too stale.
>
> In terms of releases, Calcite 1.19.0 was released at the end of March
> and includes numerous bug-fixes as well as extensive improvements in
> JSON query support.
>
> Finally, the Calcite website repository was moved to git. As a result of
> this move, all of our source repositories are now using GitBox.
>
> ## Health report:
> Activity levels on mailing lists, git and JIRA are normal for both
> Calcite and Avatica.
>
> ## PMC changes:
>
>   - Currently 19 PMC members.
>   - No new PMC members added in the last 3 months
>   - Last PMC addition was Kevin Risden on Mon Jul 09 2018
>
> ## Committer base changes:
>
>   - Currently 39 committers.
>   - New commmitters:
>  - Hongze Zhang was added as a committer on Tue Feb 19 2019
>  - Haisheng Yuan was added as a committer on Tue Mar 26 2019
>  - Zoltan Haindrich was added as a committer on Thu Jan 10 2019
>  - Stamatis Zampetakis was added as a committer on Thu Jan 31 2019
>
> ## Releases:
>
>   - 1.19.0 was released on Tue Mar 26 2019
>
> ## JIRA activity:
>
>   - 208 JIRA tickets created in the last 3 months
>   - 153 JIRA tickets closed/resolved in the last 3 months
>
> On 2/04/2019 10:18 am, Stamatis Zampetakis wrote:
> > Thanks Francis! Two minor things:
> > 3 committers -> 4 committers
> > enabling new contributors to easily submit pull requests via Github -> I
> > don't see how is this related. Contributors could submit PRs through Github
> > even before.
> >
> >
> > Στις Τρί, 2 Απρ 2019 στις 12:35 π.μ., ο/η Francis Chuang <
> > francischu...@apache.org> έγραψε:
> >
> >> Attached below is a draft of this month's board report. Please let me
> >> know if you have any additions or corrections.
> >>
> >> ## Description:
> >>
> >> Apache Calcite is a highly customizable framework for parsing and
> >> planning queries on data in a wide variety of formats. It allows
> >> database-like access, and in particular a SQL interface and advanced
> >> query optimization, for data not residing in a traditional database.
> >>
> >> Avatica is a sub-project within Calcite and provides a framework for
> >> building local and remote JDBC and ODBC database drivers. Avatica has an
> >> independent release schedule and its own repository.
> >>
> >> ## Issues:
> >> There are no issues requiring board attention at this time
> >>
> >> ## Activity:
> >> Development and mailing list activity is steady for both Calcite and its
> >> Avatica sub-project.
> >

Re: [DISCUSS] Automated website builds for Calcite projects

[Michael Mior]

I'm not sure how the builds were set up initially, but AFAIK all the
configuration is in the control panel nd not in the  repo. Would be
great to have this automated :)

--
Michael Mior
mm...@apache.org

Le dim. 31 mars 2019 à 19:07, Francis Chuang
 a écrit :
>
> While moving the website repo from SVN to Git a while back, I briefly
> mentioned the possibility of automating our website builds, so that we
> do not need to manually push the built website assets.
>
> I have done some research regarding ASF's build infrastructure and it
> turns out there is a git-website jenkins node for doing this.
> Unfortunately, there isn't much documentation for this, and the howto
> for this appears to be work in progress:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75964385
>
> Before reaching out to infra and builds I'd like to clarify my
> understanding of how Jenkins builds are currently set up for Calcite,
> before asking builds and infra for further assistance:
>
> - How were the Jenkins builds set up initially? Did INFRA set this up
> for us or was there a self-service system?
>
> - Are the ASF builds configured using exclusively using the control
> panel at https://builds.apache.org/job/Calcite-Master/ or do we have
> other configurations in the repository using configuration files similar
> to .travis.yml?
>
> Francis

Re: reuse of relational expressions

[Michael Mior]

Are you suggesting this only for UNION? I assume not, but I'm not sure I
see how the approach you're suggesting will generalize. Of course, we also
need to consider *how* to reuse the results of the expression.

The approach sounds like it could be reasonable, but I don't fully
understand what you're proposing. In any case, I'm happy to see the
discussion starting again on this!

--
Michael Mior
mm...@apache.org


Le mar. 26 mars 2019 à 08:02, Roger Shi  a
écrit :

> Hi,
>
> I'm investigating how to consider relational expression reuse in Volcano
> cost model. CALCITE-481<https://jira.apache.org/jira/browse/CALCITE-481>
> is a good start point and the JIRA description mentions integer linear
> programming, whose compute complexity is too high in some cases. So I
> propose a simpler method, and let's discuss it here.
>
> For instance, here's example a Union with two inputs.
>
>
> UnionAll
>
> input(0) ==  Filter - Scan
> input(1) ==  Filter - Scan
>
> In currently implementation the cumulative cost of Union is Cost(UnionAll)
> + Cost(Filter) * 2 + Cost(Scan) * 2. If filter and the scan is the same,
> the real cost is Cost(UnionAll) + Cost(Filter) + Cost(Scan). The key point
> is how to detect the reused expression. We could resolve it by recording
> the RelNodes used in cost calculating, and it's easy to find Filter and
> Scan is computed twice.
>
> On one hand, the method should be faster than integer linear programming.
> On the other hand it may miss the "real" optimal plan because it considers
> only local cheapest cost in subset cost computing.
>
> What do you think about the method? Comments are welcome. 🙂
>

Re: Site branch

[Michael Mior]

LGTM. Thanks for sorting this out!
--
Michael Mior
mm...@apache.org

Le mer. 27 mars 2019 à 19:12, Julian Hyde  a écrit :
>
> I saw that Stamatis just did “git push origin a9687de81:site”. That’s 
> somewhat of an improvement, because it removes the merge commit, but it 
> introduces a problem: it includes “25ffeb4ac [CALCITE-2908] Implement SQL 
> LAST_DAY function”, and that commit updates the site with a function that 
> will not be in the product until 1.20.
>
> I’m just about to push 42dce0928. That commit contains only commits that 
> change the site, cherry-picked from master. I hope people view that as an 
> improvement.
>
> Julian
>
> $ git log --graph --abbrev-commit --pretty=oneline   site origin/site --
> * 42dce0928 - (HEAD -> site) Site: Add Alibaba MaxCompute to po
> * 1939f9c68 - Suppress deprecation warning, and remove unicode
> * 819722500 - Site: Add new committers (Haisheng Yuan, Hongze Z
> * a5530e5fb - [CALCITE-2952] Add JDK 12 as tested to 1.19.0 his
> | * a9687de81 - (origin/site, origin/master) [CALCITE-2958] Upg
> | * 650d24b9a - Site: Add Alibaba MaxCompute to powered-by page
> | * ddbcd3955 - Suppress deprecation warning, and remove unicod
> | * 1f4b61989 - [CALCITE-2796] JDBC adapter should convert 'GRO
> | * 4fdf241df - In RelFieldCollation, add a "withX" copy method
> | * 90e69d418 - [CALCITE-2953] LatticeTest.testTileAlgorithm2 a
> | * 11c067f99 - Site: Add new committers (Haisheng Yuan, Hongze
> | * 35ab6c768 - [CALCITE-2952] Add JDK 12 as tested to 1.19.0 h
> | * 1b430721c - [CALCITE-574] Remove org.apache.calcite.util.Bu
> | * 81143c830 - [CALCITE-589] Extend unifyAggregates method to
> | * 25ffeb4ac - [CALCITE-2908] Implement SQL LAST_DAY function
> |/
> * 406129b97 - [CALCITE-2951] Support decorrelate subquery that
> * 2fa7fd79b - [CALCITE-2946] RelBuilder wrongly skips creation
> * ecc100ea2 - [CALCITE-2943] Materialized view rewriting logic
> * 79f432457 - [CALCITE-2942] Materialized view rewriting logic
> * 06b1894db - Site: News item for release 1.19.0 (2 days ago) <
> * b8f4edfcf - (origin/branch-1.19) [maven-release-plugin] prepa
>
>
>
>
> > On Mar 27, 2019, at 3:57 PM, Haisheng Yuan  wrote:
> >
> > +1
> >
> >
> >
> >
> >
> > Thanks~
> > Haisheng 
> > Yuan--
> > 发件人：Francis Chuang
> > 日　期：2019年03月28日 06:49:44
> > 收件人：
> > 主　题：Re: Site branch
> >
> > +1 I think this should reduce the number of "commits ahead" in the site
> > branch compared to master to 0.
> >
> > On 28/03/2019 9:46 am, Julian Hyde wrote:
> >> The site branch currently has a merge commit in it. But traditionally 
> >> after a release the site branch points to the same commit as the master 
> >> branch.
> >>
> >> So, any objections if I reset the site branch, as follows:
> >>
> >> $ git checkout site
> >> $ git reset —hard origin/branch-1.19
> >> $ git push -f origin site
> >>
> >> Then cherry-pick a couple of commits from master that need to go into the 
> >> site.
> >>
> >> To my eyes at least, that creates a simpler & clearer history.
> >>
> >> Julian
> >>
> >>
>

Re: [DISCUSS] Move site repositories from svn to gitbox

[Michael Mior]

Thanks for tracking this Kevin!

--
Michael Mior
mm...@apache.org
Le mar. 26 mars 2019 à 08:18, Kevin Risden  a écrit :
>
> Actually just followed up with infra. The gitpubsub hook should be enabled
> in the next ~hour or less when puppet runs. I will hold off on the SVN
> change and make sure the gitbox calcite-site repo is working.
>
> Kevin Risden
>
>
> On Tue, Mar 26, 2019 at 8:11 AM Kevin Risden  wrote:
>
> > Tried to push to the new gitbox site last night for the 1.19.0 release and
> > ran into some issues. The gitpubsub hook was not setup so any changes to
> > calcite-site on gitbox are not reflected on the site. I pinged INFRA to get
> > this resolved.
> >
> > I am going to push the changes to SVN as well to get the site updated.
> >
> > Kevin Risden
> >
> >
> > On Sun, Feb 17, 2019 at 5:19 PM Francis Chuang 
> > wrote:
> >
> >> @Michael, the svn repo will still be kept, but just unused. See kafka's
> >> old site: https://svn.apache.org/repos/asf/kafka/site/
> >>
> >> I have now pushed a the current working copy of our site to
> >> https://github.com/apache/calcite-site using svn export.
> >>
> >> I have also updated my ticket with infra to ask them to switch the
> >> site's publishing mechanism from svnpubsub to gitpubsub.
> >>
> >> I'll now proceed with updating the publishing instructions for our site
> >> to git.
> >>
> >> On 16/02/2019 5:37 am, Julian Hyde wrote:
> >> > Agreed, the history of the web site is not very important.
> >> >
> >> > Julian
> >> >
> >> >> On Feb 15, 2019, at 5:58 AM, Michael Mior  wrote:
> >> >>
> >> >> I think we may want to keep the old SVN repository around if this is
> >> >> the case, but I personally don't have a problem with losing history in
> >> >> the new git repo. On a related note, it would be good to find a
> >> >> process for the new repo that can work with a shallow clone so we
> >> >> don't have to have the entire history of the site to push a change.
> >> >>
> >> >> --
> >> >> Michael Mior
> >> >> mm...@apache.org
> >> >>
> >> >> Le ven. 15 févr. 2019 à 05:29, Francis Chuang
> >> >>  a écrit :
> >> >>>
> >> >>> Hey everyone,
> >> >>>
> >> >>> I have now created the calcite-site repo in Gitbox. It is now
> >> available
> >> >>> via Github and the Gitbox endpoint, but currently empty.
> >> >>>
> >> >>> I am currently trying to migrate the svn repo, but it is taking a very
> >> >>> long time and eventually timed out for me. A member of the ASF infra
> >> >>> team has also confirmed that it can take hours or days to complete
> >> [1].
> >> >>>
> >> >>> I feel that it would probably be easier if we just copy the existing
> >> >>> files from the svn repo and make that the first commit in the git
> >> repo.
> >> >>> This is what Kafka did for their migration [2].
> >> >>>
> >> >>> How important are the commits for site pushes? In my opinion it's
> >> >>> probably acceptable if we lose them and start anew with the git repo
> >> as
> >> >>> they do not document changes to our code base.
> >> >>>
> >> >>> Happy to hear your thoughts!
> >> >>>
> >> >>> Francis
> >> >>>
> >> >>> [1] https://issues.apache.org/jira/browse/INFRA-17846
> >> >>> [2]
> >> >>>
> >> https://github.com/apache/kafka-site/commit/ba6c994ca09629b047ab9175f882877ba03b92da
> >> >>>
> >> >>>> On 11/02/2019 9:00 pm, Francis Chuang wrote:
> >> >>>> Hey all,
> >> >>>>
> >> >>>> ASF project sites have the ability to use git instead of subversion
> >> as
> >> >>>> their repository for web site content [1]. It has been available
> >> since
> >> >>>> 2015 and appears to be quite stable. Quite a few other projects have
> >> >>>> also moved their websites to git and subsequently, Gitbox (for using
> >> >>>> Github as their source of truth. As an example, see the Arrow
> >> project [2].
> >> >>>>
> >> >>>> I myself would love to see this as I find gits interface and ux to be
> >> >>>> much easier to use compared to svn. It also reduces the need to
> >> context
> >> >>>> switch between Git and svn when editing and pushing the site.
> >> >>>>
> >> >>>> My overall goal is to find a way to automate the publishing and
> >> build of
> >> >>>> our websites either via Jenkins builds (there are some projects are
> >> >>>> doing this already when I searched infra) or the new Github actions
> >> [3].
> >> >>>> Having the site hosted in Git would make this process much easier to
> >> >>>> automate. I will need to get in touch with infra to clarify a few
> >> things
> >> >>>> and to see if this is feasible, but I think this is a worthwhile
> >> endeavor.
> >> >>>>
> >> >>>> How do you guys feel about moving our site's repository from svn to
> >> GitBox?
> >> >>>>
> >> >>>> Francis
> >> >>>>
> >> >>>>
> >> >>>> [1]
> >> https://blogs.apache.org/infra/entry/git_based_websites_available
> >> >>>> [2] https://issues.apache.org/jira/browse/INFRA-17655
> >> >>>> [3] https://github.com/features/actions
> >> >>>
> >>
> >>

Re: [ANNOUNCE] New committers: Hongze Zhang

[Michael Mior]

Thanks for all your work and welcome Hongze!
--
Michael Mior
mm...@apache.org

Le lun. 25 mars 2019 à 20:51, Hongze Zhang  a écrit :
>
> Thank you very much for your introduction (And no apology necessary :) ), 
> Francis ! And thank you all for the kind words.
>
> I am currently working at Tencent. During the last few months, I have been 
> focusing on providing near-standard SQL support for an existing streaming 
> compute engine. We introduce Calcite to the engine and it now works pretty 
> well.
>
> It is a great honor for me to become a committer of Apache Calcite. I also 
> look forward to contribute more in future.
>
> Thanks,
> Hongze
>
> > On Mar 26, 2019, at 07:46, Stamatis Zampetakis  wrote:
> >
> > Congrats Hongze and welcome on board.
> >
> > Calcite support for JSON is getting better every day and there are many
> > traits of you behind all that.
> > Thanks a lot for your work on Calcite and looking forward working with you.
> >
> > Best,
> > Stamatis
> >
> > Στις Δευ, 25 Μαρ 2019 στις 10:28 μ.μ., ο/η Francis Chuang <
> > francischu...@apache.org> έγραψε:
> >
> >> (Note: This was supposed to be sent last month, but I've somehow
> >> forgotten. Please accept my apologies.)
> >>
> >> Apache Calcite's Project Management Committee (PMC) has invited Hongze
> >> Zhang to become a committer, and we are pleased to announce that he has
> >> accepted.
> >>
> >> Hongze has been a consistent contributor to Calcite, being responsible
> >> for fixing various bugs and some pretty big patches. In addition, he has
> >> been reviewing PRs and participating in discussions on our mailing lists.
> >>
> >> Hongze, welcome, thank you for your contributions, and we look forward
> >> your further interactions with the community! If you wish, please feel
> >> free to tell us more about yourself and what you are working on.
> >>
> >> Francis (on behalf of the Apache Calcite PMC)
> >>
>

Re: [ANNOUNCE] New committers: Haisheng Yuan

[Michael Mior]

Congratulations and welcome Haisheng!
--
Michael Mior
mm...@apache.org

Le lun. 25 mars 2019 à 19:35, Haisheng Yuan  a écrit :
>
> Thank you for the introduction.
>
> I am currently working on Alibaba MaxCompute query engine, which uses Calcite 
> for cost-based query optimization. I will also help flink/calcite 
> integration, improve subquery unnesting, etc.
>
> It is my great honour to become a Calcite committer. Look forward to working 
> with you all to make more contributions to Calcite project.
>
> Thanks ~
> Haisheng Yuan
> --
> 发件人：Francis Chuang
> 日　期：2019年03月26日 05:24:21
> 收件人：
> 主　题：[ANNOUNCE] New committers: Haisheng Yuan
>
> Apache Calcite's Project Management Committee (PMC) has invited Haisheng
> Yuan to become a committer, and we are pleased to announce that he has
> accepted.
>
> Over the past few months, Haisheng has proposed numerous pull requests,
> reviewed PRs, participated in the mailing lists as well as filed
> numerous bugs on Jira.
>
> Haisheng, welcome, thank you for your contributions, and we look forward
> your further interactions with the community! If you wish, please feel
> free to tell us more about yourself and what you are working on.
>
> Francis (on behalf of the Apache Calcite PMC)

Re: Join, SemiJoin, Correlate

[Michael Mior]

Without fully thinking through the implications of this, I personally
like option 4. I think it's nice to be able to keep the distinction.
That said, the consensus seems to lean towards option 3 which also
sounds acceptable.
--
Michael Mior
mm...@apache.org

Le jeu. 21 mars 2019 à 14:55, Julian Hyde  a écrit :
>
> I have a few ideas for refactorings. (I’m not convinced by any of them, but 
> let me know which you like.)
>
> 1. Get rid of SemiJoinType. It is mis-named (it is not used by SemiJoin, it 
> is used by Correlate, but in a field called joinType).
>
> 2. In Correlate, use org.apache.calcite.linq4j.CorrelateJoinType. It has the 
> same set of values as SemiJoinType, but it has a better name.
>
> 3. Get rid of both SemiJoinType and CorrelateJoinType, and use JoinRelType 
> for everything. We would have to add SEMI and ANTI values. Also some methods 
> to find out whether the resulting row type contains fields from the left and 
> right inputs or just the left input.
>
> 4. Add “interface JoinLike extends BiRel” and make Join, SemiJoin and 
> Correlate implement it. It would have a methods that say whether the LHS and 
> RHS generate nulls, and whether the output row type contains columns from the 
> right input. This seems attractive because it lets Join, SemiJoin and 
> Correlate continue to be structurally different.
>
> Julian
>
>
>
>
> > On Mar 20, 2019, at 6:55 PM, Haisheng Yuan  wrote:
> >
> > SubPlan (in Postgres’ term) is a Postgres physical relational node to 
> > evaluate correlated subquery. What I mean is correlated subquery that can’t 
> > be decorrelated can’t be implemented by hashjoin or mergejoin. But it is 
> > off topic.
> >
> > Thanks ~
> > Haisheng Yuan
> > --
> > 发件人：Walaa Eldin Moustafa
> > 日　期：2019年03月21日 09:31:41
> > 收件人：
> > 抄　送：Stamatis Zampetakis
> > 主　题：Re: Re: Join, SemiJoin, Correlate
> >
> > Agreed with Stamatis. Currently: 1) Correlate is tied to IN, EXISTS,
> > NOT IN, NOT EXISTS, and 2) is used as an equivalent to nested loops
> > join. The issues here are: 1) IN, EXISTS, NOT IN, NOT EXISTS can be
> > rewritten as semi/anti joins, and 2) nested loops join is more of a
> > physical operator.
> >
> > It seems that the minimal set of logical join types are INNER, LEFT,
> > RIGHT, OUTER, SEMI, ANTI.
> >
> > So I think Calciate could have one LogicalJoin operator with an
> > attribute to specify the join type (from the above), and a number of
> > physical join operators (hash, merge, nested loops) whose
> > implementation details depend on the the join type.
> >
> > What we lose by this model is the structure of the query (whether
> > there was a sub-plan or not), but I would say that this is actually
> > what is desired from a logical representation -- to abstract away from
> > how the query is written, and how it is structured, as long as there
> > is a canonical representation. There could also be a world where both
> > models coexist (Correlate first then Decorrelate but in the light of a
> > single logical join operator?).
> >
> > @Haisheng, generally, a sub-plan can also be implemented using a
> > variant of hash or merge joins as long as we evaluate the sub-plan
> > independently (without the join predicate), but that is up to the
> > optimizer.
> >
> > Thanks,
> > Walaa.
> >
> > On Wed, Mar 20, 2019 at 5:23 PM Haisheng Yuan  
> > wrote:
> >>
> >> SemiJoinType and its relationship with JoinRelType do confuse me a little 
> >> bit.
> >>
> >> But I don’t think we should not have LogicalCorrelate. It is useful to 
> >> represent the lateral or correlated subquery (aka SubPlan in Postgres 
> >> jargon). The LogicalCorrelate can be implemented as NestLoopJoin in 
> >> Calcite, or SubPlan in Postgres’s terminology, but it can’t be implemented 
> >> as HashJoin or MergeJoin.
> >>
> >> Thanks ~
> >> Haisheng Yuan
> >> --
> >> 发件人：Stamatis Zampetakis
> >> 日　期：2019年03月21日 07:13:15
> >> 收件人：
> >> 主　题：Re: Join, SemiJoin, Correlate
> >>
> >> I have bumped into this quite a few times and I think we should really try
> >> to improve the design of the join hierarchy.
> >>
> >> From a logical point of view I think it makes sense to have the following
> >> operators:
> >> InnerJoin, LeftOuterJoin, FullOuterJoin, SemiJoin, AntiJoin, (GroupJoin)
> >>
> >> Ye

Re: Conditionally Adding Item to a Map

[Michael Mior]

The only way I can think of the top of my head is to use a CASE
statement and construct the map differently based on the value of c.
--
Michael Mior
mm...@apache.org

Le mer. 20 mars 2019 à 14:45, Shahar Cizer Kobrinsky
 a écrit :
>
> Hey All,
>
> New to Calcite, trying to express a select statement in SQL where i build a
> map.
> The thing is i want to conditionally add items to it, so for example have
> key 'c' with value from column 'c' only if the value is not null
>
> SELECT a,b, map['c',c, 'd', d] as my_map  FROM...
>
> Is there a way to conditionally have key/values in the map?
>
> Thanks!
> Shahar

Re: Druid/MongoDB Integration test failures was: [VOTE] Release apache-calcite-1.19.0 (release candidate 1)

[Michael Mior]

Thanks Kevin! Given that it seems like these failures existed in the
1.18.0 release and they do not appear to be bugs in Calcite, but
rather stale tests, I would be inclined to go ahead with the release.
--
Michael Mior
mm...@apache.org

Le lun. 18 mars 2019 à 17:58, Kevin Risden  a écrit :
>
> Was able to get the VM build and reproduces the errors:
>
> https://issues.apache.org/jira/browse/CALCITE-2932
>
> Planning to add more commentary there.
>
> Kevin Risden
>
>
> On Mon, Mar 18, 2019 at 5:45 PM Andrei Sereda  wrote:
>
> > Sure. Let me know if you need help with Druid adapter.
> >
> > On Mon, Mar 18, 2019 at 5:36 PM
> > Kevin Risden
> >  wrote:
> >
> > > Rebuilding the VM for just druid found that there is an issue with
> > > zookeeper version - 3.4.10 doesn't exist on the release mirrors anymore
> > for
> > > when installing druid. Fixing to point to 3.4.13 and see if that will let
> > > me build the test vm correctly.
> > >
> > > Kevin Risden
> > >
> > >
> > > On Mon, Mar 18, 2019 at 5:32 PM Kevin Risden  wrote:
> > >
> > > > Andrei - Thanks I just checked the results and see the same thing for
> > > > MongoDB. I see you opened CALCITE-2931 as well with a PR. Since this is
> > > > test only don't think this blocks the 1.19.0 release.
> > > >
> > > > As for Druid, looks like I'm still getting connection reset errors
> > after
> > > > rebuilding the test dataset vm. I'm going to try to rebuild the VM with
> > > > just Druid to see if its a memory issue.
> > > >
> > > > Kevin Risden
> > > >
> > > >
> > > > On Mon, Mar 18, 2019 at 4:54 PM Andrei Sereda 
> > wrote:
> > > >
> > > >> I just run Mongo tests using docker image.
> > > >>
> > > >> Failures seem to be related to key order in Bson document. Example:
> > > >>
> > > >> Expected
> > > >> {$project: {POP: '$pop', STATE: '$state'}}
> > > >>
> > > >> Actual
> > > >> {$project: {STATE: '$state', POP: '$pop'}}
> > > >>
> > > >> Those queries don't run as part of unit tests because they only work
> > in
> > > >> mongo (not fongo).
> > > >>
> > > >> I will address those inconsistencies
> > > >> (MongoAdapterTest#testGroupByAvgSumCount and
> > > >> MongoAdapterTest#testDistinctCountOrderBy)
> > > >>
> > > >> On Mon, Mar 18, 2019 at 2:05 PM
> > > >> Kevin Risden
> > > >>  wrote:
> > > >>
> > > >> > For the calcite-test-dataset vm, the docs say you can halt/up the
> > VM.
> > > It
> > > >> > turns out that Druid doesn't restart on up and MongoDB fails to
> > start
> > > >> due
> > > >> > to /var/run/mongodb missing. /var/run is symlinked to /run and /run
> > is
> > > >> > mounted as tmpfs so the folders are cleared on a restart.
> > > >> >
> > > >> > I don't know if this is what is causing the failures you saw but it
> > > was
> > > >> > causing issues with the "timeout" issues I saw. I am rebuilding the
> > VM
> > > >> and
> > > >> > checking the Druid / MongoDB integration tests individually.
> > > >> >
> > > >> > Kevin Risden
> > > >> >
> > > >> >
> > > >> > On Mon, Mar 18, 2019 at 1:43 PM Kevin Risden 
> > > >> wrote:
> > > >> >
> > > >> > > Stamatis - Can you open JIRA cases for the Druid and MongoDB
> > > >> integration
> > > >> > > test failures with details?
> > > >> > >
> > > >> > > It would be good to track them down not sure if they would block
> > the
> > > >> > > release depending on the errors. I seem to have an issue with my
> > > >> > > calcite-test-dataset vm currently since getting timeout errors for
> > > >> Mongo
> > > >> > > and Druid. I'll see if I can track them down but would be good to
> > > see
> > > >> > what
> > > >> > > the failure details are.
> > > >> > >
> > > >> > > I did not see the MySQL te

Re: [VOTE] Release apache-calcite-1.19.0 (release candidate 1)

[Michael Mior]

+1 (binding) checked hashes and signature, compiled and ran tests and
a RAT check.
--
Michael Mior
mm...@apache.org

Le ven. 15 mars 2019 à 10:38, Kevin Risden  a écrit :
>
> Hi all,
>
> I have created a build for Apache Calcite 1.19.0, release candidate 1.
>
> Thanks to everyone who has contributed to this release.
>
> Since RC 0, we have fixed the following issues:
> * [CALCITE-2925] Exclude maven-wrapper.jar from source distribution
>
> You can read the release notes here:
> https://github.com/apache/calcite/blob/branch-1.19/site/_docs/history.md
>
> The commit to be voted upon:
> https://gitbox.apache.org/repos/asf?p=calcite.git;a=commit;h=ad11340e5c5abddaa6f2729c9faa2043c4643a8d
>
> Its hash is ad11340e5c5abddaa6f2729c9faa2043c4643a8d.
>
> The artifacts to be voted on are located here:
> https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.19.0-rc1/
>
> The hashes of the artifacts are as follows:
> src.tar.gz.sha256
> 8dbe7e81d955019d78e7de270089fb42726c827f719bfd5a5d11f734fac7face
>
> A staged Maven repository is available for review at:
> https://repository.apache.org/content/repositories/orgapachecalcite-1055/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/krisden.asc
>
> Please vote on releasing this package as Apache Calcite 1.19.0.
>
> The vote is open for the next 96 hours (due to the weekend) and passes if a
> majority of
> at least three +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Calcite 1.19.0
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
> Here is my vote:
> +1 (binding)
>
> Kevin Risden

Re: [ANNOUNCE] New committer: Stamatis Zampetakis

[Michael Mior]

 No problem. I'll leave that to you then once the release is done :) Thanks!
--
Michael Mior
mm...@apache.org

Le jeu. 14 mars 2019 à 19:03, Stamatis Zampetakis  a écrit :
>
> Thanks for noticing Michael.
>
> Actually, I started doing it at some point but then there were
> inconsistencies between master, site, and svn, so I decided to do it after
> the release where everything is in line.
>
> On Thu, Mar 14, 2019, 10:15 PM Francis Chuang 
> wrote:
>
> > I also noticed Hongze was not added to the community page as well. Once
> > Kevin releases 1.19.0, we should add both of them to the page.
> >
> > On 15/03/2019 2:11 am, Michael Mior wrote:
> > > I just noticed that Stamatis was never added to the community page
> > > site. Stamatis, feel free to add yourself once the freeze for the
> > > current release is over. Otherwise, I'm happy to do so.
> > > --
> > > Michael Mior
> > > mm...@apache.org
> > >
> > > Le mer. 30 janv. 2019 à 13:01, Jesus Camacho Rodriguez
> > >  a écrit :
> > >>
> > >> Apache Calcite's Project Management Committee (PMC) has invited
> > >> Stamatis Zampetakis to become a committer, and we are pleased to
> > >> announce that he has accepted.
> > >>
> > >> Over the past few months, Stamatis has made several contributions to
> > >> Calcite and he is a very active participant in discussions in issues
> > >> and mailing lists.
> > >>
> > >> Stamatis, welcome, thank you for your contributions, and we look
> > >> forward your further interactions with the community! If you wish,
> > >> please feel free to tell us more about yourself and what you are
> > >> working on.
> > >>
> > >> Jesús (on behalf of the Apache Calcite PMC)
> >
> >

Re: [ANNOUNCE] New committer: Stamatis Zampetakis

[Michael Mior]

I just noticed that Stamatis was never added to the community page
site. Stamatis, feel free to add yourself once the freeze for the
current release is over. Otherwise, I'm happy to do so.
--
Michael Mior
mm...@apache.org

Le mer. 30 janv. 2019 à 13:01, Jesus Camacho Rodriguez
 a écrit :
>
> Apache Calcite's Project Management Committee (PMC) has invited
> Stamatis Zampetakis to become a committer, and we are pleased to
> announce that he has accepted.
>
> Over the past few months, Stamatis has made several contributions to
> Calcite and he is a very active participant in discussions in issues
> and mailing lists.
>
> Stamatis, welcome, thank you for your contributions, and we look
> forward your further interactions with the community! If you wish,
> please feel free to tell us more about yourself and what you are
> working on.
>
> Jesús (on behalf of the Apache Calcite PMC)

Re: Parantheses in RelBuilder?

[Michael Mior]

You don't explicitly create parenthesis. You just need to construct
the conditions in the right order. Just construct the two parts of
your OR separately and then create a new RexCall node with the OR
operator where you pass in each of the two RexNodes you want to
compare.
--
Michael Mior
mm...@apache.org

Le jeu. 14 mars 2019 à 07:56, Rakesh Nair  a écrit :
>
> Hello,
> Would you mind giving me some pointers on *creating parenthesis while
> building a RelNode*.
> For example, how to build RelNode for the sample query:
> SELECT * FROM `persons` WHERE *(*`SALARY` >= 10 AND `SALARY` < 20*)  *OR
> *(*`SALARY` >= 30 AND `SALARY` < 40*)*
>
> P.S. Need to know how to build the parenthesis only, everything else is
> fine..
> --
> Thanks & Regards,
> Rakesh

Re: CALCITE-2905: Maven -> Gradle: any thoughts

[Michael Mior]

Zoltan,

Please file a JIRA case for any issues you encounter when using
Eclipse. Regardless of whether we move forward with Gradle, it would
be good to have this working painlessly, hopefully with some
instructions added to the HOWTO below.

https://calcite.apache.org/docs/howto.html#setting-up-an-ide-for-contributing

--
Michael Mior
mm...@apache.org

Le lun. 11 mars 2019 à 12:17, Zoltan Haindrich  a écrit :
>
> I'm happy that this have came up - I think Gradle is much more transparent 
> than Maven in general; with no "built-in" limitations
> My experience with maven is that I'm looking for some plugin which might do 
> something similar what is needed...and depending on "lifecycle/etc" things it 
> might actually
> work...in gradle as a last resort people could write a new task and hook it 
> into the build graph - which is more straight forward.
> What I further find very usefull in Gradle; is that the build is actually 
> most of the time quite silent - messages on the screen has more meaning and 
> they don't just eat up
> space for no good reason...and it's not xml.
>
> As a matter of fact I'm also using Eclipse; and to make calcite-core up and 
> running with it: I've to remove 3 invalid source refernces; add 1 missing and 
> exclude some files
> from the buildpathso it doesn't work out of the box :)
> I will be happy to help migrating - or at least fix Eclipse support :D
> +1
>
> cheers,
> Zoltan
>
> On 3/10/19 11:49 AM, Vladimir Sitnikov wrote:
> > Muhammad>gradle supports parallel execution
> >
> > The good thing is Gradle can run tests from "core" in parallel with javadoc
> > build for "cassandra" and so on.
> >
> > Muhammad> Will this ease the project importing process to Eclipse ? This is
> > usually a
> > Muhammad> problem to me. I have to close projects to avoid displaying their
> > build
> > Muhammad> errors, define source folders, run mvn eclipse:eclipse (and some
> > say I
> > Muhammad> don't have to) and still have a couple of projects showing build
> > errors in
> > Muhammad> Eclipse.
> >
> > I don't use Eclipse, however:
> > 1) They say Gradle projects could be imported to Eclipse just fine.
> > I'm updating Apache JMeter's build system to Gradle at the moment (see
> > https://github.com/apache/jmeter/pull/448 ),
> > and you can try importing the project to see how it feels like (see gradle
> > branch https://github.com/vlsi/jmeter/tree/gradle ).
> >
> > You can try importing https://github.com/ben-manes/caffeine/ , or
> > https://github.com/spring-projects/spring-framework/ or whatever you like.
> > However I'm inclined to Kotlin-based DSL, as it has way less magic than in
> > Groovy-based scripts.
> >
> > 2) Gradle is very easy to customize. That is if default import somehow
> > fails, the build script can contain customizations to make import better.
> > For instance, you can check
> > https://github.com/ben-manes/caffeine/blob/master/gradle/eclipse.gradle
> >
> > Vladimir
> >

Re: CALCITE-2905: Maven -> Gradle: any thoughts

[Michael Mior]

I currently sit at -0 on this as I haven't found Maven too cumbersome
personally. However, I think you make some valid points. Hopefully it
won't take too long to try this and then we can see what the actual
impact on build times will be. It would also be good to hear from
others doing regular Calcite development.

--
Michael Mior
mm...@apache.org

Le dim. 10 mars 2019 à 05:35, Vladimir Sitnikov
 a écrit :
>
> Hi,
>
> I wonder what you think of migrating Maven to Gradle.
>
> I think one of the main points for having Gradle would be:
> 1) Eliminate "mvn install" for local testing. Calcite consists of
> multiple Maven modules, however Maven always uses jars from the local
> repository.
> That is if you modify a file in "core", then you can't just invoke mvn
> test from "cassandra". You have to "mvn install" "core" first.
> There are workarounds (e.g. "mvn install" all the modules every time)
>
> In Gradle, "multi-module" build feels more like "always composite
> module". In other words, even if you invoke "build" task from within
> "core" module, Gradle would find all the modules in current project,
> it would compute all the dependencies and build accordingly.
> In my opinion it makes a big difference.
>
> There's a support for cross-project incremental builds as well. I
> haven't used that, however the idea there is that one can have
> "calcite" and "drill" as different Gradle projects, however one can
> modify a file in Calcite and invoke "build" from a Drill folder. It
> would build Calcite first.
>
> 2) Maven task/plugins often fail to declare inputs/outputs. This
> causes issues like MSHARED-394: Avoid rewrite of destination in
> DefaultMavenFileFilter#filterFile when producing the same contents.
> Gradle embrases tasks authors to declare inputs outputs (e.g. files or
> just property values) and it enables the build system to track stale
> tasks.
>
> Gradle supports "buildSrc" folder which can contain code that is
> available to the buildfiles of a current project. It enables to
> express build logic in a much more sound programming language than
> XML.
>
> Vladimir

Re: JIRA Permissions to create 1.20.0 version?

[Michael Mior]

Strange. You're already an admin so you should have permissions. In
any case, I created the new release.
--
Michael Mior
mm...@apache.org

Le jeu. 7 mars 2019 à 08:47, Kevin Risden  a écrit :
>
> I was looking to move 1.19.0 issues that aren't ready to 1.20.0, but don't
> have permissions in the CALCITE JIRA project to create a new version.
>
> Can this permission be provided?
>
> Kevin Risden

Re: Where is Calcite's Roadmap?

[Michael Mior]

There is no official roadmap for Calcite. However, you may find the
most recent state of the project discussion from November
interesting[0]. If you're looking for ways to contribute, there are
plenty of feature requests and bugs which need to be worked on that
can be found on the issue tracker[1].

[0] 
https://mail-archives.apache.org/mod_mbox/calcite-dev/201811.mbox/%3CCA%2BEpF8vOgquOwR6gssQ2nL5DaedwLoPKUO-kaynXXh9UWYP4mg%40mail.gmail.com%3E
[1] https://issues.apache.org/jira/projects/CALCITE/issues/

--
Michael Mior
mm...@apache.org


Le lun. 4 mars 2019 à 21:50, vino yang  a écrit :
>
> Hi Calciter,
>
> Now, we are heavily using Calcite at work, and we are also very interested
> in participating in the
> Calcite community. To ensure that our direction is aligned with the
> community, we want to learn
> more about the community's plans.
>
> I didn't see Calcite's own Roadmap in Calcite's official website. I can
> only see Avatica's
> Roadmap[1].
>
> I don't know if it could be found elsewhere. If not, I think it would be a
> great idea to present it in
> Calcite's official website. It will help provide the right guidance to some
> potential contributors.
>
> What do you think?
>
> Best,
> Vino
>
> [1] https://calcite.apache.org/avatica/docs/roadmap.html

Re: JIRAs and Pull Requests Cleanup

[Michael Mior]

There's one other thing that I would personally find helpful. When a
committer is reviewing a PR, I propose assigning it to yourself on
GitHub. This serves two functions: 1) it makes it clear that someone
has taken responsibility for the PR, and 2) it lets you easily go back
and look at PRs you have assigned to yourself to make sure none of
them fall through the cracks.

Sometimes I have a small amount of time I can carve out to review a
couple PRs and the quicker I can find ones that aren't already "spoken
for", the better.

--
Michael Mior
mm...@apache.org

Le jeu. 28 févr. 2019 à 17:36, Julian Hyde  a écrit :
>
> Two things that we are doing that are working:
>  * Use the “pull-request-available” tag for JIRA cases;
>  * Beat down the number of open cases with the “pull-request-available” tag 
> in the lead up to the release.
>
> Let’s continue doing these.
>
> Shall we agree a target of number of open cases with “pull-request-available” 
> that we want to aim for before RC 1 of 1.19? And perhaps a date that we want 
> to achieve it by?
>
> Right now there are 92[1]. I propose that we get to 40 by Wednesday.
>
> I’m not against the PR template, or the Stale bot, but let’s have that 
> discussion after the release. Right now, there’s s**t to be shoveled, and I 
> thank those of you who have picked up a shovel. (Especially Kevin.)
>
> Julian
>
> [1] 
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20CALCITE%20%26%20labels%20%3D%20pull-request-available%20%26%20statusCategory%20%3D%20%22To%20Do%22%20
>  
> <https://issues.apache.org/jira/issues/?jql=project%20=%20CALCITE%20&%20labels%20=%20pull-request-available%20&%20statusCategory%20=%20%22To%20Do%22>
>
>
> > On Feb 28, 2019, at 2:25 PM, Kevin Risden  wrote:
> >
> > As of now, the open PRs and Calcite JIRAs are close enough to matching (96
> > vs 98). (There are a few for Avatica in the JIRA query). Thanks all those
> > that helped clean up.
> >
> > Kevin Risden
> >
> >
> > On Thu, Feb 28, 2019 at 4:21 PM Kevin Risden  wrote:
> >
> >> One thing that I think would help is to add a Github PR template [1]. We
> >> did this for Apache Knox to help make sure that PRs follow some simple
> >> guidelines. After the gitbox migration, PRs are automatically linked to
> >> JIRA (which is good). I think we just had some PRs prior to gitbox
> >> migration that didn't have the autolink part.
> >>
> >> [1]
> >> https://help.github.com/en/articles/creating-a-pull-request-template-for-your-repository
> >> [2] https://issues.apache.org/jira/browse/KNOX-1760
> >>
> >> Kevin Risden
> >>
> >>
> >> On Thu, Feb 28, 2019 at 4:07 PM Vladimir Sitnikov <
> >> sitnikov.vladi...@gmail.com> wrote:
> >>
> >>> Francis> Our problem is mainly due to PRs not being reviewed.
> >>>
> >>> Once upon a time I did try to pass over the PRs, and I added some randoms
> >>> here and there.
> >>> I'm not sure what others think of that, however we have:
> >>>
> >>> 17 "returned-with-feedback":
> >>>
> >>> https://github.com/apache/calcite/pulls?q=is%3Apr+is%3Aopen+label%3Areturned-with-feedback
> >>> 6 with pending discussion in JIRA:
> >>>
> >>> https://github.com/apache/calcite/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+label%3Adiscussion-in-jira
> >>>
> >>> I like "LGTM-will-merge-soon" as a soft warning to other committers that
> >>> "hey, either you chime in or I just commit this stuff".
> >>> We have 2 by the way:
> >>> https://github.com/apache/calcite/labels/LGTM-will-merge-soon
> >>>
> >>> Vladimir
> >>>
> >>
>

Re: Generating javadoc using Docker

[Michael Mior]

You may wish to try running mvn install in the repository first. If
this solves your problem, we should document this.
--
Michael Mior
mm...@apache.org


Le jeu. 28 févr. 2019 à 17:24, Siddharth Teotia  a écrit :
>
> Hi,
>
> I am following the instructions here
> https://github.com/apache/calcite/tree/master/site to generate java doc for
> one of my patches and running into some dependency issues. Can I get some
> help in identifying the problem?
>
> I am able to build Calcite locally. However, after running the step
> docker-compose run generate-javadoc, I see the following errors:
>
> Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.7.1:site (default-site) on
> project calcite: failed to get report for
> org.apache.maven.plugins:maven-javadoc-plugin: Failed to execute goal on
> project calcite-babel: Could not resolve dependencies for project
> org.apache.calcite:calcite-babel:jar:1.19.0-SNAPSHOT: Failure to find
> org.apache.calcite:calcite-core:jar:tests:1.19.0-SNAPSHOT
>
> I would appreciate the help.
>
> Thanks,
> Siddharth

Re: JIRAs and Pull Requests Cleanup

[Michael Mior]

Julian,

Overall, I agree with you Julian and Francis. Ideally, we would find
the resources among committers to review all PRs. My +1 for the bot
mainly came from a desire to keep the list of open PRs as ones which
are actively being worked on. Perhaps many of the same goals would be
accomplished by a one-time cleanup of existing abandoned PRs.

I think having the PRs on GitHub better managed would go a long way to
encouraging people to review them. For example, if you're reviewing a
PR with the intent of eventually merging, assign it to yourself. If
any committer could easily find a list of recent, unassigned PRs, it
would make reviewing much easier.

--
Michael Mior
mm...@apache.org

Le jeu. 28 févr. 2019 à 14:19, Julian Hyde  a écrit :
>
> -1 Using a robot to close stale PRs is solving the wrong problem.
>
> The main reason that we have a lot of open PRs is that we - as a community of 
> committers - are not putting sufficient time into reviewing. I think that we 
> are giving PR submitters poor service, and they are being very patient with 
> us.
>
> Every PR is a potential new community member, and not one but many 
> improvements to Calcite. Our goal, as committers, should be to ensure that we 
> don’t let any of those opportunities slip.
>
> We all know that there are a few poor quality PRs. They take more effort to 
> review, and bang into shape, than it would have taken the reviewer to write 
> it themselves. Comments are misspelled, and the code just looks sloppy. 
> Those, frankly, are not very high priority.
>
> But at the other end of the spectrum, there are some - many - high quality 
> PRs, where the contributor has done their homework. They are rarely perfect 
> on the first submission, but you can see that the person behind it would make 
> the community much stronger. Those PRs deserve our immediate attention.
>
> And I’ll say this just once. I get frustrated when committers prioritize PRs 
> from employees of their own company, and I get furious when they hold those 
> PRs to a lower standard than other PRs. As a committer, you need to wear your 
> Apache hat, and only your Apache hat, when reviewing code.
>
> Julian
>
>
>
>
> > On Feb 28, 2019, at 6:28 AM, Michael Mior  wrote:
> >
> > Thanks Kevin! That's an important task and one that few people are
> > willing to take on :)
> >
> > And thanks Hongze for the pointer to Stale. Especially since other
> > Apache project are already using it, I'd be inclined to have a
> > discussion on the appropriate configuration and give it a go.
> > Personally, I think the list of open PRs should things that are
> > actively being worked on. Closed PRs can always be reopened anyway, so
> > I don't think we're losing anything.
> >
> > --
> > Michael Mior
> > mm...@apache.org
> >
> > Le mer. 27 févr. 2019 à 14:36, Kevin Risden  a écrit :
> >>
> >> There are 105 open pull requests against apache/calcite repo [1]. There are
> >> only 48 Calcite JIRAs labeled with pull-request-available [2].
> >>
> >> I'm planning to go through in the next few days and make sure that we have
> >> PRs that match open JIRAs and are labeled pull-request-available. If there
> >> are PRs that are open for JIRAs that are closed, planning to close those
> >> PRs with a comment.
> >>
> >> [1] https://github.com/apache/calcite/pulls
> >> [2]
> >> https://issues.apache.org/jira/issues/?jql=project%20%3D%20CALCITE%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20pull-request-available%20ORDER%20BY%20priority%20DESC
> >>
> >> Kevin Risden
>

Re: [calcite] branch master updated: [CALCITE-2827] Allow Convention.NONE planning with VolcanoPlanner

[Michael Mior]

Laurent did answer my objection and indicate he's willing to revert.
I'm ok with the provided explanation. (Although an unrelated note,
things like "Fix checkstyle error" and "Fix grammar errors" should not
be part of the final commit message.)
--
Michael Mior
mm...@apache.org

Le mer. 27 févr. 2019 à 16:16, Julian Hyde  a écrit :
>
> Laurent,
>
> Michael had started reviewing this and had some objections. Please don’t 
> summarily commit.
>
> Julian
>
>
> > On Feb 27, 2019, at 1:07 PM, laur...@apache.org wrote:
> >
> > This is an automated email from the ASF dual-hosted git repository.
> >
> > laurent pushed a commit to branch master
> > in repository https://gitbox.apache.org/repos/asf/calcite.git
> >
> >
> > The following commit(s) were added to refs/heads/master by this push:
> > new e3a319e  [CALCITE-2827] Allow Convention.NONE planning with 
> > VolcanoPlanner
> > e3a319e is described below
> >
> > commit e3a319e1f79bc807327bd443433c50c4bbf20866
> > Author: Juhwan Kim 
> > AuthorDate: Thu Feb 21 14:31:51 2019 -0800
> >
> >[CALCITE-2827] Allow Convention.NONE planning with VolcanoPlanner
> >
> >By default, cost for Convention.NONE is infinity.
> >- Add option to allow Convention.None planning with VolcanoPlanner
> >- Add a unit test
> >
> >Change-Id: Idcee2ca923c9b10b58cc9352a390a87f899b64ee
> >
> >[CALCITE-2827] Fix checkstyle error
> >
> >Change-Id: I50702e6d50d9fe916e15b5bd42f8ec7b12bf0c6f
> >
> >[CALCITE-2827] Fix grammar errors
> >
> >Change-Id: Icca105184888ff9cc01b2b71f597f3fbd235b638
> > ---
> > .../calcite/plan/volcano/VolcanoPlanner.java   | 18 +--
> > .../plan/volcano/VolcanoPlannerTraitTest.java  | 36 
> > ++
> > 2 files changed, 52 insertions(+), 2 deletions(-)
> >
> > diff --git 
> > a/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoPlanner.java 
> > b/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoPlanner.java
> > index bf8336d..54f5049 100644
> > --- a/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoPlanner.java
> > +++ b/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoPlanner.java
> > @@ -227,6 +227,11 @@ public class VolcanoPlanner extends 
> > AbstractRelOptPlanner {
> >*/
> >   private boolean locked;
> >
> > +  /**
> > +   * Whether rels with Convention.NONE has infinite cost.
> > +   */
> > +  private boolean noneConventionHasInfiniteCost = true;
> > +
> >   private final List materializations =
> >   new ArrayList<>();
> >
> > @@ -931,13 +936,22 @@ public class VolcanoPlanner extends 
> > AbstractRelOptPlanner {
> > }
> >   }
> >
> > +  /**
> > +   * Sets whether this planner should consider rel nodes with 
> > Convention.NONE
> > +   * to have inifinte cost or not.
> > +   * @param infinite Whether to make none convention rel nodes inifite cost
> > +   */
> > +  public void setNoneConventionHasInfiniteCost(boolean infinite) {
> > +this.noneConventionHasInfiniteCost = infinite;
> > +  }
> > +
> >   public RelOptCost getCost(RelNode rel, RelMetadataQuery mq) {
> > assert rel != null : "pre-condition: rel != null";
> > if (rel instanceof RelSubset) {
> >   return ((RelSubset) rel).bestCost;
> > }
> > -if (rel.getTraitSet().getTrait(ConventionTraitDef.INSTANCE)
> > -== Convention.NONE) {
> > +if (noneConventionHasInfiniteCost
> > +&& rel.getTraitSet().getTrait(ConventionTraitDef.INSTANCE) == 
> > Convention.NONE) {
> >   return costFactory.makeInfiniteCost();
> > }
> > RelOptCost cost = mq.getNonCumulativeCost(rel);
> > diff --git 
> > a/core/src/test/java/org/apache/calcite/plan/volcano/VolcanoPlannerTraitTest.java
> >  
> > b/core/src/test/java/org/apache/calcite/plan/volcano/VolcanoPlannerTraitTest.java
> > index 66704ed..2c42b3c 100644
> > --- 
> > a/core/src/test/java/org/apache/calcite/plan/volcano/VolcanoPlannerTraitTest.java
> > +++ 
> > b/core/src/test/java/org/apache/calcite/plan/volcano/VolcanoPlannerTraitTest.java
> > @@ -258,6 +258,21 @@ public class VolcanoPlannerTraitTest {
> > assertTrue(child instanceof PhysLeafRel);
> >   }
> >
> > +  @Test public void testPlanWithNoneConvention() {
> > +VolcanoPlanner planner = new VolcanoPlanner();
> > +planner.addRelTraitDef(ConventionTraitDef.INSTANCE);
>

Re: JIRAs and Pull Requests Cleanup

[Michael Mior]

Thanks Kevin! That's an important task and one that few people are
willing to take on :)

And thanks Hongze for the pointer to Stale. Especially since other
Apache project are already using it, I'd be inclined to have a
discussion on the appropriate configuration and give it a go.
Personally, I think the list of open PRs should things that are
actively being worked on. Closed PRs can always be reopened anyway, so
I don't think we're losing anything.

--
Michael Mior
mm...@apache.org

Le mer. 27 févr. 2019 à 14:36, Kevin Risden  a écrit :
>
> There are 105 open pull requests against apache/calcite repo [1]. There are
> only 48 Calcite JIRAs labeled with pull-request-available [2].
>
> I'm planning to go through in the next few days and make sure that we have
> PRs that match open JIRAs and are labeled pull-request-available. If there
> are PRs that are open for JIRAs that are closed, planning to close those
> PRs with a comment.
>
> [1] https://github.com/apache/calcite/pulls
> [2]
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20CALCITE%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20pull-request-available%20ORDER%20BY%20priority%20DESC
>
> Kevin Risden

Re: [DISCUSS] Move gitbox notification emails to another list?

[Michael Mior]

+1 Personally, I'm not concerned about emails that have already been
sent. They're in the dev@ archive if we need them and if we need to
search two list archives for emails within this couple months that
sounds like less of a pain than forwarding all the past messages.

--
Michael Mior
mm...@apache.org

Le mer. 27 févr. 2019 à 05:13, Francis Chuang
 a écrit :
>
> Hey all,
>
> I wanted to gauge your opinions regarding the gitbox emails being sent
> to the dev@ list. I am finding these emails to be quite noisy (there is
> an email every time there is activity in our Github repos). I end up
> deleting most of these emails without reading them and I feel that there
> is a lot of noise compared to signal.
>
> How do you guys feel about moving these to another list (for example
> git...@calcite.apache.org)? As Vladimir mentioned in another thread,
> these emails are quite useful and important as they serve as a
> searchable archive of activity in our Github repos.
>
> If we do move the emails to a different list, what do we do with the
> emails that have been sent to dev@? Do we nominate someone to forward a
> copy to the new list?
>
> Francis

Re: [DISCUSS] Towards Calcite 1.19.0

[Michael Mior]

I've started to review a few myself. Although it doesn't fix the root
of the problem, it would be helpful if people who filed PRs that are
now stale could close those out. It looks like many older PRs are
abandoned and it would be nice to close out any that aren't actively
being worked on. They can always be reopened in the future.

--
Michael Mior
mm...@apache.org

Le lun. 25 févr. 2019 à 19:14, Julian Hyde  a écrit :
>
> Hey everyone.
>
> There are 108 open pull requests. What are we going to do about it?
>
> Last release I reviewed and committed dozens of pull requests, and I burned 
> out.
>
> Julian
>
>
> > On Feb 22, 2019, at 1:20 PM, Kevin Risden  wrote:
> >
> > Yea I don't mind pushing out the RC towards the end of next week (beginning
> > of March). I'm not in a rush to build the RC. I just picked Monday to try
> > to hit the target of February that was arbitrarily picked in the sand. At
> > this point, the release will most likely happen in early March.
> >
> > Kevin Risden
> >
> >
> > On Fri, Feb 22, 2019 at 4:15 PM Julian Hyde  wrote:
> >
> >> Can you do the RC towards the end of next week?
> >>
> >> I only just saw this message - 4 days after you sent it - because I’ve
> >> been fighting down a backlog of 500 Apache messages. There are some changes
> >> I would like to get into the release but I will need a few days.
> >>
> >> Julian
> >>
> >>
> >>> On Feb 18, 2019, at 9:56 AM,
> >> Kevin Risden
> >>  wrote:
> >>>
> >>> It looks like there are some PRs to be reviewed and some changes to get
> >>> merged in this week.
> >>>
> >>> How does creating the first RC on Monday Feb 25th sound? Does that give
> >>> enough time this week to get changes into Calcite 1.19.0?
> >>>
> >>> Kevin Risden
> >>>
> >>>
> >>> On Wed, Feb 13, 2019 at 11:08 AM Zoltan Haindrich  wrote:
> >>>
> >>>>
> >>>> On 2/13/19 4:24 PM, Julian Hyde wrote:
> >>>>> Sorry, there’s been a misunderstanding. Let me clarify. I didn’t say
> >>>> that your patches were too small. Or intend to imply it.
> >>>>>
> >>>>> When I said “widespread changes for no good reason” - or something like
> >>>> that - I meant changes to the RexNode format due to removing IS TRUE
> >> nodes.
> >>>>>
> >>>>> I like small patches, provided each one fixes a well defined issue and
> >>>> has appropriate tests.
> >>>>
> >>>> From now on I will try to provide a testcase when opening the jira.
> >>>> Sorry for the misunderstanding, thank you for making it clear!
> >>>>
> >>>>>
> >>>>> Julian
> >>>>>
> >>>>>> On Feb 13, 2019, at 3:40 AM, Zoltan Haindrich  wrote:
> >>>>>>
> >>>>>> Hello,
> >>>>>>
> >>>>>> In Hive I'm a little bit behind in upgrading to 1.18 and although the
> >>>> upgrade would not cause any correctness issues; but in a sense it's more
> >>>> conservative in doing some simplifications - which could be interpreted
> >> as
> >>>> regressions; if we take that into account that even the plan could get
> >>>> worse.
> >>>>>>
> >>>>>> I've a few patches almost ready - they are very small changes
> >> (actually
> >>>> Julian mentioned that they are kinda too small, so next time I will not
> >> be
> >>>> opening separate jiras for them)
> >>>>>>
> >>>>>> I will finish them and launch a custom Hive test with the latest
> >> master
> >>>> to see if there are any new issues coming from that direction.
> >>>>>> I should get the results for it by tomorrow.
> >>>>>>
> >>>>>> cheers,
> >>>>>> Zoltan
> >>>>>>
> >>>>>>
> >>>>>>> On 2/12/19 11:30 PM, Stamatis Zampetakis wrote:
> >>>>>>> I was not suggesting changing the release process. I wanted just to
> >>>>>>> highlight the fact that if the aforementioned tickets are not part of
> >>>> 1.19
> >>>>>>> I will have to create an unofficial bundle which includes them in
> >>&

Re: Supportin PostgreSQL OID casts

[Michael Mior]

There's no direct connection between the parser and the planner, so
you'll have to configure the rule sets yourself based on the parser
you're using. You could use a second rule, although you might also try
avoiding rules altogether and just using RelNode#accept on the root
node of your query and use the RexShuttle implementation to visit
every RexNode which should cover both cases.

--
Michael Mior
mm...@apache.org

Le mar. 26 févr. 2019 à 10:50, Muhammad Gelbana  a écrit :
>
> I believe
> org.apache.calcite.prepare.CalcitePrepareImpl.createPlanner(Context,
> Context, RelOptCostFactory) is the correct location to add my rule(s). But
> since this should only run when the praser is Babel, I tried to find that
> information from within the mentioned method (*createPlanner*) but I
> couldn't. Am I missing something or should I pass through such information
> to the *createPlanner* method ?
>
> Another thing, the expression may be in a "Project" too (i.e. SELECT
> typname::regproc FROM pg_catalog.pg_type WHERE typname LIKE 'bool')
> I suppose I'll have to have 2 rules. One for Project and another for Filter.
>
> Thanks,
> Gelbana
>
>
> On Tue, Feb 26, 2019 at 4:12 PM Michael Mior  wrote:
>
> > Writing a rule would certainly work. You would want to match a Filter
> > RelNode and then perhaps use an implementation of RexShuttle on the
> > condition and implement visitCall to check for appropriate casts to be
> > transformed into the subquery you want using RexSubQuery.scalar to
> > generate the new query.
> >
> > --
> > Michael Mior
> > mm...@apache.org
> >
> > Le mar. 26 févr. 2019 à 05:55, Muhammad Gelbana  a
> > écrit :
> > >
> > > I'm willing to implement running PostgreSQL queries involving OID casts
> > >
> > > *For example:*
> > > SELECT * FROM pg_attribute
> > > WHERE attrelid = 'mytable'::regclass;
> > >
> > > *Should be executed as:*
> > > SELECT * FROM pg_attribute
> > > WHERE attrelid = (SELECT oid FROM pg_class WHERE relname = 'mytable');
> > >
> > > Note that *reglass* maps to the *pg_class* table.
> > >
> > > What is the "calcite way" to implement this ? Should I write a rule ?
> > > Or should I rewrite the query before implementing it (I'm not sure where
> > is
> > > that) ?
> > >
> > > Thanks,
> > > Gelbana
> >

Re: Supportin PostgreSQL OID casts

[Michael Mior]

Writing a rule would certainly work. You would want to match a Filter
RelNode and then perhaps use an implementation of RexShuttle on the
condition and implement visitCall to check for appropriate casts to be
transformed into the subquery you want using RexSubQuery.scalar to
generate the new query.

--
Michael Mior
mm...@apache.org

Le mar. 26 févr. 2019 à 05:55, Muhammad Gelbana  a écrit :
>
> I'm willing to implement running PostgreSQL queries involving OID casts
>
> *For example:*
> SELECT * FROM pg_attribute
> WHERE attrelid = 'mytable'::regclass;
>
> *Should be executed as:*
> SELECT * FROM pg_attribute
> WHERE attrelid = (SELECT oid FROM pg_class WHERE relname = 'mytable');
>
> Note that *reglass* maps to the *pg_class* table.
>
> What is the "calcite way" to implement this ? Should I write a rule ?
> Or should I rewrite the query before implementing it (I'm not sure where is
> that) ?
>
> Thanks,
> Gelbana

Re: support more UDFs

[Michael Mior]

Issues can also be assigned to contributors and I've added you as a
contributor and assigned the issue to you. Thanks!

--
Michael Mior
mm...@apache.org

Le jeu. 21 févr. 2019 à 12:56, Ilia Gorelikhin
 a écrit :
>
> Hello! I would like to help with task 
> https://issues.apache.org/jira/browse/CALCITE-1733. Please assign it to me, 
> my jira account is isengor.
>
> Ilia Gorelikhin
> Junior Software Engineer
>
> Office: +7 812 611 10 94 x 30454   
> Cell: +7 931 257 91 89   Email: 
> ilia_gorelik...@epam.com<mailto:ilia_gorelik...@epam.com>
> Saint-Petersburg, Russia   epam.com<http://www.epam.com>
>
> CONFIDENTIALITY CAUTION AND DISCLAIMER
> This message is intended only for the use of the individual(s) or entity(ies) 
> to which it is addressed and contains information that is legally privileged 
> and confidential. If you are not the intended recipient, or the person 
> responsible for delivering the message to the intended recipient, you are 
> hereby notified that any dissemination, distribution or copying of this 
> communication is strictly prohibited. All unintended recipients are obliged 
> to delete this message and destroy any printed copies.
>

Re: Calcite Avatica - JIRA version cleanup?

[Michael Mior]

Kevin,

I'm guessing you're right that this should be cleaned up. I've added
you as an admin to the Calcite JIRA project so you should be able to
make changes yourself.

--
Michael Mior
mm...@apache.org

Le sam. 9 févr. 2019 à 09:45, Kevin Risden  a écrit :
>
> I was looking at JIRA and noticed that avatica-1.13.0 was marked as
> unreleased and there is no avatica-1.14.0 version number. There are some
> issues still tagged as avatica-1.13.0 but not resolved. I don't have
> permissions in JIRA to add the new version or mark avatica-1.13.0 as
> released.
>
> https://issues.apache.org/jira/projects/CALCITE/versions/12343546
>
> I'm not sure if this might have been missed as part of releasing Avatica
> 1.13.0? Maybe there isn't a release step for this?
>
> Kevin Risden

Re: Contribution to calcite elastic search development

[Michael Mior]

Note that in the interim, you can define a view on top of the raw
schema that will let you expand columns from the _MAP column into
tables which are more convient to query.
--
Michael Mior
mm...@apache.org

Le lun. 18 févr. 2019 à 12:21, Andrei Sereda  a écrit :
>
> Hi Kumar,
>
> Currently elastic table row type is "Map". I'm working on
> converting elastic mapping directly to RelDataType [1].
> When done, you'll be able to query document attributes directly (as in
> `select address from user` instead of `select _MAP['address'] from user`)
> (discussion here [2])
>
> Regarding missing features in elastic adapter, we can discuss them here and
> create JIRAs individually. Please let me know what functionalities you need.
>
> If you would like to contribute to Calcite a good place to start is jira
> issues (https://issues.apache.org/jira/projects/CALCITE). Read tutorial (
> https://calcite.apache.org/docs/tutorial.html) first and study CSV adapter
> code.
>
> Thanks,
> Andrei.
>
> [1]
> https://calcite.apache.org/apidocs/org/apache/calcite/rel/type/RelDataType.html
>
> [2]
> https://lists.apache.org/thread.html/4fdb9fb396730168704f6bd9d3b57a7baa5b41241f26db7c0cc84729@%3Cdev.calcite.apache.org%3E33[3]
> aa
>
> On Mon, Feb 18, 2019 at 6:15 AM Shashwat Kumar 
> wrote:
>
> > Hi all,
> >
> > I find that calcite for elasticsearch is still at very early stage and does
> > not support even select queries seamlessly. For example,
> > *SELECT id, address FROM *user needs to be done as *SELECT _MAP['id'],
> > _MAP['address'] FROM 'user'*
> >
> > I am using calcite elasticsearch in my project and would like to contribute
> > in it. I would like to know if there is any priority list of items which I
> > can take or other ongoing developments in it.
> > Kindly guide me through other necessary resources or guidelines.
> >
> > --
> > Regards
> > Shashwat Kumar
> >

Re: [DISCUSS] Move site repositories from svn to gitbox

[Michael Mior]

I think we may want to keep the old SVN repository around if this is
the case, but I personally don't have a problem with losing history in
the new git repo. On a related note, it would be good to find a
process for the new repo that can work with a shallow clone so we
don't have to have the entire history of the site to push a change.

--
Michael Mior
mm...@apache.org

Le ven. 15 févr. 2019 à 05:29, Francis Chuang
 a écrit :
>
> Hey everyone,
>
> I have now created the calcite-site repo in Gitbox. It is now available
> via Github and the Gitbox endpoint, but currently empty.
>
> I am currently trying to migrate the svn repo, but it is taking a very
> long time and eventually timed out for me. A member of the ASF infra
> team has also confirmed that it can take hours or days to complete [1].
>
> I feel that it would probably be easier if we just copy the existing
> files from the svn repo and make that the first commit in the git repo.
> This is what Kafka did for their migration [2].
>
> How important are the commits for site pushes? In my opinion it's
> probably acceptable if we lose them and start anew with the git repo as
> they do not document changes to our code base.
>
> Happy to hear your thoughts!
>
> Francis
>
> [1] https://issues.apache.org/jira/browse/INFRA-17846
> [2]
> https://github.com/apache/kafka-site/commit/ba6c994ca09629b047ab9175f882877ba03b92da
>
> On 11/02/2019 9:00 pm, Francis Chuang wrote:
> > Hey all,
> >
> > ASF project sites have the ability to use git instead of subversion as
> > their repository for web site content [1]. It has been available since
> > 2015 and appears to be quite stable. Quite a few other projects have
> > also moved their websites to git and subsequently, Gitbox (for using
> > Github as their source of truth. As an example, see the Arrow project [2].
> >
> > I myself would love to see this as I find gits interface and ux to be
> > much easier to use compared to svn. It also reduces the need to context
> > switch between Git and svn when editing and pushing the site.
> >
> > My overall goal is to find a way to automate the publishing and build of
> > our websites either via Jenkins builds (there are some projects are
> > doing this already when I searched infra) or the new Github actions [3].
> > Having the site hosted in Git would make this process much easier to
> > automate. I will need to get in touch with infra to clarify a few things
> > and to see if this is feasible, but I think this is a worthwhile endeavor.
> >
> > How do you guys feel about moving our site's repository from svn to GitBox?
> >
> > Francis
> >
> >
> > [1] https://blogs.apache.org/infra/entry/git_based_websites_available
> > [2] https://issues.apache.org/jira/browse/INFRA-17655
> > [3] https://github.com/features/actions
>

Re: new Adapter Contribution

[Michael Mior]

There is no separate project which collects Calcite adapters. You will
find a few projects around with different adapters that haven't been
accepted into Calcite for various reasons (the author doesn't desire
to, it's unlikely to be broadly useful, etc.) In general, I'd say
we're always interested in accepting well-tested code that is expected
to be generally useful.

The contribution of an adapter that no one else is likely to use may
not be accepted, partially because when any new code is added, we also
add the burden of maintaining that code. That said, even in that case,
there's nothing stopping contributors for maintaining adapters
external to the Calcite code base :)

--
Michael Mior
mm...@apache.org

Le jeu. 14 févr. 2019 à 04:57, Hanan Yehudai
 a écrit :
>
> Hi all,
> we are developing a calcite based adapter,  are there any pre-requirements to 
> contribute it as  part of calcite ?
> is there another Apache project for adapter ( like Baheer for Flink/ Spark 
> sinks )?
>
>

Re: Another Calcite-related paper accepted for SIGMOD -- "One SQL to Rule Them All"

[Michael Mior]

Excellent! Really glad to see all the work that has been happening on
streaming SQL in the Apache community get recognized.
--
Michael Mior
mm...@apache.org

Le mar. 12 févr. 2019 à 08:11, Edmon Begoli  a écrit :
>
> Dear Calcite community,
>
> I want to let you know that another significant paper featuring Calcite
> (alongside Apache Flink and Beam) has been accepted for SIGMOD 2019.
>
> The full title of the paper is:
> One SQL to Rule Them All – an Efficient and Syntactically Idiomatic
> Approach to Management of Streams and Tables. Edmon Begoli, Tyler Akidau,
> Fabian Hueske, Julian Hyde, Kathryn Knight, and Kenneth Knowles. To appear
> in Proceedings of ACM SIGMOD conference (SIGMOD ’19). ACM, New York, NY,
> USA
>
> I want to thank Julian Hyde for his contributions, and for introducing us
> to the co-authors, with special thanks to Fabian Hueske from Flink, and
> Tyler Akidau and Kenn Knowles from Beam for their outstanding contributions.
>
> Thank you,
> Edmon

Re: [DISCUSS] Towards Calcite 1.19.0

[Michael Mior]

Thanks for getting the ball rolling Kevin!
--
Michael Mior
mm...@apache.org

Le lun. 11 févr. 2019 à 09:51, Kevin Risden  a écrit :
>
> Calcite 1.18.0 was released on 2018-12 (coming up on 2 months ago). It is
> time to get the ball rolling for the Calcite 1.19.0 release since there
> have been releases every 2-3 months.
>
> Calcite currently has 32 JIRA issues tagged for 1.19.0 with 68 commits.
> Avatica currently has 2 JIRA issues tagged for avatica-1.14.0 with 8
> commits.
>
> Are there any JIRA cases that should make it into 1.19.0 but are not yet
> finished?
>
> Since there are only two minor commits to Avatica I don't think we need a
> new Avatica release before the Calcite release.
>
> Kevin Risden

Re: Pushing to site (in svn)

[Michael Mior]

Just to clarify that it is *intentional* that master and site are not
in sync. For example, the Javadoc should refer to the currently
published version of Calcite, not whatever is currently in progress.
Also, any documentation on new features or changes should only appear
when those are released, so they should stay on master.

Anything which needs to go to the site immediately which is not
impacted by a release (e.g. the addition of a new committer) should be
committed to the master branch and then cherry-picked to the site
branch. The site should *always* be deployed from the site branch.
When a release occurs, the site branch should be reset to master.
Since everything committed to site should also be committed to master,
this will be a lossless change.

Hopefully, this clarifies a couple things. I think we're all open to
suggestions on how to improve the process, but it's better than the
previous approach of carefully selecting what to commit to SVN
manually :)

--
Michael Mior
mm...@apache.org

Le lun. 11 févr. 2019 à 08:28, Stamatis Zampetakis  a écrit :
>
> Commiting to master, site, and svn is the way to go I think.
>
> It is not a problem if master and site are not in sync all the time, but it
> is a problem if
> we push directly from the master to svn. Another think to avoid is
> commiting directly to svn.
>
> Now in order to align the three branches I think it is best to wait for the
> next release (which is supposed to come out inside 2019-02)
> where everything should be in sync.
>
> Στις Κυρ, 10 Φεβ 2019 στις 5:11 μ.μ., ο/η Andrei Sereda 
> έγραψε:
>
> > Yes static site is tricky. You have to keep 3 branches in sync: master,
> > site and svn.
> >
> > I always publish from site branch (and have fewer diffs).
> >
> > Sometimes git master and site branches are only synced during release.
> >
> >
> > On Sun, Feb 10, 2019 at 9:17 AM Stamatis Zampetakis 
> > wrote:
> >
> > > @Francis: Whatever I said concerns the site branch (not the master).
> > > @Andrei: I did more or less the same thing but without using docker.
> > >
> > > It appears that the site branch in git and the published site in svn are
> > > not sync. Compare for instance:
> > >
> > > https://github.com/apache/calcite/blob/site/site/_docs/index.md
> > > https://calcite.apache.org/docs/
> > >
> > > and look for "it does not even have a favorite data format"; you will not
> > > find it in both places (this is also what the previous diff showed).
> > >
> > > It appears as things from the master branch (e.g.,
> > > fc5af78bf37a4a8b9e89c20e2a478fb4a95dbd4d) were pushed directly to the svn
> > > without passing through the site branch.
> > >
> > >
> > > Στις Κυρ, 10 Φεβ 2019 στις 3:49 π.μ., ο/η Andrei Sereda  > >
> > > έγραψε:
> > >
> > > > Try first svn checkout then build site:
> > > >
> > > > $ rm -rf target
> > > > $ svn co https://svn.apache.org/repos/asf/calcite/site target
> > > > $ docker-compose run build-site
> > > > $ cd target && svn status
> > > >
> > > >
> > > > On Sat, Feb 9, 2019 at 9:43 PM Francis Chuang <
> > francischu...@apache.org>
> > > > wrote:
> > > >
> > > > > Hey Stamatis,
> > > > >
> > > > > Are you trying to publish using the site branch? When publishing the
> > > > > site for anything other than a release, the site branch should be
> > used.
> > > > > If you're using the master branch, it's possible there are some
> > > > > documentation for a future release that should not be published.
> > > > >
> > > > > Also, can you check if some of the changes are for making files
> > > > > non-executable or executable? I recall that when I last published the
> > > > > site, some of the files became executable. I believe I should have
> > > fixed
> > > > > most of them by making them non-executable, but some might have
> > slipped
> > > > > through.
> > > > >
> > > > > Francis
> > > > >
> > > > > On 10/02/2019 10:49 am, Stamatis Zampetakis wrote:
> > > > > > It is not copyrights or headers. I get various different things.
> > For
> > > > > > instance the following:
> > > > > >
> > > > > >> svn diff docs/index.html
> > > > > > Index: docs/index.html
> > > > > > ==

Re: [DISCUSS] Move site repositories from svn to gitbox

[Michael Mior]

+1 for me as well and another +1 for automation if we can have this
happen from the "site" branch.
--
Michael Mior
mm...@apache.org

Le lun. 11 févr. 2019 à 05:01, Francis Chuang
 a écrit :
>
> Hey all,
>
> ASF project sites have the ability to use git instead of subversion as
> their repository for web site content [1]. It has been available since
> 2015 and appears to be quite stable. Quite a few other projects have
> also moved their websites to git and subsequently, Gitbox (for using
> Github as their source of truth. As an example, see the Arrow project [2].
>
> I myself would love to see this as I find gits interface and ux to be
> much easier to use compared to svn. It also reduces the need to context
> switch between Git and svn when editing and pushing the site.
>
> My overall goal is to find a way to automate the publishing and build of
> our websites either via Jenkins builds (there are some projects are
> doing this already when I searched infra) or the new Github actions [3].
> Having the site hosted in Git would make this process much easier to
> automate. I will need to get in touch with infra to clarify a few things
> and to see if this is feasible, but I think this is a worthwhile endeavor.
>
> How do you guys feel about moving our site's repository from svn to GitBox?
>
> Francis
>
>
> [1] https://blogs.apache.org/infra/entry/git_based_websites_available
> [2] https://issues.apache.org/jira/browse/INFRA-17655
> [3] https://github.com/features/actions

Re: Calcite's CBO

[Michael Mior]

Cost is always a record in Calcite, so yes. Costs from different
conventions are compared directly. It's assumed that all the costs are
expressed in a way they can be correctly compared. Again, this is not
ideal, but it's what we have now and it works relatively well. The
planner finds the cheapest plan using essentially the same algorithm
as used in the Volcano planner which I referenced earlier. You can see
the code in VolcanoPlanner#findBestExp.
--
Michael Mior
mm...@apache.org

Le ven. 8 févr. 2019 à 14:34, Lekshmi  a écrit :
>
> Hi,
>   I understand... If we are overriding the calcite's cost function in
> adapter, then should we also generate cost as record? And how does calcite
> compare the costs of different nodes in different convention? And how
> calcite generate the cheapest plan? Is there any algorithm dedicated for
> that? I would like to know more about that. Because we would like to
> develop a system in which Calcite plays the major role of optimization and
> execution. So for that we would be developing a cost function and costs
> estimated for this should be in the comparable format of calcite.
>
> Thank you so much for the support and suggestions
>
> On Thu, 7 Feb 2019, 4:07 pm Michael Mior 
> > I/O cost is always zero because Calcite itself doesn't try to estimate
> > I/O cost. As I mentioned. my previous reply, this can be overridden by
> > adapters. Row count is an estimate which starts off as the estimated
> > number of rows in the table but then is impacted by the estimated
> > filter factors of queries. So if the estimated number of rows is 100
> > and filters are applied that remove an estimated 95% and 30% of rows,
> > then the estimated rows produced by the query will be 3.5.
> >
> > --
> > Michael Mior
> > mm...@apache.org
> >
> > Le jeu. 7 févr. 2019 à 04:55, Lekshmi  a écrit :
> > >
> > > Dear Michael Mior,
> > >Thank you so much for the reply. But still, I'm confused with
> > following
> > >
> > > When I connected PostgreSQL with Calcite and execute TPCH queries,
> > provided
> > > data residing in the Postgres database, I got the cheapest plan like
> > below
> > > for TPCH query 1.
> > >
> > > EnumerableProject(l_returnflag=[$0], l_linestatus=[$1],
> > SUM_QTY=[CASE(=($3,
> > > 0), null, $2)], SUM_BASE_PRICE=[CASE(=($5, 0), null, $4)],
> > > SUM_DISC_PRICE=[CASE(=($7, 0), null, $6)], SUM_CHARGE=[CASE(=($9, 0),
> > null,
> > > $8)], AVG_QTY=[/(CASE(=($3, 0), null, $2), $3)], AVG_PRICE=[/(CASE(=($5,
> > > 0), null, $4), $5)], AVG_DISC=[/(CASE(=($11, 0), null, $10), $11)],
> > > COUNT_ORDER=[$12]): rowcount = 1.0, cumulative cost = {216.75 rows,
> > > 1870.344248356904 cpu, 0.0 io}, id = 316
> > >
> > >   EnumerableLimit(fetch=[1]): rowcount = 1.0, cumulative cost = {215.75
> > > rows, 1860.344248356904 cpu, 0.0 io}, id = 315
> > >
> > > EnumerableSort(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC]):
> > > rowcount = 10.0, cumulative cost = {214.75 rows, 1859.344248356904 cpu,
> > 0.0
> > > io}, id = 314
> > >
> > >   JdbcToEnumerableConverter: rowcount = 10.0, cumulative cost =
> > {204.75
> > > rows, 662.0 cpu, 0.0 io}, id = 313
> > >
> > > JdbcAggregate(group=[{0, 1}], SUM_QTY=[$SUM0($2)],
> > > agg#1=[COUNT($2)], SUM_BASE_PRICE=[$SUM0($3)], agg#3=[COUNT($3)],
> > > SUM_DISC_PRICE=[$SUM0($4)], agg#5=[COUNT($4)], SUM_CHARGE=[$SUM0($5)],
> > > agg#7=[COUNT($5)], agg#8=[$SUM0($6)], agg#9=[COUNT($6)],
> > > COUNT_ORDER=[COUNT()]): rowcount = 10.0, cumulative cost = {203.75 rows,
> > > 661.0 cpu, 0.0 io}, id = 312
> > >
> > >   JdbcProject(l_returnflag=[$8], l_linestatus=[$9],
> > > l_quantity=[$4], l_extendedprice=[$5], $f4=[*($5, -(1, $6))],
> > $f5=[*(*($5,
> > > -(1, $6)), +(1, $7))], l_discount=[$6]): rowcount = 100.0, cumulative
> > cost
> > > = {180.0 rows, 661.0 cpu, 0.0 io}, id = 311
> > >
> > > JdbcTableScan(table=[[TPCH, lineitem]]): rowcount = 100.0,
> > > cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 0
> > >
> > >
> > > But I didn't understand why I/O cost is always zero? And why row count is
> > > not an integer? Is row count is not actually a count? What is an I/O
> > cost?
> > > Is it count of I/O read for required input or time taken for reading
> > > required input? because i/o is also not an integer above.
> > >
> > >
> > >
> > >
> > >
> > >

Re: Broken Travis

[Michael Mior]

Thanks Kevin!
--
Michael Mior
mm...@apache.org

Le ven. 8 févr. 2019 à 16:48, Kevin Risden  a écrit :
>
> Master is backed to fixed after committing CALCITE-2836
>
> https://travis-ci.org/apache/calcite/builds/490737420
>
> Kevin Risden
>
>
> On Fri, Feb 8, 2019 at 3:47 PM Kevin Risden  wrote:
>
> > https://issues.apache.org/jira/browse/CALCITE-2836
> >
> > Kevin Risden
> >
> >
> > On Fri, Feb 8, 2019 at 3:41 PM Kevin Risden  wrote:
> >
> >> Looks like this might just be a leftover compiler plugin in calcite-plus
> >> module.
> >>
> >> https://github.com/apache/calcite/pull/1038
> >>
> >> The PR above passes with a minor change of removing the compiler-plugin
> >> to use the top level pom defined one. The comment is out of date as well.
> >>
> >> I'll open a ticket for this and cleanup the PR title/commit message.
> >>
> >> Kevin Risden
> >>
> >>
> >> On Fri, Feb 8, 2019 at 2:42 PM Kevin Risden  wrote:
> >>
> >>> From the two builds:
> >>>
> >>>- Working - Java version: 11.0.1, vendor: Oracle Corporation,
> >>>runtime: /usr/lib/jvm/java-11-openjdk-amd64
> >>>- Failed - Java version: 11.0.2, vendor: Oracle Corporation,
> >>>runtime: /usr/lib/jvm/java-11-openjdk-amd64
> >>>
> >>> Testing with this locally to see how this reproduces and can look into
> >>> fixing.
> >>>
> >>> Kevin Risden
> >>>
> >>> On Fri, Feb 8, 2019 at 2:34 PM Kevin Risden  wrote:
> >>>
> >>>> I can take a look. I poked around a bit yesterday and saw the same
> >>>> thing about the commit not changing javadocs. Not sure if JDK 11 got
> >>>> updated to have this issue. I'll see what it takes to fix.
> >>>>
> >>>> Kevin Risden
> >>>>
> >>>>
> >>>> On Fri, Feb 8, 2019 at 2:30 PM Julian Hyde  wrote:
> >>>>
> >>>>> Does someone else have time to take a look?
> >>>>>
> >>>>>
> >>>>> > On Feb 8, 2019, at 11:17 AM, Vladimir Sitnikov <
> >>>>> sitnikov.vladi...@gmail.com> wrote:
> >>>>> >
> >>>>> > Julian> Vladimir, can you please take a look?
> >>>>> >
> >>>>> > As you see, the commit did not touch "Calcite Plus", so I don't
> >>>>> really
> >>>>> > think the commit is to blame here. It looks more like JDK-8212233 to
> >>>>> > me.
> >>>>> > On top of that, Calcite Plus does not build on my machine (see
> >>>>> > CALCITE-2816 PsTableFunction fails in Russian locale), so I've a
> >>>>> > half-baked excuse there as well.
> >>>>> >
> >>>>> > So I'm inlined to tentatively decline your offer.
> >>>>> >
> >>>>> > Vladimir
> >>>>>
> >>>>>

Re: Calcite's CBO

[Michael Mior]

I/O cost is always zero because Calcite itself doesn't try to estimate
I/O cost. As I mentioned. my previous reply, this can be overridden by
adapters. Row count is an estimate which starts off as the estimated
number of rows in the table but then is impacted by the estimated
filter factors of queries. So if the estimated number of rows is 100
and filters are applied that remove an estimated 95% and 30% of rows,
then the estimated rows produced by the query will be 3.5.

--
Michael Mior
mm...@apache.org

Le jeu. 7 févr. 2019 à 04:55, Lekshmi  a écrit :
>
> Dear Michael Mior,
>Thank you so much for the reply. But still, I'm confused with following
>
> When I connected PostgreSQL with Calcite and execute TPCH queries, provided
> data residing in the Postgres database, I got the cheapest plan like below
> for TPCH query 1.
>
> EnumerableProject(l_returnflag=[$0], l_linestatus=[$1], SUM_QTY=[CASE(=($3,
> 0), null, $2)], SUM_BASE_PRICE=[CASE(=($5, 0), null, $4)],
> SUM_DISC_PRICE=[CASE(=($7, 0), null, $6)], SUM_CHARGE=[CASE(=($9, 0), null,
> $8)], AVG_QTY=[/(CASE(=($3, 0), null, $2), $3)], AVG_PRICE=[/(CASE(=($5,
> 0), null, $4), $5)], AVG_DISC=[/(CASE(=($11, 0), null, $10), $11)],
> COUNT_ORDER=[$12]): rowcount = 1.0, cumulative cost = {216.75 rows,
> 1870.344248356904 cpu, 0.0 io}, id = 316
>
>   EnumerableLimit(fetch=[1]): rowcount = 1.0, cumulative cost = {215.75
> rows, 1860.344248356904 cpu, 0.0 io}, id = 315
>
> EnumerableSort(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC]):
> rowcount = 10.0, cumulative cost = {214.75 rows, 1859.344248356904 cpu, 0.0
> io}, id = 314
>
>   JdbcToEnumerableConverter: rowcount = 10.0, cumulative cost = {204.75
> rows, 662.0 cpu, 0.0 io}, id = 313
>
> JdbcAggregate(group=[{0, 1}], SUM_QTY=[$SUM0($2)],
> agg#1=[COUNT($2)], SUM_BASE_PRICE=[$SUM0($3)], agg#3=[COUNT($3)],
> SUM_DISC_PRICE=[$SUM0($4)], agg#5=[COUNT($4)], SUM_CHARGE=[$SUM0($5)],
> agg#7=[COUNT($5)], agg#8=[$SUM0($6)], agg#9=[COUNT($6)],
> COUNT_ORDER=[COUNT()]): rowcount = 10.0, cumulative cost = {203.75 rows,
> 661.0 cpu, 0.0 io}, id = 312
>
>   JdbcProject(l_returnflag=[$8], l_linestatus=[$9],
> l_quantity=[$4], l_extendedprice=[$5], $f4=[*($5, -(1, $6))], $f5=[*(*($5,
> -(1, $6)), +(1, $7))], l_discount=[$6]): rowcount = 100.0, cumulative cost
> = {180.0 rows, 661.0 cpu, 0.0 io}, id = 311
>
> JdbcTableScan(table=[[TPCH, lineitem]]): rowcount = 100.0,
> cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 0
>
>
> But I didn't understand why I/O cost is always zero? And why row count is
> not an integer? Is row count is not actually a count? What is an I/O cost?
> Is it count of I/O read for required input or time taken for reading
> required input? because i/o is also not an integer above.
>
>
>
>
>
>
>
> Thanks and Regards
>
> Lekshmi B.G
> Email: lekshmib...@gmail.com
>
>
>
>
> On Thu, Feb 7, 2019 at 1:37 AM Michael Mior  wrote:
>
> > You're correct that it's not a single value. That said, in practice,
> > not much is actually done with. CPU or I/O cost. You'll see in
> > TableScan that computeSelfCost returns a cost which really only makes
> > use of the number of rows in the table. Individual adapters may
> > override this, but few do.
> > --
> > Michael Mior
> > mm...@apache.org
> >
> > Le mer. 6 févr. 2019 à 05:22, Lekshmi  a écrit :
> > >
> > > Hi,
> > >  In [0], they suggest, "the optimizer implementer can choose the cost to
> > be
> > > a number or record". Which does one Apache Calcite use? I found, it as a
> > > record, when I run queries in debug mode.  Is that correct? Then can you
> > > please define (rows, CPU, I/O) in Calcite? Also, when, we are connecting
> > > calcite across multiple systems, then 'i/o' should be greater than '0.0',
> > > right? Because it needs to read the results from the push-down
> > operations?
> > > Can you please explain these in a bit more detail? I appreciate your
> > > support and suggestions.
> > >
> > >
> > > Thanks and Regards
> > >
> > > Lekshmi B.G
> > > Email: lekshmib...@gmail.com
> > >
> > >
> > >
> > >
> > > On Tue, Feb 5, 2019 at 11:30 PM Michael Mior  wrote:
> > >
> > > > Calcite's optimizer is based off the Volcano optimizer[0]. In that
> > > > paper you'll find an outline of the algorithm which is basically
> > > > equivalent to what Calcite uses. Adding multiple systems doesn't
> > > > complicate things very much. The main addition used 

Re: Calcite's CBO

[Michael Mior]

You're correct that it's not a single value. That said, in practice,
not much is actually done with. CPU or I/O cost. You'll see in
TableScan that computeSelfCost returns a cost which really only makes
use of the number of rows in the table. Individual adapters may
override this, but few do.
--
Michael Mior
mm...@apache.org

Le mer. 6 févr. 2019 à 05:22, Lekshmi  a écrit :
>
> Hi,
>  In [0], they suggest, "the optimizer implementer can choose the cost to be
> a number or record". Which does one Apache Calcite use? I found, it as a
> record, when I run queries in debug mode.  Is that correct? Then can you
> please define (rows, CPU, I/O) in Calcite? Also, when, we are connecting
> calcite across multiple systems, then 'i/o' should be greater than '0.0',
> right? Because it needs to read the results from the push-down operations?
> Can you please explain these in a bit more detail? I appreciate your
> support and suggestions.
>
>
> Thanks and Regards
>
> Lekshmi B.G
> Email: lekshmib...@gmail.com
>
>
>
>
> On Tue, Feb 5, 2019 at 11:30 PM Michael Mior  wrote:
>
> > Calcite's optimizer is based off the Volcano optimizer[0]. In that
> > paper you'll find an outline of the algorithm which is basically
> > equivalent to what Calcite uses. Adding multiple systems doesn't
> > complicate things very much. The main addition used by Calcite is what
> > we call a "convention" trait that allows the optimizer to deal with
> > expressions across multiple systems. More details are available in a
> > recently published paper on Calcite [1].
> >
> > One important caveat to note is that the cost model used is not likely
> > to reflect the actual cost of query execution in many cases. It's
> > generally "good enough" in that the ordering of plans by cost will be
> > approximately correct. So although the optimal plan will be selected
> > according to the cost model, the plan which is actually the best in
> > practice may not be selected. That said, I would expect Calcite will
> > pick a plan which is generally quite close to the optimal,  but we
> > have no guarantee of this.
> >
> > [0]
> > https://pdfs.semanticscholar.org/a817/a3e74d1663d9eb35b4baf3161ab16f57df85.pdf
> > [1] https://arxiv.org/pdf/1802.10233.pdf
> >
> > --
> > Michael Mior
> > mm...@apache.org
> >
> > Le mar. 5 févr. 2019 à 15:52, Lekshmi  a écrit :
> > >
> > > Hi,
> > >I would like to know about the Calcite CBO in detail, including how it
> > > deals with global optimization when multiple processing systems are
> > > associated with it. Any documentation, pointers are much appreciated.
> > >
> > >
> > > Thanks and Regards
> > >
> > > Lekshmi B.G
> > > Email: lekshmib...@gmail.com
> >

Re: Calcite Validator Customization

[Michael Mior]

There are two main steps you'd have to take here. Firstly, you'd have
to change the operand types accepted by CONCAT in SqlStdOperatorTable.
Second, you'd have to redefine the CONCAT method in RexImpTable to
something that actually builds the string instead of just using
String.concat.

Unfortunately, I don't think you can override the behaviour of
built-in operators without making your build of Calcite.
--
Michael Mior
mm...@apache.org

Le mer. 6 févr. 2019 à 17:07, Paul Trepagnier  a écrit :
>
> I am using Calcite to try to be a federated database server for a BI tool.
> This BI tool is sending queries that I cannot change at the source.  So, I
> am trying to do some customizations within Calcite to handle these
> queries.
>
> For instance, one of the queries tries to do a sql concatenate between a
> string and an integer.  Calcite throws an exception for
> this: org.apache.calcite.sql.validate.SqlValidatorException: Cannot apply
> '||' to arguments of type ' || '. Supported form(s):
> ' || '
>
> Is there an easy way to override the behavior of the concatenate operator?
> I am new to Calcite, so I do not know what my options are for customizing
> the planning/validation behavior.  If someone is able to point me in the
> right direction, I would greatly appreciate it.
>
> Thank you for any help you can provide,
>
> Paul

Re: Calcite's CBO

[Michael Mior]

Calcite's optimizer is based off the Volcano optimizer[0]. In that
paper you'll find an outline of the algorithm which is basically
equivalent to what Calcite uses. Adding multiple systems doesn't
complicate things very much. The main addition used by Calcite is what
we call a "convention" trait that allows the optimizer to deal with
expressions across multiple systems. More details are available in a
recently published paper on Calcite [1].

One important caveat to note is that the cost model used is not likely
to reflect the actual cost of query execution in many cases. It's
generally "good enough" in that the ordering of plans by cost will be
approximately correct. So although the optimal plan will be selected
according to the cost model, the plan which is actually the best in
practice may not be selected. That said, I would expect Calcite will
pick a plan which is generally quite close to the optimal,  but we
have no guarantee of this.

[0] 
https://pdfs.semanticscholar.org/a817/a3e74d1663d9eb35b4baf3161ab16f57df85.pdf
[1] https://arxiv.org/pdf/1802.10233.pdf

--
Michael Mior
mm...@apache.org

Le mar. 5 févr. 2019 à 15:52, Lekshmi  a écrit :
>
> Hi,
>I would like to know about the Calcite CBO in detail, including how it
> deals with global optimization when multiple processing systems are
> associated with it. Any documentation, pointers are much appreciated.
>
>
> Thanks and Regards
>
> Lekshmi B.G
> Email: lekshmib...@gmail.com

Re: Calcite - Step into debug (IntelliJ)

[Michael Mior]

Daniel,

Have you checked the IntelliJ setup instructions in the HOWTO?

https://calcite.apache.org/docs/howto.html#setting-up-intellij-idea

--
Michael Mior
mm...@apache.org

Le mar. 5 févr. 2019 à 17:03, Daniel Dubovski  a écrit :
>
> Hi dear devs!
>
> I'm interested in adding and testing a new planner for academic purposes.
> I would love to be able to debug the test and attach to process so that I
> can get a feel of the flow the code goes trough.
>
> I'm having trouble doing that and was wondering if you have a setup you can
> share with a fellow (humble) developer.
>
> Thanks in advance!
> Daniel Dubovski.

Re: Error connecting Calcite to Elasticsearch

[Michael Mior]

Thanks Andrei! Allan, if you add the key as I suggested above, this
should solve your problem for now and then you should be able to
remove this after the next version of Calcite is release.
--
Michael Mior
mm...@apache.org

Le mar. 5 févr. 2019 à 11:27, Andrei Sereda  a écrit :
>
> I have removed userConfig from ElasticsearchSchemaFactory. It is not used
> anymore.
>
> https://github.com/apache/calcite/pull/1028
>
> On Mon, Feb 4, 2019 at 2:34 PM Michael Mior  wrote:
>
> > This looks like a bug. Could you try adding another key "userConfig":
> > "{}" under "operand" in your model file and see if that runs?
> > --
> > Michael Mior
> > mm...@apache.org
> >
> >
> > Le lun. 4 févr. 2019 à 14:29, Allan Keers  a
> > écrit :
> > >
> > > Hi there,
> > > I can connect my Calcite installation to the sample csv and query the
> > tables.
> > > When I try to connect to Elasticsearch I get the following error.
> > > Any help would be appreciated.
> > > Thanks,
> > > Allan Keers
> > >
> > >
> > > sqlline version 1.6.0
> > >
> > > sqlline> !connect
> > jdbc:calcite:model=/Users/Administrator/dev/calcite/elasticsearch/elasticsearch.json
> > admin admin
> > >
> > > java.lang.RuntimeException: Error instantiating
> > JsonCustomSchema(name=elasticsearch)
> > >
> > > at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:287)
> > >
> > > at
> > org.apache.calcite.model.JsonCustomSchema.accept(JsonCustomSchema.java:45)
> > >
> > > at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:208)
> > >
> > > at org.apache.calcite.model.ModelHandler.(ModelHandler.java:100)
> > >
> > > at org.apache.calcite.jdbc.Driver$1.onConnectionInit(Driver.java:104)
> > >
> > > at
> > org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:139)
> > >
> > > at sqlline.DatabaseConnection.connect(DatabaseConnection.java:130)
> > >
> > > at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:179)
> > >
> > > at sqlline.Commands.connect(Commands.java:1247)
> > >
> > > at sqlline.Commands.connect(Commands.java:1139)
> > >
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >
> > > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > >
> > > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> > >
> > > at
> > sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> > >
> > > at sqlline.SqlLine.dispatch(SqlLine.java:722)
> > >
> > > at sqlline.SqlLine.begin(SqlLine.java:540)
> > >
> > > at sqlline.SqlLine.start(SqlLine.java:264)
> > >
> > > at sqlline.SqlLine.main(SqlLine.java:195)
> > >
> > > Caused by: java.lang.NullPointerException
> > >
> > > at
> > com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:889)
> > >
> > > at
> > com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3023)
> > >
> > > at
> > org.apache.calcite.adapter.elasticsearch.ElasticsearchSchemaFactory.create(ElasticsearchSchemaFactory.java:65)
> > >
> > > at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:282)
> > >
> > > ... 18 more
> > >
> > >
> > > I'm running on Macbook Pro Mac OS Mojave 10.14.1
> > > Elasticsearch version:
> > > {
> > > "name": "i-",
> > > "cluster_name": "prod-x",
> > > "cluster_uuid": "r2G1fHbtQO6a3ft_JLbLcg",
> > > "version": {
> > > "number": "5.6.8",
> > > "build_hash": "688ecce",
> > > "build_date": "2018-02-16T16:46:30.010Z",
> > > "build_snapshot": false,
> > > "lucene_version": "6.6.1"
> > > },
> > > "tagline": "You Know, for Search"
> > > }
> > >
> > > elasticsearch.json:
> > > {
> > >   "version": "1.0",
> > >   "defaultSchema": "elasticsearch",
> > >   "schemas": [
> > > {
> > >   "type": "custom",
> > >   "name": "elasticsearch",
> > >   "factory":
> > "org.apache.calcite.adapter.elasticsearch.ElasticsearchSchemaFactory",
> > >   "operand": {
> > > "coordinates": "{'99.99.99.99': 80}",
> > > "index": "mp_mediaitems_v220190116"
> > >   }
> > > }
> > >   ]
> > > }
> > >
> >

Re: Error connecting Calcite to Elasticsearch

[Michael Mior]

This looks like a bug. Could you try adding another key "userConfig":
"{}" under "operand" in your model file and see if that runs?
--
Michael Mior
mm...@apache.org


Le lun. 4 févr. 2019 à 14:29, Allan Keers  a écrit :
>
> Hi there,
> I can connect my Calcite installation to the sample csv and query the tables.
> When I try to connect to Elasticsearch I get the following error.
> Any help would be appreciated.
> Thanks,
> Allan Keers
>
>
> sqlline version 1.6.0
>
> sqlline> !connect 
> jdbc:calcite:model=/Users/Administrator/dev/calcite/elasticsearch/elasticsearch.json
>  admin admin
>
> java.lang.RuntimeException: Error instantiating 
> JsonCustomSchema(name=elasticsearch)
>
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:287)
>
> at org.apache.calcite.model.JsonCustomSchema.accept(JsonCustomSchema.java:45)
>
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:208)
>
> at org.apache.calcite.model.ModelHandler.(ModelHandler.java:100)
>
> at org.apache.calcite.jdbc.Driver$1.onConnectionInit(Driver.java:104)
>
> at 
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:139)
>
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:130)
>
> at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:179)
>
> at sqlline.Commands.connect(Commands.java:1247)
>
> at sqlline.Commands.connect(Commands.java:1139)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
>
> at sqlline.SqlLine.dispatch(SqlLine.java:722)
>
> at sqlline.SqlLine.begin(SqlLine.java:540)
>
> at sqlline.SqlLine.start(SqlLine.java:264)
>
> at sqlline.SqlLine.main(SqlLine.java:195)
>
> Caused by: java.lang.NullPointerException
>
> at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:889)
>
> at 
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3023)
>
> at 
> org.apache.calcite.adapter.elasticsearch.ElasticsearchSchemaFactory.create(ElasticsearchSchemaFactory.java:65)
>
> at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:282)
>
> ... 18 more
>
>
> I'm running on Macbook Pro Mac OS Mojave 10.14.1
> Elasticsearch version:
> {
> "name": "i-",
> "cluster_name": "prod-x",
> "cluster_uuid": "r2G1fHbtQO6a3ft_JLbLcg",
> "version": {
> "number": "5.6.8",
> "build_hash": "688ecce",
> "build_date": "2018-02-16T16:46:30.010Z",
> "build_snapshot": false,
> "lucene_version": "6.6.1"
> },
> "tagline": "You Know, for Search"
> }
>
> elasticsearch.json:
> {
>   "version": "1.0",
>   "defaultSchema": "elasticsearch",
>   "schemas": [
> {
>   "type": "custom",
>   "name": "elasticsearch",
>   "factory": 
> "org.apache.calcite.adapter.elasticsearch.ElasticsearchSchemaFactory",
>   "operand": {
> "coordinates": "{'99.99.99.99': 80}",
> "index": "mp_mediaitems_v220190116"
>   }
> }
>   ]
> }
>

Re: Planner state vs CannotPlanException vs humans

[Michael Mior]

I'm not convinced there aren't better alternatives, so I hope we can
continue to explore this. Specifically with respect to the statement
"It means there's not enough rules to produce the node with desired
properties", it would be great if we could more clearly identify what
properties the missing node should have that are missing from the
nodes which have been previously generated.

In any case, what you have is definitely much better than what we have
now. I don't see a good reason not to commit and we can always improve
later. Thanks Vladimir

--
Michael Mior
mm...@apache.org

Le sam. 2 févr. 2019 à 12:46, Vladimir Sitnikov
 a écrit :
>
> Julian>It really bugs me that it takes a couple of hours of careful reading
> of the log to figure out that for your new Foo Convention you have a
> FooProjectRule
>
> I guess this case can be improved by searching the leaf empty subsets.
>
> I've filed https://github.com/apache/calcite/pull/1024 which does that.
> For missing EnumerableAggregateRule it produces the following message.
>
> I think it is more-or-less human-readable, and it does point to
> LogicalAggregate nodes.
>
> I'm going to commit it unless there are better alternatives.
>
> Node
> rel#169:Subset#10.ENUMERABLE.[] is not implementable. It means there's not
> enough rules to produce the node with desired properties
> There are 4 empty subsets:
> Leaf 0: rel#259:Subset#4.ENUMERABLE.[]
> 145:LogicalAggregate(group=[{0}])
>   139:LogicalProject(subset=[rel#140:Subset#1.NONE.[]], deptno=[$1])
> 82:EnumerableTableScan(subset=[rel#138:Subset#0.ENUMERABLE.[]],
> table=[[hr, emps]])
>
> Leaf 1: rel#256:Subset#1.ENUMERABLE.[0]
> 139:LogicalProject(deptno=[$1])
>   82:EnumerableTableScan(subset=[rel#138:Subset#0.ENUMERABLE.[]],
> table=[[hr, emps]])
>
> Leaf 2: rel#255:Subset#8.ENUMERABLE.[0]
> 153:LogicalAggregate(group=[{0}])
>   151:LogicalProject(subset=[rel#152:Subset#7.NONE.[]], deptno0=[$1])
> 149:LogicalFilter(subset=[rel#150:Subset#6.NONE.[]], condition=[<=($0,
> $1)])
>   147:LogicalJoin(subset=[rel#148:Subset#5.NONE.[]], condition=[>=($0,
> $1)], joinType=[inner])
> 142:LogicalProject(subset=[rel#143:Subset#3.NONE.[]], deptno=[$0])
>   84:EnumerableTableScan(subset=[rel#141:Subset#2.ENUMERABLE.[]],
> table=[[hr, depts]])
> 145:LogicalAggregate(subset=[rel#146:Subset#4.NONE.[]], group=[{0}])
>   139:LogicalProject(subset=[rel#140:Subset#1.NONE.[]], deptno=[$1])
> 82:EnumerableTableScan(subset=[rel#138:Subset#0.ENUMERABLE.[]],
> table=[[hr, emps]])
>
> Leaf 3: rel#273:Subset#8.ENUMERABLE.[]
> 153:LogicalAggregate(group=[{0}])
>   151:LogicalProject(subset=[rel#152:Subset#7.NONE.[]], deptno0=[$1])
> 149:LogicalFilter(subset=[rel#150:Subset#6.NONE.[]], condition=[<=($0,
> $1)])
>   147:LogicalJoin(subset=[rel#148:Subset#5.NONE.[]], condition=[>=($0,
> $1)], joinType=[inner])
> 142:LogicalProject(subset=[rel#143:Subset#3.NONE.[]], deptno=[$0])
>   84:EnumerableTableScan(subset=[rel#141:Subset#2.ENUMERABLE.[]],
> table=[[hr, depts]])
> 145:LogicalAggregate(subset=[rel#146:Subset#4.NONE.[]], group=[{0}])
>   139:LogicalProject(subset=[rel#140:Subset#1.NONE.[]], deptno=[$1])
> 82:EnumerableTableScan(subset=[rel#138:Subset#0.ENUMERABLE.[]],
> table=[[hr, emps]])
>
> Root: rel#169:Subset#10.ENUMERABLE.[]
> Original rel:
> LogicalProject(deptno=[$0]): rowcount = 375.0, cumulative cost = {2385.0
> rows, 1728.0 cpu, 0.0 io}, id = 137
>   LogicalJoin(condition=[=($0, $1)], joinType=[inner]): rowcount = 375.0,
> cumulative cost = {2010.0 rows, 1353.0 cpu, 0.0 io}, id = 136
> LogicalProject(deptno=[$1]): rowcount = 100.0, cumulative cost = {200.0
> rows, 201.0 cpu, 0.0 io}, id = 127
>   EnumerableTableScan(table=[[hr, emps]]): rowcount = 100.0, cumulative
> cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 82
> LogicalAggregate(group=[{0}]): rowcount = 25.0, cumulative cost =
> {1435.0 rows, 1152.0 cpu, 0.0 io}, id = 135
>   LogicalProject(deptno0=[$1]): rowcount = 250.0, cumulative cost =
> {1410.0 rows, 1152.0 cpu, 0.0 io}, id = 134
> LogicalFilter(condition=[<=($0, $1)]): rowcount = 250.0, cumulative
> cost = {1160.0 rows, 902.0 cpu, 0.0 io}, id = 133
>   LogicalJoin(condition=[>=($0, $1)], joinType=[inner]): rowcount =
> 500.0, cumulative cost = {910.0 rows, 402.0 cpu, 0.0 io}, id = 132
> LogicalProject(deptno=[$0]): rowcount = 100.0, cumulative cost
> = {200.0 rows, 201.0 cpu, 0.0 io}, id = 128
>   EnumerableTableScan(table=[[hr, depts]]): rowcount = 100.0,
> cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 84
> LogicalAggrega

Re: Release managers

[Michael Mior]

Great idea. I was intending to volunteer as RM last time, but with the
time pressure, I didn't respond soon enough. I'm happy to take the
April release (1.20).

--
Michael Mior
mm...@apache.org

Le jeu. 31 janv. 2019 à 18:54, Andrei Sereda  a écrit :
>
> Release Target date Release manager
> === === ===
> 1.192019-02
> 1.202019-04
> 1.212019-06Stamatis
> 1.222019-08   Andrei
> 1.232019-10
>
> On Thu, Jan 31, 2019 at 6:14 PM Stamatis Zampetakis 
> wrote:
>
> > Release Target date Release manager
> > === === ===
> > 1.192019-02
> > 1.202019-04
> > 1.212019-06Stamatis
> > 1.222019-08
> > 1.232019-10
> >
> > Στις Πέμ, 31 Ιαν 2019 στις 7:46 μ.μ., ο/η Julian Hyde 
> > έγραψε:
> >
> > > Calcite needs to make regular releases, and we have established a cadence
> > > of every 2 - 3 months that everyone seems to like. But to keep that
> > > running, each release needs a release manager, and finding a release
> > > manager always seems to be a chore.
> > >
> > > I wonder if we have trouble recruiting release managers because we only
> > > ask for one at a time. How about we get volunteers for the next 5
> > releases?
> > > Then everyone will be seen to be doing their fair share.
> > >
> > > Release Target date Release manager
> > > === === ===
> > > 1.192019-02
> > > 1.202019-04
> > > 1.212019-06
> > > 1.222019-08
> > > 1.232019-10
> > >
> > > I propose that frequent committers (anyone who had 2 or more fixes in
> > 1.18
> > > and 1 or 2 fixes in 1.16 or 1.17) should all step up and be release
> > manager
> > > for one of the releases this year.
> > >
> > > Julian
> > >
> > >
> >

Re: getting all the equivalent query plans of an sql query

[Michael Mior]

In general, there's no way to guarantee you get "all" the plans for a
particular query. If you want to get a variety of plans and their
associated cost, take a look at CheapestPlanVisitor in RelSubset.java.
This is used by VolcanoPlanner to build the cheapest plan at the end
of the planning phase. You could construct your own visitor which will
produce multiple alternative plans.

--
Michael Mior
mm...@apache.org

Le ven. 1 févr. 2019 à 12:41, asma zgolli  a écrit :
>
> Hello ,
>
> my use case is building the search space of a special optimizer and I which
> to general all the equivalent logical query plans after parsing. Is it
> possible with apache calcite. Can you please tell me how or direct me to
> some useful documentation / tutorial.
>
> thank you very much.
> yours sincerely
> Asma ZGOLLI
>
> PhD student in data engineering - computer science
> Email : zgollia...@gmail.com
> email alt:  asma.zgo...@univ-grenoble-alpes.fr 
> Tel : (+33) 07 52 95 04 45
> Skype : asma_zgolli

Re: Planner state vs CannotPlanException vs humans

[Michael Mior]

Thanks for surfacing that Stamatis! I recalled that popping up on the
list before, but I couldn't find it. I think that could be a really
good starting point for what Vladimir is suggesting. I would love to
see this be something distributed with Calcite (perhaps the path to a
generated graph created in a temp directory could be part of the
exception). I like the other suggested outputs Vladimir suggested as
well.

Would a good first step be taking Anton's JS and creating the
toGraphViz method Vladimir proposed? Personally I think I really like
the format of the output he used, but it's worth thinking through some
more collectively.

--
Michael Mior
mm...@apache.org

Le lun. 28 janv. 2019 à 07:43, Stamatis Zampetakis  a écrit :
>
> Hi Vladimir,
>
> Anton Haidai did a nice visualizer [1] for sets, subsets, and rels using
> Calcite's logging mechanism.
> Maybe it suits your needs.
>
> Best,
> Stamatis
>
> [1] https://github.com/anha1/apache-calcite-trace-analyzer
>
> Στις Δευ, 28 Ιαν 2019 στις 1:01 μ.μ., ο/η Vladimir Sitnikov <
> sitnikov.vladi...@gmail.com> έγραψε:
>
> > Hi,
> >
> > I've been debugging VolcanoPlanner for quite some time recently, and
> > it looks like its transparency leaves much to be desired.
> >
> > Key issues I've run into:
> > 1) It is complicated to understand "what happens" during rule execution.
> > call.transformTo(..) is used a lot, and it is really hard to keep in
> > head the IDs of rels, subsets, etc
> > 2) It takes time to draw the rels on a paper
> > 3) "CannotPlanException" is not human readable. I can read it now,
> > however it really takes time to navigate through IDs.
> >
> > I think a visual representation of the planner state would help a lot
> > for both debugging and/or crash analysis.
> > Does anyone have a tool to visualize sets, subsets, rels and friends?
> >
> > For instance:
> > A) Add String VolcanoPlanner#toGraphViz() so a dump of the current
> > planner state can be obtained and printed (e.g. during debug)
> > B) Implement automatic dump of "before and after" images for each rule
> > execution. In other words, display a source state, highlight rule
> > operands there, and display the final state. Bonus point for an
> > animated gif :)
> > C) Add a GraphViz-compatible bit to CannotPlanException, so the state
> > can be visualized.
> > D) Highlight possible root cause in CannotPlanException output. For
> > instance, empty subset is a strong indicator. In case it can be
> > related to original plan (or even SQL), then it would be great to have
> > that pointer as well.
> >
> >
> > Vladimir
> >

[jira] [Created] (CALCITE-2811) Update version of Cassandra driver

[Michael Mior (JIRA)]

Michael Mior created CALCITE-2811:
-

 Summary: Update version of Cassandra driver
 Key: CALCITE-2811
 URL: https://issues.apache.org/jira/browse/CALCITE-2811
 Project: Calcite
  Issue Type: Improvement
  Components: cassandra
Affects Versions: 1.18.0
Reporter: Michael Mior
Assignee: Michael Mior
 Fix For: 1.19.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (CALCITE-2805) Can't specify port with Cassandra adapter in connection string

[Michael Mior (JIRA)]

Michael Mior created CALCITE-2805:
-

 Summary: Can't specify port with Cassandra adapter in connection 
string
 Key: CALCITE-2805
 URL: https://issues.apache.org/jira/browse/CALCITE-2805
 Project: Calcite
  Issue Type: Bug
  Components: cassandra
Affects Versions: 1.18.0
Reporter: Michael Mior
Assignee: Michael Mior
 Fix For: 1.19.0


Since the Cassandra adapter currently expects the port to be numeric (which it 
is when parsed from JSON) it fails when specifying the port in the connection 
string since all values are represented as strings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (CALCITE-2806) Cassandra adapter doesn't allow uppercase characters in table names

[Michael Mior (JIRA)]

Michael Mior created CALCITE-2806:
-

 Summary: Cassandra adapter doesn't allow uppercase characters in 
table names
 Key: CALCITE-2806
 URL: https://issues.apache.org/jira/browse/CALCITE-2806
 Project: Calcite
  Issue Type: Bug
  Components: cassandra
Reporter: Michael Mior
Assignee: Michael Mior






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: "Patch" flag vs. "pull-request-available" label

[Michael Mior]

+1 for standardizing. No opinion on which one we use.

--
Michael Mior
mm...@apache.org


Le lun. 7 janv. 2019 à 16:13, Julian Hyde  a écrit :

> There seem to be two mechanisms to indicate that a pull-request is
> available for a JIRA case.
>
> 1. The “Patch” flag (13 issues); see
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20CALCITE%20AND%20flags%20%3D%20%22Patch%22
> <
> https://issues.apache.org/jira/issues/?jql=project%20=%20CALCITE%20AND%20flags%20=%20%22Patch%22
> >.
>
> 2. The “pull-request-available” (31 issues); see
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20CALCITE%20AND%20labels%20%3D%20%22pull-request-available%22
> <
> https://issues.apache.org/jira/issues/?jql=project%20=%20CALCITE%20AND%20labels%20=%20%22pull-request-available%22>.
>
>
> Should we standardize on one?
>
> Which is preferred?
>
> Julian
>
>

Re: Help wit Calcite

[Michael Mior]

This is the sort of situation I'm hoping the notebooks I recently published
will be useful for (link below). I haven't written up any documentation yet
but you can click through to any of the notebooks to see pre-generated
output or click one of the launch buttons to modify the code and play with
it yourself.

https://github.com/michaelmior/calcite-notebooks
--
Michael Mior
mm...@apache.org


Le lun. 7 janv. 2019 à 12:59, Tom Shashaty  a écrit :

> Hi,
>
>
>
> I am trying to find resources to help me understand Calcite better.
>
>
>
> I am mainly looking to use Calcite to convert a query based on a logical
> model and transform the query to a physical model representation that I can
> use to execute against a database.  This would be accomplished using a
> logical to Physical model mapping (metadata).
>
>
>
> As such, I would need to inspect the query and either
>
>
>
> *   Modify the existing query to use Physical model tables, attributes,
> joins etc
> *   Create a new query based on the Physical model which would be
> inferred by the logical query
>
>
>
> At the moment, I don't see many good tutorials or well explained examples.
> Any help would be greatly appreciated.
>
>
>
> Thanks,
>
>
>
> Tom
>
>

Re: Disabling of "merge commits" and "squash merge" on Github

[Michael Mior]

That comment makes it sound like it will be the authorship information for
whoever presses the button, which is not what we want. If that's the case,
keeping the button disabled makes sense to me.

--
Michael Mior
mm...@apache.org


Le ven. 4 janv. 2019 à 13:14, Vladimir Sitnikov 
a écrit :

> Michael> If the squash merge button on GitHub keeps authorship
>
> According to
> https://github.com/isaacs/github/issues/1368#issuecomment-427647139
> it will set author name/email from GitHub details.
>
> Vladimir
>

Re: Using materialized view rewrites to route queries to different backend engines

[Michael Mior]

Would this example of query optimization using Calcite help?

https://github.com/michaelmior/calcite-notebooks/blob/master/query-optimization.ipynb

If not, sharing the code you have so far and specifically pointing out the
problem you're having would be helpful.

--
Michael Mior
mm...@apache.org


Le ven. 4 janv. 2019 à 14:05, Anupam Aggarwal  a
écrit :

> Hi,
>
> I am trying to use calcite as a library/abstraction layer to front several
> DB backends and route queries  to the backend best suited for the query.
> For example for a table in hive if a subset of columns referenced in a
> query have been replicated in postgres we will reroute  the query to
> postgres (and having it execute there)
> (Note this is different than having calcite directly connect to postgres
> and push down / do stuff in memory) - In our case complete execution is
> delegated to the backend engine itself.
>
>
>
> My schema is roughly similar to the following
>
> {
>   "version": "1.0",
>   "defaultSchema": "hive",
>   "schemas": [
> {
>   "type": "jdbc",
>   "name": "hive",
>   *//hive stuff is here*
> },
> {
>   "name":"postgres",
>   "materializations":[
> {
>   "table":"bar",
>   "sql":""
> }
>   ]
> }
>   ]
>
> This helps us route queries of the form select * from hive.table to
> --> select * from postgres.bar.
>
> I was reading the code in MaterializationTest.java (unit test), The way
> this is currently done is through a jdbc:calcite connection (at connection
> time the materializations are defined) and at query execution time the
> materialized view gets used and substituted in the relnode tree. However
> the code seems tightly coupled to the jdbc calcite call.
>
> Is there any other way to get a relnode from a sql directly , call optimize
> on it to do the MV substitution and return the rewritten query (and avoid
> the jdbc call altogether)
> (I plugged some code using RelToSqlConverter in the main codepath, and I am
> able to log the rewritten query but can't think of a good way to return the
> rewritten query directly bypassing the jdbc call. In my case the result of
> execution of the query is going to be the rewrite of the query itself.
>
> I went through the dev list but wasn't able to locate a way (also checked
> out Qubole's Quark which does something similar although in a different
> way). Any pointers would be really helpful. Apologies if this is a really
> basic question
>
>
> Thanks
>
> Anupam
>

Re: Disabling of "merge commits" and "squash merge" on Github

[Michael Mior]

I wasn't advocating this, just explaining what has been done in the past.
If the squash merge button on GitHub keeps authorship, I would prefer to
keep that enabled.

--
Michael Mior
mm...@apache.org

Le ven. 4 janv. 2019 à 12:06, Julian Hyde  a écrit :

>
>
> > On Jan 4, 2019, at 6:37 AM, Michael Mior  wrote:
> >
> > What we've done in the past is to add the original author's name in
> > parentheses to the commit message. Not ideal, but better than losing the
> > attribution entirely.
>
> Let’s not do that. The author’s name must be in the git commit’s “author”
> field.
>
> Our "author’s name in parentheses” practice serves some other purposes -
> e.g. giving credit to new contributors in the release notes - but it’s not
> a panacea.
>
> Julian
>
>

Re: Disabling of "merge commits" and "squash merge" on Github

[Michael Mior]

What we've done in the past is to add the original author's name in
parentheses to the commit message. Not ideal, but better than losing the
attribution entirely. I think it would be good to make an effort to ask the
contributor to do the squash themselves but in some cases, the PR is ready
to merge but the original contributor is no longer available. This
shouldn't hold us back. That said, I don't have a particular problem with
disabling the button on GitHub.

--
Michael Mior
mm...@apache.org


Le ven. 4 janv. 2019 à 03:37, Francis Chuang  a
écrit :

> Hey Stamatis,
>
> Squash merge on Github does preserve linear history. The commits are
> squashed into a new commit and then added to master.
>
> The problem with squash commit merge on Github is that the author of the
> squashed commit is set to the person who merged the PR and the committer
> is set to Github.
>
> This is not ideal, as the original author of the commits would not be
> given credit for their work.
>
> I think the best solution would be for PR authors to squash their
> commits locally and force push to their branch once work has been
> finalized.
>
> Francis
>
> On 4/01/2019 7:11 pm, Stamatis Zampetakis wrote:
> > Hi Francis,
> >
> > I had the impression that squash merge also retains the linear history.
> > I am always performing these operations using the git client; is the
> result
> > different when using Github?
> >
> > Best,
> > Stamatis
> >
> > Στις Πέμ, 3 Ιαν 2019 στις 11:37 μ.μ., ο/η Francis Chuang <
> > francischu...@apache.org> έγραψε:
> >
> >> Yesterday, Andrei brought to my attention that my attempt to merge
> >> Sergey's PR [1] using the default merge strategy (create a merge commit)
> >> on Github broke the linear history of the calcite repository.
> >>
> >> I have since removed the commit and merged the PR using a rebase.
> >>
> >> I have also tested the merge behaviors on a test repo on Github. The
> >> "Rebase and merge" strategy will replicate what we are currently doing
> >> using the git client to preserve linear history.
> >>
> >> I have asked infra to disable the "create a merge commit" and "squash
> >> merge" strategies [2] on all our repositories.
> >>
> >> The merge button on Github should now be safe to use. In the event that
> >> there is a conflict and Github is not able to perform the rebase,
> >> committers should manually rebase the commit(s) and merge the PR using
> >> the git client.
> >>
> >> [1] https://github.com/apache/calcite/pull/772
> >> [2] https://issues.apache.org/jira/browse/INFRA-17541
> >>
> >
>
>

Re: [Review request] (CALCITE-2554) Enrich enumerable join operators with order preserving information

[Michael Mior]

Stamatis,

I left a comment on the PR. Overall it looks great to me although I wonder
if we want to resolve CALCITE-2582 first to avoid disabling the subquery
tests. Thanks!


--
Michael Mior
mm...@apache.org

Le jeu. 3 janv. 2019 à 11:09, Stamatis Zampetakis  a
écrit :

> Hello,
>
> Can somebody have a look in the PR.
>
> Enriching join operators with RelCollation information can have a big
> impact in performance of some queries since it allows dropping redundant
> sort operators which are in general costly to evaluate.
>
> Thanks in advance,
> Stamatis
>
> JIRA: https://issues.apache.org/jira/browse/CALCITE-2554
> PR: https://github.com/apache/calcite/pull/866
>

Re: [DISCUSS] Draft board report for January 2019

[Michael Mior]

Thanks Francis! My last name is spelled Mior, not Moir, but other than that
looks good to me. I have no opinion on whether the problem with PRs should
be placed under "Issues" so I'll defer to Julian's Apache experience on
that.

Technical solutions are certainly not going to solve the problem
completely, but one advantage of the Gitbox migration is that committers
now have additional permissions on GitHub, we have some extra options. A
couple things which may be helpful:

1. Using labels to categorize PRs. (I'm not sure if this would actually be
useful, but it's an option we didn't have before.)
2. The merge button on PR pages. To make use of this we would have to do
away with the policy of adding the contributor's name to the commit message
in this case. However, since the commit would be made by the author of the
PR, I don't think this is a problem.


--
Michael Mior
mm...@apache.org


Le mer. 2 janv. 2019 à 21:27, Francis Chuang  a
écrit :

> Thanks, Julian!
>
> Updated report as follows:
>
> ## Description:
>
> Apache Calcite is a highly customizable framework for parsing and
> planning queries on data in a wide variety of formats. It allows
> database-like access, and in particular a SQL interface and advanced
> query optimization, for data not residing in a traditional database.
>
> Avatica is a sub-project within Calcite and provides a framework for
> building local and remote JDBC and ODBC database drivers. Avatica has an
> independent release schedule and its own repository.
>
> ## Issues:
>
> - There are no issues requiring board attention at this time.
>
> ## Activity:
> Development and mailing list activity is steady for both Calcite and its
> Avatica sub-project.
>
> Calcite 1.18 was released towards the end of December with significant
> enhancements to the SQL dialect: JSON functions, linear regression
> functions and the WITHIN GROUP clause for aggregate functions.
>
> Avatica 1.13 was released in early December and included numerous bug
> fixes, upgraded dependencies and docker images for the HSQLDB flavor of
> Avatica.
>
> The PMC Chair was switched from Michael Moir to Francis Chuang on Dec 21
> 2018, following the project's tradition of rotating the PMC chair each
> year.
>
> We continue to have a steady amount of contributions from new and
> existing community members.
>
> All of our git repositories (calcite, calcite-avatica and
> calcite-avatica-go) were migrated to gitbox.
>
> Finally, the project received a lot of excellent contributions, but a
> number of them were not able to be merged into the Calcite 1.18 release
> due to pull requests not being reviewed in a timely manner.
>
> ## Health report:
> Activity levels on mailing lists, git and JIRA are normal for both
> Calcite and Avatica.
>
> ## PMC changes:
>
>   - Currently 19 PMC members.
>   - No new PMC members added in the last 3 months
>   - Last PMC addition was Kevin Risden on Mon Jul 09 2018
>
> ## Committer base changes:
>
>   - Currently 35 committers.
>   - No new committers added in the last 3 months
>   - Last committer addition was Andrei Sereda at Fri Sep 14 2018
>
> ## Releases:
>
>   - 1.18.0 was released on Fri Dec 21 2018
>   - avatica-1.13.0 was released on Tue Dec 04 2018
>   - avatica-go-3.2.0 was released on Mon Sep 17 2018
>
> ## JIRA activity:
>
>   - 154 JIRA tickets created in the last 3 months
>   - 105 JIRA tickets closed/resolved in the last 3 months
>
>
> On 3/01/2019 1:20 pm, Julian Hyde wrote:
> > I wouldn’t put it into “issues” because, as you say, there is nothing
> the Board can do. It’s rare to put anything into “issues” because the Board
> does not have powers to act (except in major ways, such as firing the
> entire PMC).
> >
> > I would put it into “activity”. It is normal project activity to
> identify and solve issues with our development process. The Board will be
> pleased that we have identified issues and are working on them. Board
> members, as individuals not on behalf of the Board, may well step forward
> to give us advice.
> >
> > Julian
> >
> >
> >> On Jan 2, 2019, at 6:10 PM, Francis Chuang 
> wrote:
> >>
> >> Thanks, Julian!
> >>
> >> I mentioned changing the PMC Chair and the migration to gitbox.
> >>
> >> I also added a bit about the reviewing of PRs under "issues". I am not
> quite sure what the board can do to help us, but I do hope this is
> something we can address soon for Calcite. Avatica and Avatica-Go seem to
> be fine, but they don't receive a lot of PRs.
> >>
> >> Updated report attached:
> >> ## Description:
> >

Re: Support of Regex in CalciteAssert

[Michael Mior]

+1 to what Julian said.

--
Michael Mior
mm...@apache.org


Le mer. 2 janv. 2019 à 19:57, Julian Hyde  a écrit :

> I wouldn’t do regex by default. Quite a few characters are regex meta
> characters (e.g. “(“ and “.”) and it’s a pain to have to remember to escape
> them when copy-pasting output into the test.
>
> So, my vote is for
>
> >  .returns(regexMatch("id=\\p{Graph}+; a=1")) // separate consumer
>
> I don’t have an opinion on group capturing.
>
> > On Jan 2, 2019, at 4:26 PM, Andrei Sereda  wrote:
> >
> > Hello,
> >
> >
> > As part of [CALCITE-2755](
> https://issues.apache.org/jira/browse/CALCITE-2755)
> > ([982](https://github.com/apache/calcite/pull/982)) I needed to perform
> > regex comparison of result (since it is not known in advance).
> >
> > The assertion looks like:
> >
> > ```java
> > assertThat()
> > .query("select id, a from view")
> > .returns(regexMatch("id=\\p{Graph}+; a=1"))
> > ```
> > Do you think CalciteAssert might benefit from regular expressions ?
> Should
> > it support expressions natively or as standalone Consumer implementation
> ?
> > Should it also support [group capturing](
> > https://docs.oracle.com/javase/tutorial/essential/regex/groups.html) ?
> If
> > so, what is recommended syntax ?
> >
> > ### Syntax examples
> > ```java
> > .returns(regexMatch("id=\\p{Graph}+; a=1")) // separate consumer
> > .returns("id=\\p{Graph}+; a=1") // native support
> > .returns("id=REGEX{\\p{Graph}+}; a=1") // special syntax
> > .returns("id=(\\p{Graph}+); a=$1-test") // group capturing ?
> > ```
> >
> > Regards,
> > Andrei.
>
>

[jira] [Created] (CALCITE-2760) Elasticsearch adapter can't sort on umapped fields

[Michael Mior (JIRA)]

Michael Mior created CALCITE-2760:
-

 Summary: Elasticsearch adapter can't sort on umapped fields
 Key: CALCITE-2760
 URL: https://issues.apache.org/jira/browse/CALCITE-2760
 Project: Calcite
  Issue Type: Bug
  Components: elasticsearch-adapter
Affects Versions: 1.17.0
Reporter: Michael Mior
Assignee: Julian Hyde


When trying to sort on unmapped fields, the adapter throws an exception. It 
seems like {{ElasticsearchSortRule}} should check if the field is mapped before 
it is applied so the sort can be performed by Calcite if Elasticsearch can't do 
it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Michael Mior]

Yup, that did the trick. Thanks!

--
Michael Mior
mm...@apache.org


Le ven. 28 déc. 2018 à 14:33, Julian Hyde  a écrit :

> The release news was disappearing because I’d forgotten to commit it to
> master. The problem should be fixed. When you regenerate the site using
> Jekyll, the only changes you should see will be your own.
>
> > On Dec 28, 2018, at 11:29 AM, Michael Mior  wrote:
> >
> > Sorry for the silence on this.
> >
> > Julian, I'd like to deploy the site, but when I build, I see the latest
> > release announcement disappear. Given your reminder that site and master
> > should be the same after a release, I'll try to sort through what's going
> > on later today.
> >
> > On Thu, Dec 27, 2018, 16:04 Julian Hyde  >
> >> Thanks, Francis.
> >>
> >> $ git remote set-url origin
> >> https://gitbox.apache.org/repos/asf/calcite.git <
> >> https://gitbox.apache.org/repos/asf/calcite.git>
> >>
> >> was sufficient to allow me to pull/fetch. And after I linked accounts
> >> using https://gitbox.apache.org/ <https://gitbox.apache.org/> I was
> able
> >> to also push.
> >>
> >> The “site” branch is now in sync with “master”, which is as it should
> be,
> >> right after a release.
> >>
> >> Michael, Did you know that your commit
> >>
> https://github.com/apache/calcite/commit/9d50e6d7418579c5a73d872e6aec5924ed97c239
> >> <
> >>
> https://github.com/apache/calcite/commit/9d50e6d7418579c5a73d872e6aec5924ed97c239
> >
> >> is in git but is not deployed to the site (via svn)? I’ll let you
> publish
> >> when you are ready.
> >>
> >> Julian
> >>
> >>
> >>> On Dec 25, 2018, at 11:46 PM, Francis Chuang  >
> >> wrote:
> >>>
> >>> I haven't tried pushing directly to Github yet, but did you link your
> >> Github and Apache accounts here: https://gitbox.apache.org/ ?
> >>>
> >>>> On 26/12/2018 6:35 pm, Julian Hyde wrote:
> >>>> Regarding the “site” branch. After a release, it should always point
> to
> >> the same as “master”. (It may diverge later, if there are changes made
> to
> >> the web site in master branch that we do not wish to immediately
> publish.)
> >> Sorry I forgot to push to site.
> >>>>
> >>>> Regarding gitbox. How should I set up my environment? I previously had
> >> origin as follows:
> >>>>
> >>>> $ git remote -v |grep origin
> >>>> origin   https://git-wip-us.apache.org/repos/asf/calcite.git
> >> (fetch)
> >>>> origin   https://git-wip-us.apache.org/repos/asf/calcite.git
> (push)
> >>>>
> >>>> Now I point it to GitHub, as follows:
> >>>>
> >>>> $ git remote set-url origin g...@github.com:apache/calcite.git
> >>>>
> >>>> but I am not able to push.
> >>>>
> >>>> Julian
> >>>>
> >>>>
> >>>>> On Dec 25, 2018, at 7:56 PM, Andrei Sereda  wrote:
> >>>>>
> >>>>> Just in case.
> >>>>>
> >>>>> I made some changes to elastic adapter documentation (but didn't
> deploy
> >>>>> them)  :
> >>>>> 1) c3e076c0
> >>>>> <
> >>
> https://github.com/apache/calcite/commit/c3e076c0b8bc3b4f69d90cf08f7c6a082a4fe564
> >>>
> >>>>>
> >>>>> 2) af739cbe
> >>>>> <
> >>
> https://github.com/apache/calcite/commit/af739cbef3799512d095134567ca715b1179611a
> >>>
> >>>>>
> >>>>> My changes should be in both master and site branches.
> >>>>>
> >>>>> On Tue, Dec 25, 2018 at 9:48 PM Francis Chuang <
> >> francischu...@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> When I deployed the changes to update the PMC Chair, I deployed the
> >>>>>> master branch. I checked out the site branch, but it appeared to be
> >>>>>> missing the site commits for Calcite 1.18 release and since I don't
> >>>>>> commit to Calcite often, I didn't want to cherry-pick the commits
> into
> >>>>>> the site branch and potentially break things.
> >>>>>>
> >>>>>> I think deploying the master branch should be safe

Re: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Michael Mior]

Sorry for the silence on this.

Julian, I'd like to deploy the site, but when I build, I see the latest
release announcement disappear. Given your reminder that site and master
should be the same after a release, I'll try to sort through what's going
on later today.

On Thu, Dec 27, 2018, 16:04 Julian Hyde  Thanks, Francis.
>
> $ git remote set-url origin
> https://gitbox.apache.org/repos/asf/calcite.git <
> https://gitbox.apache.org/repos/asf/calcite.git>
>
> was sufficient to allow me to pull/fetch. And after I linked accounts
> using https://gitbox.apache.org/ <https://gitbox.apache.org/> I was able
> to also push.
>
> The “site” branch is now in sync with “master”, which is as it should be,
> right after a release.
>
> Michael, Did you know that your commit
> https://github.com/apache/calcite/commit/9d50e6d7418579c5a73d872e6aec5924ed97c239
> <
> https://github.com/apache/calcite/commit/9d50e6d7418579c5a73d872e6aec5924ed97c239>
> is in git but is not deployed to the site (via svn)? I’ll let you publish
> when you are ready.
>
> Julian
>
>
> > On Dec 25, 2018, at 11:46 PM, Francis Chuang 
> wrote:
> >
> > I haven't tried pushing directly to Github yet, but did you link your
> Github and Apache accounts here: https://gitbox.apache.org/ ?
> >
> > On 26/12/2018 6:35 pm, Julian Hyde wrote:
> >> Regarding the “site” branch. After a release, it should always point to
> the same as “master”. (It may diverge later, if there are changes made to
> the web site in master branch that we do not wish to immediately publish.)
> Sorry I forgot to push to site.
> >>
> >> Regarding gitbox. How should I set up my environment? I previously had
> origin as follows:
> >>
> >> $ git remote -v |grep origin
> >> origin   https://git-wip-us.apache.org/repos/asf/calcite.git
> (fetch)
> >> origin   https://git-wip-us.apache.org/repos/asf/calcite.git (push)
> >>
> >> Now I point it to GitHub, as follows:
> >>
> >> $ git remote set-url origin g...@github.com:apache/calcite.git
> >>
> >> but I am not able to push.
> >>
> >> Julian
> >>
> >>
> >>> On Dec 25, 2018, at 7:56 PM, Andrei Sereda  wrote:
> >>>
> >>> Just in case.
> >>>
> >>> I made some changes to elastic adapter documentation (but didn't deploy
> >>> them)  :
> >>> 1) c3e076c0
> >>> <
> https://github.com/apache/calcite/commit/c3e076c0b8bc3b4f69d90cf08f7c6a082a4fe564
> >
> >>>
> >>> 2) af739cbe
> >>> <
> https://github.com/apache/calcite/commit/af739cbef3799512d095134567ca715b1179611a
> >
> >>>
> >>> My changes should be in both master and site branches.
> >>>
> >>> On Tue, Dec 25, 2018 at 9:48 PM Francis Chuang <
> francischu...@apache.org>
> >>> wrote:
> >>>
> >>>> When I deployed the changes to update the PMC Chair, I deployed the
> >>>> master branch. I checked out the site branch, but it appeared to be
> >>>> missing the site commits for Calcite 1.18 release and since I don't
> >>>> commit to Calcite often, I didn't want to cherry-pick the commits into
> >>>> the site branch and potentially break things.
> >>>>
> >>>> I think deploying the master branch should be safe for now, because
> 1.18
> >>>> was just released and there hasn't been any changes to the site for
> 1.19
> >>>> yet.
> >>>>
> >>>> On 26/12/2018 1:02 pm, Michael Mior wrote:
> >>>>> I've made commits to master to update the site but somehow my site
> branch
> >>>>> seems to be out of sync with what's deployed. If someone who has
> deployed
> >>>>> the site recently has a chance to check it out that would be great.
> >>>>>
> >>>>> --
> >>>>> Michael Mior
> >>>>> mm...@apache.org
> >>>>>
> >>>>>
> >>>>> Le mar. 25 déc. 2018 à 16:20, Michael Mior 
> a
> >>>>> écrit :
> >>>>>
> >>>>>> The migration is already done to my surprise. I'll try to update the
> >>>> site
> >>>>>> and send out an announcement to committers later today.
> >>>>>>
> >>>>>> On Tue, Dec 25, 2018, 16:15 Francis Chuang <
> francischu...@apache.org
> >>

Re: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Michael Mior]

I've made commits to master to update the site but somehow my site branch
seems to be out of sync with what's deployed. If someone who has deployed
the site recently has a chance to check it out that would be great.

--
Michael Mior
mm...@apache.org


Le mar. 25 déc. 2018 à 16:20, Michael Mior  a
écrit :

> The migration is already done to my surprise. I'll try to update the site
> and send out an announcement to committers later today.
>
> On Tue, Dec 25, 2018, 16:15 Francis Chuang  wrote:
>
>> Thanks for sorting this out, Michael! Can't wait to try it!
>>
>> On 26/12/2018 2:13 am, Michael Mior wrote:
>> > Since there have been no objections and the release is now complete,
>> I've
>> > requested the migration.
>> >
>> > https://issues.apache.org/jira/browse/INFRA-17498
>> >
>> > --
>> > Michael Mior
>> > mm...@apache.org
>> >
>> >
>> > Le ven. 7 déc. 2018 à 11:54, Daniel Gruno  a
>> écrit :
>> >
>> >> [IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE
>> >>DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS]
>> >>
>> >> Hello Apache projects,
>> >>
>> >> I am writing to you because you may have git repositories on the
>> >> git-wip-us server, which is slated to be decommissioned in the coming
>> >> months. All repositories will be moved to the new gitbox service which
>> >> includes direct write access on github as well as the standard ASF
>> >> commit access via gitbox.apache.org.
>> >>
>> >> ## Why this move? ##
>> >> The move comes as a result of retiring the git-wip service, as the
>> >> hardware it runs on is longing for retirement. In lieu of this, we
>> >> have decided to consolidate the two services (git-wip and gitbox), to
>> >> ease the management of our repository systems and future-proof the
>> >> underlying hardware. The move is fully automated, and ideally, nothing
>> >> will change in your workflow other than added features and access to
>> >> GitHub.
>> >>
>> >> ## Timeframe for relocation ##
>> >> Initially, we are asking that projects voluntarily request to move
>> >> their repositories to gitbox, hence this email. The voluntary
>> >> timeframe is between now and January 9th 2019, during which projects
>> >> are free to either move over to gitbox or stay put on git-wip. After
>> >> this phase, we will be requiring the remaining projects to move within
>> >> one month, after which we will move the remaining projects over.
>> >>
>> >> To have your project moved in this initial phase, you will need:
>> >>
>> >> - Consensus in the project (documented via the mailing list)
>> >> - File a JIRA ticket with INFRA to voluntarily move your project repos
>> >> over to gitbox (as stated, this is highly automated and will take
>> >> between a minute and an hour, depending on the size and number of
>> >> your repositories)
>> >>
>> >> To sum up the preliminary timeline;
>> >>
>> >> - December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
>> >> relocation
>> >> - January 9th -> February 6th: Mandated (coordinated) relocation
>> >> - February 7th: All remaining repositories are mass migrated.
>> >>
>> >> This timeline may change to accommodate various scenarios.
>> >>
>> >> ## Using GitHub with ASF repositories ##
>> >> When your project has moved, you are free to use either the ASF
>> >> repository system (gitbox.apache.org) OR GitHub for your development
>> >> and code pushes. To be able to use GitHub, please follow the primer
>> >> at: https://reference.apache.org/committer/github
>> >>
>> >>
>> >> We appreciate your understanding of this issue, and hope that your
>> >> project can coordinate voluntarily moving your repositories in a
>> >> timely manner.
>> >>
>> >> All settings, such as commit mail targets, issue linking, PR
>> >> notification schemes etc will automatically be migrated to gitbox as
>> >> well.
>> >>
>> >> With regards, Daniel on behalf of ASF Infra.
>> >>
>> >> PS:For inquiries, please reply to us...@infra.apache.org, not your
>> >> project's dev list :-).
>> >>
>> >>
>> >>
>>
>>

Re: PLC4X Adapter for Calcite

[Michael Mior]

Great! Thanks for sharing Julian.

--
Michael Mior
mm...@apache.org


Le mar. 25 déc. 2018 à 14:23, Julian Feinauer 
a écrit :

> Hi all,
>
> I am kind of cross posting this but I think it could be interesting for
> both communities, see my original post on the plc4x dev ML [1].
> I just finished a first implementation of a PLC4X-Calcite Adapter.
> What one can do with that is to create a Table (Scannable or Streamable)
> with values that are “scraped” regularly from PLCs.
>
> Perhaps this helps a bit in the timeseries / signal processing discussions
> we have here.
>
> If there are any questions regarding this, please feel free to ask.
> And if this would be of interest for calcite we could also duplicate the
> code to calcite, I think.
>
> Best
> Julian
>
> [1]
> https://lists.apache.org/thread.html/ea5837a2ee0ee88ffca678c553c892574e266264a0709920360fe781@%3Cdev.plc4x.apache.org%3E
>

Re: calcite git commit: Site: Update PMC chair

[Michael Mior]

Thanks for this Francis. Slipped my mind when I made the announcement.


--
Michael Mior
mm...@apache.org


Le sam. 22 déc. 2018 à 17:47,  a écrit :

> Repository: calcite
> Updated Branches:
>   refs/heads/master 5447b9ce6 -> a4bcea8c5
>
>
> Site: Update PMC chair
>
>
> Project: http://git-wip-us.apache.org/repos/asf/calcite/repo
> Commit: http://git-wip-us.apache.org/repos/asf/calcite/commit/a4bcea8c
> Tree: http://git-wip-us.apache.org/repos/asf/calcite/tree/a4bcea8c
> Diff: http://git-wip-us.apache.org/repos/asf/calcite/diff/a4bcea8c
>
> Branch: refs/heads/master
> Commit: a4bcea8c56527b179d73c826703471f1c56cec37
> Parents: 5447b9c
> Author: Francis Chuang 
> Authored: Sun Dec 23 09:46:08 2018 +1100
> Committer: Francis Chuang 
> Committed: Sun Dec 23 09:46:08 2018 +1100
>
> --
>  site/_data/contributors.yml | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> --
>
>
>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/a4bcea8c/site/_data/contributors.yml
> --
> diff --git a/site/_data/contributors.yml b/site/_data/contributors.yml
> index 1e6fcfc..a895135 100644
> --- a/site/_data/contributors.yml
> +++ b/site/_data/contributors.yml
> @@ -57,7 +57,7 @@
>apacheId: francischuang
>githubId: F21
>org: Boostport
> -  role: PMC
> +  role: PMC Chair
>  - name: Gian Merlino
>apacheId: gian
>githubId: gianm
> @@ -128,7 +128,7 @@
>apacheId: mmior
>githubId: michaelmior
>org: Rochester Institute of Technology
> -  role: PMC Chair
> +  role: PMC
>homepage: https://michael.mior.ca/
>  - name: Milinda Pathirage
>apacheId: milinda
>
>

Re: New JIRA label "pull-request-available"

[Michael Mior]

I would be -1 on removing committer status due to inactivity, but whatever
we can do to encourage continued contributions is good. Thanks for the tag
(and everything else you've been doing)! This is a poor excuse, but I've
still made use of a fairly limited portion of Calcite's code base so I find
most PRs are in code I'm really not comfortable with reviewing. As I work
on the notebooks I posted earlier, I'm hoping to expand my familiarity so
I'll be more comfortable with some other pieces.

--
Michael Mior
mm...@apache.org


Le mar. 25 déc. 2018 à 16:19, Francis Chuang  a
écrit :

> Julian, thanks for doing the legwork to get those issues tagged. I agree
> that we need to keep up with the pull requests in order to encourage
> contributors to contribute to the project.
>
> Stamatis, it's Apache policy that committer status never expires. I
> think some projects turn inactive committers into emeritus committers,
> but I don't think it's common. See
> https://www.apache.org/dev/committers.html#committer-set-term
>
> On 25/12/2018 2:35 am, Stamatis Zampetakis wrote:
> > The main problem seems to be the fact that there are not enough active
> > committers.
> >
> > Between Calcite 1.17.0 and Calcite 1.18.0 the stats regarding the reviews
> > and patches are as follows:
> > Committer Reviews Insertions(+) Deletions(-) Modifications
> > Julian Hyde 52 9039 2304 11343
> > Vladimir Sitnikov 19 865 207 1072
> > Jesus Camacho Rodriguez 7 657 292 949
> > Andrei Sereda 1 472 38 510
> > Michael Mior 3 284 18 302
> > Volodymyr Vysotskyi 1 52 19 71
> > Grand Total 83 11369 2878 14247
> >
> > Unfortunately, there is no easy way to fix this. The only measure that
> > could engage committers is to adopting a policy for removing inactive
> > members (I do not know if there is already one in place). It can be
> > something very light, just to ensure a minimum interaction with the
> > project. It will not dramatically increase the previous numbers but if
> > every committer did at least one review there wouldn't be so many open
> pull
> > requests and the work load of people in the above list could be
> > significantly smaller.
> >
> >
> >
> > Στις Κυρ, 23 Δεκ 2018 στις 1:52 π.μ., ο/η Julian Hyde 
> > έγραψε:
> >
> >> I went through JIRA and added the label "pull-request-available" to
> >> about 30 PRs that had been marked "fix for 1.18" but we didn't get
> >> to[1].
> >>
> >> I suggest that we start using the "pull-request-available" label more
> >> often.
> >>
> >> But my bigger point is that we letting our community down when we have
> >> such a large backlog of un-merged patches. The committers of this
> >> project, collectively, are not staying on top of their work load.
> >>
> >> Personally, I have been doing as much as I can, including taking on
> >> the release manager role when no one else stepped up, and personally
> >> committing a very large fraction of the 200+ commits in this release.
> >> Maintaining Calcite is NOT part of my responsibilities at my day-job,
> >> so I am doing this all in my personal time. It makes me angry that I
> >> am shouldering this much of the responsibility. Something needs to
> >> change around here.
> >>
> >> Julian
> >>
> >> [1]
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20CALCITE%20and%20labels%20%3D%20pull-request-available
> >>
>
>

Re: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Michael Mior]

The migration is already done to my surprise. I'll try to update the site
and send out an announcement to committers later today.

On Tue, Dec 25, 2018, 16:15 Francis Chuang  Thanks for sorting this out, Michael! Can't wait to try it!
>
> On 26/12/2018 2:13 am, Michael Mior wrote:
> > Since there have been no objections and the release is now complete, I've
> > requested the migration.
> >
> > https://issues.apache.org/jira/browse/INFRA-17498
> >
> > --
> > Michael Mior
> > mm...@apache.org
> >
> >
> > Le ven. 7 déc. 2018 à 11:54, Daniel Gruno  a
> écrit :
> >
> >> [IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE
> >>DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS]
> >>
> >> Hello Apache projects,
> >>
> >> I am writing to you because you may have git repositories on the
> >> git-wip-us server, which is slated to be decommissioned in the coming
> >> months. All repositories will be moved to the new gitbox service which
> >> includes direct write access on github as well as the standard ASF
> >> commit access via gitbox.apache.org.
> >>
> >> ## Why this move? ##
> >> The move comes as a result of retiring the git-wip service, as the
> >> hardware it runs on is longing for retirement. In lieu of this, we
> >> have decided to consolidate the two services (git-wip and gitbox), to
> >> ease the management of our repository systems and future-proof the
> >> underlying hardware. The move is fully automated, and ideally, nothing
> >> will change in your workflow other than added features and access to
> >> GitHub.
> >>
> >> ## Timeframe for relocation ##
> >> Initially, we are asking that projects voluntarily request to move
> >> their repositories to gitbox, hence this email. The voluntary
> >> timeframe is between now and January 9th 2019, during which projects
> >> are free to either move over to gitbox or stay put on git-wip. After
> >> this phase, we will be requiring the remaining projects to move within
> >> one month, after which we will move the remaining projects over.
> >>
> >> To have your project moved in this initial phase, you will need:
> >>
> >> - Consensus in the project (documented via the mailing list)
> >> - File a JIRA ticket with INFRA to voluntarily move your project repos
> >> over to gitbox (as stated, this is highly automated and will take
> >> between a minute and an hour, depending on the size and number of
> >> your repositories)
> >>
> >> To sum up the preliminary timeline;
> >>
> >> - December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
> >> relocation
> >> - January 9th -> February 6th: Mandated (coordinated) relocation
> >> - February 7th: All remaining repositories are mass migrated.
> >>
> >> This timeline may change to accommodate various scenarios.
> >>
> >> ## Using GitHub with ASF repositories ##
> >> When your project has moved, you are free to use either the ASF
> >> repository system (gitbox.apache.org) OR GitHub for your development
> >> and code pushes. To be able to use GitHub, please follow the primer
> >> at: https://reference.apache.org/committer/github
> >>
> >>
> >> We appreciate your understanding of this issue, and hope that your
> >> project can coordinate voluntarily moving your repositories in a
> >> timely manner.
> >>
> >> All settings, such as commit mail targets, issue linking, PR
> >> notification schemes etc will automatically be migrated to gitbox as
> >> well.
> >>
> >> With regards, Daniel on behalf of ASF Infra.
> >>
> >> PS:For inquiries, please reply to us...@infra.apache.org, not your
> >> project's dev list :-).
> >>
> >>
> >>
>
>

Re: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Michael Mior]

Since there have been no objections and the release is now complete, I've
requested the migration.

https://issues.apache.org/jira/browse/INFRA-17498

--
Michael Mior
mm...@apache.org


Le ven. 7 déc. 2018 à 11:54, Daniel Gruno  a écrit :

> [IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE
>   DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS]
>
> Hello Apache projects,
>
> I am writing to you because you may have git repositories on the
> git-wip-us server, which is slated to be decommissioned in the coming
> months. All repositories will be moved to the new gitbox service which
> includes direct write access on github as well as the standard ASF
> commit access via gitbox.apache.org.
>
> ## Why this move? ##
> The move comes as a result of retiring the git-wip service, as the
> hardware it runs on is longing for retirement. In lieu of this, we
> have decided to consolidate the two services (git-wip and gitbox), to
> ease the management of our repository systems and future-proof the
> underlying hardware. The move is fully automated, and ideally, nothing
> will change in your workflow other than added features and access to
> GitHub.
>
> ## Timeframe for relocation ##
> Initially, we are asking that projects voluntarily request to move
> their repositories to gitbox, hence this email. The voluntary
> timeframe is between now and January 9th 2019, during which projects
> are free to either move over to gitbox or stay put on git-wip. After
> this phase, we will be requiring the remaining projects to move within
> one month, after which we will move the remaining projects over.
>
> To have your project moved in this initial phase, you will need:
>
> - Consensus in the project (documented via the mailing list)
> - File a JIRA ticket with INFRA to voluntarily move your project repos
>over to gitbox (as stated, this is highly automated and will take
>between a minute and an hour, depending on the size and number of
>your repositories)
>
> To sum up the preliminary timeline;
>
> - December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
>relocation
> - January 9th -> February 6th: Mandated (coordinated) relocation
> - February 7th: All remaining repositories are mass migrated.
>
> This timeline may change to accommodate various scenarios.
>
> ## Using GitHub with ASF repositories ##
> When your project has moved, you are free to use either the ASF
> repository system (gitbox.apache.org) OR GitHub for your development
> and code pushes. To be able to use GitHub, please follow the primer
> at: https://reference.apache.org/committer/github
>
>
> We appreciate your understanding of this issue, and hope that your
> project can coordinate voluntarily moving your repositories in a
> timely manner.
>
> All settings, such as commit mail targets, issue linking, PR
> notification schemes etc will automatically be migrated to gitbox as
> well.
>
> With regards, Daniel on behalf of ASF Infra.
>
> PS:For inquiries, please reply to us...@infra.apache.org, not your
> project's dev list :-).
>
>
>

Re: [ANNOUNCE] Apache Calcite 1.18.0 released

[Michael Mior]

Thanks for serving as RM yet again Julian! Now that the release is out and
we have some slower time during the holidays, I'll ping INFRA about the
voluntary migration to the new git server.

--
Michael Mior
mm...@apache.org


Le ven. 21 déc. 2018 à 19:25, Julian Hyde  a écrit :

> The Apache Calcite team is pleased to announce the release of
> Apache Calcite 1.18.0.
>
> Calcite is a dynamic data management framework. Its cost-based
> optimizer converts queries, represented in relational algebra,
> into executable plans. Calcite supports many front-end languages
> and back-end data engines, and includes an SQL parser and the
> Avatica JDBC driver.
>
> With over 200 commits from 36 contributors, this is the largest
> Calcite release ever. To the SQL dialect, we added JSON functions and
> linear regression functions, and the WITHIN GROUP clause for aggregate
> functions; there is a new utility to recommend lattices based on past
> queries, and improvements to expression simplification, the SQL
> advisor, and the Elasticsearch and Apache Geode adapters. For more
> details, see the release notes:
>
>   https://calcite.apache.org/docs/history.html#v1-18-0
>
> The release is available here:
>
>https://calcite.apache.org/downloads/
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
>
>https://calcite.apache.org
>
> Julian Hyde, on behalf of the Apache Calcite Team
>

[ANNOUNCE] New Calcite PMC chair: Francis Chuang

[Michael Mior]

Calcite community members,

I am pleased to announce that we have a new PMC chair and VP. I have
resigned, and Francis was duly elected by the PMC and approved unanimously
by the Board.

Please join me in congratulating Francis!

-Michael

PS - This should not be indicative of a lack of continued interest in
Calcite on my part, but rather a desire on behalf of the entire PMC (myself
included) to have this responsibility regularly shared.

Re: Calcite example code

[Michael Mior]

Here's another one focused on basic query optimization

https://github.com/michaelmior/calcite-notebooks/blob/master/query-optimization.ipynb

--
Michael Mior
mm...@apache.org


Le mer. 19 déc. 2018 à 18:21, Michael Mior  a écrit :

> Yes the notebook contains the output (it's really just a JSON file). It
> doesn't necessarily have to have output, but it's much more useful if it
> does since it means whoever is viewing doesn't need to execute it. It would
> certainly be possible to use this for tests although it would require an
> installation of Python. Given how ubiquitous Python is, I don't think this
> is a huge concern, although we'd need a way of installing a couple Python
> dependencies.
>
> As Kevin mentioned, you can execute these online with Binder. I just had
> to add a Dockerfile so it could run the IJava kernel since it not the
> default. Check out the link below and you can see what the experience is
> like.
>
> https://mybinder.org/v2/gh/michaelmior/calcite-notebooks/master
>
> Just select a notebook once it loads (it may take a couple minutes). The
> experience is basically the same as what you get when running locally. The
> notebook consists of a series of "cells" which you can run individually and
> edit as you wish. This would also make it easy for people to play around
> with Calcite a little without having to install anything.
>
> --
> Michael Mior
> mm...@apache.org
>
>
> Le mer. 19 déc. 2018 à 17:22, Julian Hyde  a écrit :
>
>> For old idiots like me, can you explain how the notebook works? The file
>> you checked into GitHub, does it contain the input and output or just the
>> input? Is there a way to edit or use the notebook interactively?
>>
>> It certainly seems a better way to introduce people to examples than
>> saying “go look at this test”.
>>
>> I think quite a few of our tests could be converted into this format.
>>
>> Julian
>>
>>
>> > On Dec 19, 2018, at 10:52 AM, Michael Mior  wrote:
>> >
>> > After seeing so many people ask for example code to do certain basic
>> things
>> > in Calcite, I've been trying to find a good literate programming
>> solution
>> > for Java as I like this approach for demoing. I recently came across the
>> > IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter
>> notebooks.
>> >
>> > This is basically just a proof of concept at this point, but here's a
>> > simple example
>> >
>> >
>> https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb
>> >
>> > I'm curious what others think of this approach. If others think it
>> would be
>> > useful, I'd be happy to take suggestions on what should be included.
>> > Eventually, I'd like to get CI set up for this repository so I can
>> re-run
>> > the notebooks at will. I would then aim to check this on every release
>> so
>> > we can have a repository of code samples which we know run correctly.
>> >
>> > --
>> > Michael Mior
>> > mm...@apache.org
>>
>>

Re: Calcite example code

[Michael Mior]

Yes the notebook contains the output (it's really just a JSON file). It
doesn't necessarily have to have output, but it's much more useful if it
does since it means whoever is viewing doesn't need to execute it. It would
certainly be possible to use this for tests although it would require an
installation of Python. Given how ubiquitous Python is, I don't think this
is a huge concern, although we'd need a way of installing a couple Python
dependencies.

As Kevin mentioned, you can execute these online with Binder. I just had to
add a Dockerfile so it could run the IJava kernel since it not the default.
Check out the link below and you can see what the experience is like.

https://mybinder.org/v2/gh/michaelmior/calcite-notebooks/master

Just select a notebook once it loads (it may take a couple minutes). The
experience is basically the same as what you get when running locally. The
notebook consists of a series of "cells" which you can run individually and
edit as you wish. This would also make it easy for people to play around
with Calcite a little without having to install anything.

--
Michael Mior
mm...@apache.org


Le mer. 19 déc. 2018 à 17:22, Julian Hyde  a écrit :

> For old idiots like me, can you explain how the notebook works? The file
> you checked into GitHub, does it contain the input and output or just the
> input? Is there a way to edit or use the notebook interactively?
>
> It certainly seems a better way to introduce people to examples than
> saying “go look at this test”.
>
> I think quite a few of our tests could be converted into this format.
>
> Julian
>
>
> > On Dec 19, 2018, at 10:52 AM, Michael Mior  wrote:
> >
> > After seeing so many people ask for example code to do certain basic
> things
> > in Calcite, I've been trying to find a good literate programming solution
> > for Java as I like this approach for demoing. I recently came across the
> > IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter
> notebooks.
> >
> > This is basically just a proof of concept at this point, but here's a
> > simple example
> >
> >
> https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb
> >
> > I'm curious what others think of this approach. If others think it would
> be
> > useful, I'd be happy to take suggestions on what should be included.
> > Eventually, I'd like to get CI set up for this repository so I can re-run
> > the notebooks at will. I would then aim to check this on every release so
> > we can have a repository of code samples which we know run correctly.
> >
> > --
> > Michael Mior
> > mm...@apache.org
>
>

Calcite example code

[Michael Mior]

After seeing so many people ask for example code to do certain basic things
in Calcite, I've been trying to find a good literate programming solution
for Java as I like this approach for demoing. I recently came across the
IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter notebooks.

This is basically just a proof of concept at this point, but here's a
simple example

https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb

I'm curious what others think of this approach. If others think it would be
useful, I'd be happy to take suggestions on what should be included.
Eventually, I'd like to get CI set up for this repository so I can re-run
the notebooks at will. I would then aim to check this on every release so
we can have a repository of code samples which we know run correctly.

--
Michael Mior
mm...@apache.org

Re: Info need on the lib encryption

[Michael Mior]

Can you clarify what you mean by "encryption hooks"? Are you hoping to have
some mechanism to decrypt rows as they are read from a particular data
source? Calcite has no built-in mechanism to allow this, but you could wrap
one of the existing adapters to add encryption relatively easily.

--
Michael Mior
mm...@apache.org


Le mar. 18 déc. 2018 à 17:38, Hemakumar Gokulakannan
 a écrit :

> Hello Team,
>
> We are trying to use Apache Calcite core 1.17.0(including
> avatica-core-1.12.0, avatica-metrics-1.12.0 and calcite-linq4j-1.17.0) in
> our application, however we need some info regarding this library.  We do
> want to know whether this library contain encryption or hooks to
> encryption?
>
> Please let us know if you need more info on this.
>
> Regards,
> Hemakumar Gokulakannan
> Associate Member Technical Staff,
> TIBCO Jaspersoft Dev Server | 4800 Sugar Grove Blvd, Stafford TX - 77477
> hgoku...@tibco.com
> +1 (682)-256-4433
>

Re: [VOTE] Release apache-calcite-1.18.0 (release candidate 2)

[Michael Mior]

+1 (binding) Checked signature and hashes and compiled and ran tests.
Thanks Julian!

--
Michael Mior
mm...@apache.org


Le mar. 18 déc. 2018 à 15:00, Julian Hyde  a écrit :

> Hi all,
>
> I have created a build for Apache Calcite 1.18.0, release candidate 2.
>
> Since the previous release candidate (RC1), issues
> https://issues.apache.org/jira/browse/CALCITE-2730 and
> https://issues.apache.org/jira/browse/CALCITE-2731 have
> been fixed, and Zoltan indicates that the Hive test suite now passes.
>
> Thanks to everyone who has contributed to this release.
> You can read the release notes here:
> https://github.com/apache/calcite/blob/branch-1.18/site/_docs/history.md
>
> The commit to be voted upon:
>
> http://git-wip-us.apache.org/repos/asf/calcite/commit/6bca0b80859e86ac41ba1e342460db929cf72268
>
> Its hash is 6bca0b80859e86ac41ba1e342460db929cf72268.
>
> The artifacts to be voted on are located here:
> https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.18.0-rc2
>
> The hashes of the artifacts are as follows:
> src.tar.gz.sha256
> a04bfb1bac830e57475dffdbf5f0c3a797c358460d240f6203c929bac61b47ae
>
> A staged Maven repository is available for review at:
> https://repository.apache.org/content/repositories/orgapachecalcite-1051
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jhyde.asc
>
> Please vote on releasing this package as Apache Calcite 1.18.0.
>
> The vote is open for the next 72 hours (until 12 noon Pacific on Friday)
> and passes if a majority of at least three +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Calcite 1.18.0
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
>
> Here is my vote:
>
> +1 (binding)
>
> Julian
>

Re: Relational algebra and signal processing

[Michael Mior]

I would say a similar theory applies. Some things are different when you're
dealing with streams. Mainly joins and aggregations. Semantics are
necessarily different whenever you have operations involving more than one
row at a time from the input stream. When dealing with a relation an
aggregation is straightforward since you just consume the entire input, and
output the result of the aggregation. Since streams don't end, you need to
decide how this is handled which usually amounts to a choice of windowing
algorithm. There are a few other things to think about. The presentation
linked below from Julian Hyde has a nice overview

https://www.slideshare.net/julianhyde/streaming-sql-62376119

--
Michael Mior
mm...@apache.org


Le mar. 18 déc. 2018 à 02:28, Julian Feinauer 
a écrit :

> Hi Michael,
>
> yes, our workloads are usually in the context of streaming (but for replay
> or so we also use batch).
> But, if I understand it correctly, the same theory applies to both, tables
> ("relations") and streaming tables, or?
> I hope to find time soon to write a PLC4X - Calicte source which creates
> one or many streams based on readings from a plc.
>
> Julian
>
> Am 18.12.18, 03:19 schrieb "Michael Mior" :
>
> Perhaps you've thought of this already, but it sounds like streaming
> relational algebra could be a good fit here.
>
> https://calcite.apache.org/docs/stream.html
> --
> Michael Mior
> mm...@apache.org
>
>
> Le dim. 16 déc. 2018 à 18:39, Julian Feinauer <
> j.feina...@pragmaticminds.de>
> a écrit :
>
> > Hi Calcite-devs,
> >
> > I just had a very interesting mail exchange with Julian (Hyde) on the
> > incubator list [1]. It was about our project CRUNCH (which is mostly
> about
> > time series analyses and signal processing) and its relation to
> relational
> > algebra and I wanted to bring the discussion to this list to
> continue here.
> > We already had some discussion about how time series would work in
> calcite
> > [2] and it’s closely related to MATCH_RECOGNIZE.
> >
> > But, I have a more general question in mind, to ask the experts here
> on
> > the list.
> > I ask myself if we can see the signal processing and analysis tasks
> as
> > proper application of relational algebra.
> > Disclaimer, I’m mathematician, so I know the formals of (relational)
> > algebra pretty well but I’m lacking a lot of experience and
> knowledge in
> > the database theory. Most of my knowledge there comes from Calcites
> source
> > code and the book from Garcia-Molina and Ullman).
> >
> > So if we take, for example, a stream of signals from a sensor, then
> we can
> > of course do filtering or smoothing on it and this can be seen as a
> mapping
> > between the input relation and the output relation. But as we
> usually need
> > more than just one tuple at a time we lose many of the advantages of
> the
> > relational theory. And then, if we analyze the signal, we can again
> model
> > it as a mapping between relations, but the input relation is a “time
> > series” and the output relation consists of “events”, so these are
> in some
> > way different dimensions. In this situation it becomes mostly
> obvious where
> > the main differences between time series and relational algebra are.
> Think
> > of something simple, an event should be registered, whenever the
> signal
> > switches from FALSE to TRUE (so not for every TRUE). This could also
> be
> > modelled with MATCH_RECOGNIZE pretty easily. But, for me it feels
> > “unnatural” because we cannot use any indices (we don’t care about
> the
> > ratio of TRUE and FALSE in the DB, except for probably some very
> rough
> > outer bounds). And we are lacking the “right” information for the
> optimizer
> > like estimations on the number of analysis results.
> > It gets even more complicated when moving to continuous valued
> signals
> > (INT, DOUBLE, …), e.g., temperature readings or something.
> > If we want to analyze the number of times where we have a temperature
> > change of more than 5 degrees in under 4 hours, this should also be
> doable
> > with MATCH_RECOGNIZE but again, there is no index to help us and we
> have no
> > information for the optimizer, so it feels very “black box” for the
> > relational algebra.
> >
> > I’m not sure if you get my point, but for me, the elegance of
> relational
> > algebra was always th

Re: Relational algebra and signal processing

[Michael Mior]

Perhaps you've thought of this already, but it sounds like streaming
relational algebra could be a good fit here.

https://calcite.apache.org/docs/stream.html
--
Michael Mior
mm...@apache.org


Le dim. 16 déc. 2018 à 18:39, Julian Feinauer 
a écrit :

> Hi Calcite-devs,
>
> I just had a very interesting mail exchange with Julian (Hyde) on the
> incubator list [1]. It was about our project CRUNCH (which is mostly about
> time series analyses and signal processing) and its relation to relational
> algebra and I wanted to bring the discussion to this list to continue here.
> We already had some discussion about how time series would work in calcite
> [2] and it’s closely related to MATCH_RECOGNIZE.
>
> But, I have a more general question in mind, to ask the experts here on
> the list.
> I ask myself if we can see the signal processing and analysis tasks as
> proper application of relational algebra.
> Disclaimer, I’m mathematician, so I know the formals of (relational)
> algebra pretty well but I’m lacking a lot of experience and knowledge in
> the database theory. Most of my knowledge there comes from Calcites source
> code and the book from Garcia-Molina and Ullman).
>
> So if we take, for example, a stream of signals from a sensor, then we can
> of course do filtering or smoothing on it and this can be seen as a mapping
> between the input relation and the output relation. But as we usually need
> more than just one tuple at a time we lose many of the advantages of the
> relational theory. And then, if we analyze the signal, we can again model
> it as a mapping between relations, but the input relation is a “time
> series” and the output relation consists of “events”, so these are in some
> way different dimensions. In this situation it becomes mostly obvious where
> the main differences between time series and relational algebra are. Think
> of something simple, an event should be registered, whenever the signal
> switches from FALSE to TRUE (so not for every TRUE). This could also be
> modelled with MATCH_RECOGNIZE pretty easily. But, for me it feels
> “unnatural” because we cannot use any indices (we don’t care about the
> ratio of TRUE and FALSE in the DB, except for probably some very rough
> outer bounds). And we are lacking the “right” information for the optimizer
> like estimations on the number of analysis results.
> It gets even more complicated when moving to continuous valued signals
> (INT, DOUBLE, …), e.g., temperature readings or something.
> If we want to analyze the number of times where we have a temperature
> change of more than 5 degrees in under 4 hours, this should also be doable
> with MATCH_RECOGNIZE but again, there is no index to help us and we have no
> information for the optimizer, so it feels very “black box” for the
> relational algebra.
>
> I’m not sure if you get my point, but for me, the elegance of relational
> algebra was always this optimization stuff, which comes from declarative
> and ends in an “optimal” physical plan. And I do not see how we can use
> much of this for the examples given above.
>
> Perhaps, one solution would be to do the same as for spatial queries (or
> the JSON / JSONB support in postgres, [3]) to add specialized indices,
> statistics and optimizer rules. Then, this would make it more “relational
> algebra”-esque in the sense that there really is a possibility to apply
> transformations to a given query.
>
> What do you think? Do I see things to complicated or am I missing
> something?
>
> Julian
>
> [1]
> https://lists.apache.org/thread.html/1d5a5aae1d4f5f5a966438a2850860420b674f98b0db7353e7b476f2@%3Cgeneral.incubator.apache.org%3E
> [2]
> https://lists.apache.org/thread.html/250575a56165851ab55351b90a26eaa30e84d5bbe2b31203daaaefb9@%3Cdev.calcite.apache.org%3E
> [3] https://www.postgresql.org/docs/9.4/datatype-json.html
>
>

Re: Can’t seem to parse DDL staements

[Michael Mior]

Support for some DDL is missing. See for example
https://issues.apache.org/jira/browse/CALCITE-2663


--
Michael Mior
mm...@apache.org


Le lun. 17 déc. 2018 à 15:40, Dilip Raj Baral  a
écrit :

> Hi, Ted.
>
> Thanks for the information. That was very helpful. I was not aware of
> calcite-server sub project.
>
> Also, on a quick look I found missing support for TRUNCATE TABLE, CREATE
> PROCEDURE and CREATE FUNCTION? Are they not available for real of am I
> missing something?
>
>
> On 17 December 2018 at 11:35:19 AM, Ted Xu (frank...@gmail.com) wrote:
>
> Hi Dilip,
>
> DDL related code including statement parse and data structures are located
> in calcite-server sub-project.
>
> The unit test 'ServerParserTest' may be a good starting point to look into.
> You can find it here
>
> https://github.com/apache/calcite/blob/master/server/src/test/java/org/apache/calcite/test/ServerParserTest.java
>  .
>
> On Mon, Dec 17, 2018 at 1:34 AM Dilip Raj Baral 
> wrote:
>
> > Hi, team.
> >
> > I have been using Apache Calcite for about two weeks now for one of the
> > projects at my company. I have found it fun and pretty useful so far. I
> > have been able to parse DMLs very smoothly. However, when I tried to
> parse
> > DDLs like CREATE TABLE, ALTER TABLE, etc., the SqlParser.parseStmt() or
> > SqlParser.parseQuery() failed.
> >
> > I have detailed the specifics in the following SO question.
> >
> >
> >
> https://stackoverflow.com/questions/53801005/apache-calcite-cant-seem-to-parse-ddl-statements
> >
> > Looking forward to a suggestion.
> >
> > Best regards,
> > Dilip Raj Baral
> > https://diliprajbaral.com
> >
> >
> >
>

Fwd: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Michael Mior]

Hi all,

We have a decision to make regarding repository hosting. INFRA is retiring
git-wip-us.apache.org which Calcite is currently using. We're going to have
to move at some point in the next few months to gitbox.apache.org. The
question is whether we volunteer to do so relatively soon before we'll be
required to move. (More details in the message below.) My vote would be to
volunteer and get the move over with once the latest release is complete.
This move will have the nice advantage that we'll have write access to
GitHub directly.

Anyway, I'd suggest we come to a decision on our strategy relatively soon
so we can start thinking about what might need to change and when. Thanks!

--
Michael Mior
mm...@apache.org


-- Forwarded message -
From: Daniel Gruno 
Date: ven. 7 déc. 2018 à 11:54
Subject: [NOTICE] Mandatory relocation of Apache git repositories on
git-wip-us.apache.org
To: us...@infra.apache.org 


[IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE
  DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS]

Hello Apache projects,

I am writing to you because you may have git repositories on the
git-wip-us server, which is slated to be decommissioned in the coming
months. All repositories will be moved to the new gitbox service which
includes direct write access on github as well as the standard ASF
commit access via gitbox.apache.org.

## Why this move? ##
The move comes as a result of retiring the git-wip service, as the
hardware it runs on is longing for retirement. In lieu of this, we
have decided to consolidate the two services (git-wip and gitbox), to
ease the management of our repository systems and future-proof the
underlying hardware. The move is fully automated, and ideally, nothing
will change in your workflow other than added features and access to
GitHub.

## Timeframe for relocation ##
Initially, we are asking that projects voluntarily request to move
their repositories to gitbox, hence this email. The voluntary
timeframe is between now and January 9th 2019, during which projects
are free to either move over to gitbox or stay put on git-wip. After
this phase, we will be requiring the remaining projects to move within
one month, after which we will move the remaining projects over.

To have your project moved in this initial phase, you will need:

- Consensus in the project (documented via the mailing list)
- File a JIRA ticket with INFRA to voluntarily move your project repos
   over to gitbox (as stated, this is highly automated and will take
   between a minute and an hour, depending on the size and number of
   your repositories)

To sum up the preliminary timeline;

- December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
   relocation
- January 9th -> February 6th: Mandated (coordinated) relocation
- February 7th: All remaining repositories are mass migrated.

This timeline may change to accommodate various scenarios.

## Using GitHub with ASF repositories ##
When your project has moved, you are free to use either the ASF
repository system (gitbox.apache.org) OR GitHub for your development
and code pushes. To be able to use GitHub, please follow the primer
at: https://reference.apache.org/committer/github


We appreciate your understanding of this issue, and hope that your
project can coordinate voluntarily moving your repositories in a
timely manner.

All settings, such as commit mail targets, issue linking, PR
notification schemes etc will automatically be migrated to gitbox as
well.

With regards, Daniel on behalf of ASF Infra.

PS:For inquiries, please reply to us...@infra.apache.org, not your
project's dev list :-).

Re: [VOTE] Release apache-calcite-1.18.0 (release candidate 1)

[Michael Mior]

+1 (binding) checked hashes and signature and compiled and ran tests.
Thanks for the fixes!

--
Michael Mior
mm...@apache.org


Le jeu. 6 déc. 2018 à 03:19, Julian Hyde  a écrit :

> OK, let's try again.
>
> I have created a build for Apache Calcite 1.18.0, release candidate 1.
>
> Thanks to everyone who has contributed to this release.
> Since RC 0, we have fixed the following issues:
>
> * [CALCITE-2726] ReduceExpressionRule oversimplifies filter conditions
> containing nulls
> * [CALCITE-2670] Combine similar JSON aggregate functions in operator table
> * [CALCITE-2468] Validator throws IndexOutOfBoundsException when
> trying to infer operand type from STRUCT return type (Rong Rong)
>
> You can read the release notes here:
> https://github.com/apache/calcite/blob/branch-1.18/site/_docs/history.md
>
> The commit to be voted upon:
>
> http://git-wip-us.apache.org/repos/asf/calcite/commit/06d0526e9e58b88bb6722ae3e66789b9e60a70d1
>
> Its hash is 06d0526e9e58b88bb6722ae3e66789b9e60a70d1.
>
> The artifacts to be voted on are located here:
> https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.18.0-rc1
>
> The hashes of the artifacts are as follows:
> src.tar.gz.sha256
> 88501d9c22c5cd28c2988a1f921fa112386ac819fb1dbcc5f68504b30b8d7dd7
>
> A staged Maven repository is available for review at:
> https://repository.apache.org/content/repositories/orgapachecalcite-1050
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jhyde.asc
>
> Please vote on releasing this package as Apache Calcite 1.18.0.
>
> The vote is open for the next 108 hours (ending at 1pm Pacific on Mon 10th)
> and passes if a majority of at least three +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Calcite 1.18.0
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
>
> Here is my vote:
>
> +1 (binding)
>
> Julian
>

Re: [VOTE] Release apache-calcite-1.18.0 (release candidate 0)

[Michael Mior]

Interesting. I didn't realize the issue was Windows-only. Regardless,
hopefully this will be resolved soon but I'm not worried about it blocking
the release.

--
Michael Mior
mm...@apache.org


Le mer. 5 déc. 2018 à 11:00, Sergey Nuyanzin  a écrit :

>  Michael, thank you for link to the issue.
> I have just tried to disable CassandraAdapterTest for jdk 9+ but only on
> Windows (as on Linux it works fine)
> and it looks much better
> builds on Windows with jdk8-11 ok
> https://ci.appveyor.com/project/snuyanzin/calcite/builds/20789778
>
>
>
> On Wed, Dec 5, 2018 at 6:55 PM Zoltan Haindrich  wrote:
>
> > Hello,
> >
> > I've attached a patch for Hive to run all the tests against the RC - and
> > the results seemed promising; but a null related query is failing with
> > incorrect results.
> >
> >
> https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/ql/src/test/queries/clientpositive/in_typecheck_mixed.q#L12
> > The test seems to be broken by CALCITE-2604; but I feel that this issue
> > was hiding in the bushes - and might have just needed a more
> sophisticated
> > query prior to that patch.
> > I've reproduced the issue in Calcite and opened CALCITE-2726 to address
> it.
> >
> > cheers,
> > Zoltan
> >
> >
> > On 12/4/18 10:00 AM, Julian Hyde wrote:
> > > Hi all,
> > >
> > > I have created a build for Apache Calcite 1.18.0, release candidate 0.
> > >
> > > Thanks to everyone who has contributed to this release.
> > > With over 200 commits from 36 contributors, this is the largest
> > > Calcite release ever!
> > > You can read the release notes here:
> > >
> https://github.com/apache/calcite/blob/branch-1.18/site/_docs/history.md
> > >
> > > The commit to be voted upon:
> > >
> >
> http://git-wip-us.apache.org/repos/asf/calcite/commit/c19e458d2bdbd42dbd450143e0f41fabbfe4ec06
> > >
> > > Its hash is c19e458d2bdbd42dbd450143e0f41fabbfe4ec06.
> > >
> > > The artifacts to be voted on are located here:
> > >
> https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.18.0-rc0
> > >
> > > The hashes of the artifacts are as follows:
> > > src.tar.gz.sha256
> > > 8931ccc08863e4fe394824ead306dc89a7a78a41f898303187a91b01a723cce6
> > >
> > > A staged Maven repository is available for review at:
> > >
> https://repository.apache.org/content/repositories/orgapachecalcite-1049
> > >
> > > Release artifacts are signed with the following key:
> > > https://people.apache.org/keys/committer/jhyde.asc
> > >
> > > Please vote on releasing this package as Apache Calcite 1.18.0.
> > >
> > > The vote is open for the next 72 hours (closing at 1am Pacific on
> Friday)
> > > and passes if a majority of at least three +1 PMC votes are cast.
> > >
> > > [ ] +1 Release this package as Apache Calcite 1.18.0
> > > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> > > [ ] -1 Do not release this package because...
> > >
> > >
> > > Here is my vote:
> > >
> > > +1 (binding)
> > >
> > > Julian
> > >
> >
>
>
> --
> Best regards,
> Sergey
>

Re: [VOTE] Release apache-calcite-1.18.0 (release candidate 0)

[Michael Mior]

Thanks Sergey! CassandraUnit has known issues on Java 9 (
https://github.com/jsevellec/cassandra-unit/issues/249). Perhaps we should
disable that test on Java 9+ until this is fixed?

--
Michael Mior
mm...@apache.org


Le mer. 5 déc. 2018 à 06:53, Sergey Nuyanzin  a écrit :

>  Downloaded and checked sigs, hashes.
> Built and ran tests on Fedora 28/29 Java 8. - ok
> Built and ran tests on Windows 10 Java 8. - ok
> Built and ran tests on Windows 10 Java 9/10. - FAILED on
> CassandraAdapterTest.
>
> Here it is a link to appveyor failed on java9
>
> https://ci.appveyor.com/project/snuyanzin/calcite/builds/20784395/job/pc74d4nrskhehb0c
> The reason why it works on Windows with jdk8 but doesn't with jdk9/10 still
> not clear
>
>
> On Wed, Dec 5, 2018 at 12:38 PM HongzeZhang  wrote:
>
> > +1 (non-binding)
> >
> > - Built from tarball and ran tests (Fedora 29, OpenJDK 1.8 / 11)
> > - Built from Git tag and ran tests (Fedora 29, OpenJDK 1.8 / 11)
> > - Checked signatures and checksums
> >
> > Thanks Julian!
> >
> >
> >
> > Hongze
> >
> > From: Julian Hyde
> > Date: 2018-12-04 17:00
> > To: dev
> > Subject: [VOTE] Release apache-calcite-1.18.0 (release candidate 0)
> > Hi all,
> >
> > I have created a build for Apache Calcite 1.18.0, release candidate 0.
> >
> > Thanks to everyone who has contributed to this release.
> > With over 200 commits from 36 contributors, this is the largest
> > Calcite release ever!
> > You can read the release notes here:
> > https://github.com/apache/calcite/blob/branch-1.18/site/_docs/history.md
> >
> > The commit to be voted upon:
> >
> >
> http://git-wip-us.apache.org/repos/asf/calcite/commit/c19e458d2bdbd42dbd450143e0f41fabbfe4ec06
> >
> > Its hash is c19e458d2bdbd42dbd450143e0f41fabbfe4ec06.
> >
> > The artifacts to be voted on are located here:
> > https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.18.0-rc0
> >
> > The hashes of the artifacts are as follows:
> > src.tar.gz.sha256
> > 8931ccc08863e4fe394824ead306dc89a7a78a41f898303187a91b01a723cce6
> >
> > A staged Maven repository is available for review at:
> > https://repository.apache.org/content/repositories/orgapachecalcite-1049
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/jhyde.asc
> >
> > Please vote on releasing this package as Apache Calcite 1.18.0.
> >
> > The vote is open for the next 72 hours (closing at 1am Pacific on Friday)
> > and passes if a majority of at least three +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Calcite 1.18.0
> > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> > [ ] -1 Do not release this package because...
> >
> >
> > Here is my vote:
> >
> > +1 (binding)
> >
> > Julian
> >
>
>
> --
> Best regards,
> Sergey
>

Re: [VOTE] Release apache-calcite-1.18.0 (release candidate 0)

[Michael Mior]

+1 (binding) Downloaded and checked hash and signature. Built and ran tests
on Ubuntu 18.04 with Java 8. Thanks Julian!

--
Michael Mior
mm...@apache.org


Le mar. 4 déc. 2018 à 04:00, Julian Hyde  a écrit :

> Hi all,
>
> I have created a build for Apache Calcite 1.18.0, release candidate 0.
>
> Thanks to everyone who has contributed to this release.
> With over 200 commits from 36 contributors, this is the largest
> Calcite release ever!
> You can read the release notes here:
> https://github.com/apache/calcite/blob/branch-1.18/site/_docs/history.md
>
> The commit to be voted upon:
>
> http://git-wip-us.apache.org/repos/asf/calcite/commit/c19e458d2bdbd42dbd450143e0f41fabbfe4ec06
>
> Its hash is c19e458d2bdbd42dbd450143e0f41fabbfe4ec06.
>
> The artifacts to be voted on are located here:
> https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.18.0-rc0
>
> The hashes of the artifacts are as follows:
> src.tar.gz.sha256
> 8931ccc08863e4fe394824ead306dc89a7a78a41f898303187a91b01a723cce6
>
> A staged Maven repository is available for review at:
> https://repository.apache.org/content/repositories/orgapachecalcite-1049
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jhyde.asc
>
> Please vote on releasing this package as Apache Calcite 1.18.0.
>
> The vote is open for the next 72 hours (closing at 1am Pacific on Friday)
> and passes if a majority of at least three +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Calcite 1.18.0
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
>
> Here is my vote:
>
> +1 (binding)
>
> Julian
>

Re: Mirroring JIRA/GitHub comments

[Michael Mior]

If GH comments are mirrored to JIRA, it does mean that there would be
duplicate emails if you're subscribed to both the PR on GH and the JIRA
case. It does mean though that there would be one place where you could see
all comments. Definitely this should only be enabled if the majority thinks
it will be more productive. If it makes things worse, especially for those
who invest much more time than I such as Julian, then I would revoke this
suggestion.

--
Michael Mior
mm...@apache.org


Le mar. 4 déc. 2018 à 20:00, Julian Hyde  a écrit :

> I suspect that my practices are out of step with Michael's practices
> (and maybe other people's too). I prefer to work in JIRA as much as
> possible, using git's review mechanism only for critiques of
> individual lines. I do this because I like being able to see how a
> contribution evolved, and the reasoning behind it, long after that
> contribution has been merged. Github is not very good at preserving
> that stuff.
>
> Mirroring is all very well, but will it double the volume of email traffic?
>
> I'm feeling rather burned out because I committed over 50 of other
> people's PRs last release.
>
> Julian
>
> On Tue, Dec 4, 2018 at 3:02 PM Francis Chuang 
> wrote:
> >
> > Thanks for taking this on, Michael!
> >
> > I think this is a good idea as having to go back and forth between a
> > Github PR and JIRA issue can sometimes be a massive context switch. I
> > think Github comments are current mirrored into the appropriate JIRA
> > issue, but the reverse does not currently work.
> >
> > If possible, I think this would also be a good idea for the
> > calcite-avatica and calcite-avatica-go repositories as well.
> >
> > On 5/12/2018 6:56 am, Michael Mior wrote:
> > > This discussion came up a while back, but I think it would be really
> > > helpful to have JIRA comments mirror to the association GitHub issue.
> When
> > > reviewing PRs, I start by looking for those with no comments
> (indicating
> > > someone else has already had a look). Then I have to dig up the JIRA
> case
> > > and check if someone has reviewed there. If there are no objections,
> I'll
> > > file a ticket with INFRA.
> > >
> > > --
> > > Michael Mior
> > > mm...@apache.org
> > >
> >
>

Mirroring JIRA/GitHub comments

[Michael Mior]

This discussion came up a while back, but I think it would be really
helpful to have JIRA comments mirror to the association GitHub issue. When
reviewing PRs, I start by looking for those with no comments (indicating
someone else has already had a look). Then I have to dig up the JIRA case
and check if someone has reviewed there. If there are no objections, I'll
file a ticket with INFRA.

--
Michael Mior
mm...@apache.org

Re: [DISCUSS] Towards Avatica 1.13.0

[Michael Mior]

Thanks Francis!

--
Michael Mior
mm...@apache.org


Le dim. 2 déc. 2018 à 21:13, Francis Chuang  a
écrit :

> Avatica 1.13.0 has been tagged and released. I am holding off the formal
> announcement until the release artifacts propagate across all mirrors.
>
> The Hypersql docker image was also published successfully using docker
> hub hooks:
> - https://hub.docker.com/r/apache/calcite-avatica/
> - https://hub.docker.com/r/apache/calcite-avatica-hypersql/
>
> On 29/11/2018 8:21 am, Francis Chuang wrote:
> > Good catch, Julian! I've updated the commit to remove the comment.
> >
> > On 29/11/2018 4:41 am, Julian Hyde wrote:
> >> In docker/src/main/dockerhub/Dockerfile, is the comment line
> >>
> >># This line must be preserved. The Maven build will verify this
> >> version matches its version
> >>
> >> still accurate?
> >>
> >>> On Nov 28, 2018, at 2:12 AM, Francis Chuang
> >>>  wrote:
> >>>
> >>> Hey all,
> >>>
> >>> If there are no objections, I plan to merge my PR for CALCITE-2698
> >>> [1] in around 12 hours time and make rc0 available for voting.
> >>>
> >>> Francis
> >>>
> >>> [1] https://github.com/apache/calcite-avatica/pull/79
> >>>
> >>> On 24/11/2018 1:03 pm, Julian Hyde wrote:
> >>>> Great, thanks. Just wanted to make sure I wasn’t they only one
> >>>> seeing this.
> >>>>
> >>>>> On Nov 23, 2018, at 6:00 PM, Francis Chuang
> >>>>>  wrote:
> >>>>>
> >>>>> `mvn site` should work correctly once my PR for CALCITE-2698 is
> >>>>> merged.
> >>>>>
> >>>>> `mvn site` broke because my fix for CALCITE-2385 required
> >>>>> -DskipDockerCheck to be appended to all invocations of `mvn`. The
> >>>>> fix for CALCITE-2698 removes this check and allows docker hub to
> >>>>> pass the avatica version to be used to the dockerfile directly
> >>>>> during a build.
> >>>>>
> >>>>> On 24/11/2018 5:41 am, Julian Hyde wrote:
> >>>>>> Are you aware that  “mvn site” has been broken for several days?
> >>>>>>
> >>>>>> [ERROR] Failed to execute goal
> >>>>>> org.apache.maven.plugins:maven-site-plugin:3.7.1:site
> >>>>>> (default-site) on project avatica-parent: failed to get report
> >>>>>> for org.apache.maven.plugins:maven-javadoc-plugin: Failed to
> >>>>>> execute goal org.codehaus.gmaven:groovy-maven-plugin:2.1:execute
> >>>>>> (check-dockerhub-dockerfile-version) on project avatica-docker:
> >>>>>> Execution check-dockerhub-dockerfile-version of goal
> >>>>>> org.codehaus.gmaven:groovy-maven-plugin:2.1:execute failed:
> >>>>>> Expected Avatica version of 1.13.0-SNAPSHOT but got 1.12.0 ->
> >>>>>> [Help 1]
> >>>>>>
> >>>>>>
> >>>>>>> On Nov 22, 2018, at 2:50 PM, Francis Chuang
> >>>>>>>  wrote:
> >>>>>>>
> >>>>>>> I will be delaying the vote for rc0 for a few days as I would
> >>>>>>> like to get CALCITE-2698 [1] into this release. This change will
> >>>>>>> publish the avatica-hsqldb image to docker hub and allow the
> >>>>>>> avatica-go integration tests to switch from my unofficial
> >>>>>>> avatica-hsqldb image to the official image from Calcite.
> >>>>>>>
> >>>>>>> [1] https://issues.apache.org/jira/browse/CALCITE-2698
> >>>>>>>
> >>>>>>> On 19/11/2018 8:39 am, Francis Chuang wrote:
> >>>>>>>> Quick update for the release of Avatica 1.13.0.
> >>>>>>>>
> >>>>>>>> I was able to get releases and tests to build correctly in a
> >>>>>>>> docker container. This is now committed to the repository as
> >>>>>>>> docker-compose.yaml and docker.sh. The setup allows a committer
> >>>>>>>> to start the release by running `docker-compose run -v
> >>>>>>>> /c/Users/username/AppData/Roaming/gnupg:/.gnupg release`. The
> >>>>>>>> script prompts the user for things like the release version and
> >>

Re: [DISCUSS] Towards Calcite 1.18

[Michael Mior]

Thanks Julian! Unfortunately with family visiting this weekend, my
bandwidth is currently fairly low but I'll try to churn through some PRs
early this coming week.

On Fri, Nov 30, 2018, 23:00 Julian Hyde  It looks as if Avatica will be released soon, and as that was our main
> dependency for Calcite 1.18, we can move ahead with the release.
>
> I volunteer to be release manager for Calcite 1.18, and propose that we
> make an RC on Monday.
>
> There are a lot of PRs (107 open currently)[1]. I can’t get through them
> all myself, so I will need help reviewing & merging them. (If you are
> committer, look in the JIRA case to see whether someone has begun reviewing
> the PR. I am reviewing several right now.)
>
> Even with help, we are not going to get to all PRs. That will have to be
> OK; it has been too long since we had a release, and we just don’t have
> enough active committers to deal with the number of incoming contributions.
>
> Julian
>
> [1] https://github.com/apache/calcite/pulls
>
>
> > On Nov 6, 2018, at 4:24 AM, Michael Mior  wrote:
> >
> > Not strictly necessary I believe, but I know there's a desire to upgrade
> > Jetty in Calcite that can't happen until the Avatica release, so it would
> > be nice to see that happen first.
> >
> >
> >
> > Le lun. 5 nov. 2018 à 16:31, Francis Chuang  a
> > écrit :
> >
> >> I think we should consider releasing new versions of Calcite and Avatica
> >> soon, especially with the Thanksgiving holiday in North America and
> >> Christmas + New Year later in December/January coming up.
> >>
> >> Am I correct to assume that we need to release Avatica 1.13.0 before
> >> Calcite 1.18.0?
> >>
> >> If so, I am able to be release manager for Avatica if no one else is
> >> interested.
> >>
> >> On 21/10/2018 6:13 PM, Michael Mior wrote:
> >>> Thanks for continuing to push releases forward! Unfortunately I won't
> be
> >>> able to volunteer to be release manager this time around, but I'll try
> to
> >>> set aside some time to go through some PRs.
> >>>
> >>> On Sat, Oct 20, 2018, 02:21 Julian Hyde  wrote:
> >>>
> >>>> OK, it's now exactly 3 months since 1.17. I think it's time. I think
> >>>> we should aim for a first RC a week from today (Friday 26th October).
> >>>>
> >>>> Can we have a volunteer to be release manager?
> >>>>
> >>>> Also, there are lots of PRs to review and merge. Please help out with
> >>>> that task, committers.
> >>>>
> >>>> Julian
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Sep 20, 2018 at 11:00 AM Julian Hyde 
> wrote:
> >>>>> We’ll be sure to get PRs into the release. If you like, you can make
> >> the
> >>>> JIRA case depend on those 3 cases - that will remind us.
> >>>>> (Other contributors: You can do that also, but only if you have a PR
> >>>> that you believe is ready to submit.)
> >>>>>> On Sep 20, 2018, at 8:18 AM, Andrew Pilloud
> >>>>  wrote:
> >>>>>> Beam has a few JIRAs we'd like to see make the next release (which
> >> will
> >>>>>> enable us to replace 11k lines of code with calls to Calcite). They
> >> all
> >>>>>> have open PRs.
> >>>>>>
> >>>>>> * https://issues.apache.org/jira/browse/CALCITE-2404 Accessing
> >>>>>> structured-types is not implemented by the runtime
> >>>>>> * https://issues.apache.org/jira/browse/CALCITE-2529 linq4j should
> >>>> promote
> >>>>>> integer to floating point when generating function calls
> >>>>>> * https://issues.apache.org/jira/browse/CALCITE-2571 TRIM does not
> >>>> match
> >>>>>> the behavior of most SQL implementations
> >>>>>>
> >>>>>> Andrew
> >>>>>>
> >>>>>> On Wed, Sep 19, 2018 at 6:38 PM Kevin Risden 
> >>>> wrote:
> >>>>>>> 9/25 should be the GA release of JDK 11 according to [1]. Would be
> >>>> good to
> >>>>>>> ensure that the next release is compatible with it. We should be in
> >>>> good
> >>>>>>> shape but would be a good release note too.
> >>>>>>>
> >>>>>>

Re: Issue with extending SqlAbstractConformance

[Michael Mior]

You seem to only be showing the first line of the error. There should be a
longer message below that would probably be helpful for debugging.

--
Michael Mior
mm...@apache.org


Le lun. 26 nov. 2018 à 06:54, Devjyoti Patra  a
écrit :

> Hi, I am trying to build the parser config using the Configbuilder class
>
> SqlParser.configBuilder()
> .setUnquotedCasing(Casing.UNCHANGED)
> .setConformance(SqlConformanceEnum.DEFAULT)
> .build();
>
> But this throws an error during compile rime
>
> [*ERROR*] *  symbol:   method
> setConformance(org.apache.calcite.sql.validate.SqlConformanceEnum)*
>
> This is in Calcite 1.15.0.
>
> I wanted to extend SqlSbatractConformance class to override some settings
> for Presto. What is the right way to override the conformance or how can I
> prevent this compilation error?
>
> Thanks,
> Devjyoti
>

Re: Vote for SQLLine release 1.6.0

[Michael Mior]

Sounds excellent! Thanks to Julian (and other SQLLine committers) for
continuing to push this forward :)

--
Michael Mior
mm...@apache.org


Le mer. 21 nov. 2018 à 17:12, Julian Hyde  a écrit :

> Calcite developers and users,
>
> I have just started a vote for sqlline-1.6.0 on the SQLLine dev list. If
> you are a SQLLine committer or user, please participate. Here is the
> thread: https://groups.google.com/forum/#!topic/sqlline-dev/SWHPzpyBwv0 <
> https://groups.google.com/forum/#!topic/sqlline-dev/SWHPzpyBwv0>
>
> Note that SQLLine is not an Apache project, but our voting process is
> somewhat similar to the Apache process.
>
> SQLLine 1.6 is much more colorful and interactive than previous versions
> (we upgraded to jline3, which allowed us to add highlighting, dynamic
> prompts, and multi-line command editing) and I very much look forward to
> including it in a future Calcite release.
>
> Happy thanksgiving everyone!
>
> Julian
>
>