[VOTE] Release apache-calcite-1.23.0 (release candidate 0)

[Haisheng Yuan]

Hi all,

I have created a build for Apache Calcite 1.23.0, release
candidate 0.

Thanks to everyone who has contributed to this release.

You can read the release notes here:
https://github.com/apache/calcite/blob/calcite-1.23.0-rc0/site/_docs/history.md

The commit to be voted upon:
https://gitbox.apache.org/repos/asf?p=calcite.git;a=commit;h=edc37c0a21344a48b15877788e082c8acdc7b030

Its hash is edc37c0a21344a48b15877788e082c8acdc7b030

Tag:
https://github.com/apache/calcite/tree/calcite-1.23.0-rc0

The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.23.0-rc0
(revision 39385)

The hashes of the artifacts are as follows:
7482b0bb76e672a15bbe846f2dbdc125bd0f3d8a32abf0ea9159b5db0ab2a2d1182e19b408098ecd68d7cc9ff5d7812ea0b33e4aeac818d191b695d437fa1a94
*apache-calcite-1.23.0-src.tar.gz

A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachecalcite-1088/org/apache/calcite/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/hyuan.asc
https://dist.apache.org/repos/dist/release/calcite/KEYS

N.B.
To create the jars and test Apache Calcite: "./gradlew build".

If you do not have a Java environment available, you can run the tests
using docker. To do so, install docker and docker-compose, then run
"docker-compose run test" from the root of the directory.

Please vote on releasing this package as Apache Calcite 1.23.0.

The vote is open for the next 72 hours and passes if a majority of at
least three +1 PMC votes are cast.

[ ] +1 Release this package as Apache Calcite 1.23.0
[ ]  0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...


Here is my vote:

+1 (binding)

Thanks,
Haisheng Yuan

Re: [DISCUSS] Towards Calcite 1.23.0

[Haisheng Yuan]

Fixed. I had to send keys to keyserver in the command line.

On 2020/05/12 02:49:49, Haisheng Yuan  wrote: 
> Hi,
> I am very close to roll out RC0. But met the following error:
> Build calcite FAILURE reason:
> Execution failed for task ':closeRepository':
> io.codearte.gradle.nexus.exception.UnexpectedRepositoryState: Wrong 
> 'orgapachecalcite-1084' repository state 'open' after transition (expected 
> [closed]). Possible staging rules violation. Check repository status using 
> Nexus UI.
> 
> I logged into https://repository.apache.org/#stagingRepositories, found that 
> Signature Validation Failed.
> 
> No public key: Key with id: (3cd22abac50ddcef) was not able to be located on 
>  href="http://pool.sks-keyservers.net:11371;>http://pool.sks-keyservers.net:11371.
>  Upload your public key and try the operation again.
> 
> But the public is already uploaded to key server:
> http://pool.sks-keyservers.net:11371/pks/lookup?search=0x3CD22ABAC50DDCEF=on=vindex
> 
> Any step I may miss?
> 
> Thanks,
> Haisheng Yuan
> 
> On 2020/05/11 17:44:07, Julian Hyde  wrote: 
> > If it’s a release candidate there’s at least a 50% chance it will turn into 
> > a release. So the release notes should be in good shape. You can make 
> > tweaks afterwards.
> > 
> > How about making a “release candidate” and tell reviewers that there is no 
> > chance it will be released? You can call it RC0 if you like.
> > 
> > Even then, release notes help me review a release candidate, so you will 
> > get better feedback if you have a first iteration of release notes.
> > 
> > Can you make sure that https://github.com/apache/calcite/commit/a2b8a7a 
> >  has been applied to 
> > master? Danny assured me he’d fix the tag names but a few weeks ago it 
> > didn’t look like it had been fixed.
> > 
> > Julian
> > 
> > 
> > > On May 11, 2020, at 9:06 AM, Haisheng Yuan  wrote:
> > > 
> > > Quick question, does the RC0 need to have fully updated history.md?
> > > 
> > > On 2020/05/11 13:51:53, Haisheng Yuan  wrote: 
> > >> Oops, thanks for reminding. lol
> > >> 
> > >> On 2020/05/11 09:41:33, Stamatis Zampetakis  wrote: 
> > >>> Previously you mentioned:
> > >>> ~/vlsi-release-plugins ❯❯❯ ./gradlew prepareVote -Prc=0
> > >>> 
> > >>> Just to be sure for releasing Calcite you should execute ./gradlew
> > >>> prepareVote -Prc=0 in calcite directory (not in vlsi-release-plugins).
> > >>> 
> > >>> On Mon, May 11, 2020 at 11:03 AM Haisheng Yuan  wrote:
> > >>> 
> >  Hi Stamatis,
> >  
> >  I tried to add all the info to the property file, still no luck:
> >  
> >  signing.gnupg.keyName=3CD22ABAC50DDCEF
> >  asfCommitterId=hyuan
> >  ghTestGitSourceUsername=test
> >  ghTestGitSourcePassword=test
> >  ghTestNexusUsername=test
> >  ghTestNexusPassword=test
> >  asfGitSourceUsername=test
> >  asfGitSourcePassword=test
> >  asfNexusUsername=test
> >  asfNexusPassword=test
> >  asfGitSourceUsername=test
> >  asfGitSourcePassword=test
> >  asfSvnUsername=test
> >  asfSvnPassword=test
> >  
> >  useGpgCmd=true
> >  signing.gnupg.useLegacyGpg=true
> >  
> >  1: Task failed with an exception.
> >  ---
> >  * What went wrong:
> >  Execution failed for task ':initializeNexusStagingRepository'.
> > > Failed to find staging profile for package group: com.github.vlsi
> >  
> >  * Try:
> >  Run with --stacktrace option to get the stack trace. Run with --info or
> >  --debug option to get more log output. Run with --scan to get full 
> >  insights.
> >  
> >  ==
> >  
> >  2: Task failed with an exception.
> >  ---
> >  * What went wrong:
> >  Execution failed for task
> >  ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication'.
> > > Process 'command 'gpg'' finished with non-zero exit value 2
> >  
> >  * Try:
> >  Run with --stacktrace option to get the stack trace. Run with --info or
> >  --debug option to get more log output. Run with --scan to get full 
> >  insights.
> >  
> >  ==
> >  
> >  3: Task failed with an exception.
> >  ---
> >  * What went wrong:
> >  Execution failed for task
> >  ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'.
> > > Process 'command 'gpg'' finished with non-zero exit value 2
> >  
> >  Thanks,
> >  Haisheng
> >  
> >  On 2020/05/11 08:41:16, Stamatis Zampetakis  wrote:
> > > Hi Haisheng,
> > > 
> > > You might be missing some gradle.properties especially those required 
> > > to
> > > identify your PGP key.
> > > 
> > > Have a look in the discussion in [1]. Among other info it contains 

Re: [DISCUSS] Towards Calcite 1.23.0

[Haisheng Yuan]

Hi,
I am very close to roll out RC0. But met the following error:
Build calcite FAILURE reason:
Execution failed for task ':closeRepository':
io.codearte.gradle.nexus.exception.UnexpectedRepositoryState: Wrong 
'orgapachecalcite-1084' repository state 'open' after transition (expected 
[closed]). Possible staging rules violation. Check repository status using 
Nexus UI.

I logged into https://repository.apache.org/#stagingRepositories, found that 
Signature Validation Failed.

No public key: Key with id: (3cd22abac50ddcef) was not able to be located on http://pool.sks-keyservers.net:11371;>http://pool.sks-keyservers.net:11371.
 Upload your public key and try the operation again.

But the public is already uploaded to key server:
http://pool.sks-keyservers.net:11371/pks/lookup?search=0x3CD22ABAC50DDCEF=on=vindex

Any step I may miss?

Thanks,
Haisheng Yuan

On 2020/05/11 17:44:07, Julian Hyde  wrote: 
> If it’s a release candidate there’s at least a 50% chance it will turn into a 
> release. So the release notes should be in good shape. You can make tweaks 
> afterwards.
> 
> How about making a “release candidate” and tell reviewers that there is no 
> chance it will be released? You can call it RC0 if you like.
> 
> Even then, release notes help me review a release candidate, so you will get 
> better feedback if you have a first iteration of release notes.
> 
> Can you make sure that https://github.com/apache/calcite/commit/a2b8a7a 
>  has been applied to 
> master? Danny assured me he’d fix the tag names but a few weeks ago it didn’t 
> look like it had been fixed.
> 
> Julian
> 
> 
> > On May 11, 2020, at 9:06 AM, Haisheng Yuan  wrote:
> > 
> > Quick question, does the RC0 need to have fully updated history.md?
> > 
> > On 2020/05/11 13:51:53, Haisheng Yuan  wrote: 
> >> Oops, thanks for reminding. lol
> >> 
> >> On 2020/05/11 09:41:33, Stamatis Zampetakis  wrote: 
> >>> Previously you mentioned:
> >>> ~/vlsi-release-plugins ❯❯❯ ./gradlew prepareVote -Prc=0
> >>> 
> >>> Just to be sure for releasing Calcite you should execute ./gradlew
> >>> prepareVote -Prc=0 in calcite directory (not in vlsi-release-plugins).
> >>> 
> >>> On Mon, May 11, 2020 at 11:03 AM Haisheng Yuan  wrote:
> >>> 
>  Hi Stamatis,
>  
>  I tried to add all the info to the property file, still no luck:
>  
>  signing.gnupg.keyName=3CD22ABAC50DDCEF
>  asfCommitterId=hyuan
>  ghTestGitSourceUsername=test
>  ghTestGitSourcePassword=test
>  ghTestNexusUsername=test
>  ghTestNexusPassword=test
>  asfGitSourceUsername=test
>  asfGitSourcePassword=test
>  asfNexusUsername=test
>  asfNexusPassword=test
>  asfGitSourceUsername=test
>  asfGitSourcePassword=test
>  asfSvnUsername=test
>  asfSvnPassword=test
>  
>  useGpgCmd=true
>  signing.gnupg.useLegacyGpg=true
>  
>  1: Task failed with an exception.
>  ---
>  * What went wrong:
>  Execution failed for task ':initializeNexusStagingRepository'.
> > Failed to find staging profile for package group: com.github.vlsi
>  
>  * Try:
>  Run with --stacktrace option to get the stack trace. Run with --info or
>  --debug option to get more log output. Run with --scan to get full 
>  insights.
>  
>  ==
>  
>  2: Task failed with an exception.
>  ---
>  * What went wrong:
>  Execution failed for task
>  ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication'.
> > Process 'command 'gpg'' finished with non-zero exit value 2
>  
>  * Try:
>  Run with --stacktrace option to get the stack trace. Run with --info or
>  --debug option to get more log output. Run with --scan to get full 
>  insights.
>  
>  ==
>  
>  3: Task failed with an exception.
>  ---
>  * What went wrong:
>  Execution failed for task
>  ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'.
> > Process 'command 'gpg'' finished with non-zero exit value 2
>  
>  Thanks,
>  Haisheng
>  
>  On 2020/05/11 08:41:16, Stamatis Zampetakis  wrote:
> > Hi Haisheng,
> > 
> > You might be missing some gradle.properties especially those required to
> > identify your PGP key.
> > 
> > Have a look in the discussion in [1]. Among other info it contains also
>  the
> > complete list of properties.
> > 
> > Best,
> > Stamatis
> > 
> > [1]
> > 
>  https://lists.apache.org/thread.html/r71feef9a44c1f4ea0f2211dc8c74b25dab143d2e3215cf7ffcfd6f56%40%3Cdev.calcite.apache.org%3E
> > 
> > On Mon, May 11, 2020 at 9:55 AM Haisheng Yuan  wrote:
> > 
> >> Hi,
> >> 
> >> I saw this 

[jira] [Created] (CALCITE-3992) Beam ZetaSQL supports pure SQL user-defined aggregation functions

[Rui Wang (Jira)]

Rui Wang created CALCITE-3992:
-

 Summary: Beam ZetaSQL supports pure SQL user-defined aggregation 
functions
 Key: CALCITE-3992
 URL: https://issues.apache.org/jira/browse/CALCITE-3992
 Project: Calcite
  Issue Type: Task
Reporter: Rui Wang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3991) the required boolean should always be provided in RelSet.getOrCreateSubset()

[Botong Huang (Jira)]

Botong Huang created CALCITE-3991:
-

 Summary: the required boolean should always be provided in 
RelSet.getOrCreateSubset()
 Key: CALCITE-3991
 URL: https://issues.apache.org/jira/browse/CALCITE-3991
 Project: Calcite
  Issue Type: Bug
Reporter: Botong Huang


the required boolean should always be provided in RelSet.getOrCreateSubset(). 
Deleting the old default as well as other related code cleanup



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3990) Use a more efficient algorithm for cost propagation in Volcano planner

[James Kim (Jira)]

James Kim created CALCITE-3990:
--

 Summary: Use a more efficient algorithm for cost propagation in 
Volcano planner
 Key: CALCITE-3990
 URL: https://issues.apache.org/jira/browse/CALCITE-3990
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: James Kim


The previous method uses a recursive, depth-first approach, which can result
in repeatedly updating the cost of some relSubsets many times.

This patch moves to a breadth-first approach with a priority queue,
very similar to Djikstra's algorithm.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3989) Release Calcite 1.23.0

[Haisheng Yuan (Jira)]

Haisheng Yuan created CALCITE-3989:
--

 Summary: Release Calcite 1.23.0
 Key: CALCITE-3989
 URL: https://issues.apache.org/jira/browse/CALCITE-3989
 Project: Calcite
  Issue Type: Task
Reporter: Haisheng Yuan
Assignee: Haisheng Yuan
 Fix For: 1.23.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Calcite-Master - Build # 1753 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Calcite-Master (build #1753)

Status: Still Failing

Check console output at https://builds.apache.org/job/Calcite-Master/1753/ to 
view the results.

Calcite-Master - Build # 1752 - Failure

[Apache Jenkins Server]

The Apache Jenkins build system has built Calcite-Master (build #1752)

Status: Failure

Check console output at https://builds.apache.org/job/Calcite-Master/1752/ to 
view the results.

Re: [DISCUSS] Towards Cascades Optimizer

[Julian Hyde]

If there’s a way for all of these efforts to be committed to master, disabled 
by default, but such that they can be enabled by setting a flag, then we can at 
least avoid merge conflicts.

I hope that this “trunk-based development” approach is technically feasible. If 
the rule and metadata APIs are essentially unchanged, and the planner API is 
evolved smoothly, we should be able to do it.

From the perspective of the project and community, this is certainly a good 
problem to have. An embarrassment of riches.

From the perspective of the developers, we should their work as painless as 
possible. We, the community, also owe it them to make a timely decision about 
which effort (or efforts) we wish to support long-term.

Julian


> On May 11, 2020, at 7:04 AM, Roman Kondakov  
> wrote:
> 
> Hi Jinpeng,
> 
> Apache Calcite community has entered into the interesting situation: we
> have several concurrent efforts of building the new Cascades style
> optimizers. I can see at least three activities (correct me if I'm wrong):
> 
> 1. Haisheng's gradual rebuilding of VolcanoPlanner [1]
> 2. Jinpeng's new CascadesPlanner based on VolcanoPlanner [2]
> 3. My brand new CascadesPlanner [3]
> 
> All these activities are independent and may interfere with each other.
> It is not good. We still don't know which approach is better and which
> optimizer will be the most efficient while keeping backward compatibility.
> 
> In my opinion we must refrain from breaking changes in the master branch
> right now because it can break other optimizers.
> 
> I think we need to develop a further strategy for activities related to
> the new optimizers:
> - find out the way how we will choose the new default Cascades optimizer
> for Calcite: it should be stable, fast and backward-compatible
> optimizer. If all optimizers will prove to be stable and backward
> compatible, may be we need to choose the fastest one?
> - think about the possibility of using plug-in optimizers: it would be a
> great feature for users if they want to use their custom optimizer for
> specific tasks or research purposes. It can be configured with
> FrameworkConfig.setPlanner(new CustomPlanner()).
> 
> What do you think?
> 
> [1] https://github.com/apache/calcite/pull/1953
> [2] https://github.com/apache/calcite/pull/1950
> [3] https://github.com/apache/calcite/pull/1948
> 
> 
> -- 
> Kind Regards
> Roman Kondakov
> 
> 
> On 11.05.2020 11:12, Jinpeng Wu wrote:
>> Hi, Roman.
>> 
>> Thanks to Julian's tips, I added a CoreCascadeQuidemTest to the code. It
>> runs iq files that CoreQuidemTest would runs and uses CascadePlanner
>> instead of VolcanoPlanner to generate physical plans. Currently all tests
>> have passed. There are some plan changes but they are actually equivalent
>> plans in different forms. I‘ve also committed those plan changes as '.cas'
>> files.
>> Moreover, if you would like to set -Dcalcite.cascade=true during tests, all
>> test cases will use the new planner for planning. There would be some
>> failures, typically because there's no common ways across all tests to tell
>> whether a RelNode is physical or logical.
>> 
>> 
>> When I was running the tests, I found that Haisheng's latest changes has a
>> conflict with my PR. The new planner relies on AbstractConverter to tell
>> whether a subset canConvert/needConvert to another subset. Haisheng's
>> change will remove AbstractConverter and break this. Currently, I switch
>> off top-down trait propagation in new planner to work around. I will merge
>> these two top-down processing together later on to completely address this
>> problem.
>> One thing to recall is that, while I am writing a new planner, Haisheng
>> modified the VolcanoPlanner directly and used a flag to switch on/off the
>> new feature. I need to decide how to merge them: to keep a separated
>> planner and invoke the new trait propagation interfaces or to modified the
>> volcano planner directly. I am trying the first way now. But of course, if
>> keeping one planner is preferred, I can also take the second way.
>> 
>> 
>> Thanks,
>> Jinpeng
>> 
>> 
>> On Fri, May 1, 2020 at 6:40 AM Haisheng Yuan  wrote:
>> 
>>> Hi all,
>>> 
>>> As planned in my proposal, I opened the pull request [1] for CALCITE-3896
>>> to achieve:
>>> 1. Top-down trait request
>>> 2. Bottom-up trait derivation
>>> 3. Trait enforcement without AbstractConverter
>>> 
>>> The feature can be turned on or off by a flag, either through property
>>> config file or VolcanoPlanner set method. Since Calcite doesn't turn on
>>> AbstractConverter until latest master, I just disabled AbstractConverter by
>>> turning on top-down trait request, now all tests passed.
>>> 
>>> In our system, 99 tpcds queries' test results show almost no plan diff,
>>> but the number of relnodes created during optimization is reduced by 10~15%
>>> average (even without space pruning). I believe for other systems using
>>> VolcanoPlanner, more than 20% reduction can be expected.
>>> 
>>> It also 

Re: [DISCUSS] Towards Calcite 1.23.0

[Haisheng Yuan]

Quick question, does the RC0 need to have fully updated history.md?

On 2020/05/11 13:51:53, Haisheng Yuan  wrote: 
> Oops, thanks for reminding. lol
> 
> On 2020/05/11 09:41:33, Stamatis Zampetakis  wrote: 
> > Previously you mentioned:
> > ~/vlsi-release-plugins ❯❯❯ ./gradlew prepareVote -Prc=0
> > 
> > Just to be sure for releasing Calcite you should execute ./gradlew
> > prepareVote -Prc=0 in calcite directory (not in vlsi-release-plugins).
> > 
> > On Mon, May 11, 2020 at 11:03 AM Haisheng Yuan  wrote:
> > 
> > > Hi Stamatis,
> > >
> > > I tried to add all the info to the property file, still no luck:
> > >
> > > signing.gnupg.keyName=3CD22ABAC50DDCEF
> > > asfCommitterId=hyuan
> > > ghTestGitSourceUsername=test
> > > ghTestGitSourcePassword=test
> > > ghTestNexusUsername=test
> > > ghTestNexusPassword=test
> > > asfGitSourceUsername=test
> > > asfGitSourcePassword=test
> > > asfNexusUsername=test
> > > asfNexusPassword=test
> > > asfGitSourceUsername=test
> > > asfGitSourcePassword=test
> > > asfSvnUsername=test
> > > asfSvnPassword=test
> > >
> > > useGpgCmd=true
> > > signing.gnupg.useLegacyGpg=true
> > >
> > > 1: Task failed with an exception.
> > > ---
> > > * What went wrong:
> > > Execution failed for task ':initializeNexusStagingRepository'.
> > > > Failed to find staging profile for package group: com.github.vlsi
> > >
> > > * Try:
> > > Run with --stacktrace option to get the stack trace. Run with --info or
> > > --debug option to get more log output. Run with --scan to get full 
> > > insights.
> > >
> > > ==
> > >
> > > 2: Task failed with an exception.
> > > ---
> > > * What went wrong:
> > > Execution failed for task
> > > ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication'.
> > > > Process 'command 'gpg'' finished with non-zero exit value 2
> > >
> > > * Try:
> > > Run with --stacktrace option to get the stack trace. Run with --info or
> > > --debug option to get more log output. Run with --scan to get full 
> > > insights.
> > >
> > > ==
> > >
> > > 3: Task failed with an exception.
> > > ---
> > > * What went wrong:
> > > Execution failed for task
> > > ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'.
> > > > Process 'command 'gpg'' finished with non-zero exit value 2
> > >
> > > Thanks,
> > > Haisheng
> > >
> > > On 2020/05/11 08:41:16, Stamatis Zampetakis  wrote:
> > > > Hi Haisheng,
> > > >
> > > > You might be missing some gradle.properties especially those required to
> > > > identify your PGP key.
> > > >
> > > > Have a look in the discussion in [1]. Among other info it contains also
> > > the
> > > > complete list of properties.
> > > >
> > > > Best,
> > > > Stamatis
> > > >
> > > > [1]
> > > >
> > > https://lists.apache.org/thread.html/r71feef9a44c1f4ea0f2211dc8c74b25dab143d2e3215cf7ffcfd6f56%40%3Cdev.calcite.apache.org%3E
> > > >
> > > > On Mon, May 11, 2020 at 9:55 AM Haisheng Yuan  wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I saw this error during the dry-run:
> > > > > ~/vlsi-release-plugins ❯❯❯ ./gradlew prepareVote -Prc=0
> > > > > > Task :plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication 
> > > > > > FAILED
> > > > > > Task
> > > > >
> > > :plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication
> > > > > FAILED
> > > > > > Task :initializeNexusStagingRepository FAILED
> > > > >
> > > > > FAILURE: Build completed with 3 failures.
> > > > >
> > > > > 1: Task failed with an exception.
> > > > > ---
> > > > > * What went wrong:
> > > > > Execution failed for task
> > > > > ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication'.
> > > > > > Cannot perform signing task
> > > > > ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication' because it
> > > has
> > > > > no configured signatory
> > > > >
> > > > > * Try:
> > > > > Run with --stacktrace option to get the stack trace. Run with --info 
> > > > > or
> > > > > --debug option to get more log output. Run with --scan to get full
> > > insights.
> > > > >
> > > > >
> > > ==
> > > > >
> > > > > 2: Task failed with an exception.
> > > > > ---
> > > > > * What went wrong:
> > > > > Execution failed for task
> > > > >
> > > ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'.
> > > > > > Cannot perform signing task
> > > > >
> > > ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'
> > > > > because it has no configured signatory
> > > > >
> > > > > * Try:
> > > > > Run with --stacktrace option to get the stack trace. Run with --info 
> > > > > or
> > > > > --debug option to get more log output. Run with --scan to get full
> > > insights.
> > > > >
> > > > >
> > > 

Re: [DISCUSS] Towards Cascades Optimizer

[Roman Kondakov]

Hi Jinpeng,

Apache Calcite community has entered into the interesting situation: we
have several concurrent efforts of building the new Cascades style
optimizers. I can see at least three activities (correct me if I'm wrong):

1. Haisheng's gradual rebuilding of VolcanoPlanner [1]
2. Jinpeng's new CascadesPlanner based on VolcanoPlanner [2]
3. My brand new CascadesPlanner [3]

All these activities are independent and may interfere with each other.
It is not good. We still don't know which approach is better and which
optimizer will be the most efficient while keeping backward compatibility.

In my opinion we must refrain from breaking changes in the master branch
right now because it can break other optimizers.

I think we need to develop a further strategy for activities related to
the new optimizers:
- find out the way how we will choose the new default Cascades optimizer
for Calcite: it should be stable, fast and backward-compatible
optimizer. If all optimizers will prove to be stable and backward
compatible, may be we need to choose the fastest one?
- think about the possibility of using plug-in optimizers: it would be a
great feature for users if they want to use their custom optimizer for
specific tasks or research purposes. It can be configured with
FrameworkConfig.setPlanner(new CustomPlanner()).

What do you think?

[1] https://github.com/apache/calcite/pull/1953
[2] https://github.com/apache/calcite/pull/1950
[3] https://github.com/apache/calcite/pull/1948


-- 
Kind Regards
Roman Kondakov


On 11.05.2020 11:12, Jinpeng Wu wrote:
> Hi, Roman.
> 
> Thanks to Julian's tips, I added a CoreCascadeQuidemTest to the code. It
> runs iq files that CoreQuidemTest would runs and uses CascadePlanner
> instead of VolcanoPlanner to generate physical plans. Currently all tests
> have passed. There are some plan changes but they are actually equivalent
> plans in different forms. I‘ve also committed those plan changes as '.cas'
> files.
> Moreover, if you would like to set -Dcalcite.cascade=true during tests, all
> test cases will use the new planner for planning. There would be some
> failures, typically because there's no common ways across all tests to tell
> whether a RelNode is physical or logical.
> 
> 
> When I was running the tests, I found that Haisheng's latest changes has a
> conflict with my PR. The new planner relies on AbstractConverter to tell
> whether a subset canConvert/needConvert to another subset. Haisheng's
> change will remove AbstractConverter and break this. Currently, I switch
> off top-down trait propagation in new planner to work around. I will merge
> these two top-down processing together later on to completely address this
> problem.
> One thing to recall is that, while I am writing a new planner, Haisheng
> modified the VolcanoPlanner directly and used a flag to switch on/off the
> new feature. I need to decide how to merge them: to keep a separated
> planner and invoke the new trait propagation interfaces or to modified the
> volcano planner directly. I am trying the first way now. But of course, if
> keeping one planner is preferred, I can also take the second way.
> 
> 
> Thanks,
> Jinpeng
> 
> 
> On Fri, May 1, 2020 at 6:40 AM Haisheng Yuan  wrote:
> 
>> Hi all,
>>
>> As planned in my proposal, I opened the pull request [1] for CALCITE-3896
>> to achieve:
>> 1. Top-down trait request
>> 2. Bottom-up trait derivation
>> 3. Trait enforcement without AbstractConverter
>>
>> The feature can be turned on or off by a flag, either through property
>> config file or VolcanoPlanner set method. Since Calcite doesn't turn on
>> AbstractConverter until latest master, I just disabled AbstractConverter by
>> turning on top-down trait request, now all tests passed.
>>
>> In our system, 99 tpcds queries' test results show almost no plan diff,
>> but the number of relnodes created during optimization is reduced by 10~15%
>> average (even without space pruning). I believe for other systems using
>> VolcanoPlanner, more than 20% reduction can be expected.
>>
>> It also has top-down rule apply in mind, later can be evolved to top-down
>> rule apply and space pruning, e.g. integrating code from Jingpeng and
>> Roman's. But the interface that is exposed to user, as described in the
>> proposal, can remain the same.
>>
>> Haisheng
>>
>> [1] https://github.com/apache/calcite/pull/1953
>>
>>
>> On 2020/04/30 18:01:26, Julian Hyde  wrote:
>>> If your test cases are SQL scripts, it might be fairly straightforward
>> to port them to Quidem (.iq) files. Plenty of examples in
>> https://github.com/apache/calcite/tree/master/core/src/test/resources/sql
>> >> .
>>>
>>> Quidem files are basically SQL scripts. Expected output is embedded in
>> the script. You can run the script once, and if the output looks right,
>> overwrite the input file with the output.
>>>
>>> Julian
>>>
>>>
 On Apr 30, 2020, at 3:26 AM, Jinpeng Wu  wrote:

Re: [DISCUSS] Towards Calcite 1.23.0

[Haisheng Yuan]

Oops, thanks for reminding. lol

On 2020/05/11 09:41:33, Stamatis Zampetakis  wrote: 
> Previously you mentioned:
> ~/vlsi-release-plugins ❯❯❯ ./gradlew prepareVote -Prc=0
> 
> Just to be sure for releasing Calcite you should execute ./gradlew
> prepareVote -Prc=0 in calcite directory (not in vlsi-release-plugins).
> 
> On Mon, May 11, 2020 at 11:03 AM Haisheng Yuan  wrote:
> 
> > Hi Stamatis,
> >
> > I tried to add all the info to the property file, still no luck:
> >
> > signing.gnupg.keyName=3CD22ABAC50DDCEF
> > asfCommitterId=hyuan
> > ghTestGitSourceUsername=test
> > ghTestGitSourcePassword=test
> > ghTestNexusUsername=test
> > ghTestNexusPassword=test
> > asfGitSourceUsername=test
> > asfGitSourcePassword=test
> > asfNexusUsername=test
> > asfNexusPassword=test
> > asfGitSourceUsername=test
> > asfGitSourcePassword=test
> > asfSvnUsername=test
> > asfSvnPassword=test
> >
> > useGpgCmd=true
> > signing.gnupg.useLegacyGpg=true
> >
> > 1: Task failed with an exception.
> > ---
> > * What went wrong:
> > Execution failed for task ':initializeNexusStagingRepository'.
> > > Failed to find staging profile for package group: com.github.vlsi
> >
> > * Try:
> > Run with --stacktrace option to get the stack trace. Run with --info or
> > --debug option to get more log output. Run with --scan to get full insights.
> >
> > ==
> >
> > 2: Task failed with an exception.
> > ---
> > * What went wrong:
> > Execution failed for task
> > ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication'.
> > > Process 'command 'gpg'' finished with non-zero exit value 2
> >
> > * Try:
> > Run with --stacktrace option to get the stack trace. Run with --info or
> > --debug option to get more log output. Run with --scan to get full insights.
> >
> > ==
> >
> > 3: Task failed with an exception.
> > ---
> > * What went wrong:
> > Execution failed for task
> > ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'.
> > > Process 'command 'gpg'' finished with non-zero exit value 2
> >
> > Thanks,
> > Haisheng
> >
> > On 2020/05/11 08:41:16, Stamatis Zampetakis  wrote:
> > > Hi Haisheng,
> > >
> > > You might be missing some gradle.properties especially those required to
> > > identify your PGP key.
> > >
> > > Have a look in the discussion in [1]. Among other info it contains also
> > the
> > > complete list of properties.
> > >
> > > Best,
> > > Stamatis
> > >
> > > [1]
> > >
> > https://lists.apache.org/thread.html/r71feef9a44c1f4ea0f2211dc8c74b25dab143d2e3215cf7ffcfd6f56%40%3Cdev.calcite.apache.org%3E
> > >
> > > On Mon, May 11, 2020 at 9:55 AM Haisheng Yuan  wrote:
> > >
> > > > Hi,
> > > >
> > > > I saw this error during the dry-run:
> > > > ~/vlsi-release-plugins ❯❯❯ ./gradlew prepareVote -Prc=0
> > > > > Task :plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication FAILED
> > > > > Task
> > > >
> > :plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication
> > > > FAILED
> > > > > Task :initializeNexusStagingRepository FAILED
> > > >
> > > > FAILURE: Build completed with 3 failures.
> > > >
> > > > 1: Task failed with an exception.
> > > > ---
> > > > * What went wrong:
> > > > Execution failed for task
> > > > ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication'.
> > > > > Cannot perform signing task
> > > > ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication' because it
> > has
> > > > no configured signatory
> > > >
> > > > * Try:
> > > > Run with --stacktrace option to get the stack trace. Run with --info or
> > > > --debug option to get more log output. Run with --scan to get full
> > insights.
> > > >
> > > >
> > ==
> > > >
> > > > 2: Task failed with an exception.
> > > > ---
> > > > * What went wrong:
> > > > Execution failed for task
> > > >
> > ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'.
> > > > > Cannot perform signing task
> > > >
> > ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'
> > > > because it has no configured signatory
> > > >
> > > > * Try:
> > > > Run with --stacktrace option to get the stack trace. Run with --info or
> > > > --debug option to get more log output. Run with --scan to get full
> > insights.
> > > >
> > > >
> > ==
> > > >
> > > > 3: Task failed with an exception.
> > > > ---
> > > > * What went wrong:
> > > > Execution failed for task ':initializeNexusStagingRepository'.
> > > > > Failed to find staging profile for package group: com.github.vlsi
> > > >
> > > > Here is the gradle.properties settings:
> > > > ghTestGitSourceUsername=test
> > > > ghTestGitSourcePassword=test
> > > > 

[jira] [Created] (CALCITE-3988) Intersect in RelMdRowCount doesn't take into account 'intersect all'

[xzh_dz (Jira)]

xzh_dz created CALCITE-3988:
---

 Summary: Intersect in RelMdRowCount doesn't take into account 
'intersect all' 
 Key: CALCITE-3988
 URL: https://issues.apache.org/jira/browse/CALCITE-3988
 Project: Calcite
  Issue Type: Wish
Reporter: xzh_dz


Same as 
[https://issues.apache.org/jira/browse/CALCITE-3287|https://issues.apache.org/jira/browse/CALCITE-3287].
Intersect in RelMdRowCount.java doesn't take into account 'intersect all' .
{code:java}
public Double getRowCount(Intersect rel, RelMetadataQuery mq) {
Double rowCount = null;
for (RelNode input : rel.getInputs()) {
  Double partialRowCount = mq.getRowCount(input);
  if (rowCount == null
  || partialRowCount != null && partialRowCount < rowCount) {
rowCount = partialRowCount;
  }
}
return rowCount;
  }
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Towards Calcite 1.23.0

[Stamatis Zampetakis]

Previously you mentioned:
~/vlsi-release-plugins ❯❯❯ ./gradlew prepareVote -Prc=0

Just to be sure for releasing Calcite you should execute ./gradlew
prepareVote -Prc=0 in calcite directory (not in vlsi-release-plugins).

On Mon, May 11, 2020 at 11:03 AM Haisheng Yuan  wrote:

> Hi Stamatis,
>
> I tried to add all the info to the property file, still no luck:
>
> signing.gnupg.keyName=3CD22ABAC50DDCEF
> asfCommitterId=hyuan
> ghTestGitSourceUsername=test
> ghTestGitSourcePassword=test
> ghTestNexusUsername=test
> ghTestNexusPassword=test
> asfGitSourceUsername=test
> asfGitSourcePassword=test
> asfNexusUsername=test
> asfNexusPassword=test
> asfGitSourceUsername=test
> asfGitSourcePassword=test
> asfSvnUsername=test
> asfSvnPassword=test
>
> useGpgCmd=true
> signing.gnupg.useLegacyGpg=true
>
> 1: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task ':initializeNexusStagingRepository'.
> > Failed to find staging profile for package group: com.github.vlsi
>
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or
> --debug option to get more log output. Run with --scan to get full insights.
>
> ==
>
> 2: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task
> ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication'.
> > Process 'command 'gpg'' finished with non-zero exit value 2
>
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or
> --debug option to get more log output. Run with --scan to get full insights.
>
> ==
>
> 3: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task
> ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'.
> > Process 'command 'gpg'' finished with non-zero exit value 2
>
> Thanks,
> Haisheng
>
> On 2020/05/11 08:41:16, Stamatis Zampetakis  wrote:
> > Hi Haisheng,
> >
> > You might be missing some gradle.properties especially those required to
> > identify your PGP key.
> >
> > Have a look in the discussion in [1]. Among other info it contains also
> the
> > complete list of properties.
> >
> > Best,
> > Stamatis
> >
> > [1]
> >
> https://lists.apache.org/thread.html/r71feef9a44c1f4ea0f2211dc8c74b25dab143d2e3215cf7ffcfd6f56%40%3Cdev.calcite.apache.org%3E
> >
> > On Mon, May 11, 2020 at 9:55 AM Haisheng Yuan  wrote:
> >
> > > Hi,
> > >
> > > I saw this error during the dry-run:
> > > ~/vlsi-release-plugins ❯❯❯ ./gradlew prepareVote -Prc=0
> > > > Task :plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication FAILED
> > > > Task
> > >
> :plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication
> > > FAILED
> > > > Task :initializeNexusStagingRepository FAILED
> > >
> > > FAILURE: Build completed with 3 failures.
> > >
> > > 1: Task failed with an exception.
> > > ---
> > > * What went wrong:
> > > Execution failed for task
> > > ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication'.
> > > > Cannot perform signing task
> > > ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication' because it
> has
> > > no configured signatory
> > >
> > > * Try:
> > > Run with --stacktrace option to get the stack trace. Run with --info or
> > > --debug option to get more log output. Run with --scan to get full
> insights.
> > >
> > >
> ==
> > >
> > > 2: Task failed with an exception.
> > > ---
> > > * What went wrong:
> > > Execution failed for task
> > >
> ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'.
> > > > Cannot perform signing task
> > >
> ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'
> > > because it has no configured signatory
> > >
> > > * Try:
> > > Run with --stacktrace option to get the stack trace. Run with --info or
> > > --debug option to get more log output. Run with --scan to get full
> insights.
> > >
> > >
> ==
> > >
> > > 3: Task failed with an exception.
> > > ---
> > > * What went wrong:
> > > Execution failed for task ':initializeNexusStagingRepository'.
> > > > Failed to find staging profile for package group: com.github.vlsi
> > >
> > > Here is the gradle.properties settings:
> > > ghTestGitSourceUsername=test
> > > ghTestGitSourcePassword=test
> > > ghTestNexusUsername=test
> > > ghTestNexusPassword=test
> > >
> > >
> > > BTW, my gpg public key has been uploaded:
> > >
> https://pgp.mit.edu/pks/lookup?search=0x3CD22ABAC50DDCEF=vindex=on
> > >
> > > And the gpg fingerprint has been updated in my apache id profile at
> > > https://id.apache.org/.
> > >
> > > Any kind of help is appreciated.
> > >
> > > Thanks,
> > > 

Re: [DISCUSS] Towards Calcite 1.23.0

[Haisheng Yuan]

Hi Stamatis,

I tried to add all the info to the property file, still no luck:

signing.gnupg.keyName=3CD22ABAC50DDCEF
asfCommitterId=hyuan
ghTestGitSourceUsername=test
ghTestGitSourcePassword=test
ghTestNexusUsername=test
ghTestNexusPassword=test
asfGitSourceUsername=test
asfGitSourcePassword=test
asfNexusUsername=test
asfNexusPassword=test
asfGitSourceUsername=test
asfGitSourcePassword=test
asfSvnUsername=test
asfSvnPassword=test

useGpgCmd=true
signing.gnupg.useLegacyGpg=true

1: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':initializeNexusStagingRepository'.
> Failed to find staging profile for package group: com.github.vlsi

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

2: Task failed with an exception.
---
* What went wrong:
Execution failed for task 
':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication'.
> Process 'command 'gpg'' finished with non-zero exit value 2

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

3: Task failed with an exception.
---
* What went wrong:
Execution failed for task 
':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'.
> Process 'command 'gpg'' finished with non-zero exit value 2

Thanks,
Haisheng

On 2020/05/11 08:41:16, Stamatis Zampetakis  wrote: 
> Hi Haisheng,
> 
> You might be missing some gradle.properties especially those required to
> identify your PGP key.
> 
> Have a look in the discussion in [1]. Among other info it contains also the
> complete list of properties.
> 
> Best,
> Stamatis
> 
> [1]
> https://lists.apache.org/thread.html/r71feef9a44c1f4ea0f2211dc8c74b25dab143d2e3215cf7ffcfd6f56%40%3Cdev.calcite.apache.org%3E
> 
> On Mon, May 11, 2020 at 9:55 AM Haisheng Yuan  wrote:
> 
> > Hi,
> >
> > I saw this error during the dry-run:
> > ~/vlsi-release-plugins ❯❯❯ ./gradlew prepareVote -Prc=0
> > > Task :plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication FAILED
> > > Task
> > :plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication
> > FAILED
> > > Task :initializeNexusStagingRepository FAILED
> >
> > FAILURE: Build completed with 3 failures.
> >
> > 1: Task failed with an exception.
> > ---
> > * What went wrong:
> > Execution failed for task
> > ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication'.
> > > Cannot perform signing task
> > ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication' because it has
> > no configured signatory
> >
> > * Try:
> > Run with --stacktrace option to get the stack trace. Run with --info or
> > --debug option to get more log output. Run with --scan to get full insights.
> >
> > ==
> >
> > 2: Task failed with an exception.
> > ---
> > * What went wrong:
> > Execution failed for task
> > ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'.
> > > Cannot perform signing task
> > ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'
> > because it has no configured signatory
> >
> > * Try:
> > Run with --stacktrace option to get the stack trace. Run with --info or
> > --debug option to get more log output. Run with --scan to get full insights.
> >
> > ==
> >
> > 3: Task failed with an exception.
> > ---
> > * What went wrong:
> > Execution failed for task ':initializeNexusStagingRepository'.
> > > Failed to find staging profile for package group: com.github.vlsi
> >
> > Here is the gradle.properties settings:
> > ghTestGitSourceUsername=test
> > ghTestGitSourcePassword=test
> > ghTestNexusUsername=test
> > ghTestNexusPassword=test
> >
> >
> > BTW, my gpg public key has been uploaded:
> > https://pgp.mit.edu/pks/lookup?search=0x3CD22ABAC50DDCEF=vindex=on
> >
> > And the gpg fingerprint has been updated in my apache id profile at
> > https://id.apache.org/.
> >
> > Any kind of help is appreciated.
> >
> > Thanks,
> > Haisheng
> >
> > On 2020/05/06 02:09:11, Chunwei Lei  wrote:
> > > Hi, Haisheng, I would like to review the PR for CALCITE-3896.
> > >
> > >
> > > Best,
> > > Chunwei
> > >
> > >
> > > On Wed, May 6, 2020 at 4:50 AM Rui Wang  wrote:
> > >
> > > > Thanks for leading 1.23.0 release, Haisheng!
> > > >
> > > > I will actively contact Danny and Feng to see if we can reach a
> > conclusion
> > > > on https://github.com/apache/calcite/pull/1761 by next monday.
> > > >
> > > >
> > > > -Rui
> > > >
> > > > On Tue, May 5, 2020 at 11:31 AM Haisheng Yuan 
> > 

Re: [DISCUSS] Towards Calcite 1.23.0

[Stamatis Zampetakis]

Hi Haisheng,

You might be missing some gradle.properties especially those required to
identify your PGP key.

Have a look in the discussion in [1]. Among other info it contains also the
complete list of properties.

Best,
Stamatis

[1]
https://lists.apache.org/thread.html/r71feef9a44c1f4ea0f2211dc8c74b25dab143d2e3215cf7ffcfd6f56%40%3Cdev.calcite.apache.org%3E

On Mon, May 11, 2020 at 9:55 AM Haisheng Yuan  wrote:

> Hi,
>
> I saw this error during the dry-run:
> ~/vlsi-release-plugins ❯❯❯ ./gradlew prepareVote -Prc=0
> > Task :plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication FAILED
> > Task
> :plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication
> FAILED
> > Task :initializeNexusStagingRepository FAILED
>
> FAILURE: Build completed with 3 failures.
>
> 1: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task
> ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication'.
> > Cannot perform signing task
> ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication' because it has
> no configured signatory
>
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or
> --debug option to get more log output. Run with --scan to get full insights.
>
> ==
>
> 2: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task
> ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'.
> > Cannot perform signing task
> ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'
> because it has no configured signatory
>
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or
> --debug option to get more log output. Run with --scan to get full insights.
>
> ==
>
> 3: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task ':initializeNexusStagingRepository'.
> > Failed to find staging profile for package group: com.github.vlsi
>
> Here is the gradle.properties settings:
> ghTestGitSourceUsername=test
> ghTestGitSourcePassword=test
> ghTestNexusUsername=test
> ghTestNexusPassword=test
>
>
> BTW, my gpg public key has been uploaded:
> https://pgp.mit.edu/pks/lookup?search=0x3CD22ABAC50DDCEF=vindex=on
>
> And the gpg fingerprint has been updated in my apache id profile at
> https://id.apache.org/.
>
> Any kind of help is appreciated.
>
> Thanks,
> Haisheng
>
> On 2020/05/06 02:09:11, Chunwei Lei  wrote:
> > Hi, Haisheng, I would like to review the PR for CALCITE-3896.
> >
> >
> > Best,
> > Chunwei
> >
> >
> > On Wed, May 6, 2020 at 4:50 AM Rui Wang  wrote:
> >
> > > Thanks for leading 1.23.0 release, Haisheng!
> > >
> > > I will actively contact Danny and Feng to see if we can reach a
> conclusion
> > > on https://github.com/apache/calcite/pull/1761 by next monday.
> > >
> > >
> > > -Rui
> > >
> > > On Tue, May 5, 2020 at 11:31 AM Haisheng Yuan 
> wrote:
> > >
> > > > Yes, no problem. Will have a RC before next Monday, PDT.
> > > >
> > > > @Roman, thanks for helping.
> > > > This is the PR: https://github.com/apache/calcite/pull/1953
> > > >
> > > > Haisheng
> > > >
> > > > On 2020/05/05 14:57:48, Julian Hyde  wrote:
> > > > > Thanks for volunteering, Haisheng.
> > > > >
> > > > > Before everyone piles in with wish-lists, can we set a timescale
> for
> > > the
> > > > release?
> > > > >
> > > > > I would suggest an aggressive timescale, for example an RC a week
> from
> > > > today. Hold the RC only for mandatory cases. Other stuff gets in if
> it’s
> > > > ready.
> > > > >
> > > > > Julian
> > > > >
> > > > > > On May 5, 2020, at 5:51 AM, Ruben Q L  wrote:
> > > > > >
> > > > > > Thanks for starting the discussion, Stamatis.
> > > > > >
> > > > > > Just a reminder that there is a blocking issue [1] that needs to
> be
> > > > solved
> > > > > > (it is a regression introduced after 1.22).
> > > > > > Apart from that, it would be nice if we could include [2] in
> 1.23.
> > > > > >
> > > > > > Best regards,
> > > > > > Ruben
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/CALCITE-3926
> > > > > > [2] https://issues.apache.org/jira/browse/CALCITE-3951
> > > > > >
> > > > > >
> > > > > >
> > > > > >> Le mar. 5 mai 2020 à 12:34, Roman Kondakov
> > > 
> > > > a
> > > > > >> écrit :
> > > > > >>
> > > > > >> Hi Haisheng,
> > > > > >>
> > > > > >> I would like to review CALCITE-3896. Do you have a PR?
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Kind Regards
> > > > > >> Roman Kondakov
> > > > > >>
> > > > > >>
> > > > > >>> On 05.05.2020 06:37, Haisheng Yuan wrote:
> > > > > >>> IIRC, I am the release manager for 1.23.0.
> > > > > >>>
> > > > > >>> I think CALCITE-3896 (top-down trait request) is in good
> shape, it
> > > > will
> > > > > >> be nice if it can go into 1.23.0. Can someone 

Re: [DISCUSS] Towards Cascades Optimizer

[Jinpeng Wu]

Hi, Roman.

Thanks to Julian's tips, I added a CoreCascadeQuidemTest to the code. It
runs iq files that CoreQuidemTest would runs and uses CascadePlanner
instead of VolcanoPlanner to generate physical plans. Currently all tests
have passed. There are some plan changes but they are actually equivalent
plans in different forms. I‘ve also committed those plan changes as '.cas'
files.
Moreover, if you would like to set -Dcalcite.cascade=true during tests, all
test cases will use the new planner for planning. There would be some
failures, typically because there's no common ways across all tests to tell
whether a RelNode is physical or logical.


When I was running the tests, I found that Haisheng's latest changes has a
conflict with my PR. The new planner relies on AbstractConverter to tell
whether a subset canConvert/needConvert to another subset. Haisheng's
change will remove AbstractConverter and break this. Currently, I switch
off top-down trait propagation in new planner to work around. I will merge
these two top-down processing together later on to completely address this
problem.
One thing to recall is that, while I am writing a new planner, Haisheng
modified the VolcanoPlanner directly and used a flag to switch on/off the
new feature. I need to decide how to merge them: to keep a separated
planner and invoke the new trait propagation interfaces or to modified the
volcano planner directly. I am trying the first way now. But of course, if
keeping one planner is preferred, I can also take the second way.


Thanks,
Jinpeng


On Fri, May 1, 2020 at 6:40 AM Haisheng Yuan  wrote:

> Hi all,
>
> As planned in my proposal, I opened the pull request [1] for CALCITE-3896
> to achieve:
> 1. Top-down trait request
> 2. Bottom-up trait derivation
> 3. Trait enforcement without AbstractConverter
>
> The feature can be turned on or off by a flag, either through property
> config file or VolcanoPlanner set method. Since Calcite doesn't turn on
> AbstractConverter until latest master, I just disabled AbstractConverter by
> turning on top-down trait request, now all tests passed.
>
> In our system, 99 tpcds queries' test results show almost no plan diff,
> but the number of relnodes created during optimization is reduced by 10~15%
> average (even without space pruning). I believe for other systems using
> VolcanoPlanner, more than 20% reduction can be expected.
>
> It also has top-down rule apply in mind, later can be evolved to top-down
> rule apply and space pruning, e.g. integrating code from Jingpeng and
> Roman's. But the interface that is exposed to user, as described in the
> proposal, can remain the same.
>
> Haisheng
>
> [1] https://github.com/apache/calcite/pull/1953
>
>
> On 2020/04/30 18:01:26, Julian Hyde  wrote:
> > If your test cases are SQL scripts, it might be fairly straightforward
> to port them to Quidem (.iq) files. Plenty of examples in
> https://github.com/apache/calcite/tree/master/core/src/test/resources/sql
>  >.
> >
> > Quidem files are basically SQL scripts. Expected output is embedded in
> the script. You can run the script once, and if the output looks right,
> overwrite the input file with the output.
> >
> > Julian
> >
> >
> > > On Apr 30, 2020, at 3:26 AM, Jinpeng Wu  wrote:
> > >
> > > Sure. I will add more cases to my PR.
> > >
> > > I did not design more cases because our own product has a test
> frameworks,
> > > which contains thousands of actual user queries.
> > > Calcite's code base is quite different. I cannot just migrate cases to
> > > calcite.  So it may take some time.
> > >
> > > On Wed, Apr 29, 2020 at 4:27 AM Roman Kondakov
> 
> > > wrote:
> > >
> > >> Hi Jinpeng,
> > >>
> > >> I went through your PR and it seemed very impressive to me. It is very
> > >> similar to what I did, but you've reused many existing logic from the
> > >> Volcano planner. We should definitely stay in sync in our
> experiments. I
> > >> believe the future Cascades planner will be the kind combination of
> our
> > >> works.
> > >>
> > >> Is there any way to run tests that are close to the real system query
> > >> execution? May be with Enumerable convention, or, better, with
> > >> convention that supports distribution trait? I just want to look
> through
> > >> your planner's optimization steps more thoroughly. I've found some
> tests
> > >> in org.apache.calcite.plan.volcano package, but they use synthetic
> > >> conventions and nodes. May be I missed something.
> > >>
> > >> Thank you for sharing your work!
> > >>
> > >> --
> > >> Kind Regards
> > >> Roman Kondakov
> > >>
> > >>
> > >> On 28.04.2020 15:19, Jinpeng Wu wrote:
> > >>> Hi, Roman. It's great to see your proposal. Actually my team has also
> > >> been
> > >>> working on a cascade planner based on calcite.  And we already have
> some
> > >>> outcome as well.  Maybe we can combine our works.
> > >>>
> > >>> I've pushed my code as 

Re: [DISCUSS] Towards Calcite 1.23.0

[Haisheng Yuan]

Hi,

I saw this error during the dry-run:
~/vlsi-release-plugins ❯❯❯ ./gradlew prepareVote -Prc=0
> Task :plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication FAILED
> Task 
> :plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication
>  FAILED
> Task :initializeNexusStagingRepository FAILED

FAILURE: Build completed with 3 failures.

1: Task failed with an exception.
---
* What went wrong:
Execution failed for task 
':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication'.
> Cannot perform signing task 
> ':plugins:crlf-plugin:signCrlfPluginMarkerMavenPublication' because it has no 
> configured signatory

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

2: Task failed with an exception.
---
* What went wrong:
Execution failed for task 
':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'.
> Cannot perform signing task 
> ':plugins:checksum-dependency-plugin:signChecksum-dependencyPluginMarkerMavenPublication'
>  because it has no configured signatory

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

3: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':initializeNexusStagingRepository'.
> Failed to find staging profile for package group: com.github.vlsi

Here is the gradle.properties settings:
ghTestGitSourceUsername=test
ghTestGitSourcePassword=test
ghTestNexusUsername=test
ghTestNexusPassword=test


BTW, my gpg public key has been uploaded:
https://pgp.mit.edu/pks/lookup?search=0x3CD22ABAC50DDCEF=vindex=on

And the gpg fingerprint has been updated in my apache id profile at 
https://id.apache.org/.

Any kind of help is appreciated.

Thanks,
Haisheng

On 2020/05/06 02:09:11, Chunwei Lei  wrote: 
> Hi, Haisheng, I would like to review the PR for CALCITE-3896.
> 
> 
> Best,
> Chunwei
> 
> 
> On Wed, May 6, 2020 at 4:50 AM Rui Wang  wrote:
> 
> > Thanks for leading 1.23.0 release, Haisheng!
> >
> > I will actively contact Danny and Feng to see if we can reach a conclusion
> > on https://github.com/apache/calcite/pull/1761 by next monday.
> >
> >
> > -Rui
> >
> > On Tue, May 5, 2020 at 11:31 AM Haisheng Yuan  wrote:
> >
> > > Yes, no problem. Will have a RC before next Monday, PDT.
> > >
> > > @Roman, thanks for helping.
> > > This is the PR: https://github.com/apache/calcite/pull/1953
> > >
> > > Haisheng
> > >
> > > On 2020/05/05 14:57:48, Julian Hyde  wrote:
> > > > Thanks for volunteering, Haisheng.
> > > >
> > > > Before everyone piles in with wish-lists, can we set a timescale for
> > the
> > > release?
> > > >
> > > > I would suggest an aggressive timescale, for example an RC a week from
> > > today. Hold the RC only for mandatory cases. Other stuff gets in if it’s
> > > ready.
> > > >
> > > > Julian
> > > >
> > > > > On May 5, 2020, at 5:51 AM, Ruben Q L  wrote:
> > > > >
> > > > > Thanks for starting the discussion, Stamatis.
> > > > >
> > > > > Just a reminder that there is a blocking issue [1] that needs to be
> > > solved
> > > > > (it is a regression introduced after 1.22).
> > > > > Apart from that, it would be nice if we could include [2] in 1.23.
> > > > >
> > > > > Best regards,
> > > > > Ruben
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/CALCITE-3926
> > > > > [2] https://issues.apache.org/jira/browse/CALCITE-3951
> > > > >
> > > > >
> > > > >
> > > > >> Le mar. 5 mai 2020 à 12:34, Roman Kondakov
> > 
> > > a
> > > > >> écrit :
> > > > >>
> > > > >> Hi Haisheng,
> > > > >>
> > > > >> I would like to review CALCITE-3896. Do you have a PR?
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Kind Regards
> > > > >> Roman Kondakov
> > > > >>
> > > > >>
> > > > >>> On 05.05.2020 06:37, Haisheng Yuan wrote:
> > > > >>> IIRC, I am the release manager for 1.23.0.
> > > > >>>
> > > > >>> I think CALCITE-3896 (top-down trait request) is in good shape, it
> > > will
> > > > >> be nice if it can go into 1.23.0. Can someone help review?
> > > > >>>
> > > > >>> On 2020/05/04 22:58:25, Stamatis Zampetakis 
> > > wrote:
> > > >  Hello,
> > > > 
> > > >  Calcite 1.22.0 was released about 2 months ago (on March 5, 2020).
> > > > >> During
> > > >  the vote we said to try to have a release sooner than usual so I
> > am
> > > >  launching the discussion now hoping we could get to 1.23.0 rather
> > > soon.
> > > > 
> > > >  I was checking the status of the current release [1] and we have
> > > already
> > > >  over 100 resolved issues with 91 fixes so it is already pretty
> > big.
> > > > 
> > > >  I know that there quite a few discussions in progress so it would