Re: Re: [DISCUSS] Small contributions

[Rui Wang]

Agree with Francis's point: people are likely to move on than accumulating
enough to have a larger PR to merge, which does not help improve Calcite's
codebase. I understand it would increase the workload on Calcite committers
(more PRs to review), so at least code reviewers can always encourage
people to have a larger PR in the future, but not block existing LGTM ones.


- Rui



On Wed, Sep 25, 2019 at 7:17 PM Haisheng Yuan 
wrote:

> > most of the time, the author of the fix would  have moved on and have
> forgotten about it, resulting in the improvement falling through the
> cracks.
>
> ​Make sense. I think our current position worth reconsidering and I
>  agree with Francis.
>
> - Haisheng
>
> --
> 发件人：Francis Chuang
> 日 期：2019年09月26日 09:20:49
> 收件人：
> 主 题：Re: [DISCUSS] Small contributions
>
>  From personal experience, I think we should accept these small changes.
> I have had lots of  cases where I am reading code or documentation on
> Github and found small errors or typos that are easy to fix, so I'd edit
> directly in Github and open a PR. These changes do improve the codebase
> and fix errors that could be misleading or confuse future maintainers
> and users.
>
> It might be easy to say that we want to combine all these small changes
> into a bigger change, but most of the time, the author of the fix would
> have moved on and have forgotten about it, resulting in the improvement
> falling through the cracks. It also makes the amount of effort required
> to start contributing to the project a bit higher.
>
> With Github integration, trivial fixes like this should be easy to merge
> with a click of a button and a quick glance at the diff on Github is
> usually sufficient for review.
>
> I agree with Michael's suggestion that a JIRA should not be created for
> cases like this. They are also low-hanging fruit to improve the
> code-base and not accepting them seems like a missed opportunity to me.
>
> Francis
>
> On 26/09/2019 10:46 am, Michael Mior wrote:
> > I have mixed feelings about this, because on one hand, I'd like to
> > have these things corrected but on the other hand, we're already
> > bogged down with PRs. Perhaps a good compromise is to make it clear
> > that a JIRA should not be created and have some type of tag indicated
> > in the title of the PR. This might be a good time to create a pull
> > request template for GitHub that explains some of the policies (e.g.
> > making sure that non-trivial changes DO have a JIRA case).
> > --
> > Michael Mior
> > mm...@apache.org
> >
> > Le mer. 25 sept. 2019 à 20:42, Julian Hyde  a écrit :
> >>
> >> I noticed this exchange in https://github.com/apache/calcite/pull/1475:
> 
> >>
> >>
> >>> Q. Just curious, does Calcite accept hotfix style PR that fixes typos,
> comments, etc.?
> >>
> >>> A. As long as they are large enough. But for 1 line typo fix, it is
> not worth a specific patch, we prefer to accumulate them together.
> >>
> >> This is indeed our current position. And the reason we have given is
> that it takes considerable effort to review and commit a pull request, even
> a small one.
> >>
> >> Should we reconsider this position?
> >>
> >> Julian
>

Re: How to modify data for custom tables through Calcite.

[Juan Pan]

 @Danny Chan 



Thanks Danny, is there any document or test for me to learn `specific 
convention` more?


Regards,
Trista


 Juan Pan


panj...@apache.org
Juan Pan(Trista), Apache ShardingSphere


On 09/26/2019 12:02，Danny Chan wrote：
@Rui Wang, Yes, I wrote the Flink-sql-parser module it did support insert 
grammar well.

@Juan Pan you need the converter rules to convert all the nodes to specific 
convention you want, also specify the desired convention in the trait set of 
your planing program.

Best,
Danny Chan
在 2019年9月26日 +0800 AM6:04，Rui Wang ，写道：
Another data point is both BeamSQL and FlinkSQL support DDL by an extensive
way (and I believe it works through Avitica as well).

BeamSQL: [1]
FlinkSQL: [2]


Calcite allows add customized DDL in parser and also in implementation
schema is accessible.

[1]:
https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/codegen/includes/parserImpls.ftl#L149
[2]:
https://github.com/apache/flink/blob/master/flink-table/flink-sql-parser/src/main/codegen/data/Parser.tdd#L430

-Rui

On Wed, Sep 25, 2019 at 2:54 PM Stamatis Zampetakis 
wrote:

Hi Trista,

I think the server module is doing what you are asking for. Have a look in
ServerTest [1].
As Gelbana mentioned the implementation is based on implementations of the
ModifiableTable interface.

Best,
Stamatis

[1]

https://github.com/apache/calcite/blob/master/server/src/test/java/org/apache/calcite/test/ServerTest.java

On Wed, Sep 25, 2019 at 11:29 PM Mohamed Mohsen 
wrote:

I haven't done that before but I would start investigating from this
interface [1]. Please share your experience if you get this done.

[1] org.apache.calcite.schema.ModifiableTable


On Wed, Sep 25, 2019 at 2:00 PM Juan Pan  wrote:

Hi everyone,


Thanks for your attention. I want to know the following description is
right or not?


"Modification has only been worked on for JDBC tables, not for any
custom
tables currently.”


Query SQL on custom table is ok, so i am wondering whether i can
execute
`update/insert/delete` SQL through Calcite on custom tables.


Can anyone give me some ideas?


Really thanks for your help.


Regards,
Trista






Juan Pan


panj...@apache.org
Juan Pan(Trista), Apache ShardingSphere

Re: How to modify data for custom tables through Calcite.

[Danny Chan]

@Rui Wang, Yes, I wrote the Flink-sql-parser module it did support insert 
grammar well.

@Juan Pan you need the converter rules to convert all the nodes to specific 
convention you want, also specify the desired convention in the trait set of 
your planing program.

Best,
Danny Chan
在 2019年9月26日 +0800 AM6:04，Rui Wang ，写道：
> Another data point is both BeamSQL and FlinkSQL support DDL by an extensive
> way (and I believe it works through Avitica as well).
>
> BeamSQL: [1]
> FlinkSQL: [2]
>
>
> Calcite allows add customized DDL in parser and also in implementation
> schema is accessible.
>
> [1]:
> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/codegen/includes/parserImpls.ftl#L149
> [2]:
> https://github.com/apache/flink/blob/master/flink-table/flink-sql-parser/src/main/codegen/data/Parser.tdd#L430
>
> -Rui
>
> On Wed, Sep 25, 2019 at 2:54 PM Stamatis Zampetakis 
> wrote:
>
> > Hi Trista,
> >
> > I think the server module is doing what you are asking for. Have a look in
> > ServerTest [1].
> > As Gelbana mentioned the implementation is based on implementations of the
> > ModifiableTable interface.
> >
> > Best,
> > Stamatis
> >
> > [1]
> >
> > https://github.com/apache/calcite/blob/master/server/src/test/java/org/apache/calcite/test/ServerTest.java
> >
> > On Wed, Sep 25, 2019 at 11:29 PM Mohamed Mohsen 
> > wrote:
> >
> > > I haven't done that before but I would start investigating from this
> > > interface [1]. Please share your experience if you get this done.
> > >
> > > [1] org.apache.calcite.schema.ModifiableTable
> > >
> > >
> > > On Wed, Sep 25, 2019 at 2:00 PM Juan Pan  wrote:
> > >
> > > > Hi everyone,
> > > >
> > > >
> > > > Thanks for your attention. I want to know the following description is
> > > > right or not?
> > > >
> > > >
> > > > "Modification has only been worked on for JDBC tables, not for any
> > custom
> > > > tables currently.”
> > > >
> > > >
> > > > Query SQL on custom table is ok, so i am wondering whether i can
> > execute
> > > > `update/insert/delete` SQL through Calcite on custom tables.
> > > >
> > > >
> > > > Can anyone give me some ideas?
> > > >
> > > >
> > > > Really thanks for your help.
> > > >
> > > >
> > > > Regards,
> > > > Trista
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Juan Pan
> > > >
> > > >
> > > > panj...@apache.org
> > > > Juan Pan(Trista), Apache ShardingSphere
> > > >
> > > >
> > >
> >

[jira] [Created] (CALCITE-3373) Decorrelate boolean context IN or existential subquery directly into SEMI/ANTI join

[jin xing (Jira)]

jin xing created CALCITE-3373:
-

 Summary: Decorrelate boolean context IN or existential subquery 
directly into SEMI/ANTI join
 Key: CALCITE-3373
 URL: https://issues.apache.org/jira/browse/CALCITE-3373
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: jin xing


The current approach to generate semi-join from IN/EXISTS is as below:
 # Decorrelation
 # Transforming by SemiJoinRule to get a semi-join

A more straightforward approach is to generate the semi-join right in 
Decorrelation.

What's more, we should also have an approach to generate anti-join right in 
Decorrelation.(Currently there's no analogous rule to SemiJoinRule for 
anti-join)

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Re: [DISCUSS] Small contributions

[Haisheng Yuan]

> most of the time, the author of the fix would  have moved on and have
forgotten about it, resulting in the improvement falling through the cracks.

​Make sense. I think our current position worth reconsidering and I
 agree with Francis.

- Haisheng

--
发件人：Francis Chuang
日　期：2019年09月26日 09:20:49
收件人：
主　题：Re: [DISCUSS] Small contributions

 From personal experience, I think we should accept these small changes. 
I have had lots of  cases where I am reading code or documentation on 
Github and found small errors or typos that are easy to fix, so I'd edit 
directly in Github and open a PR. These changes do improve the codebase 
and fix errors that could be misleading or confuse future maintainers 
and users.

It might be easy to say that we want to combine all these small changes 
into a bigger change, but most of the time, the author of the fix would 
have moved on and have forgotten about it, resulting in the improvement 
falling through the cracks. It also makes the amount of effort required 
to start contributing to the project a bit higher.

With Github integration, trivial fixes like this should be easy to merge 
with a click of a button and a quick glance at the diff on Github is 
usually sufficient for review.

I agree with Michael's suggestion that a JIRA should not be created for 
cases like this. They are also low-hanging fruit to improve the 
code-base and not accepting them seems like a missed opportunity to me.

Francis

On 26/09/2019 10:46 am, Michael Mior wrote:
> I have mixed feelings about this, because on one hand, I'd like to
> have these things corrected but on the other hand, we're already
> bogged down with PRs. Perhaps a good compromise is to make it clear
> that a JIRA should not be created and have some type of tag indicated
> in the title of the PR. This might be a good time to create a pull
> request template for GitHub that explains some of the policies (e.g.
> making sure that non-trivial changes DO have a JIRA case).
> --
> Michael Mior
> mm...@apache.org
> 
> Le mer. 25 sept. 2019 à 20:42, Julian Hyde  a écrit :
>>
>> I noticed this exchange in https://github.com/apache/calcite/pull/1475: 
>> 
>>
>>
>>> Q. Just curious, does Calcite accept hotfix style PR that fixes typos, 
>>> comments, etc.?
>>
>>> A. As long as they are large enough. But for 1 line typo fix, it is not 
>>> worth a specific patch, we prefer to accumulate them together.
>>
>> This is indeed our current position. And the reason we have given is that it 
>> takes considerable effort to review and commit a pull request, even a small 
>> one.
>>
>> Should we reconsider this position?
>>
>> Julian

Re: [DISCUSS] Small contributions

[Francis Chuang]

From personal experience, I think we should accept these small changes. 
I have had lots of  cases where I am reading code or documentation on 
Github and found small errors or typos that are easy to fix, so I'd edit 
directly in Github and open a PR. These changes do improve the codebase 
and fix errors that could be misleading or confuse future maintainers 
and users.


It might be easy to say that we want to combine all these small changes 
into a bigger change, but most of the time, the author of the fix would 
have moved on and have forgotten about it, resulting in the improvement 
falling through the cracks. It also makes the amount of effort required 
to start contributing to the project a bit higher.


With Github integration, trivial fixes like this should be easy to merge 
with a click of a button and a quick glance at the diff on Github is 
usually sufficient for review.


I agree with Michael's suggestion that a JIRA should not be created for 
cases like this. They are also low-hanging fruit to improve the 
code-base and not accepting them seems like a missed opportunity to me.


Francis

On 26/09/2019 10:46 am, Michael Mior wrote:

I have mixed feelings about this, because on one hand, I'd like to
have these things corrected but on the other hand, we're already
bogged down with PRs. Perhaps a good compromise is to make it clear
that a JIRA should not be created and have some type of tag indicated
in the title of the PR. This might be a good time to create a pull
request template for GitHub that explains some of the policies (e.g.
making sure that non-trivial changes DO have a JIRA case).
--
Michael Mior
mm...@apache.org

Le mer. 25 sept. 2019 à 20:42, Julian Hyde  a écrit :


I noticed this exchange in https://github.com/apache/calcite/pull/1475: 




Q. Just curious, does Calcite accept hotfix style PR that fixes typos, 
comments, etc.?



A. As long as they are large enough. But for 1 line typo fix, it is not worth a 
specific patch, we prefer to accumulate them together.


This is indeed our current position. And the reason we have given is that it 
takes considerable effort to review and commit a pull request, even a small one.

Should we reconsider this position?

Julian

Re: [DISCUSS] Small contributions

[Michael Mior]

I have mixed feelings about this, because on one hand, I'd like to
have these things corrected but on the other hand, we're already
bogged down with PRs. Perhaps a good compromise is to make it clear
that a JIRA should not be created and have some type of tag indicated
in the title of the PR. This might be a good time to create a pull
request template for GitHub that explains some of the policies (e.g.
making sure that non-trivial changes DO have a JIRA case).
--
Michael Mior
mm...@apache.org

Le mer. 25 sept. 2019 à 20:42, Julian Hyde  a écrit :
>
> I noticed this exchange in https://github.com/apache/calcite/pull/1475: 
> 
>
>
> > Q. Just curious, does Calcite accept hotfix style PR that fixes typos, 
> > comments, etc.?
>
> > A. As long as they are large enough. But for 1 line typo fix, it is not 
> > worth a specific patch, we prefer to accumulate them together.
>
> This is indeed our current position. And the reason we have given is that it 
> takes considerable effort to review and commit a pull request, even a small 
> one.
>
> Should we reconsider this position?
>
> Julian

[DISCUSS] Small contributions

[Julian Hyde]

I noticed this exchange in https://github.com/apache/calcite/pull/1475: 



> Q. Just curious, does Calcite accept hotfix style PR that fixes typos, 
> comments, etc.?

> A. As long as they are large enough. But for 1 line typo fix, it is not worth 
> a specific patch, we prefer to accumulate them together.

This is indeed our current position. And the reason we have given is that it 
takes considerable effort to review and commit a pull request, even a small one.

Should we reconsider this position?

Julian

Re: How to modify data for custom tables through Calcite.

[Rui Wang]

Another data point is both BeamSQL and FlinkSQL support DDL by an extensive
way (and I believe it works through Avitica as well).

BeamSQL: [1]
FlinkSQL: [2]


Calcite allows add customized DDL in parser and also in implementation
schema is accessible.

[1]:
https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/codegen/includes/parserImpls.ftl#L149
[2]:
https://github.com/apache/flink/blob/master/flink-table/flink-sql-parser/src/main/codegen/data/Parser.tdd#L430

-Rui

On Wed, Sep 25, 2019 at 2:54 PM Stamatis Zampetakis 
wrote:

> Hi Trista,
>
> I think the server module is doing what you are asking for. Have a look in
> ServerTest [1].
> As Gelbana mentioned the implementation is based on implementations of the
> ModifiableTable interface.
>
> Best,
> Stamatis
>
> [1]
>
> https://github.com/apache/calcite/blob/master/server/src/test/java/org/apache/calcite/test/ServerTest.java
>
> On Wed, Sep 25, 2019 at 11:29 PM Mohamed Mohsen 
> wrote:
>
> > I haven't done that before but I would start investigating from this
> > interface [1]. Please share your experience if you get this done.
> >
> > [1] org.apache.calcite.schema.ModifiableTable
> >
> >
> > On Wed, Sep 25, 2019 at 2:00 PM Juan Pan  wrote:
> >
> > > Hi everyone,
> > >
> > >
> > > Thanks for your attention. I want to know the following description is
> > > right or not?
> > >
> > >
> > > "Modification has only been worked on for JDBC tables, not for any
> custom
> > > tables currently.”
> > >
> > >
> > > Query SQL on custom table is ok, so i am wondering whether i can
> execute
> > > `update/insert/delete` SQL through Calcite on custom tables.
> > >
> > >
> > > Can anyone give me some ideas?
> > >
> > >
> > > Really thanks for your help.
> > >
> > >
> > > Regards,
> > > Trista
> > >
> > >
> > >
> > >
> > >
> > >
> > >  Juan Pan
> > >
> > >
> > > panj...@apache.org
> > > Juan Pan(Trista), Apache ShardingSphere
> > >
> > >
> >
>

Re: How to modify data for custom tables through Calcite.

[Stamatis Zampetakis]

Hi Trista,

I think the server module is doing what you are asking for. Have a look in
ServerTest [1].
As Gelbana mentioned the implementation is based on implementations of the
ModifiableTable interface.

Best,
Stamatis

[1]
https://github.com/apache/calcite/blob/master/server/src/test/java/org/apache/calcite/test/ServerTest.java

On Wed, Sep 25, 2019 at 11:29 PM Mohamed Mohsen  wrote:

> I haven't done that before but I would start investigating from this
> interface [1]. Please share your experience if you get this done.
>
> [1] org.apache.calcite.schema.ModifiableTable
>
>
> On Wed, Sep 25, 2019 at 2:00 PM Juan Pan  wrote:
>
> > Hi everyone,
> >
> >
> > Thanks for your attention. I want to know the following description is
> > right or not?
> >
> >
> > "Modification has only been worked on for JDBC tables, not for any custom
> > tables currently.”
> >
> >
> > Query SQL on custom table is ok, so i am wondering whether i can execute
> > `update/insert/delete` SQL through Calcite on custom tables.
> >
> >
> > Can anyone give me some ideas?
> >
> >
> > Really thanks for your help.
> >
> >
> > Regards,
> > Trista
> >
> >
> >
> >
> >
> >
> >  Juan Pan
> >
> >
> > panj...@apache.org
> > Juan Pan(Trista), Apache ShardingSphere
> >
> >
>

Re: Trivial query simplification

[Enrico Olivelli]

Il mer 25 set 2019, 23:38 Stamatis Zampetakis  ha
scritto:

> Hi Enrico,
>
> The ReduceExpressionsRule.FILTER_INSTANCE is using the simplifier so if it
> works
> correctly I don't think there is anything more to be done.
>

Fine.

Cheers
Enrico

>
> Best,
> Stamatis
>
> On Wed, Sep 25, 2019 at 5:31 PM Enrico Olivelli 
> wrote:
>
> > Thank you for your feedback.
> >
> > Actually ReduceExpressionsRule.FILTER_INSTANCE fixes the problem.
> >
> > RelOptPlanner optPlanner = cluster.getPlanner();
> > optPlanner.addRule(ReduceExpressionsRule.FILTER_INSTANCE);
> >
> > This code I had was wrong:
> >  final FrameworkConfig config = Frameworks.newConfigBuilder()
> > .parserConfig(SQL_PARSER_CONFIG)
> > .defaultSchema(subSchema)
> > .traitDefs(TRAITS)
> > .programs(Programs.ofRules(Programs.RULE_SET))
> > .build();
> >
> > as the 'programs' property of FrameworkConfig is applied only if you are
> > using Planner#transform .
> >
> > @Julian do I have to create a JIRA case for RexSimplify ? Honestly I
> don't
> > have enough knowledge to explain the problem and create a meaning full
> > issue.
> >
> > Is it work to add ReduceExpressionsRule.FILTER_INSTANCE to the default
> > rules ?
> >
> > As far as I can see the "default rules" are in CalcitePrepareImpl, is
> that
> > correct?
> >
> >
> > Cheers
> > Enrico
> >
> >
> > Il giorno mar 24 set 2019 alle ore 20:01 Julian Hyde 
> ha
> > scritto:
> >
> > > A few thoughts on this.
> > >
> > >
> > > 0. I am surprised that this simplification does not happen, and I think
> > we
> > > should do it. Specifically, "filter(x = 0 and x is not null)" should
> > > simplify to “filter(x = 0)”.
> > >
> > > 1. Enrico, please log a JIRA case now, and transcribe the key points of
> > > this discussion into the JIRA case when the discussion has finished.
> > >
> > > 2. The most obvious way to achieve this is via simplification, i.e.
> > > RexSimplify, which occurs when building expressions via RelBuilder. It
> > does
> > > not require planner rules.
> > >
> > > 3. An algorithm to achieve this would be to gather the implied
> predicates
> > > as we go through a conjunction. After generating the condition “x = 0”
> we
> > > arrive at “x is not null”. We know that “x = 0” holds, therefore we
> could
> > > also deduce that “x is not null” holds (we could also deduce other
> > > conditions, such as “x < 100” holds).
> > >
> > > 4. Another way to achieve this is via
> > > ReduceExpressionsRule.FILTER_INSTANCE. The algorithm would be as
> follows.
> > > First note the condition “x = 0” and therefore constant-reduce x to 0
> in
> > > the expression “0 is not null”. This algorithm has similar problems to
> > the
> > > algorithm in 2 - it passes along the conjunctive predicates in strict
> > order
> > > and therefore only works if they are in a particular order. Also, as a
> > > planner rule, this algorithm is more expensive, so would not be applied
> > as
> > > early/often as the RexSimplify implementation.
> > >
> > > 5. We could also simplify “filter(x is not null and x = 0)” to
> “filter(x
> > =
> > > 0)”, and people would reasonably expect that we would. But that is a
> more
> > > complex algorithm because it would require, for instance, re-ordering
> > > predicates. In the past, we have discussed re-ordering predicates as
> part
> > > of simplification; I am cautious about doing it because it would
> affect a
> > > lot of existing plans (and tests). There is no perfect order for
> > > predicates, so we might come back in 6 months and want to change the
> > order
> > > again. Better to sort them during simplification but then spit them out
> > in
> > > the original order.
> > >
> > > Julian
> > >
> > >
> > >
> > > > On Sep 24, 2019, at 8:34 AM, Enrico Olivelli 
> > > wrote:
> > > >
> > > > Il giorno mar 24 set 2019 alle ore 13:45 XING JIN <
> > > jinxing.co...@gmail.com >
> > > > ha scritto:
> > > >
> > > >> "v = 1 and v is null"
> > > >> cannot be simplified to "v = 1" not matter v is nullable or not
> > nullable
> > > >>
> > > >> If you really mean that "v is not null", I made below test case in
> > > >> RelOptRulesTest.java for illustration:
> > > >>
> > > >>
> > > >> // mgr is nullable
> > > >>  @Test public void testDEV() throws Exception {
> > > >>HepProgram program = new HepProgramBuilder()
> > > >>  .addRuleInstance(ReduceExpressionsRule.FILTER_INSTANCE)
> > > >>  .build();
> > > >>
> > > >>final String sql = "select deptno"
> > > >>  + " from emp"
> > > >>  + " where mgr = 10 and mgr is not null";
> > > >>checkPlanning(new HepPlanner(program), sql);
> > > >>  }
> > > >>
> > > >> The plan is
> > > >> LogicalProject(DEPTNO=[$7])
> > > >>  LogicalFilter(condition=[=($3, 10)])
> > > >>LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> > > >>
> > > >> Enrico ~ you may try ReduceExpressionsRule.FILTER_INSTANCE
> > > >>
> > > >
> > > >
> > > > @XING JIN

Re: Trivial query simplification

[Stamatis Zampetakis]

Hi Enrico,

The ReduceExpressionsRule.FILTER_INSTANCE is using the simplifier so if it
works
correctly I don't think there is anything more to be done.

Best,
Stamatis

On Wed, Sep 25, 2019 at 5:31 PM Enrico Olivelli  wrote:

> Thank you for your feedback.
>
> Actually ReduceExpressionsRule.FILTER_INSTANCE fixes the problem.
>
> RelOptPlanner optPlanner = cluster.getPlanner();
> optPlanner.addRule(ReduceExpressionsRule.FILTER_INSTANCE);
>
> This code I had was wrong:
>  final FrameworkConfig config = Frameworks.newConfigBuilder()
> .parserConfig(SQL_PARSER_CONFIG)
> .defaultSchema(subSchema)
> .traitDefs(TRAITS)
> .programs(Programs.ofRules(Programs.RULE_SET))
> .build();
>
> as the 'programs' property of FrameworkConfig is applied only if you are
> using Planner#transform .
>
> @Julian do I have to create a JIRA case for RexSimplify ? Honestly I don't
> have enough knowledge to explain the problem and create a meaning full
> issue.
>
> Is it work to add ReduceExpressionsRule.FILTER_INSTANCE to the default
> rules ?
>
> As far as I can see the "default rules" are in CalcitePrepareImpl, is that
> correct?
>
>
> Cheers
> Enrico
>
>
> Il giorno mar 24 set 2019 alle ore 20:01 Julian Hyde  ha
> scritto:
>
> > A few thoughts on this.
> >
> >
> > 0. I am surprised that this simplification does not happen, and I think
> we
> > should do it. Specifically, "filter(x = 0 and x is not null)" should
> > simplify to “filter(x = 0)”.
> >
> > 1. Enrico, please log a JIRA case now, and transcribe the key points of
> > this discussion into the JIRA case when the discussion has finished.
> >
> > 2. The most obvious way to achieve this is via simplification, i.e.
> > RexSimplify, which occurs when building expressions via RelBuilder. It
> does
> > not require planner rules.
> >
> > 3. An algorithm to achieve this would be to gather the implied predicates
> > as we go through a conjunction. After generating the condition “x = 0” we
> > arrive at “x is not null”. We know that “x = 0” holds, therefore we could
> > also deduce that “x is not null” holds (we could also deduce other
> > conditions, such as “x < 100” holds).
> >
> > 4. Another way to achieve this is via
> > ReduceExpressionsRule.FILTER_INSTANCE. The algorithm would be as follows.
> > First note the condition “x = 0” and therefore constant-reduce x to 0 in
> > the expression “0 is not null”. This algorithm has similar problems to
> the
> > algorithm in 2 - it passes along the conjunctive predicates in strict
> order
> > and therefore only works if they are in a particular order. Also, as a
> > planner rule, this algorithm is more expensive, so would not be applied
> as
> > early/often as the RexSimplify implementation.
> >
> > 5. We could also simplify “filter(x is not null and x = 0)” to “filter(x
> =
> > 0)”, and people would reasonably expect that we would. But that is a more
> > complex algorithm because it would require, for instance, re-ordering
> > predicates. In the past, we have discussed re-ordering predicates as part
> > of simplification; I am cautious about doing it because it would affect a
> > lot of existing plans (and tests). There is no perfect order for
> > predicates, so we might come back in 6 months and want to change the
> order
> > again. Better to sort them during simplification but then spit them out
> in
> > the original order.
> >
> > Julian
> >
> >
> >
> > > On Sep 24, 2019, at 8:34 AM, Enrico Olivelli 
> > wrote:
> > >
> > > Il giorno mar 24 set 2019 alle ore 13:45 XING JIN <
> > jinxing.co...@gmail.com >
> > > ha scritto:
> > >
> > >> "v = 1 and v is null"
> > >> cannot be simplified to "v = 1" not matter v is nullable or not
> nullable
> > >>
> > >> If you really mean that "v is not null", I made below test case in
> > >> RelOptRulesTest.java for illustration:
> > >>
> > >>
> > >> // mgr is nullable
> > >>  @Test public void testDEV() throws Exception {
> > >>HepProgram program = new HepProgramBuilder()
> > >>  .addRuleInstance(ReduceExpressionsRule.FILTER_INSTANCE)
> > >>  .build();
> > >>
> > >>final String sql = "select deptno"
> > >>  + " from emp"
> > >>  + " where mgr = 10 and mgr is not null";
> > >>checkPlanning(new HepPlanner(program), sql);
> > >>  }
> > >>
> > >> The plan is
> > >> LogicalProject(DEPTNO=[$7])
> > >>  LogicalFilter(condition=[=($3, 10)])
> > >>LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> > >>
> > >> Enrico ~ you may try ReduceExpressionsRule.FILTER_INSTANCE
> > >>
> > >
> > >
> > > @XING JIN
> > > thank you.
> > > I am with Calcite 1.19 and VolcanoPlanner
> > >
> > > Original query:
> > >
> > > select * from pippo where n1 is null AND n1 is not null
> > >
> > > Logical plan:
> > >
> > > LogicalFilter(condition=[AND(IS NULL($1), IS NOT NULL($1))]): rowcount
> =
> > > 1.35, cumulative cost = {7.35 rows, 13.0 cpu, 0.0 io}, id = 48
> > >
> > >

Calcite-Master - Build # 1355 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Calcite-Master (build #1355)

Status: Still Failing

Check console output at https://builds.apache.org/job/Calcite-Master/1355/ to 
view the results.

Re: How to modify data for custom tables through Calcite.

[Mohamed Mohsen]

I haven't done that before but I would start investigating from this
interface [1]. Please share your experience if you get this done.

[1] org.apache.calcite.schema.ModifiableTable


On Wed, Sep 25, 2019 at 2:00 PM Juan Pan  wrote:

> Hi everyone,
>
>
> Thanks for your attention. I want to know the following description is
> right or not?
>
>
> "Modification has only been worked on for JDBC tables, not for any custom
> tables currently.”
>
>
> Query SQL on custom table is ok, so i am wondering whether i can execute
> `update/insert/delete` SQL through Calcite on custom tables.
>
>
> Can anyone give me some ideas?
>
>
> Really thanks for your help.
>
>
> Regards,
> Trista
>
>
>
>
>
>
>  Juan Pan
>
>
> panj...@apache.org
> Juan Pan(Trista), Apache ShardingSphere
>
>

[DISCUSS] [CALCITE-3340] Table Function TUMBLE as an operator in SqlStdOperator

[Rui Wang]

Hi community,

In order to support TUMBLE as a TABLE function (see CALCITE-3272), the
first step is to add a new operator in SqlStdOperatorTable(which is
CALCITE-3340).  Note that new operator will need a name, which is the
syntax used in a query. For example, GROUP BY TUMBE(...) will lead to a
TUMBLE operator created with a name "TUMBLE".

Thus, for the new operator for TUMBLE as built-in table function, we also
need to specify a name for it. And there are options: still use "TUMBLE",
or use a new name, e.g. "TUMBLE_TABLE".

For readers' convenience, start from here I will name GROUP BY TUMBLE as
the old operator, and new table function one as the new operator.

Option 1 that reuse "TUMBLE" as new operator's name:
This is called operator overloading in Calcite because Calcite seems two
operators are overloading by checking their names. It turns out, unlike
other built-in operators, old TUMBLE is not recognized by Parser directly:
Parser leaves it as UnresolvedFunction to validator. Then it also turns out
validator is not favor of overloaded built-in operators: overloaded
built-in operators will lead to "function not found" because more than one
built-in operators are found from operator table for TUMBLE based on an
operator name lookup approach. If more than one built-in operator is found
by validator, validator will just not match either one.

I am now convinced by many of my exploration, testing, code reading and
prototyping that the resolution for option 1, is to code *both* old and new
TUMBLE direclty in Parser, and overrite "SqlOperator,deriveType" for *both*
old and new operators.

You could check one of my prototype:
https://github.com/apache/calcite/pull/1457


For option 2 that use a new name for new operator's name, e.g.
"TUMBLE_TALBE"
Because validator treats two operators as operator overloading if they have
the same name, using another name is ok. It would be very straighforward to
let the new operator go through parse, validate, rel steps if using a new
name (I have proven it in an old verrsion of PR1457
 but unfortunetly I did a
force push so that version was overwritten and it cannot be seen now).



For me it seems like option 2 makes sense and require less work. However I
do want to hear more opinions from community. Please let me know if you
have any thought.


-Rui

[jira] [Created] (CALCITE-3372) Add a rule to transform a semijoin into a join on top of aggregate

[Haisheng Yuan (Jira)]

Haisheng Yuan created CALCITE-3372:
--

 Summary: Add a rule to transform a semijoin into a join on top of 
aggregate
 Key: CALCITE-3372
 URL: https://issues.apache.org/jira/browse/CALCITE-3372
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: Haisheng Yuan


{code:java}
Semijoin
 |- TS1
 +- TS2
{code}
can be transformed into
{code:java}
Project
  +-InnerJoin
  |- TS1
  +- Aggregate
+ TS2
{code}

We may benefit from the new plan by join relation reordering if TS2 is large 
even after aggregation.





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Trivial query simplification

[Enrico Olivelli]

Thank you for your feedback.

Actually ReduceExpressionsRule.FILTER_INSTANCE fixes the problem.

RelOptPlanner optPlanner = cluster.getPlanner();
optPlanner.addRule(ReduceExpressionsRule.FILTER_INSTANCE);

This code I had was wrong:
 final FrameworkConfig config = Frameworks.newConfigBuilder()
.parserConfig(SQL_PARSER_CONFIG)
.defaultSchema(subSchema)
.traitDefs(TRAITS)
.programs(Programs.ofRules(Programs.RULE_SET))
.build();

as the 'programs' property of FrameworkConfig is applied only if you are
using Planner#transform .

@Julian do I have to create a JIRA case for RexSimplify ? Honestly I don't
have enough knowledge to explain the problem and create a meaning full
issue.

Is it work to add ReduceExpressionsRule.FILTER_INSTANCE to the default
rules ?

As far as I can see the "default rules" are in CalcitePrepareImpl, is that
correct?


Cheers
Enrico


Il giorno mar 24 set 2019 alle ore 20:01 Julian Hyde  ha
scritto:

> A few thoughts on this.
>
>
> 0. I am surprised that this simplification does not happen, and I think we
> should do it. Specifically, "filter(x = 0 and x is not null)" should
> simplify to “filter(x = 0)”.
>
> 1. Enrico, please log a JIRA case now, and transcribe the key points of
> this discussion into the JIRA case when the discussion has finished.
>
> 2. The most obvious way to achieve this is via simplification, i.e.
> RexSimplify, which occurs when building expressions via RelBuilder. It does
> not require planner rules.
>
> 3. An algorithm to achieve this would be to gather the implied predicates
> as we go through a conjunction. After generating the condition “x = 0” we
> arrive at “x is not null”. We know that “x = 0” holds, therefore we could
> also deduce that “x is not null” holds (we could also deduce other
> conditions, such as “x < 100” holds).
>
> 4. Another way to achieve this is via
> ReduceExpressionsRule.FILTER_INSTANCE. The algorithm would be as follows.
> First note the condition “x = 0” and therefore constant-reduce x to 0 in
> the expression “0 is not null”. This algorithm has similar problems to the
> algorithm in 2 - it passes along the conjunctive predicates in strict order
> and therefore only works if they are in a particular order. Also, as a
> planner rule, this algorithm is more expensive, so would not be applied as
> early/often as the RexSimplify implementation.
>
> 5. We could also simplify “filter(x is not null and x = 0)” to “filter(x =
> 0)”, and people would reasonably expect that we would. But that is a more
> complex algorithm because it would require, for instance, re-ordering
> predicates. In the past, we have discussed re-ordering predicates as part
> of simplification; I am cautious about doing it because it would affect a
> lot of existing plans (and tests). There is no perfect order for
> predicates, so we might come back in 6 months and want to change the order
> again. Better to sort them during simplification but then spit them out in
> the original order.
>
> Julian
>
>
>
> > On Sep 24, 2019, at 8:34 AM, Enrico Olivelli 
> wrote:
> >
> > Il giorno mar 24 set 2019 alle ore 13:45 XING JIN <
> jinxing.co...@gmail.com >
> > ha scritto:
> >
> >> "v = 1 and v is null"
> >> cannot be simplified to "v = 1" not matter v is nullable or not nullable
> >>
> >> If you really mean that "v is not null", I made below test case in
> >> RelOptRulesTest.java for illustration:
> >>
> >>
> >> // mgr is nullable
> >>  @Test public void testDEV() throws Exception {
> >>HepProgram program = new HepProgramBuilder()
> >>  .addRuleInstance(ReduceExpressionsRule.FILTER_INSTANCE)
> >>  .build();
> >>
> >>final String sql = "select deptno"
> >>  + " from emp"
> >>  + " where mgr = 10 and mgr is not null";
> >>checkPlanning(new HepPlanner(program), sql);
> >>  }
> >>
> >> The plan is
> >> LogicalProject(DEPTNO=[$7])
> >>  LogicalFilter(condition=[=($3, 10)])
> >>LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> >>
> >> Enrico ~ you may try ReduceExpressionsRule.FILTER_INSTANCE
> >>
> >
> >
> > @XING JIN
> > thank you.
> > I am with Calcite 1.19 and VolcanoPlanner
> >
> > Original query:
> >
> > select * from pippo where n1 is null AND n1 is not null
> >
> > Logical plan:
> >
> > LogicalFilter(condition=[AND(IS NULL($1), IS NOT NULL($1))]): rowcount =
> > 1.35, cumulative cost = {7.35 rows, 13.0 cpu, 0.0 io}, id = 48
> >
> >EnumerableTableScan(table=[[herd, pippo]]): rowcount = 6.0, cumulative
> > cost = {6.0 rows, 7.0 cpu, 0.0 io}, id = 47
> >
> > Final Plan:
> >
> > BindableTableScan(table=[[herd, pippo]], filters=[[AND(IS NULL($1), IS
> NOT
> > NULL($1))]]): rowcount = 6.0, cumulative cost = {0.03 rows, 0.035 cpu,
> 0.0
> > io}, id = 59
> >
> >
> > It seems that ReduceExpressionsRule.FILTER_INSTANCE does not have any
> > effect.
> > May it be a problem of 1.19 or VolcanoPlanner ?
> >
> > This is my "program" now:
> >
> > 

How to modify data for custom tables through Calcite.

[Juan Pan]

Hi everyone,


Thanks for your attention. I want to know the following description is right or 
not?


"Modification has only been worked on for JDBC tables, not for any custom 
tables currently.”


Query SQL on custom table is ok, so i am wondering whether i can execute 
`update/insert/delete` SQL through Calcite on custom tables.


Can anyone give me some ideas?


Really thanks for your help.


Regards,
Trista






 Juan Pan


panj...@apache.org
Juan Pan(Trista), Apache ShardingSphere

[jira] [Created] (CALCITE-3371) revise doc

[xzh_dz (Jira)]

xzh_dz created CALCITE-3371:
---

 Summary: revise doc
 Key: CALCITE-3371
 URL: https://issues.apache.org/jira/browse/CALCITE-3371
 Project: Calcite
  Issue Type: Wish
Reporter: xzh_dz


h1. revise doc

{[@link|https://github.com/link] ProjectToCalcRule} creates this from a 
{[@link|https://github.com/link] LogicalProject}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)