Re: Is there a mechanism for constant folding in Calcite?

[Stamatis Zampetakis]

Hi Ian,

As Alessandro correctly said, the engine relies on scalar subqueries being
unnested. Rules are not applied to RexSubquery expressions.
In a relatively recent discussion [1] somebody else also expressed the need
to retain nesting in scalar subqueries.
I think that if somebody pushes this forward it could be something that
could be incorporated to the core.

Best,
Stamatis

[1]
https://lists.apache.org/thread.html/r17369e6319e2d810fdeb99bdc6fd90cf5dfd5648fb2f85589b4cc12d%40%3Cdev.calcite.apache.org%3E

On Tue, Oct 26, 2021 at 8:57 PM Alessandro Solimando <
alessandro.solima...@gmail.com> wrote:

> Hi Ian,
> regarding commutativity/associativity I think this ML discussion
> <
> https://lists.apache.org/thread.html/r68b60538222d01a3cf065e44f8dbad7c23d042350d9cd6b7b52ee811%40%3Cdev.calcite.apache.org%3E
> >
> could
> be relevant and it has some pointers.
>
> For what concerns decorrelation, I think that most of Calcite's code relies
> on subqueries being decorrelated, like in the example you have cited.
> I recall some ML discussions where this topic has been elaborated but I
> can't find any right now.
>
> Best regards,
> Alessandro
>

Re: Is there a mechanism for constant folding in Calcite?

[Alessandro Solimando]

Hi Ian,
regarding commutativity/associativity I think this ML discussion

could
be relevant and it has some pointers.

For what concerns decorrelation, I think that most of Calcite's code relies
on subqueries being decorrelated, like in the example you have cited.
I recall some ML discussions where this topic has been elaborated but I
can't find any right now.

Best regards,
Alessandro

Re: Is there a mechanism for constant folding in Calcite?

[Ian Bertolacci]

Thanks! 
I've just tried it out, and it does work for lot of our cases.

But I've noticed that it doesn't fold expressions that could be reordered under 
associativity rules.
For example `+( $1, +( 10, 20 ) )` is folded to `+( $1, 30 )`, but the 
left-deep equivalent `+( +( $1, 10 ), 20 )` is not folded.
I assume that ReduceExpressionsRule does not attempt any kind of reordering of 
associative/commutative operators? (It could be a large search space, I know)
Are there other rules that we can/should apply before ReduceExpressionsRule 
that would allow it to fold the expression `$1 + 10 + 20` regardless of the 
nesting?

Additionally, ReduceExpressionsRule doesn't seem to handle scalar subquery 
expressions.
I think the idea is/was that scalar subquery expressions would be converted to 
joins by the *_SUB_QUERY_TO_CORRELATE CoreRules.
However, for our purposes we retain un-correlated scalar subqueries.
It seems that RexProgramBuilder.RegisterShuttle/RegisterInputShuttle don't 
override the visitSubQuery method and return a RexInputRef as it does for all 
the other expression types.
As a result, when RegisterShuttle/RegisterInputShuttle's visitSubQuery method 
is invoked on the scalar subquery expression, nothing happens, no RexInputRef 
is created for it, and it simply returns the original RexSubQuery.
Then RexProgramBuilder attempts to cast that node as a RexInputRef, and throws 
"org.apache.calcite.rex.RexSubQuery cannot be cast to 
org.apache.calcite.rex.RexLocalRef at 
org.apache.calcite.rex.RexProgramBuilder.registerInput(RexProgramBuilder.java:296)"
(At least, this is what happens in 1.26.0, I haven't checked new versions).
I figure this is something we are just gonna have to work around on our end?

Thanks!
-Ian J. Bertolacci

On 10/25/21, 10:12 PM, "Julian Hyde"  wrote:

Chunwei is correct.

If there are expressions that you would expect to constant-folded by that 
rule that aren’t, please log a bug.

> On Oct 25, 2021, at 7:10 PM, Chunwei Lei  wrote:
> 
> Hi, Ian.
> 
> ReduceExpressionsRule is always used to do constant folding.
> 
> 
> Best,
> Chunwei
> 
> 
> On Tue, Oct 26, 2021 at 4:34 AM Ian Bertolacci
>  wrote:
> 
>> Howdy,
>> Does Calcite have any mechanism for applying constant folding to RexNodes
>> in a query?
>> 
>> We’ve been wondering why expressions like `1 + 2` don’t get folded into
>> the constant `3`.
>> We’re aware of RexSimplify, but it only does constant folding for Boolean
>> expressions (i.e. logical and (some) comparison operations)
>> And even with RexSimplify, we find that it still does not perform some
>> (what we would consider trivial) optimizations on SEARCH expressions.
>> For example `SEARCH(123:BIGINT, Sarg[123L:BIGINT]:BIGINT)` should get
>> folded into `true`, but isn’t.
>> (But that’s a separate conversation).
>> 
>> So: is constant folding on all expressions something that already exists
>> in Calcite?
>> If so, what should we be looking at to configure Calcite to apply 
constant
>> folding?
>> 
>> Thanks!
>> -Ian J. Bertolacci
>> 

Re: Is there a mechanism for constant folding in Calcite?

[Julian Hyde]

Chunwei is correct.

If there are expressions that you would expect to constant-folded by that rule 
that aren’t, please log a bug.

> On Oct 25, 2021, at 7:10 PM, Chunwei Lei  wrote:
> 
> Hi, Ian.
> 
> ReduceExpressionsRule is always used to do constant folding.
> 
> 
> Best,
> Chunwei
> 
> 
> On Tue, Oct 26, 2021 at 4:34 AM Ian Bertolacci
>  wrote:
> 
>> Howdy,
>> Does Calcite have any mechanism for applying constant folding to RexNodes
>> in a query?
>> 
>> We’ve been wondering why expressions like `1 + 2` don’t get folded into
>> the constant `3`.
>> We’re aware of RexSimplify, but it only does constant folding for Boolean
>> expressions (i.e. logical and (some) comparison operations)
>> And even with RexSimplify, we find that it still does not perform some
>> (what we would consider trivial) optimizations on SEARCH expressions.
>> For example `SEARCH(123:BIGINT, Sarg[123L:BIGINT]:BIGINT)` should get
>> folded into `true`, but isn’t.
>> (But that’s a separate conversation).
>> 
>> So: is constant folding on all expressions something that already exists
>> in Calcite?
>> If so, what should we be looking at to configure Calcite to apply constant
>> folding?
>> 
>> Thanks!
>> -Ian J. Bertolacci
>> 

Re: Is there a mechanism for constant folding in Calcite?

[Chunwei Lei]

Hi, Ian.

ReduceExpressionsRule is always used to do constant folding.


Best,
Chunwei


On Tue, Oct 26, 2021 at 4:34 AM Ian Bertolacci
 wrote:

> Howdy,
> Does Calcite have any mechanism for applying constant folding to RexNodes
> in a query?
>
> We’ve been wondering why expressions like `1 + 2` don’t get folded into
> the constant `3`.
> We’re aware of RexSimplify, but it only does constant folding for Boolean
> expressions (i.e. logical and (some) comparison operations)
> And even with RexSimplify, we find that it still does not perform some
> (what we would consider trivial) optimizations on SEARCH expressions.
> For example `SEARCH(123:BIGINT, Sarg[123L:BIGINT]:BIGINT)` should get
> folded into `true`, but isn’t.
> (But that’s a separate conversation).
>
> So: is constant folding on all expressions something that already exists
> in Calcite?
> If so, what should we be looking at to configure Calcite to apply constant
> folding?
>
> Thanks!
> -Ian J. Bertolacci
>

Is there a mechanism for constant folding in Calcite?

[Ian Bertolacci]

Howdy,
Does Calcite have any mechanism for applying constant folding to RexNodes in a 
query?

We’ve been wondering why expressions like `1 + 2` don’t get folded into the 
constant `3`.
We’re aware of RexSimplify, but it only does constant folding for Boolean 
expressions (i.e. logical and (some) comparison operations)
And even with RexSimplify, we find that it still does not perform some (what we 
would consider trivial) optimizations on SEARCH expressions.
For example `SEARCH(123:BIGINT, Sarg[123L:BIGINT]:BIGINT)` should get folded 
into `true`, but isn’t.
(But that’s a separate conversation).

So: is constant folding on all expressions something that already exists in 
Calcite?
If so, what should we be looking at to configure Calcite to apply constant 
folding?

Thanks!
-Ian J. Bertolacci