It does seem to be something that RelBuilder could do. (RexSimplify can’t
really do it, because it doesn’t know how the expression is being used.)
It’s also worth discovering why the CAST was added in the first place. It
doesn’t seem to be helpful. I think we should strive to eliminate all of
the slightly unhelpful things that Calcite does; those things can add up
and cause major inefficiencies in the planning process and/or sub-optimal
plans.
Julian
On Mar 24, 2020, at 1:47 AM, Zoltan Haindrich <k...@rxd.hu> wrote:
Hey,
That's a great diagnosis :)
I would guess that newCondition became non-nullable for some reason
(rexSimplify runs under RexProgramBuilder so it might be able to narrow the
nullability)
you could try invoking simplify.simplifyPreservingType() on it to see
if that would help.
I know it's necessary to preserve the nullability when simplifying a
boolean expression in project columns, but as for condition in Filter/Calc,
may be we can omit the
nullability?
I think that could probably work - we can't change the nullability on
project columns because those could be referenced (and the reference also
has the type) ; but for filter/join conditions we don't need to care with
it.
It seems we already have a "matchnullability" in ReduceExpressionsRule
; for FILTER/JOIN we should probably turn that off... :)
cheers,
Zoltan
On 3/24/20 9:15 AM, Shuo Cheng wrote:
Hi Zoltan,
I encountered the problem when running TPC tests, and have not
reproduced it in Calcite master.
But I figured it out how the problem is produced:
There is semi join with the condition：AND(EXPANDED_INDF1,
EXPANDED_INDF2), type of AND is BOOLEAN with nullable `true`
After JoinPushExpressionsRule -->> join condition: AND(INDF1, INDF2),
type of AND is BOOLEAN with nullable `true`
After SemiJoinProjectTransposeRule --> Join condition:
CAST(AND(INDF1, INDF2)), type of AND is BOOLEAN with nullable `false`
Just as what you suspected, It's in `SemiJoinProjectTransposeRule`
where forced type correction is added by `RexProgramBuilder#addCondition`,
which will call `RexSimplify#simplifyPreservingType` before registering an
expression.
I know it's necessary to preserve the nullability when simplifying a
boolean expression in project columns, but as for condition in Filter/Calc,
may be we can omit the nullability?
Best Regards,
Shuo
On Tue, Mar 24, 2020 at 3:35 PM Zoltan Haindrich <k...@rxd.hu <mailto:
k...@rxd.hu>> wrote:
Hey Shuo!
I think that simplification should been made on join conditions -
I've done a quick check; and it seems to be working for me.
I suspected that it will be either a missing call to RexSimplify
for some reason - or it is added by a forced return type correction: IIRC
there are some cases in which
the
RexNode type should retained after simplification.
Is this reproducible on current master; could you share a testcase?
cheers,
Zoltan
On 3/24/20 7:28 AM, Shuo Cheng wrote:
> Hi, Julian, That's what we do as a workaround way. we remove
CAST which are
> only widening nullability as what CALCITE-2695 does before
applying
> hash-join/sort-merge-join rule, such that equiv predicate can be
split
> out. I'm not sure whether it's properly for Calcite to do the
'convert
> back' job, for example, simplify the join condition when create
a Join; Or
> maybe let other systems what use Calcite to do the "convert
back" job as an
> optimization? What do you think?
>
> On Tue, Mar 24, 2020 at 2:04 PM Julian Hyde <
jhyde.apa...@gmail.com <mailto:jhyde.apa...@gmail.com>> wrote:
>
>> Or convert it back to a not-nullable BOOLEAN? The join
condition treats
>> UNKNOWN the same as FALSE, and besides UNKNOWN will never
occur, so the
>> conditions with and without the CAST are equivalent.
>>
>> Julian
>>
>>> On Mar 23, 2020, at 9:34 PM, Shuo Cheng <njucs...@gmail.com
<mailto:njucs...@gmail.com>> wrote:
>>>
>>> Hi all,
>>>
>>> Considering the Join condition 'CAST(IS_NOT_DISTINCT_FROM($1,
$2),
>>> BOOLEAN)', which cast the non-nullable BOOLEAN to nullable
BOOLEAN,
>> Calcite
>>> can not split out equiv predicate, thus some join operation
like hash
>> join
>>> / sort merge join may not be used. Maybe we can
>>> expand RelOptUtil#splitJoinCondition to support this scenario?
>>
>
