[jira] [Created] (CALCITE-6492) Support aggregate functions which could process DISTINCT natively
Zoltan Haindrich created CALCITE-6492: - Summary: Support aggregate functions which could process DISTINCT natively Key: CALCITE-6492 URL: https://issues.apache.org/jira/browse/CALCITE-6492 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich This could be usefull if the execution engine natively supports some distinct aggregations natively - there is no rewrite necessary for these functions. Currently there is support [SqlAggFunction#getDistinctOptionality|https://github.com/apache/calcite/blob/0deab6f7e0cb4ec63eae8b59477d6f0fadfd11e8/core/src/main/java/org/apache/calcite/sql/SqlAggFunction.java#L187-L189] - which have overlaps with this - possibly the closest would be to set it to *IGNORED* if its supported natively...however * that's a bit misleading as its not IGNORED; but supported... * there is also [checkArgument|https://github.com/apache/calcite/blob/0deab6f7e0cb4ec63eae8b59477d6f0fadfd11e8/core/src/main/java/org/apache/calcite/rel/core/AggregateCall.java#L125] which ensures that *distinct* is not accepted in tht case. More or less the end result would be to also enhance AggregateExpandDistinctAggregatesRule with the ability to ignore aggregates. note: In Druid * if approximationCountDistinct is disabled ; that [enables a calcite rule which rewrites *all* disitnct aggregates|https://github.com/apache/druid/blob/master/sql/src/main/java/org/apache/druid/sql/calcite/planner/CalciteRulesManager.java#L496-L503] * in the meantime there are also some aggregate functions which support *distinct* natively like [string_agg](https://github.com/apache/druid/blob/c9aae9d8e683c0cc9c4687e526b8270f744c57c2/sql/src/main/java/org/apache/druid/sql/calcite/aggregation/builtin/StringSqlAggregator.java#L154) - which doesn't need any rewrites -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6473) HAVING clauses may not contain window functions
Zoltan Haindrich created CALCITE-6473: - Summary: HAVING clauses may not contain window functions Key: CALCITE-6473 URL: https://issues.apache.org/jira/browse/CALCITE-6473 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich according to the standard: {code} The shall not contain a without an intervening . {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6435) SqlToRel conversion of IN expressions may lead to incorrect simplifications
Zoltan Haindrich created CALCITE-6435: - Summary: SqlToRel conversion of IN expressions may lead to incorrect simplifications Key: CALCITE-6435 URL: https://issues.apache.org/jira/browse/CALCITE-6435 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich the query must have the following features: * not all columns are selected ** to enable `RelFieldTrimmer` to start a cycle * two equivalent eq filters ** one in `IN` form (`ename in ( 'Sebastian' )`) ** a regular `=` (`ename = 'Sebastian'`) * an unrelated filter like `deptno < 100` the optimizer should more-or-less start with the `RelFieldTrimmer` the issue happens like: * at parse time both literals are parsed as `CHAR(n)` * the number of values in the `IN` is below `inSubqueryThreshold` - so it gets converted to a set of `=` filters ** expression is converted to OR form ** during conversion [SqlToRelConverter#ensureSqlType|#ensureSqlType]([https://github.com/apache/calcite/blob/fb15511e76c660cbd440578421645ebe63941bf7/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L1777-L1779]) is called *** which skips the conversion for `CHAR` / `VARCHAR` * the `=` filter goes thru the "regular" rex conversion - which involves calling `rexBuilder#ensureType` * the filter condition contains `ename = 'Sebastian'` twice; however the types differ * `RelFieldTrimmer` start a change cycle ; which induces the simplification of the filter condition * `RexSimplify` is executed with predicate elimination disabled (this will be important) * simplification compares the two literals with [equals]([https://github.com/apache/calcite/blob/fb15511e76c660cbd440578421645ebe63941bf7/core/src/main/java/org/apache/calcite/rex/RexSimplify.java#L1685]) and returns `false` workarounds: * disable the conversion by setting `inSubqueryThreshold` to `1` * run a rule which executes `RexSimplify` with predicate elimination enabled earlier than the trimmer (ex: `ReduceExpressionsRule`) ** I think this bug remained hidden because this might happen easily testcase for `RelOptRulesTest` {code:java} @Test void testIncorrectInType() { final String sql = "select ename from emp " + " where ename in ( 'Sebastian' ) and ename = 'Sebastian' and deptno < 100"; sql(sql) .withTrim(true) .withRule() .checkUnchanged(); } {code} results in plan {code:java} LogicalProject(ENAME=[$0]) LogicalValues(tuples=[[]]) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6020) SqlToRelConverter should not replace windowed SUM with equivalent expression using SUM0
Zoltan Haindrich created CALCITE-6020: - Summary: SqlToRelConverter should not replace windowed SUM with equivalent expression using SUM0 Key: CALCITE-6020 URL: https://issues.apache.org/jira/browse/CALCITE-6020 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich {{SqlToRelConverter}} replaces {{SUM}} with {{SUM0}} around [here|https://github.com/apache/calcite/blob/e1991e08a225ef08c2402ab35c310d88fff3c222/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5885] This might have been needed at some point in the past - but I think it will be better to leave it as {{SUM}} - as in case there is no {{SUM0}} in the system that will be replaced with a {{COALESCE(SUM(...) , 0 )}} to provide it - as see [here|https://github.com/apache/calcite/blob/e1991e08a225ef08c2402ab35c310d88fff3c222/core/src/test/java/org/apache/calcite/rel/rel2sql/RelToSqlConverterTest.java#L1288] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5953) AggregateCaseToFilterRule may make inaccurate SUM transformation
Zoltan Haindrich created CALCITE-5953: - Summary: AggregateCaseToFilterRule may make inaccurate SUM transformation Key: CALCITE-5953 URL: https://issues.apache.org/jira/browse/CALCITE-5953 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich consider: `sum(case when x = 1 then 2 else 0 end) as b` notice that this expression may only be null if there are no rows in the table `AggregateCaseToFilterRule` rewrites the above expression to `sum(1) filter (where x=2)` which broadens when it could be `null` to when there are no matches to the filter * `A` is `0` correctly in this case; but I think it will be still `0` in case there are 0 input rows * The result for `B` supposed to be `0` but since there are no matches by the filter it becomes `null` * `C` is not touched ``` # Convert CASE to FILTER without matches select sum(case when x = 1 then 1 else 0 end) as a, sum(case when x = 1 then 2 else 0 end) as b, sum(case when x = 1 then 3 else -1 end) as c from (values 0, null, 0, 2) as t(x); +---+---++ | A | B | C | +---+---++ | 0 | 0 | -4 | +---+---++ (1 row) !ok EnumerableCalc(expr#0..2=[{inputs}], expr#3=[CAST($t0):INTEGER], A=[$t3], B=[$t1], C=[$t2]) EnumerableAggregate(group=[{}], A=[COUNT() FILTER $1], B=[SUM($2) FILTER $3], C=[SUM($0)]) EnumerableCalc(expr#0=[{inputs}], expr#1=[1], expr#2=[=($t0, $t1)], expr#3=[3], expr#4=[-1], expr#5=[CASE($t2, $t3, $t4)], expr#6=[IS TRUE($t2)], expr#7=[2], $f2=[$t5], $f3=[$t6], $f4=[$t7], $f5=[$t6]) EnumerableValues(tuples=[[{ 0 }, { null }, { 0 }, { 2 }]]) !plan ``` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-4006) Add ordered-set aggregate functions
Zoltan Haindrich created CALCITE-4006: - Summary: Add ordered-set aggregate functions Key: CALCITE-4006 URL: https://issues.apache.org/jira/browse/CALCITE-4006 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich * mode * percentile_cont * percentile_disc -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3887) Filter and Join conditions may not need to retain nullability during simplifications
Zoltan Haindrich created CALCITE-3887: - Summary: Filter and Join conditions may not need to retain nullability during simplifications Key: CALCITE-3887 URL: https://issues.apache.org/jira/browse/CALCITE-3887 Project: Calcite Issue Type: Improvement Components: core Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Reported by [~icshuo]: that there are situation in which join conditions are placed inside a redundant `CAST`. https://mail-archives.apache.org/mod_mbox/calcite-dev/202003.mbox/%3CCABYySnvtrMFWaBqM7_FgKedXELdXeCqciRJV7G4MyYN07HUOnQ%40mail.gmail.com%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3215) Simplification may not fully simplify IsNotNull expressions
Zoltan Haindrich created CALCITE-3215: - Summary: Simplification may not fully simplify IsNotNull expressions Key: CALCITE-3215 URL: https://issues.apache.org/jira/browse/CALCITE-3215 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich CALCITE-2929 have added a safety check to avoid simplifying problematic cases. The safety check apparently misses some kinds, for example: {{UNARY_PLUS}} {code} @Test public void testIsNullSimplificationWithUnaryPlus() { RexNode expr = isNotNull(coalesce(unaryPlus(vInt(1)), vIntNotNull(0))); RexNode s = simplify.simplifyUnknownAs(expr, RexUnknownAs.UNKNOWN); assertThat(expr.isAlwaysTrue(), is(true)); assertThat(s, is(trueLiteral)); } {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (CALCITE-2929) Simplification of IS NULL checks are incorrectly assuming that CAST-s are possible
Zoltan Haindrich created CALCITE-2929: - Summary: Simplification of IS NULL checks are incorrectly assuming that CAST-s are possible Key: CALCITE-2929 URL: https://issues.apache.org/jira/browse/CALCITE-2929 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Example: {code} with ax(s) as (values ('xxx'),(cast(null as character varying))) select cast(s as int) IS NULL from ax; {code} returns a result set; which evaluates however 'xxx' is not a valid integer; so an error should be recieved instead Another class of almost the same issue: {code} select cast('xxx' as int) IS NULL; {code} is also problematic; in that case the not nullability is deduced from the fact that the literal's type is not nullable -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2886) Simplification of AND expressions should push negations earlier
Zoltan Haindrich created CALCITE-2886: - Summary: Simplification of AND expressions should push negations earlier Key: CALCITE-2886 URL: https://issues.apache.org/jira/browse/CALCITE-2886 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich * currently simplification decomposes the AND operands * simplifies all operands * runs main simplification * (re) simplifies all operands which are negated Idea is to make the recursion as a first step; which could also push negation further down; and skip the simplification at end completely - as its not connected to a structural change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2852) Simplification: enable traversal of unknown UDFs
Zoltan Haindrich created CALCITE-2852: - Summary: Simplification: enable traversal of unknown UDFs Key: CALCITE-2852 URL: https://issues.apache.org/jira/browse/CALCITE-2852 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Currently simplification concentrates on basic constructs (and/or/=/...), but if it encounters an UDF which is not known; the recursion is stopped. For example the following expression is not simplified: {code:java} @Test public void testSimplifyRecurseIntoArithmetics() { checkSimplify( plus(literal(1), case_( falseLiteral, literal(1), trueLiteral, literal(2), literal(3))), "+(1, 2)"); } {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2851) Simplification: track visited nodes in paranoid mode
Zoltan Haindrich created CALCITE-2851: - Summary: Simplification: track visited nodes in paranoid mode Key: CALCITE-2851 URL: https://issues.apache.org/jira/browse/CALCITE-2851 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Idea is to have an object which could track all the visited nodes, goal is to identify if we have rules which are about to re-visit a previously visited node again. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2848) Simplifying a case statement's first branch should disregard its safety
Zoltan Haindrich created CALCITE-2848: - Summary: Simplifying a case statement's first branch should disregard its safety Key: CALCITE-2848 URL: https://issues.apache.org/jira/browse/CALCITE-2848 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Example: a very simple CASE which is not get rewritten because the first condition branch contains an unknown UDF {code:sql} CASE WHEN to_date('2001-11-11') = '2000' THEN true ELSE false END {code} This expression right now retained as is; because the "to_date" was an unknown UDF; but since the first branch of a CASE is always evaluated and that could unlock the rewrite of this case to it's boolean form. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2842) Computing digest of IN expressions leads to Exceptions
Zoltan Haindrich created CALCITE-2842: - Summary: Computing digest of IN expressions leads to Exceptions Key: CALCITE-2842 URL: https://issues.apache.org/jira/browse/CALCITE-2842 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Julian Hyde Testcase: {code} /* RexProgramTest */ @Test public void testInDigest() { RexNode e = in(vInt(), literal(1), literal(2)); assertThat(e.toString(), is("IN(?0.int0, 1, 2)")); } /* RexProgramBuilder */ protected RexNode in(Iterable nodes) { return rexBuilder.makeCall(SqlStdOperatorTable.IN, ImmutableList.copyOf(nodes)); } {code} The exception is due to that it tries to look at the other operand https://github.com/apache/calcite/blob/883666929478aabe07ee5b9e572c43a6f1a703e2/core/src/main/java/org/apache/calcite/rex/RexCall.java#L115 [SIMPLE_BINARY_OPS|https://github.com/apache/calcite/blob/883666929478aabe07ee5b9e572c43a6f1a703e2/core/src/main/java/org/apache/calcite/rex/RexCall.java#L64] contains [COMPARISION|https://github.com/apache/calcite/blob/883666929478aabe07ee5b9e572c43a6f1a703e2/core/src/main/java/org/apache/calcite/sql/SqlKind.java#L1248] which also contains "IN" this was introduced in CALCITE-2454 a few days ago; bug not released -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2840) Simplification may use narrower UnknownAs modes during AND/OR simplification
Zoltan Haindrich created CALCITE-2840: - Summary: Simplification may use narrower UnknownAs modes during AND/OR simplification Key: CALCITE-2840 URL: https://issues.apache.org/jira/browse/CALCITE-2840 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Some parts of this was already working in 1.17 ; but 1.18 turned out to be a more conservative in this sense. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2841) Simplification: push negation into Case expression
Zoltan Haindrich created CALCITE-2841: - Summary: Simplification: push negation into Case expression Key: CALCITE-2841 URL: https://issues.apache.org/jira/browse/CALCITE-2841 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Julian Hyde -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2839) Simplify x = true to x and x = false to not x
Zoltan Haindrich created CALCITE-2839: - Summary: Simplify x = true to x and x = false to not x Key: CALCITE-2839 URL: https://issues.apache.org/jira/browse/CALCITE-2839 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Julian Hyde Testcase for RexProgramTest {code} @Test public void testEqTrue() { checkSimplify3(eq(vBool(1), trueLiteral), "?0.bool1", "?0.bool1", "?0.bool1"); } {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2838) Simplification: Remove redundant IS TRUE checks
Zoltan Haindrich created CALCITE-2838: - Summary: Simplification: Remove redundant IS TRUE checks Key: CALCITE-2838 URL: https://issues.apache.org/jira/browse/CALCITE-2838 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich In case simplifcation is already processing in unknownAsFalse mode, {{expr IS TRUE}} is redundant - and may just prevent further optimizations from happening: {code} @Test public void testRedundantIsTrue() { // in case of unknownAsFalse // x is TRUE <=> x checkSimplify3(isTrue(vBool(1)), "IS TRUE(?0.bool1)", "?0.bool1", "IS TRUE(?0.bool1)"); } {code} there are some further possibilities {code} (unknownAsTrue) x is NOT FALSE <=> x (unknownAsTrue) x is FALSE <=> not x {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2802) Druid adapter: Usage of range conditions like "2010-01-01 < timestamp" leads to incorrect results
Zoltan Haindrich created CALCITE-2802: - Summary: Druid adapter: Usage of range conditions like "2010-01-01 < timestamp" leads to incorrect results Key: CALCITE-2802 URL: https://issues.apache.org/jira/browse/CALCITE-2802 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Timestamp range conditions when the timestamp is on left hand side work correctly; however when the literal is on the left hand side results are missing. {code} @Test public void testRangeCalc() { final Fixture2 f = new Fixture2(); checkDateRange(f, f.and( f.le(f.timestampLiteral(2011, Calendar.JANUARY, 1), f.t), f.le(f.t, f.timestampLiteral(2012, Calendar.FEBRUARY, 2))), is("[2011-01-01T00:00:00.000Z/2012-02-02T00:00:00.001Z]")); } {code} Fail: {code} java.lang.AssertionError: Expected: is "[2011-01-01T00:00:00.000Z/2012-02-02T00:00:00.001Z]" but: was "[1900-01-01T00:00:00.000Z/2011-01-01T00:00:00.001Z]" {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2731) Case statements are simplified with unsafe conditions
Zoltan Haindrich created CALCITE-2731: - Summary: Case statements are simplified with unsafe conditions Key: CALCITE-2731 URL: https://issues.apache.org/jira/browse/CALCITE-2731 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich In CALCITE-1413 we were cautious to not simplify {code} CASE a=0 THEN 0 ELSE 1/a END {code} and it seemed good; however: {{RexProgramBuilder}} is also invokes simplify - but the protection at that point is not able to work, because it sees only references: {code} CASE $1 THEN $2 ELSE $3 END {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2695) Simplify casts which are only widening nullability
Zoltan Haindrich created CALCITE-2695: - Summary: Simplify casts which are only widening nullability Key: CALCITE-2695 URL: https://issues.apache.org/jira/browse/CALCITE-2695 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich There are cases in which casts are added; but they are not neccessary. {code} @Test public void testSimplifyRedundantCast() { RexNode expr = cast(isTrue(vBoolNotNull()), tBoolean(true)); assertThat(expr.getType().isNullable(), is(true)); RexNode result = simplify.simplifyUnknownAs(expr, RexUnknownAs.UNKNOWN); assertThat(result.getType().isNullable(), is(false)); } {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2688) Improve diagnosability when return type could not be inferred.
Zoltan Haindrich created CALCITE-2688: - Summary: Improve diagnosability when return type could not be inferred. Key: CALCITE-2688 URL: https://issues.apache.org/jira/browse/CALCITE-2688 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Currently when the return type inferenence fails; a {{null}} is returned; while leads to an NPE very quickly - the resulting Exception is not informative at all. {code} @Test(expected = IllegalArgumentException.class) public void checkNoCommonReturnTypeException() { coalesce(vVarchar(1), vInt(2)); } {code} example NPE: {code} java.lang.NullPointerException at java.util.Objects.requireNonNull(Objects.java:203) at org.apache.calcite.rex.RexCall.(RexCall.java:59) at org.apache.calcite.rex.RexBuilder.makeCall(RexBuilder.java:242) at org.apache.calcite.rex.RexBuilder.makeCall(RexBuilder.java:254) at org.apache.calcite.test.RexProgramBuilderBase.coalesce(RexProgramBuilderBase.java:300) at org.apache.calcite.test.RexProgramTest.checkNoCommonReturnTypeException(RexProgramTest.java:753) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2687) Is distinct from could lead to Exceptions in ReduceExpressionRule
Zoltan Haindrich created CALCITE-2687: - Summary: Is distinct from could lead to Exceptions in ReduceExpressionRule Key: CALCITE-2687 URL: https://issues.apache.org/jira/browse/CALCITE-2687 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Test: {code} @Test public void testReduceConstants4() throws Exception { HepProgram program = new HepProgramBuilder() .addRuleInstance(ReduceExpressionsRule.PROJECT_INSTANCE) .addRuleInstance(ReduceExpressionsRule.FILTER_INSTANCE) .addRuleInstance(ReduceExpressionsRule.JOIN_INSTANCE) .build(); String sql = "select e.mgr is not distinct from f.mgr from emp e join emp f on (e.mgr=f.mgr) where e.mgr is null"; sql(sql).with(program) .check(); } {code} Unfortunately the exception doesn't have a stacktrace by default; just some type: {code} org.apache.calcite.adapter.enumerable.RexToLixTranslator$AlwaysNull {code} Translation is not possible at this point: https://github.com/apache/calcite/blob/d32ee5c320938b5c34ce09df2276c9570c27a301/core/src/main/java/org/apache/calcite/adapter/enumerable/RexToLixTranslator.java#L746 the underlying rexNode is: {code} CAST(null):INTEGER NOT NULL {code} which is problematic -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2685) Bridge cases in which Calcite's type resolution is more stricter than Hive.
Zoltan Haindrich created CALCITE-2685: - Summary: Bridge cases in which Calcite's type resolution is more stricter than Hive. Key: CALCITE-2685 URL: https://issues.apache.org/jira/browse/CALCITE-2685 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Calcite is more stricter w.r.t common types than Hive. For example in case a CASE with different types on the branches is not something Calcite likes: {code} CASE WHEN cond1 THEN booleanCol WHEN cond2 THEN stringCol WHEN cond3 THEN floatCol ELSE doubleCol END {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2675) ReduceExpressionRule may leave behind altered types w.r.t nullability
Zoltan Haindrich created CALCITE-2675: - Summary: ReduceExpressionRule may leave behind altered types w.r.t nullability Key: CALCITE-2675 URL: https://issues.apache.org/jira/browse/CALCITE-2675 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich If a simplification could happen after some [ReduceExpression rewrite|https://github.com/apache/calcite/blob/fcc8bf7f44f92efb3c9a1e1f51ffc1a09cab27b9/core/src/main/java/org/apache/calcite/rel/rules/ReduceExpressionsRule.java#L794]; the simplification result may have a slightly different type in nullability. {code} @Test public void testReduceCaseNullabilityChange() throws Exception { HepProgram program = new HepProgramBuilder() .addRuleInstance(ReduceExpressionsRule.FILTER_INSTANCE) .addRuleInstance(ReduceExpressionsRule.PROJECT_INSTANCE) .build(); try (Hook.Closeable a = Hook.REL_BUILDER_SIMPLIFY.add(Hook.propertyJ(false))) { checkPlanning(program, "select case when empno = 1 then 1 when 1 IS NOT NULL then 2 else null end as qx " + "from emp"); } {code} Exposed by CALCITE-1413 changes; I'm not sure if there is any other variations for which the same could happen. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2652) Fix Rel2Sql conversion in case the join condition contains a boolean column reference
Zoltan Haindrich created CALCITE-2652: - Summary: Fix Rel2Sql conversion in case the join condition contains a boolean column reference Key: CALCITE-2652 URL: https://issues.apache.org/jira/browse/CALCITE-2652 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich If the join condition contains a boolean column reference in an {{AND}} it leads to an exception. Reproducible with the following testcase - by adding it to RelToSqlConvertTest {code} @Test public void testBooleanColInOn() { final String sql = "SELECT 1 from emps join emp on (emp.deptno = emps.empno and manager)"; final String expected = "..."; sql(sql).schema(CalciteAssert.SchemaSpec.POST).ok(expected); } {code} exception is: {code} java.lang.RuntimeException: While invoking method 'public org.apache.calcite.rel.rel2sql.SqlImplementor$Result org.apache.calcite.rel.rel2sql.RelToSqlConverter.visit(org.apache.calcite.rel.core.Project)' at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:527) at org.apache.calcite.rel.rel2sql.RelToSqlConverter.dispatch(RelToSqlConverter.java:103) at org.apache.calcite.rel.rel2sql.RelToSqlConverter.visitChild(RelToSqlConverter.java:109) at org.apache.calcite.rel.rel2sql.RelToSqlConverterTest.toSql(RelToSqlConverterTest.java:136) at org.apache.calcite.rel.rel2sql.RelToSqlConverterTest.access$1(RelToSqlConverterTest.java:134) at org.apache.calcite.rel.rel2sql.RelToSqlConverterTest$Sql.exec(RelToSqlConverterTest.java:2893) at org.apache.calcite.rel.rel2sql.RelToSqlConverterTest$Sql.ok(RelToSqlConverterTest.java:2868) at org.apache.calcite.rel.rel2sql.RelToSqlConverterTest.testBooleanColInOn(RelToSqlConverterTest.java:998) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:89) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:41) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:541) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:763) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:463) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:209) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524) ... 30 more Caused by: java.lang.RuntimeException: While invoking method 'public org.apache.calcite.rel.rel2sql.SqlImplementor$Result org.apache.calcite.rel.rel2sql.RelToSqlConverter.visit(org.apache.calcite.rel.core.Join)' at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:527) at org.apache.calcite.rel.rel2sql.RelToSqlConverter.dispatch(RelToSqlConverter.java:103) at org.apache.calcite.rel.rel2sql.RelToSqlConverter.visitChild(RelToSqlConverter.java:109) at org.apache.calcite.rel.rel2sql.RelToSqlConverter.visit
[jira] [Created] (CALCITE-2632) Add hashCode and equals implementations to RexNode
Zoltan Haindrich created CALCITE-2632: - Summary: Add hashCode and equals implementations to RexNode Key: CALCITE-2632 URL: https://issues.apache.org/jira/browse/CALCITE-2632 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Right now RexNode doesn't have any equals or hashCode functions; which makes it rely on the default implementation. But when we are writing simplification logics we sometimes forget to use {{toString()}} during comparisions and may try to rely on pure equals: * there is a [Set of RexNode-s|https://github.com/apache/calcite/blob/5b16e23dff03e5eaed80642ae91e28ebf806e6b0/core/src/main/java/org/apache/calcite/rex/RexSimplify.java#L1104] during {{AND}} simplification and in [RexUtil as well|https://github.com/apache/calcite/blob/5b16e23dff03e5eaed80642ae91e28ebf806e6b0/core/src/main/java/org/apache/calcite/rex/RexUtil.java#L321] * I've by mistake just written rexNode.equals(otherRexNode) during the implementation of CALCITE-1413 * I've just bumped into the same thing...that [RexUtil.andNot|https://github.com/apache/calcite/blob/5b16e23dff03e5eaed80642ae91e28ebf806e6b0/core/src/main/java/org/apache/calcite/rex/RexUtil.java#L1888] is also rely on itand I think those comparisions go back a while (~3years at least) ; and a bug is not appeared from it because this comparision is in most cases false. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2631) Address small issues in case simplification
Zoltan Haindrich created CALCITE-2631: - Summary: Address small issues in case simplification Key: CALCITE-2631 URL: https://issues.apache.org/jira/browse/CALCITE-2631 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich followup of CALCITE-1413 https://github.com/apache/calcite/commit/b470a0cd4572c9f6c4c0e9b51926b97c5af58d3f#comments to address the following things: * fuse branch removal logics with case branch simplification * postpone condition simplification during branch compaction removal to avoid re-simplification of the same subtree if multiple branches are removed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2611) Code generation fails if one side of an or contains unknown
Zoltan Haindrich created CALCITE-2611: - Summary: Code generation fails if one side of an or contains unknown Key: CALCITE-2611 URL: https://issues.apache.org/jira/browse/CALCITE-2611 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Julian Hyde This is rather strange...it needs also an and below which contains the unknown. Running the following query: {code} !connect jdbc:calcite:model=target/test-classes/model.json admin admin select (EMPID = 3 and unknown) or ( EMPID = 3) from emps; {code} results in a compilation exception; the main problem is that there is an interesting field: {code} static final Object $L4J$C$_null = !null; {code} full exception: {code} 0: jdbc:calcite:model=target/test-classes/mod> select (cast(EMPID as integer) = 3 and unknown) or not ( cast(EMPID as integer) = 3) from emps; Error: Error while executing SQL "select (cast(EMPID as integer) = 3 and unknown) or not ( cast(EMPID as integer) = 3) from emps": Error while compiling generated Java code: org.apache.calcite.DataContext root; public org.apache.calcite.linq4j.Enumerable bind(final org.apache.calcite.DataContext root0) { root = root0; final org.apache.calcite.rel.RelNode v1stashed = (org.apache.calcite.rel.RelNode) root.get("v1stashed"); final org.apache.calcite.interpreter.Interpreter interpreter = new org.apache.calcite.interpreter.Interpreter( root, v1stashed); return new org.apache.calcite.linq4j.AbstractEnumerable(){ public org.apache.calcite.linq4j.Enumerator enumerator() { return new org.apache.calcite.linq4j.Enumerator(){ public final org.apache.calcite.linq4j.Enumerator inputEnumerator = interpreter.enumerator(); public void reset() { inputEnumerator.reset(); } public boolean moveNext() { return inputEnumerator.moveNext(); } public void close() { inputEnumerator.close(); } public Object current() { final Integer inp5_ = (Integer) ((Object[]) inputEnumerator.current())[5]; return inp5_ != null && inp5_.intValue() != 3 ? Boolean.TRUE : inp5_ != null || $L4J$C$_null || inp5_ == null ? (Boolean) null : Boolean.FALSE; } static final Object $L4J$C$_null = !null; }; } }; } public Class getElementType() { return java.lang.Boolean.class; } (state=,code=0) {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2606) Return type inference for and/or may identify result type incorrectly
Zoltan Haindrich created CALCITE-2606: - Summary: Return type inference for and/or may identify result type incorrectly Key: CALCITE-2606 URL: https://issues.apache.org/jira/browse/CALCITE-2606 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich In case in an AND expression a null constant appears earlier than a nullable boolean; the return type is set to NULL instead of boolean. https://github.com/apache/calcite/blob/c39bfaa02a06ac91575076a6e74f29863923f5eb/core/src/main/java/org/apache/calcite/sql/type/ReturnTypes.java#L198 The problem can be reproduced; however I'm not sure if it would happen naturally or not - I've discovered it during enhancing case simplification. {code} @Test public void testUsageOfConstantNull() { RexLiteral constantNull = rexBuilder.constantNull(); RexNode node1 = or(constantNull, vBool()); assertThat(node1.getType(), is(vBool().getType())); RexNode node2 = or(vBoolNotNull(), constantNull); assertThat(node2.getType(), is(vBool().getType())); RexNode node3 = or(constantNull, constantNull); assertThat(node3.getType(), is(constantNull.getType())); } {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2445) RexSimplify x=x
Zoltan Haindrich created CALCITE-2445: - Summary: RexSimplify x=x Key: CALCITE-2445 URL: https://issues.apache.org/jira/browse/CALCITE-2445 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Julian Hyde -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2444) Rel2Sql: support c IN (1,2)
Zoltan Haindrich created CALCITE-2444: - Summary: Rel2Sql: support c IN (1,2) Key: CALCITE-2444 URL: https://issues.apache.org/jira/browse/CALCITE-2444 Project: Calcite Issue Type: Bug Components: jdbc-adapter Affects Versions: 1.17.0 Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich It is not possible to convert an Rel tree into an sql string if it contains a filter which has an IN: example: `select * from emp where empno in (10,20)` note: Calcite's sql parser translates INs into either ORs or into a subquery. The following testcase can be added to RelToSqlConverterTest to reproduce the issue: {code} @Test public void testHiveIn() { // this can't be tested using "sql" because Calcite's sql parser replaces INs with ORs or subqueries. final RelBuilder builder = RelBuilder.create(RelBuilderTest.config().build()); RexBuilder rexBuilder = builder.getRexBuilder(); RelNode root = builder.scan("EMP") .filter(rexBuilder.makeCall(SqlStdOperatorTable.IN, builder.field("DEPTNO"), builder.literal(20), builder.literal(21))) .build(); RelNode rel = root; SqlDialect dialect = SqlDialect.DatabaseProduct.HIVE.getDialect(); final RelToSqlConverter converter = new RelToSqlConverter(dialect); final SqlNode sqlNode = converter.visitChild(0, rel).asStatement(); String sqlStr = sqlNode.toSqlString(dialect).getSql(); assertEquals("select * from emp where deptno in (10,20)", sqlStr); } {code} The exception is raised because calcite expects that every IN is a subquery(because of sql2rel rewrites them) {code} Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171] at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524) ~[calcite-core-1.17.0.jar:1.17.0] at org.apache.calcite.rel.rel2sql.RelToSqlConverter.dispatch(RelToSqlConverter.java:103) ~[calcite-core-1.17.0.jar:1.17.0] at org.apache.calcite.rel.rel2sql.RelToSqlConverter.visitChild(RelToSqlConverter.java:109) ~[calcite-core-1.17.0.jar:1.17.0] at org.apache.calcite.rel.rel2sql.RelToSqlConverter.visit(RelToSqlConverter.java:173) ~[calcite-core-1.17.0.jar:1.17.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171] at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524) ~[calcite-core-1.17.0.jar:1.17.0] ... 61 more Caused by: java.lang.ClassCastException: org.apache.calcite.rex.RexCall cannot be cast to org.apache.calcite.rex.RexSubQuery at org.apache.calcite.rel.rel2sql.SqlImplementor$Context.toSql(SqlImplementor.java:540) ~[calcite-core-1.17.0.jar:1.17.0] at org.apache.calcite.rel.rel2sql.RelToSqlConverter.visit(RelToSqlConverter.java:166) ~[calcite-core-1.17.0.jar:1.17.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171] at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524) ~[calcite-core-1.17.0.jar:1.17.0] at org.apache.calcite.rel.rel2sql.RelToSqlConverter.dispatch(RelToSqlConverter.java:103) ~[calcite-core-1.17.0.jar:1.17.0] at org.apache.calcite.rel.rel2sql.RelToSqlConverter.visitChild(RelToSqlConverter.java:109) ~[calcite-core-1.17.0.jar:1.17.0] at org.apache.calcite.rel.rel2sql.RelToSqlConverter.visit(RelToSqlConverter.java:173) ~[calcite-core-1.17.0.jar:1.17.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171] at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:52
[jira] [Created] (CALCITE-2401) Improve RelMdPredicates performance
Zoltan Haindrich created CALCITE-2401: - Summary: Improve RelMdPredicates performance Key: CALCITE-2401 URL: https://issues.apache.org/jira/browse/CALCITE-2401 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich In CALCITE-2384 a simplify feature was disabled - to prevent performance degradation of RelMdPredicates I think that the algorithm which tries out all the combination of the variables could be probably improved to while it's doing the same - it doesn't run RexSimplify that many times. for more details see my comment in the other ticket [here|https://issues.apache.org/jira/browse/CALCITE-2384?focusedCommentId=16530021&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16530021] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2338) Reconsider the externally visible api of RexSimplify
Zoltan Haindrich created CALCITE-2338: - Summary: Reconsider the externally visible api of RexSimplify Key: CALCITE-2338 URL: https://issues.apache.org/jira/browse/CALCITE-2338 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Currently many lower level simplification methods are visible. While I was porting CALCITE-2247 to a branch near 1.16 I've bumped into an issue because of the fact that by calling a specific simplify method may leave out some otherwise applicable simplifications. For master there is already an extra safety feature by the presence of CALCITE-2205; it seems like using less entry points may even lead to better simplifications - by changing this; a filter have been removed. https://github.com/kgyrtkirk/calcite/commit/2e29a659792f6bd9419dc0f97bf5a3bdd9f6f2cc -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2326) Generalize p(x) IS TRUE/FALSE/UNKNOWN handling in RexSimplify
Zoltan Haindrich created CALCITE-2326: - Summary: Generalize p(x) IS TRUE/FALSE/UNKNOWN handling in RexSimplify Key: CALCITE-2326 URL: https://issues.apache.org/jira/browse/CALCITE-2326 Project: Calcite Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich Currently only IS TRUE is handle by the unknownAsFalse field variable. The main goal would be to extend the logic to also handle IS FALSE cases (and possibly IS UNKNOWN too) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2240) Predicate push into CASE statment doesn't happen in some cases
Zoltan Haindrich created CALCITE-2240: - Summary: Predicate push into CASE statment doesn't happen in some cases Key: CALCITE-2240 URL: https://issues.apache.org/jira/browse/CALCITE-2240 Project: Calcite Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Julian Hyde [CaseShuttle|https://github.com/apache/calcite/blob/8139acb78b100483fbafb078fb82d07f921abef0/core/src/main/java/org/apache/calcite/rel/rules/ReduceExpressionsRule.java#L562] may make the optimization but the functions return value doesn't depend on that. For the following query the =1 is not pushed into the case. {code} select empno from emp where case when sal > 1000 then empno else sal end = 1 {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)