[jira] [Updated] (CALCITE-3368) 'is null' expression in SQL may be optimized incorrectly in the underlying implementation
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated CALCITE-3368:
Summary: 'is null' expression in SQL may be optimized incorrectly in the
underlying implementation (was: Nonequivalent simplification happens in
RexSimplify)
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation
> -
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Danny Chan
>Priority: Major
>
> When I test a Fink sql to check overflow situation like
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found the ' when (f0 + f1) is null' has been optimized by Calcite, and
> the optimization may be incorrect.
>
> The Calcite's simplification logic of isNull expression in SQL will convert
> from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL.
> And this issue come from Flink （FLINK-14030）.
>
> [~danny0405]， Could you have a look at this?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated CALCITE-3368:
Description:
When I test a Fink sql to check overflow situation like
{code:java}
select
case when (f0 + f1) is null then 'null' else 'not null' end
from testTable
{code}
, I found the ' when (f0 + f1) is null' has been optimized by Calcite, and the
optimization may be incorrect.
The Calcite's simplification logic of isNull expression in SQL will convert
from
*"f(operand0, operand1) IS NULL"* to
*"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
is ANY。
This simplification leads to the expression will not calculate the real
value of *f(operand0, operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
{code:java}
//org.apache.calcite.rex.RexSimplify.java
private RexNode simplifyIsNull(RexNode a) {
// Simplify the argument first,
// call ourselves recursively to see whether we can make more progress.
// For example, given
// "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
// argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
a = simplify(a, UNKNOWN);
if (!a.getType().isNullable() && isSafeExpression(a)) {
return rexBuilder.makeLiteral(false);
}
if (RexUtil.isNull(a)) {
return rexBuilder.makeLiteral(true);
}
if (a.getKind() == SqlKind.CAST) {
return null;
}
switch (Strong.policy(a.getKind())) {
case NOT_NULL:
return rexBuilder.makeLiteral(false);
case ANY:
// "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
// to "operand0 IS NULL OR operand1 IS NULL"
final List operands = new ArrayList<>();
for (RexNode operand : ((RexCall) a).getOperands()) {
final RexNode simplified = simplifyIsNull(operand);
if (simplified == null) {
operands.add(
rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
} else {
operands.add(simplified);
}
}
return RexUtil.composeDisjunction(rexBuilder, operands, false);
case AS_IS:
default:
return null;
}
}{code}
And most of calculating SqlKinds are assigned *Policy.ANY* at present.
{code:java}
//org.apache.calcite.plan.Strong.java
public static Policy policy(SqlKind kind) {
return MAP.getOrDefault(kind, Policy.AS_IS);
}
map.put(SqlKind.PLUS, Policy.ANY);
map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
map.put(SqlKind.MINUS, Policy.ANY);
map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
map.put(SqlKind.TIMES, Policy.ANY);
map.put(SqlKind.DIVIDE, Policy.ANY);
* that operator evaluates to null. */
public enum Policy {
/** This kind of expression is never null. No need to look at its arguments,
* if it has any. */
NOT_NULL,
/** This kind of expression has its own particular rules about whether it
* is null. */
CUSTOM,
/** This kind of expression is null if and only if at least one of its
* arguments is null. */
ANY,
/** This kind of expression may be null. There is no way to rewrite. */
AS_IS,
}{code}
It may be an obvious nonequivalent simplification in SQL.
And this issue come from Flink （FLINK-14030）.
[~danny0405]， Could you have a look at this?
was:
When I test a Fink sql to check overflow situation
{code:java}
select
case when (f0 + f1) is null then 'null' else 'not null' end
from testTable
{code}
, I found the ' when (f0 + f1) is null' has been simplified by Calcite, and
the simplification may be incorrect.
The Calcite's simplification logic of isNull expression in SQL will convert
from
*"f(operand0, operand1) IS NULL"* to
*"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
is ANY。
This simplification leads to the expression will not calculate the real
value of *f(operand0, operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
{code:java}
//org.apache.calcite.rex.RexSimplify.java
private RexNode simplifyIsNull(RexNode a) {
// Simplify the argument first,
// call ourselves recursively to see whether we can make more progress.
// For example, given
// "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
// argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
a = simplify(a, UNKNOWN);
if (!a.getType().isNullable() && isSafeExpression(a)) {
return rexBuilder.makeLiteral(false);
}
if (RexUtil.isNull(a)) {
return rexBuilder.makeLiteral(true);
}
if (a.getKind() == SqlKind.CAST) {
return null;
}
switch (Strong.policy(a.getKind())) {
case NOT_NULL:
return rexBuilder.makeLiteral(false);
case ANY:
// "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
// to "operand0 IS NULL OR operand1 IS NULL"
final List operands = new ArrayList<>();
for (RexNode operand : ((RexCall) a).getOperands()) {
final RexNode simplified = simplifyIsNull(operand);
if (simplified == null) {
operands.add(
rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
} else {
operands.add(simplified);
}
}
return
[jira] [Updated] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated CALCITE-3368:
Description:
When I test a Fink sql to check overflow situation
{code:java}
select
case when (f0 + f1) is null then 'null' else 'not null' end
from testTable
{code}
, I found the ' when (f0 + f1) is null' has been simplified by Calcite, and
the simplification may be incorrect.
The Calcite's simplification logic of isNull expression in SQL will convert
from
*"f(operand0, operand1) IS NULL"* to
*"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
is ANY。
This simplification leads to the expression will not calculate the real
value of *f(operand0, operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
{code:java}
//org.apache.calcite.rex.RexSimplify.java
private RexNode simplifyIsNull(RexNode a) {
// Simplify the argument first,
// call ourselves recursively to see whether we can make more progress.
// For example, given
// "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
// argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
a = simplify(a, UNKNOWN);
if (!a.getType().isNullable() && isSafeExpression(a)) {
return rexBuilder.makeLiteral(false);
}
if (RexUtil.isNull(a)) {
return rexBuilder.makeLiteral(true);
}
if (a.getKind() == SqlKind.CAST) {
return null;
}
switch (Strong.policy(a.getKind())) {
case NOT_NULL:
return rexBuilder.makeLiteral(false);
case ANY:
// "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
// to "operand0 IS NULL OR operand1 IS NULL"
final List operands = new ArrayList<>();
for (RexNode operand : ((RexCall) a).getOperands()) {
final RexNode simplified = simplifyIsNull(operand);
if (simplified == null) {
operands.add(
rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
} else {
operands.add(simplified);
}
}
return RexUtil.composeDisjunction(rexBuilder, operands, false);
case AS_IS:
default:
return null;
}
}{code}
And most of calculating SqlKinds are assigned *Policy.ANY* at present.
{code:java}
//org.apache.calcite.plan.Strong.java
public static Policy policy(SqlKind kind) {
return MAP.getOrDefault(kind, Policy.AS_IS);
}
map.put(SqlKind.PLUS, Policy.ANY);
map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
map.put(SqlKind.MINUS, Policy.ANY);
map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
map.put(SqlKind.TIMES, Policy.ANY);
map.put(SqlKind.DIVIDE, Policy.ANY);
* that operator evaluates to null. */
public enum Policy {
/** This kind of expression is never null. No need to look at its arguments,
* if it has any. */
NOT_NULL,
/** This kind of expression has its own particular rules about whether it
* is null. */
CUSTOM,
/** This kind of expression is null if and only if at least one of its
* arguments is null. */
ANY,
/** This kind of expression may be null. There is no way to rewrite. */
AS_IS,
}{code}
It may be an obvious nonequivalent simplification in SQL.
And this issue come from Flink （FLINK-14030）.
[~danny0405]， Could you have a look at this?
was:
when I test a Fink sql to check overflow situation like
{code:java}
select
case when (f0 + f1) is null then 'null' else 'not null' end
from testTable
{code}
, I found the ' when (f0 + f1) is null' has been simplified by Calcite, and
the simplification may be incorrect.
The Calcite's simplification logic of isNull expression in SQL will convert
from
*"f(operand0, operand1) IS NULL"* to
*"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
is ANY。
This simplification leads to the expression will not calculate the real
value of *f(operand0, operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
{code:java}
//org.apache.calcite.rex.RexSimplify.java
private RexNode simplifyIsNull(RexNode a) {
// Simplify the argument first,
// call ourselves recursively to see whether we can make more progress.
// For example, given
// "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
// argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
a = simplify(a, UNKNOWN);
if (!a.getType().isNullable() && isSafeExpression(a)) {
return rexBuilder.makeLiteral(false);
}
if (RexUtil.isNull(a)) {
return rexBuilder.makeLiteral(true);
}
if (a.getKind() == SqlKind.CAST) {
return null;
}
switch (Strong.policy(a.getKind())) {
case NOT_NULL:
return rexBuilder.makeLiteral(false);
case ANY:
// "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
// to "operand0 IS NULL OR operand1 IS NULL"
final List operands = new ArrayList<>();
for (RexNode operand : ((RexCall) a).getOperands()) {
final RexNode simplified = simplifyIsNull(operand);
if (simplified == null) {
operands.add(
rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
} else {
operands.add(simplified);
}
}
return
[jira] [Commented] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937422#comment-16937422
]
Leonard Xu commented on CALCITE-3368:
-
[~julianhyde]
Sorry for the confused tile.
How about the title " 'is null' expression in SQL may be optimized incorrectly
in the underlying implementation " ?
> Nonequivalent simplification happens in RexSimplify
> ---
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Danny Chan
>Priority: Major
>
> when I test a Fink sql to check overflow situation like
>
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found the ' when (f0 + f1) is null' has been simplified by Calcite, and
> the simplification may be incorrect.
>
> The Calcite's simplification logic of isNull expression in SQL will convert
> from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL.
> And this issue come from Flink （FLINK-14030）.
>
> [~danny0405]， Could you have a look at this?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3370) Emulate NULL direction for MSSQL
[ https://issues.apache.org/jira/browse/CALCITE-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde resolved CALCITE-3370. -- Fix Version/s: 1.22.0 Resolution: Fixed Fixed in [0c8cbacd|https://github.com/apache/calcite/commit/0c8cbacd0f30b5df15b7cb93c0ab2be5d0324372]; thanks for the PR, [~jswett]! > Emulate NULL direction for MSSQL > > > Key: CALCITE-3370 > URL: https://issues.apache.org/jira/browse/CALCITE-3370 > Project: Calcite > Issue Type: Task >Reporter: Justin Swett >Priority: Minor > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 10m > Remaining Estimate: 0h > > MSSQL cannot handle NULLS LAST/FIRST and should emulate the NULLS LAST/FIRST > in order by, i.e. > --for nulls last > select [column] from [table] ORDER BY CASE WHEN [column] IS NULL THEN 1 ELSE > 0 END, [column]" > --for nulls first > select [column] from [table] ORDER BY CASE WHEN [column] IS NULL THEN 0 ELSE > 1 END, [column]" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3323) Handle arbitrary/unknown functions that have ordinary syntax
[ https://issues.apache.org/jira/browse/CALCITE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde resolved CALCITE-3323. -- Fix Version/s: 1.22.0 Resolution: Fixed Fixed in [ed692fd4|https://github.com/apache/calcite/commit/ed692fd4f2f00261348ecac779cb1a6d544d052e]; thanks for the PR, [~fib-seq]! > Handle arbitrary/unknown functions that have ordinary syntax > > > Key: CALCITE-3323 > URL: https://issues.apache.org/jira/browse/CALCITE-3323 > Project: Calcite > Issue Type: New Feature >Reporter: Ryan Fu >Priority: Major > Fix For: 1.22.0 > > > Add a strategy where if a function has ordinary function syntax and we don't > recognize it we assume that it can take any argument types and returns a > result of unknown type. > We will still need to change the parser to handle functions with non-standard > syntax (e.g. DATEADD). And it's a good idea to explicitly add commonly used > non-standard functions (e.g. MD5, CONCAT).” > [19/Sept/2019] Functions have to have ordinary function syntax [e.g. > foo(arg0, ...)]. Current implementation idea is to: > * Create a known/unknown flag for use within the validator > * Create a more permissive SqlUnresolvedFunction to be passed through > validator > ** Allow OperandTypes to be variadic -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3370) Emulate NULL direction for MSSQL
[ https://issues.apache.org/jira/browse/CALCITE-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CALCITE-3370: Labels: pull-request-available (was: ) > Emulate NULL direction for MSSQL > > > Key: CALCITE-3370 > URL: https://issues.apache.org/jira/browse/CALCITE-3370 > Project: Calcite > Issue Type: Task >Reporter: Justin Swett >Priority: Minor > Labels: pull-request-available > > MSSQL cannot handle NULLS LAST/FIRST and should emulate the NULLS LAST/FIRST > in order by, i.e. > --for nulls last > select [column] from [table] ORDER BY CASE WHEN [column] IS NULL THEN 1 ELSE > 0 END, [column]" > --for nulls first > select [column] from [table] ORDER BY CASE WHEN [column] IS NULL THEN 0 ELSE > 1 END, [column]" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chan reassigned CALCITE-3368:
---
Assignee: Danny Chan
> Nonequivalent simplification happens in RexSimplify
> ---
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Danny Chan
>Priority: Major
>
> when I test a Fink sql to check overflow situation like
>
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found the ' when (f0 + f1) is null' has been simplified by Calcite, and
> the simplification may be incorrect.
>
> The Calcite's simplification logic of isNull expression in SQL will convert
> from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL.
> And this issue come from Flink （FLINK-14030）.
>
> [~danny0405]， Could you have a look at this?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde updated CALCITE-3368:
-
Fix Version/s: (was: 1.22.0)
> Nonequivalent simplification happens in RexSimplify
> ---
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Priority: Major
>
> when I test a Fink sql to check overflow situation like
>
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found the ' when (f0 + f1) is null' has been simplified by Calcite, and
> the simplification may be incorrect.
>
> The Calcite's simplification logic of isNull expression in SQL will convert
> from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL.
> And this issue come from Flink （FLINK-14030）.
>
> [~danny0405]， Could you have a look at this?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937340#comment-16937340
]
Julian Hyde commented on CALCITE-3368:
--
I have removed the 'fix version', unless someone is committing to fix this.
> Nonequivalent simplification happens in RexSimplify
> ---
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Priority: Major
>
> when I test a Fink sql to check overflow situation like
>
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found the ' when (f0 + f1) is null' has been simplified by Calcite, and
> the simplification may be incorrect.
>
> The Calcite's simplification logic of isNull expression in SQL will convert
> from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL.
> And this issue come from Flink （FLINK-14030）.
>
> [~danny0405]， Could you have a look at this?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937338#comment-16937338
]
Julian Hyde edited comment on CALCITE-3368 at 9/25/19 2:09 AM:
---
A SQL user would not understand what "Nonequivalent simplification happens in
RexSimplify" means. Neither do I, in fact.
When you title a JIRA case, imagine that you're writing the release notes.
Because your description will go directly into the release notes.
was (Author: julianhyde):
A SQL user would not understand what "Nonequivalent simplification happens in
RexSimplify" means. Neither do I, in fact.
> Nonequivalent simplification happens in RexSimplify
> ---
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Priority: Major
> Fix For: 1.22.0
>
>
> when I test a Fink sql to check overflow situation like
>
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found the ' when (f0 + f1) is null' has been simplified by Calcite, and
> the simplification may be incorrect.
>
> The Calcite's simplification logic of isNull expression in SQL will convert
> from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL.
> And this issue come from Flink （FLINK-14030）.
>
> [~danny0405]， Could you have a look at this?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937338#comment-16937338
]
Julian Hyde commented on CALCITE-3368:
--
A SQL user would not understand what "Nonequivalent simplification happens in
RexSimplify" means. Neither do I, in fact.
> Nonequivalent simplification happens in RexSimplify
> ---
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Priority: Major
> Fix For: 1.22.0
>
>
> when I test a Fink sql to check overflow situation like
>
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found the ' when (f0 + f1) is null' has been simplified by Calcite, and
> the simplification may be incorrect.
>
> The Calcite's simplification logic of isNull expression in SQL will convert
> from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL.
> And this issue come from Flink （FLINK-14030）.
>
> [~danny0405]， Could you have a look at this?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated CALCITE-3368:
Description:
when I test a Fink sql to check overflow situation like
{code:java}
select
case when (f0 + f1) is null then 'null' else 'not null' end
from testTable
{code}
, I found the ' when (f0 + f1) is null' has been simplified by Calcite, and
the simplification may be incorrect.
The Calcite's simplification logic of isNull expression in SQL will convert
from
*"f(operand0, operand1) IS NULL"* to
*"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
is ANY。
This simplification leads to the expression will not calculate the real
value of *f(operand0, operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
{code:java}
//org.apache.calcite.rex.RexSimplify.java
private RexNode simplifyIsNull(RexNode a) {
// Simplify the argument first,
// call ourselves recursively to see whether we can make more progress.
// For example, given
// "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
// argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
a = simplify(a, UNKNOWN);
if (!a.getType().isNullable() && isSafeExpression(a)) {
return rexBuilder.makeLiteral(false);
}
if (RexUtil.isNull(a)) {
return rexBuilder.makeLiteral(true);
}
if (a.getKind() == SqlKind.CAST) {
return null;
}
switch (Strong.policy(a.getKind())) {
case NOT_NULL:
return rexBuilder.makeLiteral(false);
case ANY:
// "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
// to "operand0 IS NULL OR operand1 IS NULL"
final List operands = new ArrayList<>();
for (RexNode operand : ((RexCall) a).getOperands()) {
final RexNode simplified = simplifyIsNull(operand);
if (simplified == null) {
operands.add(
rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
} else {
operands.add(simplified);
}
}
return RexUtil.composeDisjunction(rexBuilder, operands, false);
case AS_IS:
default:
return null;
}
}{code}
And most of calculating SqlKinds are assigned *Policy.ANY* at present.
{code:java}
//org.apache.calcite.plan.Strong.java
public static Policy policy(SqlKind kind) {
return MAP.getOrDefault(kind, Policy.AS_IS);
}
map.put(SqlKind.PLUS, Policy.ANY);
map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
map.put(SqlKind.MINUS, Policy.ANY);
map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
map.put(SqlKind.TIMES, Policy.ANY);
map.put(SqlKind.DIVIDE, Policy.ANY);
* that operator evaluates to null. */
public enum Policy {
/** This kind of expression is never null. No need to look at its arguments,
* if it has any. */
NOT_NULL,
/** This kind of expression has its own particular rules about whether it
* is null. */
CUSTOM,
/** This kind of expression is null if and only if at least one of its
* arguments is null. */
ANY,
/** This kind of expression may be null. There is no way to rewrite. */
AS_IS,
}{code}
It may be an obvious nonequivalent simplification in SQL.
And this issue come from Flink （FLINK-14030）.
[~danny0405]， Could you have a look at this?
was:
when I test a Fink sql to check overflow situation like
{code:java}
select
case when (f0 + f1) is null then 'null' else 'not null' end
from testTable
{code}
, I found the ' when (f0 + f1) is null' has been simplified by Calcite, and
the simplification may be incorrect.
The Calcite's simplification logic of isNull expression in SQL will convert
from
*"f(operand0, operand1) IS NULL"* to
*"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
is ANY。
This simplification will not calculate the real value of *f(operand0,
operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
{code:java}
//org.apache.calcite.rex.RexSimplify.java
private RexNode simplifyIsNull(RexNode a) {
// Simplify the argument first,
// call ourselves recursively to see whether we can make more progress.
// For example, given
// "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
// argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
a = simplify(a, UNKNOWN);
if (!a.getType().isNullable() && isSafeExpression(a)) {
return rexBuilder.makeLiteral(false);
}
if (RexUtil.isNull(a)) {
return rexBuilder.makeLiteral(true);
}
if (a.getKind() == SqlKind.CAST) {
return null;
}
switch (Strong.policy(a.getKind())) {
case NOT_NULL:
return rexBuilder.makeLiteral(false);
case ANY:
// "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
// to "operand0 IS NULL OR operand1 IS NULL"
final List operands = new ArrayList<>();
for (RexNode operand : ((RexCall) a).getOperands()) {
final RexNode simplified = simplifyIsNull(operand);
if (simplified == null) {
operands.add(
rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
} else {
operands.add(simplified);
}
}
return
[jira] [Commented] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937324#comment-16937324
]
Leonard Xu commented on CALCITE-3368:
-
[~julianhyde] Thank you for your reply，I have updated the description.
> Nonequivalent simplification happens in RexSimplify
> ---
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Priority: Major
> Fix For: 1.22.0
>
>
> when I test a Fink sql to check overflow situation like
>
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found the ' when (f0 + f1) is null' has been simplified by Calcite, and
> the simplification may be incorrect.
>
> The Calcite's simplification logic of isNull expression in SQL will convert
> from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL.
> And this issue come from Flink （FLINK-14030）.
>
> [~danny0405]， Could you have a look at this?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated CALCITE-3368:
Description:
when I test a Fink sql to check overflow situation like
{code:java}
select
case when (f0 + f1) is null then 'null' else 'not null' end
from testTable
{code}
, I found the ' when (f0 + f1) is null' has been simplified by Calcite, and
the simplification may be incorrect.
The Calcite's simplification logic of isNull expression in SQL will convert
from
*"f(operand0, operand1) IS NULL"* to
*"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
is ANY。
This simplification will not calculate the real value of *f(operand0,
operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
{code:java}
//org.apache.calcite.rex.RexSimplify.java
private RexNode simplifyIsNull(RexNode a) {
// Simplify the argument first,
// call ourselves recursively to see whether we can make more progress.
// For example, given
// "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
// argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
a = simplify(a, UNKNOWN);
if (!a.getType().isNullable() && isSafeExpression(a)) {
return rexBuilder.makeLiteral(false);
}
if (RexUtil.isNull(a)) {
return rexBuilder.makeLiteral(true);
}
if (a.getKind() == SqlKind.CAST) {
return null;
}
switch (Strong.policy(a.getKind())) {
case NOT_NULL:
return rexBuilder.makeLiteral(false);
case ANY:
// "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
// to "operand0 IS NULL OR operand1 IS NULL"
final List operands = new ArrayList<>();
for (RexNode operand : ((RexCall) a).getOperands()) {
final RexNode simplified = simplifyIsNull(operand);
if (simplified == null) {
operands.add(
rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
} else {
operands.add(simplified);
}
}
return RexUtil.composeDisjunction(rexBuilder, operands, false);
case AS_IS:
default:
return null;
}
}{code}
And most of calculating SqlKinds are assigned *Policy.ANY* at present.
{code:java}
//org.apache.calcite.plan.Strong.java
public static Policy policy(SqlKind kind) {
return MAP.getOrDefault(kind, Policy.AS_IS);
}
map.put(SqlKind.PLUS, Policy.ANY);
map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
map.put(SqlKind.MINUS, Policy.ANY);
map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
map.put(SqlKind.TIMES, Policy.ANY);
map.put(SqlKind.DIVIDE, Policy.ANY);
* that operator evaluates to null. */
public enum Policy {
/** This kind of expression is never null. No need to look at its arguments,
* if it has any. */
NOT_NULL,
/** This kind of expression has its own particular rules about whether it
* is null. */
CUSTOM,
/** This kind of expression is null if and only if at least one of its
* arguments is null. */
ANY,
/** This kind of expression may be null. There is no way to rewrite. */
AS_IS,
}{code}
It may be an obvious nonequivalent simplification in SQL.
And this issue come from Flink （FLINK-14030）.
[~danny0405]， Could you have a look at this?
was:
I found that Calcite's simplification logic of isNull expression will convert
from
*"f(operand0, operand1) IS NULL"* to
*"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
is ANY
When I do some overflow tests 。
This simplification will not calculate the real value of *f(operand0,
operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
{code:java}
//org.apache.calcite.rex.RexSimplify.java
private RexNode simplifyIsNull(RexNode a) {
// Simplify the argument first,
// call ourselves recursively to see whether we can make more progress.
// For example, given
// "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
// argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
a = simplify(a, UNKNOWN);
if (!a.getType().isNullable() && isSafeExpression(a)) {
return rexBuilder.makeLiteral(false);
}
if (RexUtil.isNull(a)) {
return rexBuilder.makeLiteral(true);
}
if (a.getKind() == SqlKind.CAST) {
return null;
}
switch (Strong.policy(a.getKind())) {
case NOT_NULL:
return rexBuilder.makeLiteral(false);
case ANY:
// "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
// to "operand0 IS NULL OR operand1 IS NULL"
final List operands = new ArrayList<>();
for (RexNode operand : ((RexCall) a).getOperands()) {
final RexNode simplified = simplifyIsNull(operand);
if (simplified == null) {
operands.add(
rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
} else {
operands.add(simplified);
}
}
return RexUtil.composeDisjunction(rexBuilder, operands, false);
case AS_IS:
default:
return null;
}
}{code}
And most of calculating SqlKinds are assigned *Policy.ANY* at present.
{code:java}
//org.apache.calcite.plan.Strong.java
public static Policy policy(SqlKind kind) {
return
[jira] [Updated] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated CALCITE-3368:
Description:
I found that Calcite's simplification logic of isNull expression will convert
from
*"f(operand0, operand1) IS NULL"* to
*"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
is ANY
When I do some overflow tests 。
This simplification will not calculate the real value of *f(operand0,
operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
{code:java}
//org.apache.calcite.rex.RexSimplify.java
private RexNode simplifyIsNull(RexNode a) {
// Simplify the argument first,
// call ourselves recursively to see whether we can make more progress.
// For example, given
// "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
// argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
a = simplify(a, UNKNOWN);
if (!a.getType().isNullable() && isSafeExpression(a)) {
return rexBuilder.makeLiteral(false);
}
if (RexUtil.isNull(a)) {
return rexBuilder.makeLiteral(true);
}
if (a.getKind() == SqlKind.CAST) {
return null;
}
switch (Strong.policy(a.getKind())) {
case NOT_NULL:
return rexBuilder.makeLiteral(false);
case ANY:
// "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
// to "operand0 IS NULL OR operand1 IS NULL"
final List operands = new ArrayList<>();
for (RexNode operand : ((RexCall) a).getOperands()) {
final RexNode simplified = simplifyIsNull(operand);
if (simplified == null) {
operands.add(
rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
} else {
operands.add(simplified);
}
}
return RexUtil.composeDisjunction(rexBuilder, operands, false);
case AS_IS:
default:
return null;
}
}{code}
And most of calculating SqlKinds are assigned *Policy.ANY* at present.
{code:java}
//org.apache.calcite.plan.Strong.java
public static Policy policy(SqlKind kind) {
return MAP.getOrDefault(kind, Policy.AS_IS);
}
map.put(SqlKind.PLUS, Policy.ANY);
map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
map.put(SqlKind.MINUS, Policy.ANY);
map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
map.put(SqlKind.TIMES, Policy.ANY);
map.put(SqlKind.DIVIDE, Policy.ANY);
* that operator evaluates to null. */
public enum Policy {
/** This kind of expression is never null. No need to look at its arguments,
* if it has any. */
NOT_NULL,
/** This kind of expression has its own particular rules about whether it
* is null. */
CUSTOM,
/** This kind of expression is null if and only if at least one of its
* arguments is null. */
ANY,
/** This kind of expression may be null. There is no way to rewrite. */
AS_IS,
}{code}
It should be an obvious nonequivalent simplification.
And this issue come from Flink （FLINK-14030）.
[~danny0405]， Could you have a look at this?
was:
I found that Calcite's simplification logic of isNull expression will convert
from
*"f(operand0, operand1) IS NULL"* to
*"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
is ANY
When I do some overflow tests 。
This simplification will not calculate the real vale of *f(operand0,
operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
{code:java}
//org.apache.calcite.rex.RexSimplify.java
private RexNode simplifyIsNull(RexNode a) {
// Simplify the argument first,
// call ourselves recursively to see whether we can make more progress.
// For example, given
// "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
// argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
a = simplify(a, UNKNOWN);
if (!a.getType().isNullable() && isSafeExpression(a)) {
return rexBuilder.makeLiteral(false);
}
if (RexUtil.isNull(a)) {
return rexBuilder.makeLiteral(true);
}
if (a.getKind() == SqlKind.CAST) {
return null;
}
switch (Strong.policy(a.getKind())) {
case NOT_NULL:
return rexBuilder.makeLiteral(false);
case ANY:
// "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
// to "operand0 IS NULL OR operand1 IS NULL"
final List operands = new ArrayList<>();
for (RexNode operand : ((RexCall) a).getOperands()) {
final RexNode simplified = simplifyIsNull(operand);
if (simplified == null) {
operands.add(
rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
} else {
operands.add(simplified);
}
}
return RexUtil.composeDisjunction(rexBuilder, operands, false);
case AS_IS:
default:
return null;
}
}{code}
And most of calculating SqlKinds are assigned *Policy.ANY* at present.
{code:java}
//org.apache.calcite.plan.Strong.java
public static Policy policy(SqlKind kind) {
return MAP.getOrDefault(kind, Policy.AS_IS);
}
map.put(SqlKind.PLUS, Policy.ANY);
map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
map.put(SqlKind.MINUS, Policy.ANY);
map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
map.put(SqlKind.TIMES, Policy.ANY);
map.put(SqlKind.DIVIDE,
[jira] [Commented] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937206#comment-16937206
]
Julian Hyde commented on CALCITE-3368:
--
Please change the description so that it makes sense to someone who knows only
SQL, not Calcite's internals.
> Nonequivalent simplification happens in RexSimplify
> ---
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Priority: Major
> Fix For: 1.22.0
>
>
> I found that Calcite's simplification logic of isNull expression will convert
> from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY
> When I do some overflow tests 。
> This simplification will not calculate the real vale of *f(operand0,
> operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It should be an obvious nonequivalent simplification.
> And this issue come from Flink （FLINK-14030）.
>
> [~danny0405]， Could you have a look at this?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3370) Emulate NULL direction for MSSQL
Justin Swett created CALCITE-3370: - Summary: Emulate NULL direction for MSSQL Key: CALCITE-3370 URL: https://issues.apache.org/jira/browse/CALCITE-3370 Project: Calcite Issue Type: Task Reporter: Justin Swett MSSQL cannot handle NULLS LAST/FIRST and should emulate the NULLS LAST/FIRST in order by, i.e. --for nulls last select [column] from [table] ORDER BY CASE WHEN [column] IS NULL THEN 1 ELSE 0 END, [column]" --for nulls first select [column] from [table] ORDER BY CASE WHEN [column] IS NULL THEN 0 ELSE 1 END, [column]" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CALCITE-3369) In LatticeSuggester, recommend lattices based on UNION queries
[
https://issues.apache.org/jira/browse/CALCITE-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde reassigned CALCITE-3369:
Assignee: Julian Hyde
> In LatticeSuggester, recommend lattices based on UNION queries
> --
>
> Key: CALCITE-3369
> URL: https://issues.apache.org/jira/browse/CALCITE-3369
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Julian Hyde
>Priority: Major
>
> In LatticeSuggester, recommend lattices based on UNION, EXCEPT and INTERSECT
> queries. Currently such queries are ignored.
> Given the query
> {code:java}
> select * from t1 join t2
> union
> select * from t2 join t3;{code}
> suggester should generate the same lattice(s) as if it had been given two
> separate queries
> {code:java}
> select * from t1 join t2;
> select * from t2 join t3; {code}
> Which may be a single lattice t1 - t2 - t3, or might be two lattices t1 - t2,
> t2 - t3.
> Same for EXCEPT (MINUS), INTERSECT, UNION ALL, etc.
> If the set-op is internal, I'm not sure what to do, e.g.
> {code:java}
> select *
> from sales
> join (select * from good_product
> union
> select * from bad_product) using (product_id){code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3369) In LatticeSuggester, recommend lattices based on UNION queries
[
https://issues.apache.org/jira/browse/CALCITE-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde updated CALCITE-3369:
-
Fix Version/s: 1.22.0
> In LatticeSuggester, recommend lattices based on UNION queries
> --
>
> Key: CALCITE-3369
> URL: https://issues.apache.org/jira/browse/CALCITE-3369
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.22.0
>
>
> In LatticeSuggester, recommend lattices based on UNION, EXCEPT and INTERSECT
> queries. Currently such queries are ignored.
> Given the query
> {code:java}
> select * from t1 join t2
> union
> select * from t2 join t3;{code}
> suggester should generate the same lattice(s) as if it had been given two
> separate queries
> {code:java}
> select * from t1 join t2;
> select * from t2 join t3; {code}
> Which may be a single lattice t1 - t2 - t3, or might be two lattices t1 - t2,
> t2 - t3.
> Same for EXCEPT (MINUS), INTERSECT, UNION ALL, etc.
> If the set-op is internal, I'm not sure what to do, e.g.
> {code:java}
> select *
> from sales
> join (select * from good_product
> union
> select * from bad_product) using (product_id){code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3369) In LatticeSuggester, recommend lattices based on UNION queries
Julian Hyde created CALCITE-3369:
Summary: In LatticeSuggester, recommend lattices based on UNION
queries
Key: CALCITE-3369
URL: https://issues.apache.org/jira/browse/CALCITE-3369
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In LatticeSuggester, recommend lattices based on UNION, EXCEPT and INTERSECT
queries. Currently such queries are ignored.
Given the query
{code:java}
select * from t1 join t2
union
select * from t2 join t3;{code}
suggester should generate the same lattice(s) as if it had been given two
separate queries
{code:java}
select * from t1 join t2;
select * from t2 join t3; {code}
Which may be a single lattice t1 - t2 - t3, or might be two lattices t1 - t2,
t2 - t3.
Same for EXCEPT (MINUS), INTERSECT, UNION ALL, etc.
If the set-op is internal, I'm not sure what to do, e.g.
{code:java}
select *
from sales
join (select * from good_product
union
select * from bad_product) using (product_id){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3365) Don't require use of JdbcSchema in QuerySqlStatisticProvider
[ https://issues.apache.org/jira/browse/CALCITE-3365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde resolved CALCITE-3365. -- Resolution: Fixed Fixed in [3005ff97|https://github.com/apache/calcite/commit/3005ff97ddb6173ac339b45affb0b4da5d406345]. Thanks [~lindseycat]! > Don't require use of JdbcSchema in QuerySqlStatisticProvider > > > Key: CALCITE-3365 > URL: https://issues.apache.org/jira/browse/CALCITE-3365 > Project: Calcite > Issue Type: Improvement >Reporter: Lindsey Meyer >Assignee: Julian Hyde >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We're trying to pass in schema data manually to process queries for BigQuery, > but that causes problems in the QuerySqlStatisticProvider because it requires > a JdbcSchema in order to get the dialect and dataSource, as well as using a > Jdbc Builder. > I propose refactoring QuerySqlStatisticProvider to grab the dialect and > dataSource using unwrap on the schema instead, so that any schema that > implements unwrap methods for these types will work with > QuerySqlStatisticProvider. As well as using a generic RelBuilder, instead of > a JDBC_BUILDER. > cc [~julianhyde] since we have been discussing this > I've opened a PR at https://github.com/apache/calcite/pull/1467 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3340) Make TUMBLE be accepted as an operand for "FROM TABLE()" in validator
[
https://issues.apache.org/jira/browse/CALCITE-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937076#comment-16937076
]
Rui Wang commented on CALCITE-3340:
---
I see.
OK, so I will try more ideas to see if I can keep both operators of TUMBLE. The
new idea of making TUMBLE created in parser worth to try.
If it turns out that we cannot keep both operators with the same name, at least
I have proven by PR that having a different name for new TUMBLE works (maybe
just TUMBLE_table, or TUMBLE_table_function).
> Make TUMBLE be accepted as an operand for "FROM TABLE()" in validator
> -
>
> Key: CALCITE-3340
> URL: https://issues.apache.org/jira/browse/CALCITE-3340
> Project: Calcite
> Issue Type: Sub-task
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The goal of this JIRA is to generate a logical plan for the following query:
> {code:java}
> SELECT *
> FROM TABLE(TUMBLE(
> TABLE ORDERS,
> INTERVAL '10' MINUTE))
> {code}
> This SQL query does not have DESCRIPTOR included, which is being tracked and
> discussed by CALCITE-3339.
> I expect we should generate a logical plan from this query that is similar to
> the following:
> {code:java}
> LogicalProject(ROWTIME=[$0], ID=[$1], PRODUCT=[$2], UNITS=[$3], wstart=[$4],
> wend=[$5])
> LogicalTableFunctionScan(invocation=[TUMBLE($3, 6:INTERVAL MINUTE)],
> rowType=[RecordType(TIMESTAMP(0) ROWTIME, INTEGER ID, VARCHAR(10) PRODUCT,
> INTEGER UNITS, TIMESTAMP(0) wstart, TIMESTAMP(0) wend)])
> LogicalProject(ROWTIME=[$0], ID=[$1], PRODUCT=[$2], UNITS=[$3])
> LogicalTableScan(table=[[ORINOCO, ORDERS]])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3365) Don't require use of JdbcSchema in QuerySqlStatisticProvider
[ https://issues.apache.org/jira/browse/CALCITE-3365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937068#comment-16937068 ] Lindsey Meyer commented on CALCITE-3365: Looks good to me, thank you [~julianhyde]!! > Don't require use of JdbcSchema in QuerySqlStatisticProvider > > > Key: CALCITE-3365 > URL: https://issues.apache.org/jira/browse/CALCITE-3365 > Project: Calcite > Issue Type: Improvement >Reporter: Lindsey Meyer >Assignee: Julian Hyde >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 10m > Remaining Estimate: 0h > > We're trying to pass in schema data manually to process queries for BigQuery, > but that causes problems in the QuerySqlStatisticProvider because it requires > a JdbcSchema in order to get the dialect and dataSource, as well as using a > Jdbc Builder. > I propose refactoring QuerySqlStatisticProvider to grab the dialect and > dataSource using unwrap on the schema instead, so that any schema that > implements unwrap methods for these types will work with > QuerySqlStatisticProvider. As well as using a generic RelBuilder, instead of > a JDBC_BUILDER. > cc [~julianhyde] since we have been discussing this > I've opened a PR at https://github.com/apache/calcite/pull/1467 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3340) Make TUMBLE be accepted as an operand for "FROM TABLE()" in validator
[
https://issues.apache.org/jira/browse/CALCITE-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937060#comment-16937060
]
Julian Hyde commented on CALCITE-3340:
--
I really don't know.
> Make TUMBLE be accepted as an operand for "FROM TABLE()" in validator
> -
>
> Key: CALCITE-3340
> URL: https://issues.apache.org/jira/browse/CALCITE-3340
> Project: Calcite
> Issue Type: Sub-task
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The goal of this JIRA is to generate a logical plan for the following query:
> {code:java}
> SELECT *
> FROM TABLE(TUMBLE(
> TABLE ORDERS,
> INTERVAL '10' MINUTE))
> {code}
> This SQL query does not have DESCRIPTOR included, which is being tracked and
> discussed by CALCITE-3339.
> I expect we should generate a logical plan from this query that is similar to
> the following:
> {code:java}
> LogicalProject(ROWTIME=[$0], ID=[$1], PRODUCT=[$2], UNITS=[$3], wstart=[$4],
> wend=[$5])
> LogicalTableFunctionScan(invocation=[TUMBLE($3, 6:INTERVAL MINUTE)],
> rowType=[RecordType(TIMESTAMP(0) ROWTIME, INTEGER ID, VARCHAR(10) PRODUCT,
> INTEGER UNITS, TIMESTAMP(0) wstart, TIMESTAMP(0) wend)])
> LogicalProject(ROWTIME=[$0], ID=[$1], PRODUCT=[$2], UNITS=[$3])
> LogicalTableScan(table=[[ORINOCO, ORDERS]])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-3340) Make TUMBLE be accepted as an operand for "FROM TABLE()" in validator
[
https://issues.apache.org/jira/browse/CALCITE-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937044#comment-16937044
]
Rui Wang edited comment on CALCITE-3340 at 9/24/19 5:49 PM:
I can see many of the functions are coded into parser (e.g. SUBSTRING, TRIM,
etc.), and they are not looked up by name.
However, based on my test and my read from code, both old TUMBLE(the group
function) and new TUMBLE(table function) don't follow that, thus these two
operators are looked up by name(and that's why same "TUMBLE" as operator name
leads to operator overloading and I always observed ). Either same SqlKind
(e.g. SqlKind.TUMBLE) or different SqlKind(e.g. assign OTHER_FUNCTION to new
TUMBLE operator) does not change the lookup name. Also, "TUMBLE" is not a
keyword in parser.jj
So do you suggest I code both TUMBLE into parser, like other built-in
operators? I can give a try on this idea.
was (Author: amaliujia):
I can see many of the functions are coded into parser (e.g. SUBSTRING, TRIM,
etc.), and they are not looked up by name.
However, based on my test and my read from code, both old TUMBLE(the group
function) and new TUMBLE(table function) don't follow that, thus these two
operators are looked up by name(and that's why same "TUMBLE" as operator name
leads to operator overloading and I always observed ). Either same SqlKind
(e.g. SqlKind.TUMBLE) or different SqlKind(e.g. assign OTHER_FUNCTION to new
TUMBLE operator) does not change the lookup name.
So do you suggest I code both TUMBLE into parser, like other built-in
operators? I can give a try on this idea.
> Make TUMBLE be accepted as an operand for "FROM TABLE()" in validator
> -
>
> Key: CALCITE-3340
> URL: https://issues.apache.org/jira/browse/CALCITE-3340
> Project: Calcite
> Issue Type: Sub-task
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The goal of this JIRA is to generate a logical plan for the following query:
> {code:java}
> SELECT *
> FROM TABLE(TUMBLE(
> TABLE ORDERS,
> INTERVAL '10' MINUTE))
> {code}
> This SQL query does not have DESCRIPTOR included, which is being tracked and
> discussed by CALCITE-3339.
> I expect we should generate a logical plan from this query that is similar to
> the following:
> {code:java}
> LogicalProject(ROWTIME=[$0], ID=[$1], PRODUCT=[$2], UNITS=[$3], wstart=[$4],
> wend=[$5])
> LogicalTableFunctionScan(invocation=[TUMBLE($3, 6:INTERVAL MINUTE)],
> rowType=[RecordType(TIMESTAMP(0) ROWTIME, INTEGER ID, VARCHAR(10) PRODUCT,
> INTEGER UNITS, TIMESTAMP(0) wstart, TIMESTAMP(0) wend)])
> LogicalProject(ROWTIME=[$0], ID=[$1], PRODUCT=[$2], UNITS=[$3])
> LogicalTableScan(table=[[ORINOCO, ORDERS]])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-3340) Make TUMBLE be accepted as an operand for "FROM TABLE()" in validator
[
https://issues.apache.org/jira/browse/CALCITE-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937044#comment-16937044
]
Rui Wang edited comment on CALCITE-3340 at 9/24/19 5:45 PM:
I can see many of the functions are coded into parser (e.g. SUBSTRING, TRIM,
etc.), and they are not looked up by name.
However, based on my test and my read from code, both old TUMBLE(the group
function) and new TUMBLE(table function) don't follow that, thus these two
operators are looked up by name(and that's why same "TUMBLE" as operator name
leads to operator overloading and I always observed ). Either same SqlKind
(e.g. SqlKind.TUMBLE) or different SqlKind(e.g. assign OTHER_FUNCTION to new
TUMBLE operator) does not change the lookup name.
So do you suggest I code both TUMBLE into parser, like other built-in
operators? I can give a try on this idea.
was (Author: amaliujia):
I can see many of the functions are coded into parser (e.g. SUBSTRING, TRIM,
etc.), and they are not looked up by name.
However, based on my test and my read from code, both old TUMBLE(the group
function) and new TUMBLE(table function) don't follow that, thus these two
operators are looked up by name(and that's why same "TUMBLE" as operator name
leads to operator overloading and I always observed ). Either same SqlKind
(e.g. SqlKind.TUMBLE) or different SqlKind(e.g. assign OTHER_FUNCTION to new
TUMBLE operator) does not change the lookup name.
So do you suggest I code both TUMBLE into parser, like other built-in operators?
> Make TUMBLE be accepted as an operand for "FROM TABLE()" in validator
> -
>
> Key: CALCITE-3340
> URL: https://issues.apache.org/jira/browse/CALCITE-3340
> Project: Calcite
> Issue Type: Sub-task
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The goal of this JIRA is to generate a logical plan for the following query:
> {code:java}
> SELECT *
> FROM TABLE(TUMBLE(
> TABLE ORDERS,
> INTERVAL '10' MINUTE))
> {code}
> This SQL query does not have DESCRIPTOR included, which is being tracked and
> discussed by CALCITE-3339.
> I expect we should generate a logical plan from this query that is similar to
> the following:
> {code:java}
> LogicalProject(ROWTIME=[$0], ID=[$1], PRODUCT=[$2], UNITS=[$3], wstart=[$4],
> wend=[$5])
> LogicalTableFunctionScan(invocation=[TUMBLE($3, 6:INTERVAL MINUTE)],
> rowType=[RecordType(TIMESTAMP(0) ROWTIME, INTEGER ID, VARCHAR(10) PRODUCT,
> INTEGER UNITS, TIMESTAMP(0) wstart, TIMESTAMP(0) wend)])
> LogicalProject(ROWTIME=[$0], ID=[$1], PRODUCT=[$2], UNITS=[$3])
> LogicalTableScan(table=[[ORINOCO, ORDERS]])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3340) Make TUMBLE be accepted as an operand for "FROM TABLE()" in validator
[
https://issues.apache.org/jira/browse/CALCITE-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937044#comment-16937044
]
Rui Wang commented on CALCITE-3340:
---
I can see many of the functions are coded into parser (e.g. SUBSTRING, TRIM,
etc.), and they are not looked up by name.
However, based on my test and my read from code, both old TUMBLE(the group
function) and new TUMBLE(table function) don't follow that, thus these two
operators are looked up by name(and that's why same "TUMBLE" as operator name
leads to operator overloading and I always observed ). Either same SqlKind
(e.g. SqlKind.TUMBLE) or different SqlKind(e.g. assign OTHER_FUNCTION to new
TUMBLE operator) does not change the lookup name.
So do you suggest I code both TUMBLE into parser, like other built-in operators?
> Make TUMBLE be accepted as an operand for "FROM TABLE()" in validator
> -
>
> Key: CALCITE-3340
> URL: https://issues.apache.org/jira/browse/CALCITE-3340
> Project: Calcite
> Issue Type: Sub-task
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The goal of this JIRA is to generate a logical plan for the following query:
> {code:java}
> SELECT *
> FROM TABLE(TUMBLE(
> TABLE ORDERS,
> INTERVAL '10' MINUTE))
> {code}
> This SQL query does not have DESCRIPTOR included, which is being tracked and
> discussed by CALCITE-3339.
> I expect we should generate a logical plan from this query that is similar to
> the following:
> {code:java}
> LogicalProject(ROWTIME=[$0], ID=[$1], PRODUCT=[$2], UNITS=[$3], wstart=[$4],
> wend=[$5])
> LogicalTableFunctionScan(invocation=[TUMBLE($3, 6:INTERVAL MINUTE)],
> rowType=[RecordType(TIMESTAMP(0) ROWTIME, INTEGER ID, VARCHAR(10) PRODUCT,
> INTEGER UNITS, TIMESTAMP(0) wstart, TIMESTAMP(0) wend)])
> LogicalProject(ROWTIME=[$0], ID=[$1], PRODUCT=[$2], UNITS=[$3])
> LogicalTableScan(table=[[ORINOCO, ORDERS]])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated CALCITE-3368:
Description:
I found that Calcite's simplification logic of isNull expression will convert
from
*"f(operand0, operand1) IS NULL"* to
*"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
is ANY
When I do some overflow tests 。
This simplification will not calculate the real vale of *f(operand0,
operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
{code:java}
//org.apache.calcite.rex.RexSimplify.java
private RexNode simplifyIsNull(RexNode a) {
// Simplify the argument first,
// call ourselves recursively to see whether we can make more progress.
// For example, given
// "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
// argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
a = simplify(a, UNKNOWN);
if (!a.getType().isNullable() && isSafeExpression(a)) {
return rexBuilder.makeLiteral(false);
}
if (RexUtil.isNull(a)) {
return rexBuilder.makeLiteral(true);
}
if (a.getKind() == SqlKind.CAST) {
return null;
}
switch (Strong.policy(a.getKind())) {
case NOT_NULL:
return rexBuilder.makeLiteral(false);
case ANY:
// "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
// to "operand0 IS NULL OR operand1 IS NULL"
final List operands = new ArrayList<>();
for (RexNode operand : ((RexCall) a).getOperands()) {
final RexNode simplified = simplifyIsNull(operand);
if (simplified == null) {
operands.add(
rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
} else {
operands.add(simplified);
}
}
return RexUtil.composeDisjunction(rexBuilder, operands, false);
case AS_IS:
default:
return null;
}
}{code}
And most of calculating SqlKinds are assigned *Policy.ANY* at present.
{code:java}
//org.apache.calcite.plan.Strong.java
public static Policy policy(SqlKind kind) {
return MAP.getOrDefault(kind, Policy.AS_IS);
}
map.put(SqlKind.PLUS, Policy.ANY);
map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
map.put(SqlKind.MINUS, Policy.ANY);
map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
map.put(SqlKind.TIMES, Policy.ANY);
map.put(SqlKind.DIVIDE, Policy.ANY);
* that operator evaluates to null. */
public enum Policy {
/** This kind of expression is never null. No need to look at its arguments,
* if it has any. */
NOT_NULL,
/** This kind of expression has its own particular rules about whether it
* is null. */
CUSTOM,
/** This kind of expression is null if and only if at least one of its
* arguments is null. */
ANY,
/** This kind of expression may be null. There is no way to rewrite. */
AS_IS,
}{code}
It should be an obvious nonequivalent simplification.
And this issue come from Flink （FLINK-14030）.
[~danny0405]， Could you have a look at this?
was:
When I do some overflow test ，I found that Calcite's simplification logic of
isNull expression will convert from
*"f(operand0, operand1) IS NULL"* to
*"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
is ANY 。
This simplification will not calculate the real vale of *f(operand0,
operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
{code:java}
//org.apache.calcite.rex.RexSimplify.java
private RexNode simplifyIsNull(RexNode a) {
// Simplify the argument first,
// call ourselves recursively to see whether we can make more progress.
// For example, given
// "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
// argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
a = simplify(a, UNKNOWN);
if (!a.getType().isNullable() && isSafeExpression(a)) {
return rexBuilder.makeLiteral(false);
}
if (RexUtil.isNull(a)) {
return rexBuilder.makeLiteral(true);
}
if (a.getKind() == SqlKind.CAST) {
return null;
}
switch (Strong.policy(a.getKind())) {
case NOT_NULL:
return rexBuilder.makeLiteral(false);
case ANY:
// "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
// to "operand0 IS NULL OR operand1 IS NULL"
final List operands = new ArrayList<>();
for (RexNode operand : ((RexCall) a).getOperands()) {
final RexNode simplified = simplifyIsNull(operand);
if (simplified == null) {
operands.add(
rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
} else {
operands.add(simplified);
}
}
return RexUtil.composeDisjunction(rexBuilder, operands, false);
case AS_IS:
default:
return null;
}
}{code}
And most of calculating SqlKinds are assigned *Policy.ANY* at present.
{code:java}
//org.apache.calcite.plan.Strong.java
public static Policy policy(SqlKind kind) {
return MAP.getOrDefault(kind, Policy.AS_IS);
}
map.put(SqlKind.PLUS, Policy.ANY);
map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
map.put(SqlKind.MINUS, Policy.ANY);
map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
map.put(SqlKind.TIMES, Policy.ANY);
map.put(SqlKind.DIVIDE,
[jira] [Created] (CALCITE-3368) Nonequivalent simplification happens in RexSimplify
Leonard Xu created CALCITE-3368:
---
Summary: Nonequivalent simplification happens in RexSimplify
Key: CALCITE-3368
URL: https://issues.apache.org/jira/browse/CALCITE-3368
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.21.0
Reporter: Leonard Xu
Fix For: 1.22.0
When I do some overflow test ，I found that Calcite's simplification logic of
isNull expression will convert from
*"f(operand0, operand1) IS NULL"* to
*"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
is ANY 。
This simplification will not calculate the real vale of *f(operand0,
operand1)* (eg..'f0 + 'f1 ）which maybe overflow.
{code:java}
//org.apache.calcite.rex.RexSimplify.java
private RexNode simplifyIsNull(RexNode a) {
// Simplify the argument first,
// call ourselves recursively to see whether we can make more progress.
// For example, given
// "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
// argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
a = simplify(a, UNKNOWN);
if (!a.getType().isNullable() && isSafeExpression(a)) {
return rexBuilder.makeLiteral(false);
}
if (RexUtil.isNull(a)) {
return rexBuilder.makeLiteral(true);
}
if (a.getKind() == SqlKind.CAST) {
return null;
}
switch (Strong.policy(a.getKind())) {
case NOT_NULL:
return rexBuilder.makeLiteral(false);
case ANY:
// "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
// to "operand0 IS NULL OR operand1 IS NULL"
final List operands = new ArrayList<>();
for (RexNode operand : ((RexCall) a).getOperands()) {
final RexNode simplified = simplifyIsNull(operand);
if (simplified == null) {
operands.add(
rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
} else {
operands.add(simplified);
}
}
return RexUtil.composeDisjunction(rexBuilder, operands, false);
case AS_IS:
default:
return null;
}
}{code}
And most of calculating SqlKinds are assigned *Policy.ANY* at present.
{code:java}
//org.apache.calcite.plan.Strong.java
public static Policy policy(SqlKind kind) {
return MAP.getOrDefault(kind, Policy.AS_IS);
}
map.put(SqlKind.PLUS, Policy.ANY);
map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
map.put(SqlKind.MINUS, Policy.ANY);
map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
map.put(SqlKind.TIMES, Policy.ANY);
map.put(SqlKind.DIVIDE, Policy.ANY);
* that operator evaluates to null. */
public enum Policy {
/** This kind of expression is never null. No need to look at its arguments,
* if it has any. */
NOT_NULL,
/** This kind of expression has its own particular rules about whether it
* is null. */
CUSTOM,
/** This kind of expression is null if and only if at least one of its
* arguments is null. */
ANY,
/** This kind of expression may be null. There is no way to rewrite. */
AS_IS,
}{code}
It should be an obvious nonequivalent simplification.
And this issue come from Flink
（[FLINK-14030|https://issues.apache.org/jira/browse/FLINK-14030]）.
[~danny0405] could you have a look at this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3159) Distinct can be removed for MIN/MAX/BIT_OR/BIT_AND aggregate functions
[
https://issues.apache.org/jira/browse/CALCITE-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chan resolved CALCITE-3159.
-
Assignee: Danny Chan (was: Julian Hyde)
Resolution: Fixed
Fixed in
[3abb974|https://github.com/apache/calcite/commit/3abb974588194e2e531aa43daa79c4dfd18ef25f],
thanks for your PR, [~x1q1j1] !
> Distinct can be removed for MIN/MAX/BIT_OR/BIT_AND aggregate functions
> --
>
> Key: CALCITE-3159
> URL: https://issues.apache.org/jira/browse/CALCITE-3159
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Assignee: Danny Chan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> For the following query:
> {code:java}
> select a, min(distinct b), bit_or(distinct c) from foo group by a;
> {code}
> Currently Calcite still preserve the distinct for these aggregate functions,
> but DISTINCT is not meaningful with MIN/MAX and is available for ISO
> compatibility only. We can safely remove distinct and get more optimization
> opportunities.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3159) Distinct can be removed for MIN/MAX/BIT_OR/BIT_AND aggregate functions
[
https://issues.apache.org/jira/browse/CALCITE-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chan updated CALCITE-3159:
Fix Version/s: 1.21.0
> Distinct can be removed for MIN/MAX/BIT_OR/BIT_AND aggregate functions
> --
>
> Key: CALCITE-3159
> URL: https://issues.apache.org/jira/browse/CALCITE-3159
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Assignee: Danny Chan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> For the following query:
> {code:java}
> select a, min(distinct b), bit_or(distinct c) from foo group by a;
> {code}
> Currently Calcite still preserve the distinct for these aggregate functions,
> but DISTINCT is not meaningful with MIN/MAX and is available for ISO
> compatibility only. We can safely remove distinct and get more optimization
> opportunities.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3159) Distinct can be removed for MIN/MAX/BIT_OR/BIT_AND aggregate functions
[
https://issues.apache.org/jira/browse/CALCITE-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chan updated CALCITE-3159:
Affects Version/s: 1.20.0
> Distinct can be removed for MIN/MAX/BIT_OR/BIT_AND aggregate functions
> --
>
> Key: CALCITE-3159
> URL: https://issues.apache.org/jira/browse/CALCITE-3159
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.20.0
>Reporter: Haisheng Yuan
>Assignee: Danny Chan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> For the following query:
> {code:java}
> select a, min(distinct b), bit_or(distinct c) from foo group by a;
> {code}
> Currently Calcite still preserve the distinct for these aggregate functions,
> but DISTINCT is not meaningful with MIN/MAX and is available for ISO
> compatibility only. We can safely remove distinct and get more optimization
> opportunities.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3159) Distinct can be removed for MIN/MAX/BIT_OR/BIT_AND aggregate functions
[
https://issues.apache.org/jira/browse/CALCITE-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936517#comment-16936517
]
Kurt Young commented on CALCITE-3159:
-
Seems this issue had been fixed in 1.21.0, could you please update the status?
> Distinct can be removed for MIN/MAX/BIT_OR/BIT_AND aggregate functions
> --
>
> Key: CALCITE-3159
> URL: https://issues.apache.org/jira/browse/CALCITE-3159
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Assignee: Julian Hyde
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> For the following query:
> {code:java}
> select a, min(distinct b), bit_or(distinct c) from foo group by a;
> {code}
> Currently Calcite still preserve the distinct for these aggregate functions,
> but DISTINCT is not meaningful with MIN/MAX and is available for ISO
> compatibility only. We can safely remove distinct and get more optimization
> opportunities.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3239) Calc#accept(RexShuttle shuttle) does not update rowType
[
https://issues.apache.org/jira/browse/CALCITE-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-3239.
Fix Version/s: 1.22.0
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/be01394210102ea83dc83f95ce5a59c60525b8d3.
> Calc#accept(RexShuttle shuttle) does not update rowType
> ---
>
> Key: CALCITE-3239
> URL: https://issues.apache.org/jira/browse/CALCITE-3239
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> In current code, {{Calc#accept(RexShuttle shuttle)}} keep the original
> {{rowType}}, even when the expression list is modified by the shuttle.
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Calc.java#L175
>
> It might be wrong and a new {{rowType}} should be derived.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3367) Add AntiJoinRule to convert project-filter-join-aggregate into anti-join
[ https://issues.apache.org/jira/browse/CALCITE-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated CALCITE-3367: -- Description: This Jira proposes to add AntiJoinRule to convert project-filter-join-aggregate into anti-join. The idea is from SemiJoinRule. This issue was found when I resolve CALCITE-3363 in [https://github.com/apache/calcite/pull/1466] I failed to construct an anti-join operator from SQL string. was: This Jira proposes to add AntiJoinRule to convert project-filter-join-aggregate into anti-join This issue was found when I resolve CALCITE-3363 in https://github.com/apache/calcite/pull/1466 I failed to construct an anti-join operator from SQL string. > Add AntiJoinRule to convert project-filter-join-aggregate into anti-join > > > Key: CALCITE-3367 > URL: https://issues.apache.org/jira/browse/CALCITE-3367 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: jin xing >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira proposes to add AntiJoinRule to convert > project-filter-join-aggregate into anti-join. > The idea is from SemiJoinRule. > This issue was found when I resolve CALCITE-3363 in > [https://github.com/apache/calcite/pull/1466] > I failed to construct an anti-join operator from SQL string. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3367) Add AntiJoinRule to convert project-filter-join-aggregate into anti-join
[ https://issues.apache.org/jira/browse/CALCITE-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated CALCITE-3367: -- Description: This Jira proposes to add AntiJoinRule to convert project-filter-join-aggregate into anti-join This issue was found when I resolve CALCITE-3363 in https://github.com/apache/calcite/pull/1466 I failed to construct an anti-join operator from SQL string. was:This Jira proposes to add AntiJoinRule to convert project-filter-join-aggregate into anti-join > Add AntiJoinRule to convert project-filter-join-aggregate into anti-join > > > Key: CALCITE-3367 > URL: https://issues.apache.org/jira/browse/CALCITE-3367 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: jin xing >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira proposes to add AntiJoinRule to convert > project-filter-join-aggregate into anti-join > This issue was found when I resolve CALCITE-3363 in > https://github.com/apache/calcite/pull/1466 > I failed to construct an anti-join operator from SQL string. -- This message was sent by Atlassian Jira (v8.3.4#803005)
