[jira] [Created] (CALCITE-4847) Parse SQL with BigQuery-style quoted identifiers and character literals
Julian Hyde created CALCITE-4847:
Summary: Parse SQL with BigQuery-style quoted identifiers and
character literals
Key: CALCITE-4847
URL: https://issues.apache.org/jira/browse/CALCITE-4847
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Parse SQL with BigQuery-style quoted identifiers and character literals.
BigQuery quotes identifiers using backticks, escaping interior backticks using
backslash. In CALCITE-4767 we added {{Quoting.BACK_TICK_BACKSLASH}} to
distinguish this style from what MySQL does, namely {{Quoting.BACK_TICK}}.
BigQuery quotes character literals using double quotes, escaping interior
double quotes using backslash. In CALCITE-4767 we added
{{Quoting.DOUBLE_QUOTE_BACKSLASH}} to distinguish this style from
{{Quoting.DOUBLE_QUOTE}}.
After this change, we should be able to parse the following query if we invoke
the parser with {{lex=BIG_QUERY}} or {{dialect=BIG_QUERY}}:
{code}
SELECT "a \"quoted\" char literal"
FROM `a \`quoted\` table`
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4835) Release Calcite 1.28.0
Julian Hyde created CALCITE-4835: Summary: Release Calcite 1.28.0 Key: CALCITE-4835 URL: https://issues.apache.org/jira/browse/CALCITE-4835 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Release Calcite 1.28.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4815) Avatica's Travis fails with 'LICENSE-like files are missing' error
Julian Hyde created CALCITE-4815:
Summary: Avatica's Travis fails with 'LICENSE-like files are
missing' error
Key: CALCITE-4815
URL: https://issues.apache.org/jira/browse/CALCITE-4815
Project: Calcite
Issue Type: Bug
Components: avatica
Reporter: Julian Hyde
Some configurations of Avatica's Travis CI job fail with the following error.
{noformat}
> Task :standalone-server:classes
> Task :standalone-server:getLicenses FAILED
> Task :server:checkstyleTest
Build [0;1mcalcite-avatica [0;1;31mFAILURE[0;1m[0m reason:
Execution [0;1;31mfailed[0m for task
'[0;1m:standalone-server:getLicenses[0m':
LICENSE-like files are missing
==
MIT
* org.checkerframework:checker-qual:2.11.1
at
com.github.vlsi.gradle.license.GatherLicenseTask.run(GatherLicenseTask.kt:417)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
at
org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':standalone-server:getLicenses'.
> LICENSE-like files are missing
==
MIT
* org.checkerframework:checker-qual:2.11.1
{noformat}
@vlsi, Do you have any idea what's going on? Avatica doesn't even use
checkerframework. You can see the build output (for a few days) at
[https://app.travis-ci.com/github/julianhyde/calcite-avatica/builds/238877621].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4795) In class SqlBasicCall, make the "operands" field private.
Julian Hyde created CALCITE-4795:
Summary: In class SqlBasicCall, make the "operands" field private.
Key: CALCITE-4795
URL: https://issues.apache.org/jira/browse/CALCITE-4795
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In {{class SqlBasicCall}}, the {{operands}}
[field|https://github.com/apache/calcite/blob/4bc916619fd286b2c0cc4d5c653c96a68801d74e/core/src/main/java/org/apache/calcite/sql/SqlBasicCall.java#L34]
is a {{public}} array. This seems crazy to me – any client might be writing
into that field at any time. I propose to make the field private.
This presents some compatibility problems, because people might be depending on
the field. So I propose a quick deprecation and removal:
* In release 1.28 (the next release, as I write this) the field and the
{{public SqlNode[] getOperands()}} method will be marked deprecated. We will
mirror into another field, {{private final List operandList =
Arrays.asList(operands);}} We can replace all uses of the {{operands}} field in
Calcite with uses of the new {{operandList}} field.
* In release 1.29 (the release after next) the {{operands}} field and the
{{getOperands()}} method will be removed. People can operate using
{{List getOperandList()}} and {{setOperand(int, SqlNode)}} methods
that are inherited from {{SqlCall}}.
After the field is a private list, we could consider making it an immutable
list. The list would be copied when people call {{setOperand}}, but would not
need to be cloned when the {{SqlBasicCall}} is created or cloned.
This case completes the work started in CALCITE-147.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4789) Build is broken on Guava versions < 21
Julian Hyde created CALCITE-4789: Summary: Build is broken on Guava versions < 21 Key: CALCITE-4789 URL: https://issues.apache.org/jira/browse/CALCITE-4789 Project: Calcite Issue Type: Bug Environment: The build is currently broken on Guava versions 20 and lower. We claim in the release notes to support Guava 19 and higher. As part of this change, enable testing in CI of the lowest and highest supported Guava version. (We used to do this, but we stopped when we moved from Maven to Gradle. I still don't know how to set the Guava version from the Gradle command line.) Reporter: Julian Hyde -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4763) EXISTS_AGG, an aggregate function that returns whether count is positive
Julian Hyde created CALCITE-4763:
Summary: EXISTS_AGG, an aggregate function that returns whether
count is positive
Key: CALCITE-4763
URL: https://issues.apache.org/jira/browse/CALCITE-4763
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add {{EXISTS_AGG}}, an aggregate function that returns whether count is greater
than zero.
Thus {{EXISTS_AGG(*)}} is equivalent to {{COUNT(*) > 0}}, and {{EXISTS_AGG(c)}}
is equivalent to {{COUNT(c) > 0}}.
{{EXISTS_AGG}} would mainly be of use internally. Since it produces a
{{BOOLEAN}} value, we can use the value directly from an {{Aggregate}} without
an intervening {{Project}}. It also captures the fact that we don't care how
many rows were produced.
See also {{TRUE_AGG}}, CALCITE-4334.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4762) Upgrade to Avatica 1.19
Julian Hyde created CALCITE-4762: Summary: Upgrade to Avatica 1.19 Key: CALCITE-4762 URL: https://issues.apache.org/jira/browse/CALCITE-4762 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Upgrade Calcite to Avatica version 1.19. Currently version 1.19 has not been released. 1.19 fixes at least one issue whose test cases are attached as PRs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4737) Add Volcano visualizer for debugging
Julian Hyde created CALCITE-4737: Summary: Add Volcano visualizer for debugging Key: CALCITE-4737 URL: https://issues.apache.org/jira/browse/CALCITE-4737 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Add Volcano visualizer for debugging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4728) Parse and validate procedural code (such as SQL/PSM, PL/SQL, PL/pgSQL, T-SQL)
Julian Hyde created CALCITE-4728:
Summary: Parse and validate procedural code (such as SQL/PSM,
PL/SQL, PL/pgSQL, T-SQL)
Key: CALCITE-4728
URL: https://issues.apache.org/jira/browse/CALCITE-4728
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Parse and validate procedural code (such as the SQL standard's SQL/PSM,
Oracle's PL/SQL, PostgreSQL's PL/pgSQL, MSSQL's T-SQL).
This would entail:
* Extensions to the SQL parser. (I'm not sure whether this would be the core
parser or an extended parser such as Babel.)
* AST classes (sub-classes of {{SqlNode}}) for functions, procedures, blocks,
variable and parameter declarations, variable assignment, if-then-else, loop,
and so forth.
* Extensions to the validator to validate blocks.
* Extensions to the validator to validate a SQL statement that is inside a
block. (Variables and parameters are in scope.)
Optional:
* Extend Interpreter to execute functions, procedures, blocks.
* Some means to convert a block into executable Java code.
* Extend {{SqlDialect}} so that ASTs of functions, procedures and blocks can
be emitted using the syntax of particular dialects.
The languages are sufficiently similar that we can use the same AST classes for
all.
I don't think we need to extend {{RelNode}} to represent blocks. (It's not a
good fit, since a block does not evaluate to a relation.) Possibly we would
create some data structure to represent a validated block (e.g. the type of
each variable; each use of a variable points to the variable's definition).
{{SqlToRelConverter}} would create this data structure in order "freeze" the
state of the validator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4723) Check whether JDBC adapter generates "GROUP BY ()" against Oracle, DB2, MSSQL
Julian Hyde created CALCITE-4723:
Summary: Check whether JDBC adapter generates "GROUP BY ()"
against Oracle, DB2, MSSQL
Key: CALCITE-4723
URL: https://issues.apache.org/jira/browse/CALCITE-4723
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Oracle, DB2 and MSSQL have non-standard semantics for "GROUP BY ()". Standard
behavior is to always return one "grand total" row, but [Oracle, DB2 and MSSQL
return no rows if the input is
empty|https://blog.jooq.org/2018/05/25/how-to-group-by-nothing-in-sql/].
Calcite's semantics is that "GROUP BY ()" always returns one row, and the JDBC
adapter currently assumes that all back ends have the same semantics. On back
ends that have different semantics, some queries might be giving incorrect
results.
I suggest the following remedy:
* Add a {{SqlDialect}} method {{boolean omitGrandTotalOnEmptyInput()}}
* Run the test suite, and see whether we ever generate "GROUP BY ()" on one of
the affected dialects. Try to write a test case where we do this.
* Modify the dialects to generate safe SQL in these cases (possibly "GROUP BY
()", or possibly something else). As the above article notes, it is
particularly difficult to find SQL that works for MSSQL, because it bumps into
the no-constants rule (see CALCITE-4702)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4720) Obsolete the Collect relational operator, using Aggregate and ARRAY_AGG (and new aggregate functions MULTISET_AGG and MAP_AGG) instead
Julian Hyde created CALCITE-4720:
Summary: Obsolete the Collect relational operator, using Aggregate
and ARRAY_AGG (and new aggregate functions MULTISET_AGG and MAP_AGG) instead
Key: CALCITE-4720
URL: https://issues.apache.org/jira/browse/CALCITE-4720
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
The {{Collect}} relational operator converts a multi-row relation into a
relation with a single row and a column whose type is {{MULTISET}}.
But it is difficult to generalize it; we would like to:
* Generating multiple rows, one for each group key, rather than a single row
for the whole relation;
* Generate an {{ARRAY}} or {{MAP}} rather than a {{MULTISET};
* Generate a collection of scalars rather than a collection of records if the
input is a single column (e.g. {{INTEGER MULTISET}} rather than {{ROW(INTEGER
i) MULTISET}})
And, it is difficult to maintain; it is a minor RelNode that has only 2
implementations (that I know of) and I'm sure that there are bugs and missing
support in SqlToRelConverter and the RelOptRule library.
We can achieve the same using the {{Aggregate}} operator and the {{ARRAY_AGG}}
aggregate function. We would need new aggregate functions (let's call them
{{MULTISET_AGG}} and {{MAP_AGG}}) for the {{MULTISET}} and {{MAP}} types.
Then we can obsolete {{Collect}}, and make current code paths use {{Aggregate}}
instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4719) Add variants of RexSubQuery that collect sub-queries into MULTISET, ARRAY and MAP collections
Julian Hyde created CALCITE-4719:
Summary: Add variants of RexSubQuery that collect sub-queries into
MULTISET, ARRAY and MAP collections
Key: CALCITE-4719
URL: https://issues.apache.org/jira/browse/CALCITE-4719
Project: Calcite
Issue Type: Bug
Environment: Add variants of {{RexSubQuery}} that collect sub-queries
into {{MULTISET}}, {{ARRAY}} and {{MAP}} collections. We currently use
{{RexSubQuery}} for scalar sub-query, {{EXISTS}}, {{IN}} and some others; this
allows us to defer conversion: convert via rewrites ({{RelOptRule}}) rather
than in {{SqlToRelConverter}}. The same benefits would apply to {{MULTISET}},
{{ARRAY}} and {{MAP}} sub-queries.
Examples:
{code}
SELECT deptno, MULTISET(SELECT * FROM Emp WHERE deptno = d.deptno)
FROM Dept AS d
SELECT deptno, ARRAY(SELECT * FROM Emp WHERE deptno = d.deptno)
FROM Dept AS d
SELECT deptno, MAP(SELECT empno, job FROM Emp WHERE deptno = d.deptno)
FROM Dept AS d
{code}
Reporter: Julian Hyde
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4711) RexProgramBuilder should not simplify
Julian Hyde created CALCITE-4711:
Summary: RexProgramBuilder should not simplify
Key: CALCITE-4711
URL: https://issues.apache.org/jira/browse/CALCITE-4711
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
{{RexProgramBuilder}} currently simplifies expressions even if no
{{RexSimplifier}} instance is supplied in its constructor: the method
{{registerInternal}} creates a {{RexSimplifier}}.
This simplification used to be beneficial. For instance, as of
[4ef9f467|https://github.com/apache/calcite/commits/4ef9f46757528d21c510eb8bd171fa04ba86e36d],
{{RexProgramBuilder}} can simplify "x AND y AND x" to "x AND y". But now we
have RexSimplifier, which is applied pretty much any time a RexNode is included
in a RelNode via a RelBuilder method.
Simplify is now called in too many places, and I suspect that the running time
is multiplying. Consider: {{RexSimplify.simplifyCast}} calls
{{RexExecutor.reduce}} to reduce constants. To do this, {{RexExecutor}} uses
{{RexProgramBuilder}} to generate a program (which will then be executed to
yield the reduced value), and the RexProgramBuilder calls simplify again. It is
ridiculous to call simplify at so many levels. It's time that we reduced the
responsibilities of {{RexProgramBuilder}} to just building programs.
(It is possible that, once a {{RexProgram}} has been created, we can see some
patterns in the flattened expressions that would not be apparent to
{{RexSimplify}}. If this happens, we could have a simplify step that works on
{{Calc}} after its program has been generated.)
After this change, {{RexProgramBuilder}} will simplify only if a
{{RexSimplify}} is passed in the constructor. {{RexProgramBuilder}} will no
longer be able to simplify "x AND y AND x". We will add a test to ensure that
{{RelBuilder.filter}} can do this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4707) Optimize incremental maintenance of materialized views
Julian Hyde created CALCITE-4707:
Summary: Optimize incremental maintenance of materialized views
Key: CALCITE-4707
URL: https://issues.apache.org/jira/browse/CALCITE-4707
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Optimize incremental maintenance of materialized views, when we know what DML
has occurred since the last build.
The goal here is to develop an algebraic approach (i.e. new relational
operators and transformation rules) that will generate an optimal query for
maintaining the view.
Consider a materialized view
{code}
CREATE TABLE EmpsByDeptno AS
SELECT deptno, COUNT(*) AS c, SUM(sal) AS ss
FROM Emps
GROUP BY deptno;
{code}
We built it at time T1, when the value of {{Emps}} was {{Emps1}}, and it had
the value {{EmpsByDeptno1}}.
It is now T2, and since then, new employees have been created in the
{{NewEmps}} table. Thus {{Emps2 = Emps1 UNION ALL NewEmps}}. We could build a
new MV,
{code}
CREATE TABLE EmpsByDeptno2 AS
SELECT deptno, COUNT(*) AS c, SUM(sal) AS ss
FROM (SELECT * FROM Emps1
UNION ALL
SELECT * FROM NewEmps)
GROUP BY deptno;
{code}but we would prefer to generate a DML command to modify {{EmpsByDeptno}}
in place:{code}
MERGE INTO EmpsByDeptno AS e
USING (SELECT deptno, COUNT(*) AS c, SUM(sal) AS ss
FROM NewEmps) AS de
ON de.deptno = e.deptno
WHEN MATCHED THEN
UPDATE SET c = c + de.c, ss = ss + de.ss
WHEN NOT MATCHED THEN
INSERT (deptno, c, ss)
{code}
We propose two new relational operators:
* {{Diff(S, R)}} computes the difference between two relations. It has a same
schema as R and S but with an additional column {{delta}}. Rows in S but not in
R are called insertions, and have delta=1. Rows that are in R but not in S are
called deletions, and have delta=-1.
* {{Patch(R, P)}} applies
The relational operators are named by analogy with the UNIX utilities {{diff}}
and {{patch}}. (But {{Diff}}'s arguments are reversed compared to {{diff}}.)
Consider:
{code}
grep r /usr/share/dict/words > r.txt
grep s /usr/share/dict/words > s.txt
diff r.txt s.txt > p.patch
patch -p1 r.txt < p.patch
cmp r.txt s.txt. # r and s are now identical
{code}
The relation {code}R = Patch(S, Diff(R, S)){code} always holds. {{Diff}}
computes the difference between R and S, and when {{Patch}} applies that
difference to {{S}} you get {{R}}.
{{Patch}} and {{Diff}} are intended by be generalizations of set operators. If
there are only additions, i.e. {{R}} is a superset of {{S}}, then you can
substitute {{Minus}} for {{Diff}}, and {{Union}} for {{Patch}}:
{code}
R = Union(S, Minus(R, S))
{code}
So, how do these relational operators solve our original problem?
An {{INSERT}} statement is assignment to a relation of itself union some new
rows. ({{INSERT INTO R SELECT * FROM P}} is {{R R UNION P}}).
A {{MERGE}} statement is similar to {{INSERT}} but allows updates.
The {{MERGE}} query above is represented as {code}EmpsByDeptno
Patch(EmpsByDeptno, Diff(NewEmpsByDeptno, EmpsByDeptno)){code}
Lastly we need relational transformation rules to optimize the expressions into
per-key updates.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4687) Add LIMIT to WITHIN GROUP clause of aggregate functions
Julian Hyde created CALCITE-4687:
Summary: Add LIMIT to WITHIN GROUP clause of aggregate functions
Key: CALCITE-4687
URL: https://issues.apache.org/jira/browse/CALCITE-4687
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add LIMIT to WITHIN GROUP clause of aggregate functions. LIMIT is not in the
SQL standard, but it is useful, and is not hard to implement.
The following query computes the 3 highest paid employees in each department:
{code:java}
SELECT deptno, ARRAY_AGG(sal) WITHIN GROUP (ORDER BY sal DESC LIMIT 3)
FROM Emp
GROUP BY deptno {code}
It can be implemented efficiently (using a merge sort that discards all but the
top 3 rows in each key, at each pass).
Note that BigQuery does not support the {{WITHIN GROUP}} clause, but in the
{{ARRAY_AGG}} function, the {{ORDER BY}} and {{LIMIT}} sub-clauses appear
within the parentheses, like this: {{ARRAY_AGG(sal ORDER BY sal DESC LIMIT
3)}}. In Calcite, you can use either syntax for {{ARRAY_AGG}},
{{ARRAY_CONCAT_AGG}}, {{GROUP_CONCAT}}, {{STRING_AGG}} functions; we should add
{{LIMIT}} in both.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4671) Allow Avatica connect-string properties in any case (lowerCamel, UPPER_SNAKE, etc.)
Julian Hyde created CALCITE-4671:
Summary: Allow Avatica connect-string properties in any case
(lowerCamel, UPPER_SNAKE, etc.)
Key: CALCITE-4671
URL: https://issues.apache.org/jira/browse/CALCITE-4671
Project: Calcite
Issue Type: Bug
Components: avatica
Reporter: Julian Hyde
Avatica's built-in properties are mostly lower_snake case (e.g.
{{truststore_password}}; the one exception being {{timeZone}} in lowerCamel;
see [client
reference|https://calcite.apache.org/avatica/docs/client_reference.html]);
Calcite's built-in properties are lowerCamel case (e.g.
[approximateTopN|https://calcite.apache.org/javadocAggregate/org/apache/calcite/config/CalciteConnectionProperty.html#APPROXIMATE_TOP_N];
see [Calcite JDBC driver
reference|https://calcite.apache.org/docs/adapter.html#jdbc-connect-string-parameters%C2%A0]).
Avatica's properties are inherited by Calcite, so there is bound to be
confusion.
Avatica's connect string parser (also used in Calcite) should allow any case.
Thus {{truststore_password}}, {{truststorePassword}}, {{TRUSTSTORE_PASSWORD}},
{{TruststorePassword}} are all acceptable. {{trustStorePassword}} is not
acceptable, because "truststore" is one word.
In Calcite and Avatica doc, add a note that properties can are accepted in any
case.
Change Avatica's properties to lowerCamel case in both doc and code. (The old
forms, e.g. "truststore_password", will continue to work.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4649) Add AggregateExpandGroupingSetsRule, a planner rule that removes GROUPING SETS
Julian Hyde created CALCITE-4649:
Summary: Add AggregateExpandGroupingSetsRule, a planner rule that
removes GROUPING SETS
Key: CALCITE-4649
URL: https://issues.apache.org/jira/browse/CALCITE-4649
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add {{class AggregateExpandGroupingSetsRule}}, a planner rule that removes
{{GROUPING SETS}}.
Some databases do not support {{GROUPING SETS}}. (For example, BigQuery
supports {{GROUP BY ROLLUP}} but not general {{GROUPING SETS}}.) But {{GROUPING
SETS}} is a useful construct, increasingly used in user SQL, and also generated
by rules (e.g. {{AggregateExpandWithinDistinctRule}}).
The proposed rule would match an {{Aggregate}} whose {{groupSets}} field has
more than one element, and would convert it to an {{Aggregate}} over a {{Join}}
whose right-hand side is a list of integers, the group ids active in the query.
Calls to the {{GROUP_ID}} field would be replaced by references to the group id
column.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4629) RexNormalize.hashCode() throws NullPointerException
Julian Hyde created CALCITE-4629:
Summary: RexNormalize.hashCode() throws NullPointerException
Key: CALCITE-4629
URL: https://issues.apache.org/jira/browse/CALCITE-4629
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
{{RexNormalize.hashCode()}} throws {{NullPointerException}}. The stack is as
follows:
{noformat}
[java.util.Objects.requireNonNull(java/util/Objects.java:203)]:
Java::JavaLang::NullPointerException:
java.util.Objects.requireNonNull(java/util/Objects.java:203)
org.apache.calcite.rex.RexNormalize.hashCode(org/apache/calcite/rex/RexNormalize.java:127)
org.apache.calcite.rex.RexCall.hashCode(org/apache/calcite/rex/RexCall.java:297)
java.util.HashMap.hash(java/util/HashMap.java:339)
java.util.HashMap.put(java/util/HashMap.java:612)
java.util.HashSet.add(java/util/HashSet.java:220)
org.apache.calcite.rex.RexUtil.addOr(org/apache/calcite/rex/RexUtil.java:1370)
org.apache.calcite.rex.RexUtil.flattenOr(org/apache/calcite/rex/RexUtil.java:1356)
org.apache.calcite.rex.RexUtil.composeDisjunction(org/apache/calcite/rex/RexUtil.java:1330)
org.apache.calcite.rex.RexUtil.composeDisjunction(org/apache/calcite/rex/RexUtil.java:1320)
org.apache.calcite.tools.RelBuilder.or(org/apache/calcite/tools/RelBuilder.java:721)
{noformat}
It seems likely that this was caused by commit
[6a9832c|https://github.com/julianhyde/calcite/commit/6a9832c14e7c7c02a0176920b87d15dab474300a].
I don't have a repro case, and I can't figure out what might have caused this.
The problem occurs in 1.27.0 RC 0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4628) If SqlImplementor fails, include the RelNode in the exception
Julian Hyde created CALCITE-4628:
Summary: If SqlImplementor fails, include the RelNode in the
exception
Key: CALCITE-4628
URL: https://issues.apache.org/jira/browse/CALCITE-4628
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Fix For: 1.28.0
If {{SqlImplementor}} fails, include the {{RelNode}} in the exception. (For
example, we recently encountered a {{java.lang.ArrayIndexOutOfBoundsException}}
where the problem was in the {{RelNode}} but the {{RelNode}} was not in the
stack.)
Add a catch block that calls
[Util.throwAsRuntime|https://github.com/apache/calcite/blob/f1309fa3af6826e73377e54081160228eb7ab951/core/src/main/java/org/apache/calcite/util/Util.java#L961]
to the {{SqlImplementor.visitRoot}} method. That method seems to be a widely
used entry point.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4609) WITH query using AVG throws AssertionError: type mismatch
Julian Hyde created CALCITE-4609:
Summary: WITH query using AVG throws AssertionError: type mismatch
Key: CALCITE-4609
URL: https://issues.apache.org/jira/browse/CALCITE-4609
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
The query
{code:sql}
WITH EmpAnalytics AS (
SELECT deptno, job, AVG(sal) AS avg_sal
FROM Emp
GROUP BY deptno, job)
SELECT job, avg(avg_sal) AS avg_sal2
FROM EmpAnalytics
WHERE deptno = 30
GROUP BY job{code}
gives error
{noformat}
java.lang.AssertionError: type mismatch:
ref:
DECIMAL(19, 2)
input:
DECIMAL(7, 2)
at org.apache.calcite.util.Litmus$1.fail(Litmus.java:32)
at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:2209)
at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:129)
at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:61)
at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:114)
at org.apache.calcite.rel.core.Project.isValid(Project.java:219)
at org.apache.calcite.rel.core.Project.(Project.java:98)
at
org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:69)
at
org.apache.calcite.rel.logical.LogicalProject.create(LogicalProject.java:126)
at
org.apache.calcite.rel.logical.LogicalProject.create(LogicalProject.java:114)
at
org.apache.calcite.rel.core.RelFactories$ProjectFactoryImpl.createProject(RelFactories.java:178)
at org.apache.calcite.tools.RelBuilder.project_(RelBuilder.java:1645)
at org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:1417)
at org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:1389)
at org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:1378)
at
org.apache.calcite.rel.rules.AggregateRemoveRule.onMatch(AggregateRemoveRule.java:120)
{noformat}
Here is a patch that reproduces:
{noformat}
diff --git a/core/src/test/resources/sql/misc.iq
b/core/src/test/resources/sql/misc.iq
index ba5ce1053..d036da3d2 100644
--- a/core/src/test/resources/sql/misc.iq
+++ b/core/src/test/resources/sql/misc.iq
@@ -1065,6 +1065,16 @@ Expression 'DEPTNO' is not being grouped
!use scott
+WITH EmpAnalytics as (
+ SELECT deptno, job, AVG(sal) AS avg_sal
+ FROM "scott".emp
+ GROUP BY deptno, job)
+SELECT job, AVG(avg_sal) AS avg_sal2
+FROM EmpAnalytics
+WHERE deptno = 30
+GROUP BY job;
+!ok
+
# ORDER BY expression with SELECT DISTINCT
select distinct deptno, job
from "scott".emp
{noformat}
If you run the same query from SQLLine, you get a different error, but I think
they are probably related:
{noformat}
Error while applying rule ProjectMergeRule, args
[rel#406:LogicalProject.NONE.[](input=RelSubset#301,exprs=[$1, $2, CASE(IS NOT
NULL($2), 1:BIGINT, 0:BIGINT)]),
rel#362:LogicalProject.NONE.[](input=RelSubset#361,exprs=[$1, $0, $2])]
(state=,code=0)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4597) Allow RelNodes to have an empty row type (zero fields)
Julian Hyde created CALCITE-4597:
Summary: Allow RelNodes to have an empty row type (zero fields)
Key: CALCITE-4597
URL: https://issues.apache.org/jira/browse/CALCITE-4597
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add an option, {{EmptyRowTypePolicy}}, to allow creation of {{RelNode}}s whose
row type is empty. That is, contains zero fields.
There are three values:
* {{FORBIDDEN}} - Calcite prevents empty row type. (For example, planner and
{{RelBuilder}} throw if they see one.) Rules must not produce empty row types.
Rules can assume that they will not encounter empty row types.
* {{DISCOURAGED}} - Empty row types are discouraged. (Planner and
{{RelBuilder}} will not throw if they see one.) Rules must not fail if they
encounter an empty row type. Rules should not produce empty row types (with
reasonable exceptions, such as if the input has an empty row type).
* {{ALLOWED}} - Empty row types are OK. All rules should handle {{RelNode}}s
with empty row types, and it's OK if they generate {{RelNode}}s with empty row
types.
The current policy is effectively {{DISCOURAGED}}. We try not to create empty
RelNodes, but we don't check, and they crop up occasionally.
After this change, and for a few releases, the policy will be {{DISCOURAGED}}
by default, but we will run tests in all three modes. All rules must run in all
modes.
At some point in the future, we will change the default policy to {{ALLOWED}}.
All rules must continue to run in all modes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4594) Interpreter returns wrong result when Values has zero fields
Julian Hyde created CALCITE-4594: Summary: Interpreter returns wrong result when Values has zero fields Key: CALCITE-4594 URL: https://issues.apache.org/jira/browse/CALCITE-4594 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Interpreter returns wrong result when Values has zero fields. It returns zero rows rather than N rows of zero columns. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4593) DiffRepository tests should fail if XML resources are not in alphabetical order
Julian Hyde created CALCITE-4593:
Summary: DiffRepository tests should fail if XML resources are not
in alphabetical order
Key: CALCITE-4593
URL: https://issues.apache.org/jira/browse/CALCITE-4593
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Tests that use XML resources managed via {{class DiffRepository}} should fail
if the XML resources are not in alphabetical order.
First, some background. Quite a few tests store resources in an XML file,
accessed via {{class DiffRepository}}. For example, {{RelOptRulesTest}} stores
the plan before and after the rule(s) that it is testing are fired. The
resources are organized by test case name, enclosed in {{}} elements.
Contributors have a tendency to add test resources at the end of the file. But
this makes the end of the file a hot-spot for conflicts.
Therefore the best practice is to put the resources into alphabetical order.
{{DiffRepository}} tries to help with this, by generating an {{_actual.xml}}
file with the new resource in inserted in its right alphabetical position, but
contributors somehow miss this, and write the file by hand. So, conflicts.
With this change, a test will fail if resources are out of order. The message
will look something like this:
{noformat}
java.lang.IllegalArgumentException: expected 10 test cases to be out of order,
but there were 11; here are the new ones:
"testAggregateRemove6"
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051)
at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
at
com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4964)
at org.apache.calcite.test.DiffRepository.lookup(DiffRepository.java:808)
at
org.apache.calcite.test.RelOptRulesTest.getDiffRepos(RelOptRulesTest.java:190){noformat}
Why does it say "expected 10 test case to be out of order, but there were 11"?
Because resource files are not currently in total order: they were not sorted
to start with, and we don't want to destroy history by sorting them. But to
solve the conflict problem, we only need *new* test cases to be in order. So,
this change adds a list of exceptions - test cases that are known to be out of
order - and {{DiffRepository}} will not complain if those test cases are out of
order. Over time, the sort order of resource files will get better, or at least
will not get any worse.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4591) RelRunner should throw SQLException if prepare fails
Julian Hyde created CALCITE-4591:
Summary: RelRunner should throw SQLException if prepare fails
Key: CALCITE-4591
URL: https://issues.apache.org/jira/browse/CALCITE-4591
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
{{interface RelRunner}} allows statements to be prepared from a {{RelNode}}
tree rather than via SQL. A {{RelRunner}} is typically obtained from a JDBC
connection, by calling {{connection.unwrap(RelRunner.class)}}. As such,
{{RelRunner}} is an extension to the JDBC driver, and therefore the
{{RelRunner.prepare(RelNode)}} method should throw {{SQLException}}, rather
than {{RuntimeException}} as it does today.
This change adds a new method
{code}
PreparedStatement prepareStatement(RelNode) throws SQLException
{code}
and marks the current {{prepare}} method as deprecated. (As {{RelRunner}} is
marked "experimental", we could have removed or changed the old method without
notice, but chose not to.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4576) Release process should not overwrite LICENSE file
Julian Hyde created CALCITE-4576:
Summary: Release process should not overwrite LICENSE file
Key: CALCITE-4576
URL: https://issues.apache.org/jira/browse/CALCITE-4576
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
The release process should generate a source distribution whose {{LICENSE}}
file has different contents from the {{LICENSE}} file stored in Git.
Rationale: Source distributions should be a mirror of the contents of Git at
the time of the release. (Possibly a few files are omitted -- see CALCITE-4575
for example -- but files should not be generated/compiled.) Users expect that
the LICENSE file they see in Git is the same as the LICENSE file in the source
distribution.
One possible implementation of this is as follows. The release process should
not overwrite the {{LICENSE}} file when generating the source distribution. It
should generate a {{LICENSE}} file (possibly from a {{LICENSE.template}} file),
check that the one in source control is identical, and fail if it is not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4575) Remove Gradle wrapper from source distribution
Julian Hyde created CALCITE-4575:
Summary: Remove Gradle wrapper from source distribution
Key: CALCITE-4575
URL: https://issues.apache.org/jira/browse/CALCITE-4575
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Apache release policy does not allow binary files in source releases, and in
particular not JAR files. Calcite and Avatica source code includes the Gradle
wrapper, which requires {{gradle/wrapper/gradle-wrapper.jar}}. The policy
[exempts a few build tools, but the Gradle wrapper is not among
them||https://lists.apache.org/thread.html/rd1aabe5052b5bedf3eceebd331f878b92a8ade6d4ca170f845d5db37%40%3Clegal-discuss.apache.org%3E].
A request to expand the list (see LEGAL-288) was denied.
In my opinion, the list should be expanded to include {{gradle-wrapper.jar}},
but that is not current policy.
I propose to solve this by removing the Gradle wrapper from the source
distribution and amending {{site/_docs/howto.md}} to explain that people
building from a source distribution (not from Git) need to install a particular
version of Gradle first. The list of files to be removed from the source
distribution is as follows:
* {{gradlew}}
* {{gradlew.bat}}
* {{gradle/wrapper/gradle-wrapper.jar}}
* {{gradle/wrapper/gradle-wrapper.properties}}
These files would not be removed from Git.
In future, anyone upgrading Gradle would also need to edit the version in
{{site/_docs/howto.md}}.
(Note that OFBIZ-10145 is another solution to this problem. In my opinion, it
is an inferior solution for our case. It is more complicated for people
building from Git, and would require us to maintain the shell scripts that
replace the functionality of {{gradle-wrapper.jar}}.)
This change needs to be applied to both Calcite and Avatica.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4564) Initialization context for non-static user-defined functions (UDFs)
Julian Hyde created CALCITE-4564:
Summary: Initialization context for non-static user-defined
functions (UDFs)
Key: CALCITE-4564
URL: https://issues.apache.org/jira/browse/CALCITE-4564
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
I propose to allow user-defined functions (UDFs) to read from an initialization
context during construction. The initialization context would be a new Java
{{interface UdfInitializer}} that provides, among other things, a type factory
and the values of the arguments to the function call whose values are literals.
The purpose of this feature is to allow functions to do more work at
initialization time and less work on each invocation. Suppose I wanted to write
a UDF {{regexMatch(pattern, string)}} that matches Java regular expressions. If
{{pattern}} is a literal, I would like to create an instance of the function
object that calls {{Pattern.compile(pattern)}} in its constructor and stores
the resulting {{Pattern}} object as a field. Each invocation of the function
can use that {{Pattern}} object, and does not have to pay the cost of
compilation.
In order to use this feature, a UDF class would have a public constructor with
a single argument that is a {{UdfInitializer}}. The method that invokes the
function, conventionally called {{eval}}, must be non-static.
This feature is optional. A UDF that has a public constructor with zero
arguments (which is the current contract for non-static UDFs) will continue to
work. [class
MyPlusFunction|https://github.com/apache/calcite/blob/4bc916619fd286b2c0cc4d5c653c96a68801d74e/core/src/test/java/org/apache/calcite/util/Smalls.java#L429]
is an example of this kind of UDF.
This feature would apply to all UDFs, including table functions (i.e. those
whose argument are tables or which return tables) and aggregate functions.
The initialization context would not affect type derivation aspects of the
function. The return type, operand types, and so forth, will already have been
derived during validate time, and is complete well before any code is generated
or executed. If you want to control type derivation, you should create your own
sub-class of {{SqlOperator}}, as today.
There are some implementation challenges:
* The code generator will need to generate an instance of {{UdfInitializer}}
for each UDF call that occurs in the query. Some data structures that are
readily available at validate time (e.g. {{RexCall}}) are not easily re-created
at run time, so we should be conservative what information is available via
{{UdfInitializer}}.
* The code generator must ensure that those instances are constructed exactly
once during the execution of the query; those instances should not be variables
in the {{execute}} method, but should instead be fields, or perhaps static
fields, in the generated class.
* This functionality needs to work through both the interpreter ({{Bindable}}
convention) and generated code ({{Enumerable}} convention).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4562) Improve simplification of "expression IS TRUE"
Julian Hyde created CALCITE-4562:
Summary: Improve simplification of "expression IS TRUE"
Key: CALCITE-4562
URL: https://issues.apache.org/jira/browse/CALCITE-4562
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
When simplifying "expression IS TRUE", RexSimplify should simplify "expression"
in an "unknown as FALSE" context, and this should allow more simplifications
than at present.
Currently "(x LIKE '%') IS TRUE" simplifies to "(x IS NOT NULL) IS TRUE", but
after this change will simplify to "x IS NOT NULL". The following test
illustrates:
{code}
final RexNode ref = input(tVarchar(true, 10), 0);
checkSimplify(isTrue(like(ref, literal("%"))), "IS NOT NULL($0)");
{code}
Similarly "IS FALSE", "IS NOT TRUE", etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4559) Create 'interface RexRule', a modular rewrite for row-expressions
Julian Hyde created CALCITE-4559:
Summary: Create 'interface RexRule', a modular rewrite for
row-expressions
Key: CALCITE-4559
URL: https://issues.apache.org/jira/browse/CALCITE-4559
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
We propose to add {{class RexRule}}, a rewrite rule for row-expressions
({{class RexNode}}).
{{class RexRule}} is analogous to how {{class RelRule}} (and the older {{class
RelOptRule}}) operates on relational expressions ({{interface RelNode}}). Also,
{{class RexRuleProgram}} is analogous to {{HepProgram}} and {{VolcanoPlanner}}
(it indexes rules so that we do not have to try every rule against every part
of the expression). And a rule describes which operands it matches using
{{RexRule.describe(RexRule.OperandBuilder)}}, similar to calling
{{RelRule.Config.operandSupplier().apply()}}.
The advantages of {{RexRule}} are similar to {{RelRule}}: rules can be defined
in a modular way, can be documented and tested individually, and can be enabled
individually.
The rules could be applied in various ways. {{RelBuilder.Config}} could contain
a {{RexRuleProgram}} that would be applied every time an expression is
simplified by a {{RelBuilder}}. There could also be a sub-class of {{interface
RelShuttle}} that applies the rules to every {{RexNode}} in a tree (e.g. inside
{{Filter}}, {{Project}} and {{Join}}).
I don't yet know whether, or how, rules might support 3-valued boolean logic
({{RexUnknownAs}}). For example, a rule that simplifies "x = x" to "TRUE" is
valid in an "unknownAsFalse" context (e.g. as top-level of {{Filter}}
condition), but not in an "unknownAsUnknown" context (e.g. in {{Project}}
expression).
This case is related to CALCITE-3470 (making relational and row-expression
rules more similar, as in CockroachDB), but would deliver an API rather than a
textual DSL.
I am working on a prototype that illustrates the key interfaces and
demonstrates a couple of rules and tests for them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4552) Interpreter does not close resources held by its Nodes on close
Julian Hyde created CALCITE-4552: Summary: Interpreter does not close resources held by its Nodes on close Key: CALCITE-4552 URL: https://issues.apache.org/jira/browse/CALCITE-4552 Project: Calcite Issue Type: Bug Environment: When closing Interpreter, close resources held by its Nodes. Currently, if a node is based on a JdbcTableScan, and the Interpreter's results have not been read to completion, the statement will still be open, and the JDBC connection will not be returned to the pool. This can cause a hang, waiting for a connection, if you allocate and deallocate lots of Interpreter instances. Reporter: Julian Hyde -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4547) Support Java 16
Julian Hyde created CALCITE-4547: Summary: Support Java 16 Key: CALCITE-4547 URL: https://issues.apache.org/jira/browse/CALCITE-4547 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Support building and running on [Java 16|https://jdk.java.net/16/] (OpenJDK 16). This will require an [upgrade to Gradle 7|https://melix.github.io/blog/2021/03/gradle-java16.html], because Gradle 6 will support Java 16. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4524) Make some fields non-nullable (SqlSelect.selectList, DataContext.getTypeFactory)
Julian Hyde created CALCITE-4524:
Summary: Make some fields non-nullable (SqlSelect.selectList,
DataContext.getTypeFactory)
Key: CALCITE-4524
URL: https://issues.apache.org/jira/browse/CALCITE-4524
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Make some fields non-nullable: {{SqlSelect.selectList}},
{{DataContext.getTypeFactory()}}, {{DataContext.getQueryProvider()}}.
Add {{class DataContexts}}, with a few useful implementations of {{interface
DataContext}}.
Resolve some other TODOs relating to nullability.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4520) In ImmutableBeans, nullable properties become not-nullable in project that does not use checkerframework
Julian Hyde created CALCITE-4520:
Summary: In ImmutableBeans, nullable properties become
not-nullable in project that does not use checkerframework
Key: CALCITE-4520
URL: https://issues.apache.org/jira/browse/CALCITE-4520
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In ImmutableBeans, nullable properties become not-nullable in project that does
not use checkerframework. In my Morel project, I get the following error:
{noformat}
java.lang.ExceptionInInitializerError
at
org.apache.calcite.prepare.CalcitePrepareImpl.(CalcitePrepareImpl.java:154)
at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:182)
at org.apache.calcite.tools.RelBuilder.create(RelBuilder.java:225)
at net.hydromatic.morel.foreign.Calcite.(Calcite.java:42)
at
net.hydromatic.morel.foreign.Calcite$CalciteMap.(Calcite.java:64)
at net.hydromatic.morel.foreign.Calcite.withDataSets(Calcite.java:56)
at net.hydromatic.morel.Ml.assertEvalSame(Ml.java:319)
at net.hydromatic.morel.AlgebraTest.checkEqual(AlgebraTest.java:234)
at net.hydromatic.morel.AlgebraTest.testNative(AlgebraTest.java:230)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
Caused by: java.lang.IllegalArgumentException: property
'org.apache.calcite.adapter.enumerable.EnumerableLimitSortRule$Config#Description'
is required and has no default value
at
org.apache.calcite.util.ImmutableBeans.lambda$makeDef$0(ImmutableBeans.java:168)
at
org.apache.calcite.util.ImmutableBeans$BeanImpl.invoke(ImmutableBeans.java:480)
at com.sun.proxy.$Proxy15.description(Unknown Source)
at org.apache.calcite.plan.RelRule.(RelRule.java:115)
at
org.apache.calcite.adapter.enumerable.EnumerableLimitSortRule.(EnumerableLimitSortRule.java:35)
at
org.apache.calcite.adapter.enumerable.EnumerableLimitSortRule$Config.toRule(EnumerableLimitSortRule.java:57)
at
java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:719)
at
org.apache.calcite.util.ImmutableBeans.lambda$makeDef$3(ImmutableBeans.java:299)
at
org.apache.calcite.util.ImmutableBeans$BeanImpl.invoke(ImmutableBeans.java:480)
at com.sun.proxy.$Proxy15.toRule(Unknown Source)
at
org.apache.calcite.adapter.enumerable.EnumerableRules.(EnumerableRules.java:90)
... 31 more
{noformat}
The property {{EnumerableLimitSortRule.Config.Description}} is declared as
nullable (because it has the annotation
{{org.checkerframework.checker.nullness.qual.Nullable}}) but that annotation
does not come through when the class is used in the Morel project, and
therefore the property becomes non-nullable.
This bug was introduced by the [~vlsi]'s change CALCITE-4284.
We should not require client projects to use checkerframework, but if they
don't, they will get this error when they reference nullable properties.
I think that nullable properties should be indicated by an annotation that we
control - go back to
[jira] [Created] (CALCITE-4506) Upgrade SQLLine to 1.10.0
Julian Hyde created CALCITE-4506: Summary: Upgrade SQLLine to 1.10.0 Key: CALCITE-4506 URL: https://issues.apache.org/jira/browse/CALCITE-4506 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Upgrade SQLLine to 1.10.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4497) In RelBuilder, support windowed aggregate functions (OVER)
Julian Hyde created CALCITE-4497:
Summary: In RelBuilder, support windowed aggregate functions (OVER)
Key: CALCITE-4497
URL: https://issues.apache.org/jira/browse/CALCITE-4497
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In {{RelBuilder}}, support windowed aggregate functions (OVER). Currently, you
have to write code like this (from
[testAggregatedWindowFunction|https://github.com/apache/calcite/blob/f1da65504e598928cf77aa6a7244552692ae2529/core/src/test/java/org/apache/calcite/rel/rel2sql/RelToSqlConverterTest.java#L830]):
{code}
b.scan("EMP")
.project(b.field("SAL"))
.project(
b.alias(
b.getRexBuilder().makeOver(
b.getTypeFactory().createSqlType(SqlTypeName.INTEGER),
SqlStdOperatorTable.RANK, ImmutableList.of(),
ImmutableList.of(),
ImmutableList.of(
new RexFieldCollation(b.field("SAL"),
ImmutableSet.of())),
RexWindowBounds.UNBOUNDED_PRECEDING,
RexWindowBounds.UNBOUNDED_FOLLOWING,
true, true, false, false, false),
"rank"))
{code}
but potentially you could write this:
{code}
b.scan("EMP")
.project(b.field("SAL"))
.project(
b.aggregateCall(SqlStdOperatorTable.RANK)
.over()
.rowsUnbounded()
.sort(b.field("SAL))
.as("rank"))
{code}
{{class RelBuilder}} would need a new inner {{interface OverCall}}, and {{class
RelBuilder.AggCall}} would need a new method {{OverCall over()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4496) Measure columns ("SELECT ... AS MEASURE")
Julian Hyde created CALCITE-4496:
Summary: Measure columns ("SELECT ... AS MEASURE")
Key: CALCITE-4496
URL: https://issues.apache.org/jira/browse/CALCITE-4496
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In multi-dimensional languages such as MDX, DAX, Tableau, you can define
calculations in your models that can be re-evaluated in other dimensional
contexts. (The models are often called cubes, and the calculations are often
called measures.)
In SQL, the model is a view (or a sub-query in the FROM clause) but the columns
are just values. Suppose you have a private {{Employees}} table, a
{{Departments}} view that rolls {{Employees}} up to department level and has an
{{averageSalary}} column. Now suppose you wish to roll up {{averageSalary}} to
the region level. The values that went into {{averageSalary}} are not available
to you, either directly or indirectly, so the best you can do is to
average-the-averages.
In this proposed (and experimental) feature, you can define a special kind of
column - a measure - in the SELECT list of a view (or sub-query in a FROM
clause), and it remains a calculation. When a query uses a measure column, the
calculation is re-evaluated in the context of that query.
To some extent, this breaches the "black box" property of SQL views. Hitherto,
a SQL view can be replaced with a table that has the same contents, and all
queries that use that view will return the same results. That property no
longer holds. But the view remains a useful "hiding" abstraction, and the rows
that compose that view cannot be viewed directly.
Like dimensional models, measures in SQL would allow high-level abstractions
such as key-performance indicators (KPIs) to be shared and composed. Unlike
dimensional models, the models remain relational, namely, it is still possible
to enumerate and count the rows in a model.
Consider the following view and query that uses it:
{code:sql}
CREATE VIEW EmpSummary AS
SELECT deptno,
job,
AVG(sal) AS avg_sal,
AVG(sal) AS MEASURE avg_sal_measure,
COUNT(*) + 1 AS MEASURE count_plus_one_measure
FROM Emp
GROUP BY deptno, job;
SELECT deptno,
AVG(avg_sal) AS a1,
AGGREGATE(avg_sal_measure) AS a2,
AGGREGATE(count_plus_one_measure) AS c1
FROM EmpSummary
GROUP BY deptno;{code}
Note that there is a special aggregate function, {{AGGREGATE}}, that rolls up
measures. Columns {{a1}} and {{a2}} will contain different values; the first
averages the averages, and the second computes the average from the raw data.
Column {{c1}} will return the number of employees in each department plus one,
not rolling up the "plus one" for each distinct job in the department.
This is just a brief sketch illustrating the purpose of measures. This feature
is experimental, the syntax will no doubt change, and much of the semantics
(for example, what expressions are valid as measures, whether measures remain
measures they appear in the SELECT clause of an enclosing query, and what is
the "context" in which a measure is evaluated) need to be ironed out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4486) UNIQUE predicate
Julian Hyde created CALCITE-4486:
Summary: UNIQUE predicate
Key: CALCITE-4486
URL: https://issues.apache.org/jira/browse/CALCITE-4486
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Implement the UNIQUE predicate, as specified by the SQL standard.
Example:
{code:java}
UNIQUE (SELECT publishedIn FROM Book
WHERE authorId = 3)
NOT UNIQUE (SELECT publishedIn FROM BOOK
WHERE authorId = 3)
{code}
{{UNIQUE}} returns {{FALSE}} if the query returns two or more rows that do not
include NULL values and are equal. Like {{EXISTS}}, it never returns
{{UNKNOWN}}.
More details at
[JOOQ|https://www.jooq.org/doc/3.0/manual/sql-building/conditional-expressions/unique-predicate/].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4484) Add UNIQUE_VALUE(x) aggregate function, that throws if x is not unique
Julian Hyde created CALCITE-4484:
Summary: Add UNIQUE_VALUE(x) aggregate function, that throws if x
is not unique
Key: CALCITE-4484
URL: https://issues.apache.org/jira/browse/CALCITE-4484
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add a {{UNIQUE_VALUE\(x)}} aggregate function, that throws if {{x}} is not
unique.
{{UNIQUE_VALUE\(x)}} would throw if {{x}} has values [1, 2], or has values [1,
NULL]; but would not throw if x has values [1, 1, 1] or [] or [NULL, NULL].
Like {{ANY_VALUE}} it behaves as if {{RESPECT NULLS}} is specified.
There are similar functions:
* {{ANY_VALUE\(x)}} non-deterministically picks a value. (It is present in
BigQuery, MySQL, Snowflake, MSSQL and perhaps others.)
* {{SINGLE_VALUE\(x)}} returns the value of x if there is just one value (e.g.
[1] or [NULL]), NULL if there are no values, throws if there is more than one
value (e.g. [NULL, NULL] or [1, 1, 1] or [1, 2]). {{SINGLE_VALUE}} is in
Calcite, no other DBs that I am aware of, not documented, but available through
SQL. Calcite uses it internally to enforce scalar sub-queries.
BigQuery has an internal function "{{$ANY_AND_CHECK\(x)}}" that is equivalent
to {{UNIQUE_VALUE\(x)}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4483) Add WITHIN DISTINCT clause for aggregate functions
Julian Hyde created CALCITE-4483:
Summary: Add WITHIN DISTINCT clause for aggregate functions
Key: CALCITE-4483
URL: https://issues.apache.org/jira/browse/CALCITE-4483
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add a {{WITHIN DISTINCT}} clause to aggregate functions, allowing duplicate
rows to be eliminated before entering the function.
This feature is non-standard, and in fact does not exist in any database I am
aware of.
It is related to {{DISTINCT}}, and is in fact a generalization of it.
{{DISTINCT}} can always be rewritten in terms of {{WITHIN DISTINCT}}. For
example, {{SUM(DISTINCT sal)}} is equivalent to {{SUM(sal) WITHIN DISTINCT
(sal)}}.
Consider the query
{noformat}
SELECT SUM(age),
SUM(DISTINCT age),
SUM(age) WITHIN DISTINCT (name)
FROM Friends{noformat}
where {{Friends}} has the rows
{noformat}
name age job
== === ==
Julian 16 Programmer
Dick15
Anne13 Car wash
George 15 Lifeguard
George 15 Dog walker
Timmy4
{noformat}
Note that there are two rows for George, because she has two jobs.
The values of the columns are as follows:
* {{SUM(age)}} has the value (16 + 15 + 13 + 15 + 15 + 4) = 78;
* {{SUM(DISTINCT age)}} has the value (16 + 15 + 13 + 4) = 48;
* {{SUM(age) WITHIN DISTINCT (name)}} has the value (16 + 15 + 13 + 15 + 4) =
63.
{{WITHIN DISTINCT}} has treated the two 15 values for George as one value, but
has still counted the 15 for Dick separately.
The {{WITHIN DISTINCT}} clause can be useful to prevent double-counting when
duplicates have been added via a many-to-one join.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4478) In interpreter, support infinite relations
Julian Hyde created CALCITE-4478:
Summary: In interpreter, support infinite relations
Key: CALCITE-4478
URL: https://issues.apache.org/jira/browse/CALCITE-4478
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In interpreter, support infinite relations. Currently, if you execute
{code}select *
from table("s"."fibonacci"())
limit 6{code}(by modifying
{{InterpreterTest.testInterpretNilaryTableFunction()}} slightly) you get an
{{OutOfMemoryError}}.
The reason is that {{TableFunctionScanNode}} tries to run to completion before
handing over control to {{SortNode}} (which performs the {{LIMIT 6}}). The
intermediate result is an unbounded list ({{ArrayDeque}}).
The solution would be for each operator to yield every so often.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4477) In interpreter, support table-valued functions
Julian Hyde created CALCITE-4477:
Summary: In interpreter, support table-valued functions
Key: CALCITE-4477
URL: https://issues.apache.org/jira/browse/CALCITE-4477
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In interpreter, support table-valued functions. Currently if you try to run a
plan that contains table-valued functions in the interpreter, you get an error
like this:
{noformat}
java.lang.AssertionError: interpreter: no implementation for class
org.apache.calcite.rel.logical.LogicalTableFunctionScanjava.lang.AssertionError:
interpreter: no implementation for class
org.apache.calcite.rel.logical.LogicalTableFunctionScan
at
org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:462)
at org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:45)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4449) When converting Sarg 'x IS NULL OR x NOT IN (1, 2)', generate better SQL
Julian Hyde created CALCITE-4449:
Summary: When converting Sarg 'x IS NULL OR x NOT IN (1, 2)',
generate better SQL
Key: CALCITE-4449
URL: https://issues.apache.org/jira/browse/CALCITE-4449
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
When converting the Sarg
{code:java}
x IS NULL OR x NOT IN (1, 2){code}
to SQL, Calcite currently generates
{code:java}
x IS NULL OR x < 1 OR x > 1 AND x < 2 OR x > 2{code}
This is incorrect (because {{AND}} has higher precedence than {{OR}}) and also
ugly. Adding parentheses makes it correct:
{code:java}
x IS NULL OR (x < 1 OR x > 1) AND (x < 2 OR x > 2){code}
But it's still ugly. Using {{NOT IN}} or {{<>}} would be ideal:
{code:java}
x IS NULL OR x <> 1 AND x <> 2{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4446) Implement three-valued logic for SEARCH operator
Julian Hyde created CALCITE-4446:
Summary: Implement three-valued logic for SEARCH operator
Key: CALCITE-4446
URL: https://issues.apache.org/jira/browse/CALCITE-4446
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Implement three-valued logic for SEARCH operator.
Consider the expression {{x IN (10, 20)}}, which we might represent as
{{SEARCH(x, SARG(10, 20))}}. Suppose we invoke this with a value of {{NULL}}
for {{x}}. Do we want it to return UNKNOWN, FALSE or TRUE? The answer is: all
of the above.
Here are the 3 variants:
* {{SEARCH(10, 20, UNKNOWN AS TRUE)}}: {{x IS NULL OR x IN (10, 20)}}
TRUE
* {{SEARCH(10, 20, UNKNOWN AS UNKNOWN)}}: {{x IN (10, 20)}} UNKNOWN
* {{SEARCH(10, 20, UNKNOWN AS FALSE)}}: {{x IS NOT NULL AND (x IN (10, 20))}}
FALSE
Currently {{class Sarg}} has a field {{boolean containsNull}} which deals with
the first two cases. Changing {{boolean containsNull}} to {{RexUnknownAs
unknownAs}} (which has 3 values) will allow us to represent the third. The new
representation is symmetrical under negation, which de Morgan's law suggests is
a good thing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4434) Cannot implement 'CASE row WHEN row ...'
Julian Hyde created CALCITE-4434:
Summary: Cannot implement 'CASE row WHEN row ...'
Key: CALCITE-4434
URL: https://issues.apache.org/jira/browse/CALCITE-4434
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Cannot implement 'CASE row WHEN row ...'.
For example, the test
{noformat}
diff --git a/core/src/test/resources/sql/misc.iq
b/core/src/test/resources/sql/misc.iq
index 3c945e2cc..84ef67b6c 100644
--- a/core/src/test/resources/sql/misc.iq
+++ b/core/src/test/resources/sql/misc.iq
@@ -1065,6 +1065,47 @@ Expression 'DEPTNO' is not being grouped
!use scott
+# [CALCITE-] Cannot implement 'CASE row WHEN row ...'
+SELECT deptno, job,
+ CASE (deptno, job)
+ WHEN (20, 'CLERK') THEN 1
+ WHEN (30, 'SALESMAN') THEN 2
+ ELSE 3
+ END AS x
+FROM "scott".emp
+WHERE empno < 7600;
+++--+---+
+| DEPTNO | JOB | X |
+++--+---+
+| 20 | CLERK| 1 |
+| 20 | MANAGER | 3 |
+| 30 | SALESMAN | 2 |
+| 30 | SALESMAN | 2 |
+++--+---+
+(4 rows)
+!ok
+
+# Equivalent to previous
+SELECT deptno, job,
+ CASE
+ WHEN deptno = 20 AND job = 'CLERK' THEN 1
+ WHEN deptno = 30 AND job = 'SALESMAN' THEN 2
+ ELSE 3
+ END AS x
+FROM "scott".emp
+WHERE empno < 7600;
+++--+---+
+| DEPTNO | JOB | X |
+++--+---+
+| 20 | CLERK| 1 |
+| 20 | MANAGER | 3 |
+| 30 | SALESMAN | 2 |
+| 30 | SALESMAN | 2 |
+++--+---+
+(4 rows)
+
+!ok
+
{noformat}
fails with the following stack trace:
{noformat}
Unable to implement EnumerableCalc(expr#0..7=[{inputs}], expr#8=[ROW($t7,
$t2)], expr#9=[CAST($t8):RecordType(INTEGER EXPR$0, VARCHAR(9) EXPR$1) NOT
NULL], expr#10=[20], expr#11=['CLERK'], expr#12=[ROW($t10, $t11)],
expr#13=[CAST($t12):RecordType(INTEGER EXPR$0, VARCHAR(9) EXPR$1) NOT NULL],
expr#14=[=($t9, $t13)], expr#15=[1], expr#16=[30], expr#17=['SALESMAN'],
expr#18=[ROW($t16, $t17)], expr#19=[CAST($t18):RecordType(INTEGER EXPR$0,
VARCHAR(9) EXPR$1) NOT NULL], expr#20=[=($t9, $t19)], expr#21=[2], expr#22=[3],
expr#23=[CASE($t14, $t15, $t20, $t21, $t22)], expr#24=[7600], expr#25=[<($t0,
$t24)], DEPTNO=[$t7], JOB=[$t2], X=[$t23], $condition=[$t25]): rowcount = 7.0,
cumulative cost = {21.0 rows, 435.0 cpu, 0.0 io}, id = 49495
EnumerableTableScan(table=[[scott, EMP]]): rowcount = 14.0, cumulative cost =
{14.0 rows, 15.0 cpu, 0.0 io}, id = 49458
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163)
at
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
at net.hydromatic.quidem.Quidem.checkResult(Quidem.java:322)
...
Caused by: java.lang.IllegalStateException: Unable to implement
EnumerableCalc(expr#0..7=[{inputs}], expr#8=[ROW($t7, $t2)],
expr#9=[CAST($t8):RecordType(INTEGER EXPR$0, VARCHAR(9) EXPR$1) NOT NULL],
expr#10=[20], expr#11=['CLERK'], expr#12=[ROW($t10, $t11)],
expr#13=[CAST($t12):RecordType(INTEGER EXPR$0, VARCHAR(9) EXPR$1) NOT NULL],
expr#14=[=($t9, $t13)], expr#15=[1], expr#16=[30], expr#17=['SALESMAN'],
expr#18=[ROW($t16, $t17)], expr#19=[CAST($t18):RecordType(INTEGER EXPR$0,
VARCHAR(9) EXPR$1) NOT NULL], expr#20=[=($t9, $t19)], expr#21=[2], expr#22=[3],
expr#23=[CASE($t14, $t15, $t20, $t21, $t22)], expr#24=[7600], expr#25=[<($t0,
$t24)], DEPTNO=[$t7], JOB=[$t2], X=[$t23], $condition=[$t25]): rowcount = 7.0,
cumulative cost = {21.0 rows, 435.0 cpu, 0.0 io}, id = 49495
EnumerableTableScan(table=[[scott, EMP]]): rowcount = 14.0, cumulative cost =
{14.0 rows, 15.0 cpu, 0.0 io}, id = 49458
at
org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.implementRoot(EnumerableRelImplementor.java:114)
at
org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:113)
at
org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.implement(CalcitePrepareImpl.java:1130)
...
Suppressed: java.lang.NullPointerException: SqlTypeFamily for
RecordType(INTEGER EXPR$0, VARCHAR(9) EXPR$1)
at java.base/java.util.Objects.requireNonNull(Objects.java:348)
at
org.apache.calcite.adapter.enumerable.RexToLixTranslator.scaleIntervalToNumber(RexToLixTranslator.java:943)
at
org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateCast(RexToLixTranslator.java:593)
at
org.apache.calcite.adapter.enumerable.RexImpTable$CastImplementor.implementSafe(RexImpTable.java:2511)
at
org.apache.calcite.adapter.enumerable.RexImpTable$AbstractRexCallImplementor.genValueStatement(RexImpTable.java:2972)
at
[jira] [Created] (CALCITE-4433) Add UNPIVOT operator to SQL
Julian Hyde created CALCITE-4433:
Summary: Add UNPIVOT operator to SQL
Key: CALCITE-4433
URL: https://issues.apache.org/jira/browse/CALCITE-4433
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add {{UNPIVOT}} operator to SQL. (We added {{PIVOT}} in CALCITE-3752, and
{{UNPIVOT}} is the inverse.)
Oracle has {{UNPIVOT}}. For
[example|https://oracle-base.com/articles/11g/pivot-and-unpivot-operators-11gr1#unpivot]:
{noformat}
SELECT * FROM unpivot_test;
ID CUSTOMER_ID PRODUCT_CODE_A PRODUCT_CODE_B PRODUCT_CODE_C
PRODUCT_CODE_D
-- --- -- -- --
--
1 101 10 20 30
2 102 4050
3 103 60 70 80
90
4 104100
4 rows selected.
SQL> SELECT *
FROM unpivot_test
UNPIVOT (quantity FOR product_code IN (product_code_a AS 'A', product_code_b AS
'B', product_code_c AS 'C', product_code_d AS 'D'));
ID CUSTOMER_ID P QUANTITY
-- --- - --
1 101 A 10
1 101 B 20
1 101 C 30
2 102 A 40
2 102 C 50
3 103 A 60
3 103 B 70
3 103 C 80
3 103 D 90
4 104 A100
10 rows selected.
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4427) Make SUBSTRING operator comply with ISO standard SQL
Julian Hyde created CALCITE-4427:
Summary: Make SUBSTRING operator comply with ISO standard SQL
Key: CALCITE-4427
URL: https://issues.apache.org/jira/browse/CALCITE-4427
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Make {{SUBSTRING}} operator comply with ISO standard SQL. It currently complies
with BigQuery standard SQL, which gives different behavior when start is
negative. See discussion in CALCITE-4408.
Add an option to get the BigQuery behavior, for those who want it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4405) Search/Sarg does not handle UNKNOWN correctly
Julian Hyde created CALCITE-4405: Summary: Search/Sarg does not handle UNKNOWN correctly Key: CALCITE-4405 URL: https://issues.apache.org/jira/browse/CALCITE-4405 Project: Calcite Issue Type: Bug Reporter: Julian Hyde The internal SEARCH operator (introduced in CALCITE-4163 along with the Sarg literal type) does not handle UNKNOWN correctly. However, we do not currently have a SQL test case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4394) When generating code for a function call, take the inferred operand type into account
Julian Hyde created CALCITE-4394:
Summary: When generating code for a function call, take the
inferred operand type into account
Key: CALCITE-4394
URL: https://issues.apache.org/jira/browse/CALCITE-4394
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
When generating code for a function call, take the inferred operand type into
account. If we don't do this, the Java code we generate for {{CONCAT(CAST(NULL
AS ANY), 'x')}} does not compile, because the first parameter is of type
Object; it needs to be String (based on the inferred operand type of VARCHAR).
We achieve this by using {{SqlOperator.operandTypeInference}} at validate time,
storing the list of inferred operand types for the call, then using that list
(if available) during code generation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4389) Calls to ROW and anonymous row operators sometimes print too many spaces
Julian Hyde created CALCITE-4389:
Summary: Calls to ROW and anonymous row operators sometimes print
too many spaces
Key: CALCITE-4389
URL: https://issues.apache.org/jira/browse/CALCITE-4389
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Calls to {{ROW}} and anonymous row operators sometimes print too many spaces.
For example, "{{VALUES (1, TRUE)}}" is converted to SQL as
{noformat}
VALUES (1, TRUE)
{noformat}
(Note two spaces after "VALUES".)
The cause is that the argument to VALUES is initially represented as a call to
the ROW operator, "ROW (1, TRUE)", and later converted to a call to the
anonymous row operator. The anonymous row operator, whose name is " " (one
space), does not to be preceded by white space, but the one white space " " has
already been printed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4383) In RelBuilder, optimize Union(all, Values, Values) to Values
Julian Hyde created CALCITE-4383:
Summary: In RelBuilder, optimize Union(all, Values, Values) to
Values
Key: CALCITE-4383
URL: https://issues.apache.org/jira/browse/CALCITE-4383
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In {{RelBuilder}}, optimize {{Union(all, Values, Values)}} to {{Values}}.
Similarly optimize {{Project(Values)}} to {{Values}} if the projected
expressions are all literals.
This pattern occurs frequently, if someone writes {{SELECT 1, 'a' UNION ALL
SELECT 2, 'b'}} for instance.
Enable this rewrite with a new property {{boolean
RelBuilder.Config.simplifyValues()}}, default true.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4380) Make class SqlNodeList implement List
Julian Hyde created CALCITE-4380:
Summary: Make class SqlNodeList implement List
Key: CALCITE-4380
URL: https://issues.apache.org/jira/browse/CALCITE-4380
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Make {{class SqlNodeList}} implement {{List}}. Currently it implements
{{Iterable}}. Supporting the List interface would make the size
available, and that might allow users of SqlNodeList to pre-allocate space.
The downside is that a few places might have different overloads on {{SqlNode}}
and {{Collection}} or {{List}}, and these would become
ambiguous because {{SqlNodeList}} implements both.
Another benefit is that we can remove many of the calls to {{List
SqlNodeList.toList()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4370) Ensure that all aggregate functions work in windowed aggregates
Julian Hyde created CALCITE-4370:
Summary: Ensure that all aggregate functions work in windowed
aggregates
Key: CALCITE-4370
URL: https://issues.apache.org/jira/browse/CALCITE-4370
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Ensure that all aggregate functions work in windowed aggregates.
I made a simple modification:
{noformat}
diff --git a/core/src/test/java/org/apache/calcite/sql/test/SqlTests.java
b/core/src/test/java/org/apache/calcite/sql/test/SqlTests.java
index 5404121fd..5a218538a 100644
--- a/core/src/test/java/org/apache/calcite/sql/test/SqlTests.java
+++ b/core/src/test/java/org/apache/calcite/sql/test/SqlTests.java
@@ -115,7 +115,7 @@ public static String getTypeString(RelDataType sqlType) {
public static String generateAggQuery(String expr, String[] inputValues) {
StringBuilder buf = new StringBuilder();
-buf.append("SELECT ").append(expr).append(" FROM ");
+buf.append("SELECT ").append(expr).append(" OVER () FROM ");
if (inputValues.length == 0) {
buf.append("(VALUES 1) AS t(x) WHERE false");
} else {
{noformat}
and then ran {{SqlOperatorBaseTest}}, and got errors or failures in tests:
* {{COUNT}} disgregards {{DISTINCT}});
* {{COUNTIF}}, {{STRING_AGG}}, {{APPROX_COUNT_DISTINCT}}, and {{BIT_XOR}} are
not implemented for windowed functions;
* {{STDDEV_SAMP}} returns zero for an empty window but should return null;
* {{JSON_ARRAY_AGG}} cannot even parse {{OVER}};
* miscellaneous other problems for {{BIT_AND}} {{ARRAY_AGG}}, {{COLLECT}}.
We should fix these problems, and ensure that {{SqlOperatorTest}} runs each
aggregate function with {{OVER}} as well as {{GROUP BY}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4366) LatticeSuggester should treat UNION / VALUES sub-expressions as tables
Julian Hyde created CALCITE-4366:
Summary: LatticeSuggester should treat UNION / VALUES
sub-expressions as tables
Key: CALCITE-4366
URL: https://issues.apache.org/jira/browse/CALCITE-4366
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
LatticeSuggester should treat UNION / VALUES sub-expressions as tables, and
generate a LatticeTable for each.
If those expressions occur as a common-table expression (CTE; i.e. in the WITH
clause) the LatticeTable should have the name and column names of that CTE.
For example, given the query
{code}
WITH SubDept (id, name) AS (SELECT deptno, dname FROM Dept)
SELECT *
FROM Emp AS e
JOIN SubDept AS d ON e.deptno = d.id
{code}
{{LatticeSuggester}} should generate {{LatticeTable(Emp (empno, deptno, ename,
sal))}} and {{LatticeTable(SubDept (id, name))}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4349) Support GROUP_CONCAT aggregate function for MySQL
Julian Hyde created CALCITE-4349:
Summary: Support GROUP_CONCAT aggregate function for MySQL
Key: CALCITE-4349
URL: https://issues.apache.org/jira/browse/CALCITE-4349
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Support GROUP_CONCAT aggregate function for MySQL. Here is the syntax:
{noformat}
GROUP_CONCAT([DISTINCT] expr [,expr ...]
[ORDER BY {unsigned_integer | col_name | expr}
[ASC | DESC] [,col_name ...]]
[SEPARATOR str_val])
{noformat}
It is analogous to {{LISTAGG}} (see CALCITE-2754) (and also to BigQuery and
PostgreSQL's {{STRING_AGG}}, see CALCITE-4335). For example, the query
{code}
SELECT deptno, GROUP_CONCAT(ename ORDER BY empno SEPARATOR ';')
FROM Emp
GROUP BY deptno
{code}
is equivalent to (and in Calcite's algebra would be desugared to)
{code}
SELECT deptno, LISTAGG(ename, ';') WITHIN GROUP (ORDER BY empno)
FROM Emp
GROUP BY deptno
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4335) Aggregate functions for BigQuery
Julian Hyde created CALCITE-4335:
Summary: Aggregate functions for BigQuery
Key: CALCITE-4335
URL: https://issues.apache.org/jira/browse/CALCITE-4335
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add aggregate functions for BigQuery
([spec|https://cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions]):
{{ARRAY_AGG}}, {{ARRAY_CONCAT_AGG}}, {{COUNTIF}}, {{LOGICAL_AND}},
{{LOGICAL_OR}}, {{STRING_AGG}}.
Calcite already has {{ANY_VALUE}}, {{BIT_AND}}, {{BIT_OR}}, {{BIT_XOR}},
{{MAX}}, {{MIN}}, {{SUM}} aggregate functions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4334) TRUE_AGG, an aggregate function that always returns TRUE
Julian Hyde created CALCITE-4334:
Summary: TRUE_AGG, an aggregate function that always returns TRUE
Key: CALCITE-4334
URL: https://issues.apache.org/jira/browse/CALCITE-4334
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
It would be useful to have an (internal) aggregate function that has no
arguments and returns a constant value, regardless of how many rows are in the
group. We suggest {{TRUE_AGG}}, which always returns the {{BOOLEAN}} value
{{TRUE}}.
If there are reasons why it would be better to return other constants (such as
0, 1, FALSE or the empty string) we will consider them.
For example, when rewriting sub-queries (see SubQueryRemoveRule) we introduce
add "true as indicator" to the SELECT clause of sub-queries. It can be used to
detect rows generated by an outer join. If it is an aggregate query, we would
have to write "min(true) as indicator", which necessitates an extra {{Project}}
below the {{Aggregate}} to provide the "true" value.
Another example. We would like to make {{RelBuilder.aggregate(groupKey())}}
throw when given an empty group key and no aggregate calls. (Because it would
create an {{Aggregate}} that has zero fields, and that is problematic elsewhere
in Calcite.) But we would also like a pattern to generate a constant single-row
relational expression. So, {{RelBulder.aggregate(groupKey(),
aggregateCall(TRUE_AGG))}} should generate {{VALUES TRUE}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4324) sqlline is broken
Julian Hyde created CALCITE-4324:
Summary: sqlline is broken
Key: CALCITE-4324
URL: https://issues.apache.org/jira/browse/CALCITE-4324
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
After
[532f903f|https://github.com/apache/calcite/commit/532f903fe495d741053619c13a51537e57dcd619],
the {{buildSqllineClasspath}} gradle task is broken:
{noformat}
$ ./gradlew :buildSqllineClasspath
> Configure project :
Building Apache Calcite 1.27.0-SNAPSHOT
> Task :buildSqllineClasspath FAILED
Build calcite FAILURE reason:
Execution failed for task ':buildSqllineClasspath':
org.gradle.api.internal.artifacts.ivyservice.DefaultLenientConfiguration$ArtifactResolveException:
Could not resolve all files for configuration ':sqllineClasspath'.
at
org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.rethrowFailure(DefaultConfiguration.java:1276)
at
org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.access$1700(DefaultConfiguration.java:142)
{noformat}
This causes the {{./sqlline}} shell script to fail.
[~vlsi], Can you look into this, please.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4323) View with ORDER BY throws AssertionError during view expansion
Julian Hyde created CALCITE-4323:
Summary: View with ORDER BY throws AssertionError during view
expansion
Key: CALCITE-4323
URL: https://issues.apache.org/jira/browse/CALCITE-4323
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
If you have a view that has an ORDER BY clause, and try to execute a query on
that view, SqlToRelConverter throws AssertionError during view expansion. For
example,
{noformat}
create view v as select * from "EMPLOYEES" order by "deptno";
select * from V;
java.lang.AssertionError
at
org.apache.calcite.sql2rel.SqlToRelConverter.requiredCollation(SqlToRelConverter.java:634)
at
org.apache.calcite.sql2rel.SqlToRelConverter.requiredCollation(SqlToRelConverter.java:629)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:578)
at
org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.expandView(CalcitePrepareImpl.java:1072)
at
org.apache.calcite.plan.ViewExpanders$1.expandView(ViewExpanders.java:52)
at
org.apache.calcite.schema.impl.ViewTable.expandView(ViewTable.java:127)
at org.apache.calcite.schema.impl.ViewTable.toRel(ViewTable.java:120)
at
org.apache.calcite.prepare.RelOptTableImpl.toRel(RelOptTableImpl.java:285)
at
org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3605)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2522)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2160)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2109)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2066)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:662)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:643)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3458)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:569)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:242)
{noformat}
I suspected that it would be a problem for when the view does not project the
sort column(s), but it seems to be a problem even if the columns are projected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4305) Implicit column alias for single-column UNNEST
Julian Hyde created CALCITE-4305:
Summary: Implicit column alias for single-column UNNEST
Key: CALCITE-4305
URL: https://issues.apache.org/jira/browse/CALCITE-4305
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Single-column UNNEST with a single alias should assign that alias to both a
table and the unique column. For example, PostgreSQL returns a column called
'unnest' in the first, and 'fruit' in all of the rest:
{code}
select * from UNNEST(array ['apple', 'banana', 'pear']);
select * from UNNEST(array ['apple', 'banana', 'pear']) as fruit;
select * from UNNEST(array ['apple', 'banana', 'pear']) as t(fruit);
select t.* from UNNEST(array ['apple', 'banana', 'pear']) as t(fruit);
select fruit.* from UNNEST(array ['apple', 'banana', 'pear']) as fruit;
{code}
Thus {{FROM UNNEST(...) AS x}} is creating a table alias {{x}} and a column
alias {{x}}.
This only happens for UNNEST. When aliasing other single-column relations the
column name is retained, such as a SELECT-FROM-UNNEST sub-query as follows:
{code}
SELECT fruit.*
FROM (SELECT * FROM UNNEST(array ['apple', 'banana', 'pear']) as x) as fruit;
x
==
apple
banana
pear
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4298) Avoid disabling hostname verification on HTTPS connections
Julian Hyde created CALCITE-4298: Summary: Avoid disabling hostname verification on HTTPS connections Key: CALCITE-4298 URL: https://issues.apache.org/jira/browse/CALCITE-4298 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Avoid disabling hostname verification on HTTPS connections. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4294) Use JTS rather than ESRI as the underlying library for geospatial (ST_) functions
Julian Hyde created CALCITE-4294: Summary: Use JTS rather than ESRI as the underlying library for geospatial (ST_) functions Key: CALCITE-4294 URL: https://issues.apache.org/jira/browse/CALCITE-4294 Project: Calcite Issue Type: Bug Components: spatial Reporter: Julian Hyde The geospatial functions are currently implemented using the ESRI library. We should consider using JTS instead. AT the time we started work on geospatial the JTS did not have a suitable license, but this is no longer the case. I gather that JTS is a superior library. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4271) RelBuilder.in should allow duplicate values
Julian Hyde created CALCITE-4271:
Summary: RelBuilder.in should allow duplicate values
Key: CALCITE-4271
URL: https://issues.apache.org/jira/browse/CALCITE-4271
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Currently, {{RelBuilder.in(e, constant(1), constant(2), constant(1))}} throws,
as follows:
{noformat}
java.lang.IllegalArgumentException: Ranges may not overlap, but received [1..1]
and [1..1]
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:146)
at
com.google.common.collect.ImmutableRangeSet$Builder.add(ImmutableRangeSet.java:585)
at org.apache.calcite.rex.RexBuilder.toSarg(RexBuilder.java:1374)
at org.apache.calcite.rex.RexBuilder.makeIn(RexBuilder.java:1310)
at org.apache.calcite.tools.RelBuilder.in(RelBuilder.java:612)
{noformat}
Since the SQL expression {{e IN (1, 2, 1)}} is valid, the corresponding call to
{{RelBuilder.in}} should be valid also.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4266) JDBC adapter throws UnsupportedOperationException if query contains range predicate on columns from sub-query
Julian Hyde created CALCITE-4266:
Summary: JDBC adapter throws UnsupportedOperationException if
query contains range predicate on columns from sub-query
Key: CALCITE-4266
URL: https://issues.apache.org/jira/browse/CALCITE-4266
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
JDBC adapter throws {{UnsupportedOperationException}} if query contains range
predicate on columns from sub-query. The query
{code}
SELECT CASE WHEN v.g IN (0, 1) THEN 0 ELSE 1 END
FROM (SELECT * FROM "foodmart"."customer") AS c,
(SELECT 0 AS g) AS v
GROUP BY v.g
{code}
contains a range (by virtue of the {{IN (0, 1)}}) and throws
{noformat}
java.lang.UnsupportedOperationException
at
org.apache.calcite.rel.rel2sql.SqlImplementor$Result$1.implementor(SqlImplementor.java:1597)
at
org.apache.calcite.rel.rel2sql.SqlImplementor$Context.lambda$toSql$0(SqlImplementor.java:862)
at
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
{noformat}
The problem is that, in order to simplify a Sarg (see CALCITE-4173), the
{{implementor()}} is being called in a context where it was not called before.
The solution is to implement the {{SqlImplementor.Context.implementor()}}
method in more sub-classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4265) Improve error message when CAST to unknown type
Julian Hyde created CALCITE-4265:
Summary: Improve error message when CAST to unknown type
Key: CALCITE-4265
URL: https://issues.apache.org/jira/browse/CALCITE-4265
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Improve error message when {{CAST}} to unknown type. If you add the following
test case to {{RelToSqlConverterTest}}, you get an
{{UnsupportedOperationException}} during validation due to the unknown datatype
{{SIGNED}}:
{code}
@Test void testSelectFromUnionGrouping() {
String query = "SELECT CASE WHEN \"t3\".\"groupingVal\" IN (0, 1) THEN
\"t\".\"fname\" ELSE NULL END AS name, "
+ "CASE WHEN \"t3\".\"groupingVal\" = 0 THEN \"t\".\"birthdate\" ELSE
NULL END AS birthdate, AVG(\"t\".\"__f2\") AS "
+ "\"__f2\", CAST(CASE WHEN \"t3\".\"groupingVal\" = 0 THEN 0 ELSE 1 END AS
SIGNED) AS \"__f3\", CAST(CASE WHEN"
+ " \"t3\".\"groupingVal\" IN (0, 1) THEN 0 ELSE 1 END + CASE WHEN
\"t3\".\"groupingVal\" = 0 THEN 0 ELSE 2 "
+ "END AS SIGNED) AS \"__f4\"\n"
+ "FROM (SELECT \"fname\", \"birthdate\", \"total_children\" + 2 AS
\"__f2\"\n"
+ "FROM \"foodmart\".\"customer\") AS \"t\",\n"
+ "(SELECT 0 AS \"groupingVal\"\n"
+ "UNION ALL\n"
+ "SELECT 1 AS \"groupingVal\"\n"
+ "UNION ALL\n"
+ "SELECT 2 AS \"groupingVal\") AS \"t3\"\n"
+ "GROUP BY CASE WHEN \"t3\".\"groupingVal\" IN (0, 1) THEN
\"t\".\"fname\" ELSE NULL END, CASE WHEN \"t3\""
+ ".\"groupingVal\" = 0 THEN \"t\".\"birthdate\" ELSE NULL END,
\"t3\".\"groupingVal\"";
sql(query).ok("--");
}{code} throws
{noformat}
java.lang.UnsupportedOperationException: class
org.apache.calcite.sql.SqlIdentifier: SIGNED
at org.apache.calcite.util.Util.needToImplement(Util.java:1074)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.getValidatedNodeType(SqlValidatorImpl.java:1655)
at
org.apache.calcite.sql.SqlUserDefinedTypeNameSpec.deriveType(SqlUserDefinedTypeNameSpec.java:49)
at
org.apache.calcite.sql.SqlDataTypeSpec.deriveType(SqlDataTypeSpec.java:222)
at
org.apache.calcite.sql.SqlDataTypeSpec.deriveType(SqlDataTypeSpec.java:209)
at
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5868)
at
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5753)
at
org.apache.calcite.sql.SqlDataTypeSpec.accept(SqlDataTypeSpec.java:186)
{noformat}
If you change {{SIGNED}} to {{INTEGER}} the exception goes away.
Clearly this test case is not minimal. There could probably be a 1 line test
case for {{SqlValidatorTest}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4262) In SqlToRelConverter, use RelBuilder for creating all relational expressions
Julian Hyde created CALCITE-4262: Summary: In SqlToRelConverter, use RelBuilder for creating all relational expressions Key: CALCITE-4262 URL: https://issues.apache.org/jira/browse/CALCITE-4262 Project: Calcite Issue Type: Bug Reporter: Julian Hyde -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4259) Support JDK 15
Julian Hyde created CALCITE-4259: Summary: Support JDK 15 Key: CALCITE-4259 URL: https://issues.apache.org/jira/browse/CALCITE-4259 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Fix For: 1.26.0 JDK 15 is now GA. We should support it (and OpenJDK 15) as a build- and run-time platform. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4254) ImmutableBeans should make an immutable copy of lists, sets, maps
Julian Hyde created CALCITE-4254:
Summary: ImmutableBeans should make an immutable copy of lists,
sets, maps
Key: CALCITE-4254
URL: https://issues.apache.org/jira/browse/CALCITE-4254
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
{{ImmutableBeans}} should make an immutable copy of values of type {{List}},
{{Set}} or {{Map}} when they are provided via a setter method. For example,
this is how it should work:
{code:java}
MyBean bean = ImmutableBeans.create(MyBean.class);
List evens = Arrays.asList(2, 4, 6);
bean = bean.withNumbers(evens);
assertThat(bean.getNumbers(), equalTo(evens));
assertThat(bean.getNumbers(), isInstanceOf(ImmutableList.class));
{code}
Note that {{evens}} has been converted to an {{ImmutableList}} with the same
contents.
You can control this behavior by setting {{makeImmutable = false}} when you
declare the property:
{code:java}
interface MyBean {
@ImmutableBean.Property(makeImmutable = false)
List getNumbers();
MyBean withNumbers(List numbers);
}
{code}
By default, {{makeImmutable}} is true. This will change behavior of a very few
existing beans that were assuming that the value that was put in would be the
value that came out (see a couple of changes in {{VolcanoPlannerTest}}), but
for the vast majority, immutability is the desired behavior.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4248) Deprecate SqlParser.ConfigBuilder
Julian Hyde created CALCITE-4248:
Summary: Deprecate SqlParser.ConfigBuilder
Key: CALCITE-4248
URL: https://issues.apache.org/jira/browse/CALCITE-4248
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Deprecate {{SqlParser.ConfigBuilder}}; for each property 'xxx', add a method
{{withXxx}} to {{SqlParser.Config}} that creates a copy of the {{Config}}.
Therefore {{Config}} can be used everywhere that {{ConfigBuilder}} used to be
used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4247) When parsing SQL in BigQuery dialect, character literals may be enclosed in single- or double-quotes, and use backslashes as escapes
Julian Hyde created CALCITE-4247:
Summary: When parsing SQL in BigQuery dialect, character literals
may be enclosed in single- or double-quotes, and use backslashes as escapes
Key: CALCITE-4247
URL: https://issues.apache.org/jira/browse/CALCITE-4247
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
When parsing SQL in BigQuery dialect, character literals may be enclosed in
single- or double-quotes, and use backslashes as escapes. In standard SQL, and
most dialects, character literals are enclosed in single-quotes only, and use
single-quotes as the escape character.
For example:
{noformat}
'Let''s call him "Elvis"!' /* valid in Oracle, PostgreSQL, etc. */
'Let\'s call him "Elvis"!' /* valid in BigQuery */
"Let's call him \"Elvis\"!" /* valid in BigQuery */
{noformat}
To control this feature, we add
{code}
enum CharLiteralStyle { STANDARD, BQ_SINGLE, BQ_DOUBLE }
{code}
and to {{interface SqlParser.Config}} we add method {{Set
charLiteralStyles()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4246) When parsing SQL in BigQuery dialect, allow unquoted table names to contain hyphens
Julian Hyde created CALCITE-4246:
Summary: When parsing SQL in BigQuery dialect, allow unquoted
table names to contain hyphens
Key: CALCITE-4246
URL: https://issues.apache.org/jira/browse/CALCITE-4246
Project: Calcite
Issue Type: Bug
Components: babel
Reporter: Julian Hyde
When parsing SQL in BigQuery dialect, allow unquoted table names to contain
hyphens. This feature was [added in May
2020|https://www.reddit.com/r/bigquery/comments/fgk31y/new_in_bigquery_no_more_backticks_around_table/].
For these purposes, "table names" are considered to be the first identifier
after FROM, JOIN, INSERT, UPDATE, DELETE, TABLE without an intervening SELECT
or SET. (Yeah, the rules are rather rigid, but to implement it we have to shift
lexical states in the parser, and we can only do that based on what tokens
we've seen. Hopefully the rules will suffice.)
Valid identifiers with hyphens:
* {{select * from bigquery-public-data.foo}}
* {{insert into bigquery-public-data.foo values (1)}}
* {{delete from bigquery-public-data.foo}}
* {{select * from dept cross join bigquery-public-data.foo}}
Invalid identifiers with hyphens:
* {{select foo-bar from dept}} # column names cannot have hyphens
* select * from dept as foo-bar}} # table aliases cannot have hyphens
* {{select foo-bar.deptno from dept as `foo-bar`}} # ditto
* {{select * from foo.bar-baz.bump}} # only first part of table name
In all of the above 'invalid' cases, you can solve by enclosing the identifier
with back-ticks, e.g. {{select `foo-bar` from dept}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4238) Don't require each sub-project parser to have its own list of non-reserved keywords
Julian Hyde created CALCITE-4238:
Summary: Don't require each sub-project parser to have its own
list of non-reserved keywords
Key: CALCITE-4238
URL: https://issues.apache.org/jira/browse/CALCITE-4238
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Since CALCITE-2405 each sub-parser has its own list of non-reserved keywords.
This is good, because it allows the sub-parser to make keywords non-reserved
(e.g. in DRILL-1065 Drill wanted {{EXEC}} [to be
non-reserved|https://issues.apache.org/jira/browse/DRILL-1065?focusedCommentId=14903669=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14903669])
but it still requires each parser to have a list of non-reserved keywords (see
e.g. [the list in Babel
parser|https://github.com/apache/calcite/blob/eb4011bb4b9d962c2f8e6767cbee8c23435ef939/babel/src/main/codegen/config.fmpp#L37]).
And if we add a keyword to Calcite, we usually have to modify the list in each
sub-parser.
This proposal would move the list of default non-reserved keywords to a shared
config file. This would reduce the amount of code in sub-parsers, and remove
the need to edit them when we add a keyword to the base parser.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4230) In Babel for BigQuery, split quoted table names that contain dots
Julian Hyde created CALCITE-4230:
Summary: In Babel for BigQuery, split quoted table names that
contain dots
Key: CALCITE-4230
URL: https://issues.apache.org/jira/browse/CALCITE-4230
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In Babel for BigQuery, split quoted table names that contain dots. The
following three are equivalent:
{code}
select * from `bigquery-public-data`.`samples`.`natality`;
select * from `bigquery-public-data.samples.natality`;
select * from bigquery-public-data.samples.natality;
{code}
After this change, Babel will return the same parse tree for all 3 queries.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4220) In SqlToRelConverter, use RelBuilder for creating Aggregate
Julian Hyde created CALCITE-4220:
Summary: In SqlToRelConverter, use RelBuilder for creating
Aggregate
Key: CALCITE-4220
URL: https://issues.apache.org/jira/browse/CALCITE-4220
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In {{SqlToRelConverter}} use {{RelBuilder}} for creating {{Aggregate}}.
Currently we call LogicalAggregate.create directly, and that misses some
optimizations.
Add config option {{RelBuilder.Config.skipAggOnUniqueKey()}} to disable the
optimization that we don't create an {{Aggregate}} if the input is already
unique on the GROUP BY key. Without it, some tests in {{SqlToRelConverterTest}}
become trivial.
Also, that optimization is not valid if there are multiple group sets. (Because
we should output multiple rows, and Project only returns one.) So disable it in
that case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4190) OR simplification incorrectly loses term
Julian Hyde created CALCITE-4190:
Summary: OR simplification incorrectly loses term
Key: CALCITE-4190
URL: https://issues.apache.org/jira/browse/CALCITE-4190
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Simplification of OR expressions incorrectly loses a term. For example,
{code}(0 < a and a <= 10) or a is null or (8 < a and a < 12) or a >= 15{code}
is simplified to
{code}as is null or (0 < a and a <= 10) or (8 < a and a < 12){code}
(Moving {{a is null}} to the front is expected, but removing {{a >= 15}} is
wrong.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4186) Add ST_Covers, ST_CoveredBy, ST_Relate spatial functions
Julian Hyde created CALCITE-4186:
Summary: Add ST_Covers, ST_CoveredBy, ST_Relate spatial functions
Key: CALCITE-4186
URL: https://issues.apache.org/jira/browse/CALCITE-4186
Project: Calcite
Issue Type: Bug
Components: spatial
Reporter: Julian Hyde
Add {{ST_Covers}}, {{ST_CoveredBy}}, {{ST_Relate}} spatial functions.
There are [subtle differences between ST_Contains and
ST_Covers|http://lin-ear-th-inking.blogspot.com/2007/06/subtleties-of-ogc-covers-spatial.html];
{{ST_Covers}} is usually what you want, and is easier to compute.
Extend the Hilbert spatial index (added in CALCITE-1861) to handle
{{ST_Covers}}, {{ST_CoveredBy}}, {{ST_ContainsPropertly}} as it does
{{ST_DWithin}} and {{ST_Contains}}.
It might make sense to also implement {{ST_ExteriorRing}} so that we can
implement the [PostGIS example|https://postgis.net/docs/ST_Covers.html].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4173) Add internal SEARCH operator and Sarg literal, replacing use of IN in RexCall
Julian Hyde created CALCITE-4173:
Summary: Add internal SEARCH operator and Sarg literal, replacing
use of IN in RexCall
Key: CALCITE-4173
URL: https://issues.apache.org/jira/browse/CALCITE-4173
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add internal SEARCH operator and Sarg literal, replacing use of IN in RexCall.
Recently we started to allow IN in RexCalls to represent lists of constant
values; previously it had only been allowed in SqlCall. The use is confusing,
because a RexSubQuery is a sub-class of RexCall that may also use IN as its
operator.
More important, we would like to be able to represent more general search
arguments in a single RexCall. Examples:
* {{x BETWEEN 3 AND 10}}
* {{x > 3 AND x <= 10}}
* {{x NOT BETWEEN 3 AND 10}}
* {{x IS NULL OR x BETWEEN 3 AND 100 AND x != 50 OR x IN (200, 300, 400)}}
* {{x > 5 AND x < 10 AND x IN (3, 8, 10, 20)}}
All of these can be converted to sets of ranges, where each range has a lower
or upper bound (or both), bounds can be open or closed, plus a flag to say
whether NULL is an allowed value. Guava's RangeSet is an efficient
implementation of range sets.
This change would create a new class {{Sarg}} to represent a range set as a
literal. The new internal SEARCH operator tests whether an operand belongs to
the range set.
A RexCall to SEARCH is converted back to SQL, typically an IN or OR.
This change would obsolete the use of IN in {{RexCall}} to represent a fixed
list of values. (This is a breaking change, but when we first allowed IN in
{{RexCall}}, in CALCITE-2444, it was only intended to be for
{{RelToSqlConverter}}. Most people still believe that [IN is not
allowed|https://lists.apache.org/thread.html/e1c5d56ecca7c1bc3608344ceac9b209bb8100fbca1c1928feb9cce7%40%3Cdev.calcite.apache.org%3E].)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4170) Improve simplification of "<>" predicates
Julian Hyde created CALCITE-4170: Summary: Improve simplification of "<>" predicates Key: CALCITE-4170 URL: https://issues.apache.org/jira/browse/CALCITE-4170 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Improve simplification of "<>" (not equals) predicates. Today we can simplify "a = 1 or a <> 1" to "true" but we cannot simplify "a <> 1 or a = 1" to "true". The problem is this. The algorithm for simplifying "or" builds a up a list of predicates as it works left to right. After dealing with "a = 1" RexSimplifier has a predicate "a = 1" and therefore it can simplify the next predicate to anything that returns the right result over the possible inputs - which will not include a = 1 - and therefore 'a <> 1' can be simplified to 'true'. But today, we can't take that approach with "a <> 1 or a = 1" because "a <> 1" cannot be represented as a single range. It needs two ranges - less than 1 or greater than 1. The fix is therefore to allow predicates to be range sets not just ranges. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4154) Add a rule to merge a Project onto an Aggregate
Julian Hyde created CALCITE-4154:
Summary: Add a rule to merge a Project onto an Aggregate
Key: CALCITE-4154
URL: https://issues.apache.org/jira/browse/CALCITE-4154
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add a rule to merge a {{Project}} onto an {{Aggregate}}. In practice, it means
that aggregate functions that are not used in the Project are removed from the
Aggregate.
For example,
{code:java}
SELECT deptno, sum_sal
FROM (
SELECT deptno, job, SUM(sal) AS sum_sal, MIN(ename) AS min_ename
FROM emp
GROUP BY deptno, job)
{code}
becomes
{code:java}
SELECT deptno, sum_sal
FROM (
SELECT deptno, job, SUM(sal) AS sum_sal
FROM emp
GROUP BY deptno, job)
{code}
Note that {{min_ename}} is removed from the {{Aggregate}} (because it is not
used by the {{Project}}). {{job}} is not used, but it remains because it is a
{{GROUP BY}} key.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4137) Checkstyle should ensure that every class has a Javadoc comment
Julian Hyde created CALCITE-4137:
Summary: Checkstyle should ensure that every class has a Javadoc
comment
Key: CALCITE-4137
URL: https://issues.apache.org/jira/browse/CALCITE-4137
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Checkstyle should ensure that every class, interface, enum, package has a
Javadoc comment. And that javadoc types (e.g {{@param}} and {{@return}}) are
not empty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4128) Remove dependency of File adapter on Demo CSV adapter
Julian Hyde created CALCITE-4128: Summary: Remove dependency of File adapter on Demo CSV adapter Key: CALCITE-4128 URL: https://issues.apache.org/jira/browse/CALCITE-4128 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Remove dependency of File adapter on Demo CSV adapter. The dependency should be in the other direction. We want the File adapter to have lots of capabilities, and the Demo CSV adapter to be easy to understand. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4117) Release Calcite 1.24.0
Julian Hyde created CALCITE-4117: Summary: Release Calcite 1.24.0 Key: CALCITE-4117 URL: https://issues.apache.org/jira/browse/CALCITE-4117 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Assignee: Chunwei Lei Fix For: 1.24.0 Release Calcite 1.24.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4089) In Babel, allow 'CAST(integer AS DATE)' even though it is illegal in Calcite SQL
Julian Hyde created CALCITE-4089:
Summary: In Babel, allow 'CAST(integer AS DATE)' even though it is
illegal in Calcite SQL
Key: CALCITE-4089
URL: https://issues.apache.org/jira/browse/CALCITE-4089
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In Babel, allow 'CAST(integer AS DATE)', even though it is illegal in Calcite
SQL (and standard SQL).
Map it to valid standard SQL. BigQuery {{CAST(n AS DATE)}} is equivalent to
Calcite {{DATE '1970-01-01' + n * INTERVAL '1' DAY}}.
Also, add functions {{DATE_FROM_UNIX_DATE(INTEGER)}},
{{UNIX_SECONDS(TIMESTAMP)}}, {{UNIX_MILLIS(TIMESTAMP)}},
{{UNIX_MICROS(TIMESTAMP)}}, {{TIMESTAMP_SECONDS(BIGINT)}},
{{TIMESTAMP_MILLIS(BIGINT)}}, {{TIMESTAMP_MICROS(BIGINT)}} in the BigQuery
library. They are well-named and well-specified, so they might be useful to
people working in dialects besides BigQuery. {{DATE_FROM_UNIX_DATE(0)}} should
return {{DATE '1970-01-01'}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4087) Hoist, a utility to replace literals in a SQL string with placeholders
Julian Hyde created CALCITE-4087:
Summary: Hoist, a utility to replace literals in a SQL string with
placeholders
Key: CALCITE-4087
URL: https://issues.apache.org/jira/browse/CALCITE-4087
Project: Calcite
Issue Type: New Feature
Components: babel
Affects Versions: 1.23.0
Reporter: Julian Hyde
Add "Hoist", a utility to replace literals in a SQL string with placeholders.
By default it uses Calcite's parser, but you can configure to use different
quoting, casing, and also to use a different parser such as Babel for other
dialects of SQL.
It generates a data structure, {{Hoisted}}, that contains a map of where the
literals occur within the SQL string. The Hoisted.substitute method generates
an alternative SQL string, replacing the literals with a custom string.
It does not call {{SqlNode.unparse}}, and therefore does not lose any of the
original formatting.
For more examples, see tests added to {{SqlParserTest}} and {{BabelParserTest}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4072) Avatica build fails with various license errors with Guava 22.0 or higher
Julian Hyde created CALCITE-4072:
Summary: Avatica build fails with various license errors with
Guava 22.0 or higher
Key: CALCITE-4072
URL: https://issues.apache.org/jira/browse/CALCITE-4072
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
On Guava 22.0 or higher, Avatica build gives various license errors:
{noformat}
> Task :tck:getLicenses FAILED
Build calcite-avatica FAILURE reason:
Execution failed for task ':tck:getLicenses':
LICENSE-like files are missing
==
MIT
* org.codehaus.mojo:animal-sniffer-annotations:1.14
at
com.github.vlsi.gradle.license.GatherLicenseTask.run(GatherLicenseTask.kt:417)
at
org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
at
org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:49)
{noformat}
JDK does not seem to be a factor: occurs on JDK 8, 11, 14.
The workaround is to disable various license-checking tasks on the command-line:
{noformat}
./gradlew clean build -x :tck:getLicenses -x :tck:license -x
:tck:licenseCopySpec -x :standalone-server:getLicenses -x
:standalone-server:license -x :standalone-server:licenseCopySpec
{noformat}
I had expected to be able to change Guava version using
{{-Dguava.version=22.0}} on the gradle command line. But I had to edit
{{gradle.properties}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4061) Build should fail if Calcite code uses deprecated APIs
Julian Hyde created CALCITE-4061: Summary: Build should fail if Calcite code uses deprecated APIs Key: CALCITE-4061 URL: https://issues.apache.org/jira/browse/CALCITE-4061 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Calcite build should fail if Calcite code uses deprecated APIs. This includes external APIs (e.g. Guava methods) and internal APIs (e.g. a deprecated class in the org.apache.calcite.util package). This used to occur when the build was powered by Maven. Compared to a policy where code is allowed to use deprecated APIs for a 'grace period', this policy has a number of advantages. One is that Calcite devs (or users) can upgrade dependencies at short notice, with no code changes. Another is that it forces people who are doing internal refactoring (replacing one internal Calcite API with another) to finish the job. [~vlsi] Can you take this please? I can't figure out how to set javac's '-Xlint:deprecated -Werror' from Gradle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4038) Refactor RexVisitor, RexBiVisitor, RelOptUtil.InputFinder
Julian Hyde created CALCITE-4038:
Summary: Refactor RexVisitor, RexBiVisitor, RelOptUtil.InputFinder
Key: CALCITE-4038
URL: https://issues.apache.org/jira/browse/CALCITE-4038
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Various improvements to {{RexVisitor}}, {{RexBiVisitor}},
{{RelOptUtil.InputFinder}} classes.
1. Make mutable field {{RelOptUtil.InputFinder.inputBitSet}} private. The field
still exists, public and deprecated, but it shadows a new private field. Rather
than calling {{InputFinder.inputBitSet.build()}} you should now call
{{InputFinder.build()}}.
2. In {{RexVisitor}} and {{RexBiVisitor}} add methods {{visitList}} and
{{visitEach}} (the former returns a list, and the latter returns void).
3. Deprecate {{RexShuttle.apply}} in favor of {{visitList}}.
4. Add {{RexWindowBound.accept(RexBiVisitor)}}.
5. For {{RexBiVisitor}} add a method {{visitEachIndexed}}, the
index passed as the 'payload' argument.
6. Add abstract implementations of {{RexBiVisitor}}: {{RexUnaryBiVisitor}} and
{{RexBiVisitorImpl}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3967) Publish components "calcite" and "calcite-examples"
Julian Hyde created CALCITE-3967: Summary: Publish components "calcite" and "calcite-examples" Key: CALCITE-3967 URL: https://issues.apache.org/jira/browse/CALCITE-3967 Project: Calcite Issue Type: Bug Components: build Affects Versions: 1.22.0 Reporter: Julian Hyde Until release 1.21 we published components "calcite" and "calcite-examples" as part of the release. In 1.22 these components were not published. I suspect that they were left out when we migrated from Maven to Gradle. One consequence is that if you [search Maven Central for 'apache calcite'|https://search.maven.org/search?q=apache%20calcite] the latest version appears to be 1.21. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3923) Refactor how planner rules are parameterized
Julian Hyde created CALCITE-3923:
Summary: Refactor how planner rules are parameterized
Key: CALCITE-3923
URL: https://issues.apache.org/jira/browse/CALCITE-3923
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
People often want different variants of planner rules. An example is
{{FilterJoinRule}}, which has a 'boolean smart’ parameter, a predicate (which
returns whether to pull up filter conditions), operands (which determine the
precise sub-classes of {{RelNode}} that the rule should match) and a
{{RelBuilderFactory}} (which controls the type of {{RelNode}} created by this
rule).
Suppose you have an instance of {{FilterJoinRule}} and you want to change
{{smart}} from true to false. The {{smart}} parameter is immutable (good!) but
you can’t easily create a clone of the rule because you don’t know the values
of the other parameters. Your instance might even be (unbeknownst to you) a
sub-class with extra parameters and a private constructor.
So, my proposal is to put all of the config information of a {{RelOptRule}}
into a single {{config}} parameter that contains all relevant properties. Each
sub-class of {{RelOptRule}} would have one constructor with just a ‘config’
parameter. Each config knows which sub-class of {{RelOptRule}} to create.
Therefore it is easy to copy a config, change one or more properties, and
create a new rule instance.
Adding a property to a rule’s config does not require us to add or deprecate
any constructors.
The operands are part of the config, so if you have a rule that matches a
{{EnumerableFilter}} on an {{EnumerableJoin}} and you want to make it match an
{{EnumerableFilter}} on an {{EnumerableNestedLoopJoin}}, you can easily create
one with one changed operand.
The config is immutable and self-describing, so we can use it to automatically
generate a unique description for each rule instance.
(See the email thread [[DISCUSS] Refactor how planner rules are
parameterized|https://lists.apache.org/thread.html/rfdf6f9b7821988bdd92b0377e3d293443a6376f4773c4c658c891cf9%40%3Cdev.calcite.apache.org%3E].)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3897) Windowed aggregate with bracket (ROWS or RANGE) without ORDER BY is invalid
Julian Hyde created CALCITE-3897:
Summary: Windowed aggregate with bracket (ROWS or RANGE) without
ORDER BY is invalid
Key: CALCITE-3897
URL: https://issues.apache.org/jira/browse/CALCITE-3897
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
I believe that windowed aggregate with bracket (ROWS or RANGE) without ORDER BY
is invalid. If so, we should throw.
[~danny0405]
[reports|https://github.com/apache/calcite/pull/1896#discussion_r403420946]
that the syntax is invalid on MSSQL 2017.
Someone should check the SQL standard. If it is invalid in the SQL standard
(but only if) the validator should give an error.
I am a little ambivalent. A missing {{ORDER BY}} is basically sorting by zero
columns, an of course it's not deterministic, but even with {{ORDER BY}}, the
ordering may not be deterministic (e.g. apply {{ORDER BY deptno}} to the
{{Emp}} table). So we should go with the SQL standard.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3892) Make junit test classes and methods non-public
Julian Hyde created CALCITE-3892: Summary: Make junit test classes and methods non-public Key: CALCITE-3892 URL: https://issues.apache.org/jira/browse/CALCITE-3892 Project: Calcite Issue Type: Bug Reporter: Julian Hyde In junit 4 and earlier versions, junit classes and methods must be public. In junit 5 (jupiter) this is no longer the case. This change makes classes and methods package-protected where possible (i.e. removes the "public" modifier). It makes tests a bit more concise, and gives us more freedom to refactor tests in future. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3841) Change downloads page to use downloads.apache.org
Julian Hyde created CALCITE-3841:
Summary: Change downloads page to use downloads.apache.org
Key: CALCITE-3841
URL: https://issues.apache.org/jira/browse/CALCITE-3841
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Infra has
[decided|https://lists.apache.org/thread.html/rcd2864e75e417597d342b8eb83080eb2d7a0cafea84fd4521a4d9cfd%40%3Cusers.infra.apache.org%3E]
(login required for that email link) to deprecate
[www.apache.org/dist|https://www.apache.org/dist] and move downloads to
[https://downloads.apache.org|https://downloads.apache.org].
On [Calcite's downloads page|https://calcite.apache.org/downloads/], we need to
change the 'digest' link from (for example)
{{https://www.apache.org/dist/calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz.sha256}}
to
{{https://downloads.apache.org/calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz.sha256}},
and similarly the 'gpg' link.
I believe that the 'tar' link can remain as
{{https://www.apache.org/dyn/closer.lua?filename=calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz=download}}
for the latest release and
{{https://archive.apache.org/dist/calcite/apache-calcite-1.20.0/apache-calcite-1.20.0-src.tar.gz}}
for older releases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3823) Do not use String.replaceAll
Julian Hyde created CALCITE-3823:
Summary: Do not use String.replaceAll
Key: CALCITE-3823
URL: https://issues.apache.org/jira/browse/CALCITE-3823
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
We use JDK's {{String.replaceAll(String regex, String replacement)}} about 60
times in the code, but most of these should probably use
{{String.replace(CharSequence target, CharSequence replacement)}}.
{{replaceAll}} uses regex semantics, which is slow (it compiles a pattern each
call) and probably not desired behavior.
Consider adding to {{forbidden-apis/signatures.txt}} to prevent it from being
accidentally used in future.
See tweet "[String.replaceAll is the most invisible java perf issue I end up
finding out of
profiles|https://twitter.com/t3rmin4t0r/status/1232353433371336704];.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3822) Source distribution must not contain fonts under SIL OFL 1.1 license (category B)
Julian Hyde created CALCITE-3822: Summary: Source distribution must not contain fonts under SIL OFL 1.1 license (category B) Key: CALCITE-3822 URL: https://issues.apache.org/jira/browse/CALCITE-3822 Project: Calcite Issue Type: Bug Reporter: Julian Hyde The source distribution contains "font-awesome:fonts" which is licensed as http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License). OFL 1.1 is "category B" (see LEGAL-112). We need to remove "web and desktop font files", probably under site/fonts. Code and icons are OK. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3815) Add missing SQL standard aggregate functions: EVERY, SOME, INTERSECTION
Julian Hyde created CALCITE-3815:
Summary: Add missing SQL standard aggregate functions: EVERY,
SOME, INTERSECTION
Key: CALCITE-3815
URL: https://issues.apache.org/jira/browse/CALCITE-3815
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add missing SQL standard aggregate functions: EVERY, SOME, INTERSECTION.
Examples:
{code}
SELECT deptno,
EVERY(sal < 3000) AS es, SOME(sal < 1000) AS ss,
EVERY(comm < 500) AS ec, SOME(comm < 500) AS sc
FROM emp
GROUP BY deptno;
++---+---+---+--+
| DEPTNO | ES | SS | EC | SC |
++---+---+---+--+
| 10 | FALSE | FALSE | | |
| 20 | FALSE | TRUE | | |
| 30 | TRUE | TRUE | FALSE | TRUE |
++---+---+---+--+
{code}
{{EVERY}} and {{SOME}} can be implemented by translating to {{MIN}}, {{MAX}}:
* {{EVERY(condition)}} is equivalent to {{MIN(condition)}};
* {{SOME(condition)}} is equivalent to {{MAX(condition)}}
where {{condition}} is a {{BOOLEAN}} expression, possibly allowing {{NULL}}
values).
{{INTERSECTION}} computes the intersection of collections (arrays and
multisets). (Compare to {{FUSION}}, which computes the union of collections.)
{{FUSION}} is in the operator table but there is no code-generation for it.
This task should implement {{FUSION}} and {{INTERSECTION}} so that we can run
queries in Enumerable mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3814) Support JDK 14
Julian Hyde created CALCITE-3814:
Summary: Support JDK 14
Key: CALCITE-3814
URL: https://issues.apache.org/jira/browse/CALCITE-3814
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
JDK 14 will be released on 2020/03/17.
I tried building Calcite on an early access build and got an error from Gradle:
{noformat}
$ java -version
openjdk version "14" 2020-03-17
OpenJDK Runtime Environment (build 14+36-1461)
OpenJDK 64-Bit Server VM (build 14+36-1461, mixed mode, sharing)
$ ./gradlew build
Starting a Gradle Daemon, 1 busy and 1 incompatible Daemons could not be
reused, use --status for details
java.lang.NoClassDefFoundError: Could not initialize class
org.codehaus.groovy.vmplugin.v7.Java7
at
org.codehaus.groovy.vmplugin.VMPluginFactory.(VMPluginFactory.java:43)
at
org.codehaus.groovy.reflection.GroovyClassValueFactory.(GroovyClassValueFactory.java:35)
at org.codehaus.groovy.reflection.ClassInfo.(ClassInfo.java:107)
...
$ ./gradlew clean --stacktrace
FAILURE: Build failed with an exception.
* What went wrong:
Could not initialize class org.codehaus.groovy.runtime.InvokerHelper
* Try:
Run with --info or --debug option to get more log output. Run with --scan to
get full insights.
* Exception is:
java.lang.NoClassDefFoundError: Could not initialize class
org.codehaus.groovy.runtime.InvokerHelper
at
org.gradle.internal.extensibility.DefaultExtraPropertiesExtension.(DefaultExtraPropertiesExtension.java:29)
at
org.gradle.internal.extensibility.DefaultConvention.(DefaultConvention.java:49)
at
org.gradle.internal.extensibility.ExtensibleDynamicObject.(ExtensibleDynamicObject.java:60)
...
{noformat}
Problem appears to be https://github.com/gradle/gradle/issues/10248, which is
related to (caused by) GROOVY-9211, and alrso related to NETBEANS-3876.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3811) JDBC adapter generates SQL with invalid field names if Filter's row type is different from its input
Julian Hyde created CALCITE-3811:
Summary: JDBC adapter generates SQL with invalid field names if
Filter's row type is different from its input
Key: CALCITE-3811
URL: https://issues.apache.org/jira/browse/CALCITE-3811
Project: Calcite
Issue Type: Bug
Components: jdbc-adapter
Affects Versions: 1.21.0
Reporter: Julian Hyde
Fix For: 1.22.0
JDBC adapter generates SQL with invalid field names if a {{Filter}} object's
row type is different from its input. Usually a {{Filter}} object's row type is
identical to its input's row type (because that is how it derives its row type
on construction), but if you call {{RelNode.replaceInput}} after construction
it is possible to make the field names inconsistent.
Here is an example of the wrong SQL generated:
{code}
SELECT D AS `emps.deptno`
FROM (
SELECT DEPTNO AS D2, COUNT(*) AS `emps.count`
FROM `scott`.EMP
GROUP BY DEPTNO
HAVING COUNT(*) < 2) AS `t1`
{code}
Note that the sub-query returns a column {{D2}} but the outer {{SELECT}}
expression refers to {{D}}, because that was the name in the row type of the
{{Filter}} (which became the {{HAVING}} clause in the generated SQL).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3774) Add option to RelBuilder to prevent it from merging projects
Julian Hyde created CALCITE-3774:
Summary: Add option to RelBuilder to prevent it from merging
projects
Key: CALCITE-3774
URL: https://issues.apache.org/jira/browse/CALCITE-3774
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add option to {{RelBuilder}} to prevent it from merging projects. Currently if
you call {{RelBuilder.project}} and the input is a {{Project}} it will merge
the expressions. This is usually a good idea, but sometimes it creates very
complex expressions.
There is an existing method {{RelBuilder.shouldMergeProject()}} but by default
it returns true, and in order to change it you have to sub-class
{{RelBuilder}}, which is not easy to do.
I propose to add a property {{RelBuilder.Config.mergeBloat}}, default 0, which
would prevent creating a project that has more complexity than the two projects
that went into it.
Example 1:
* Input 1: {{Project(a+b+c+d AS w, b+c+d+e AS x, c+d+e+f AS y, d+e+f+g AS z)}}
followed by
* Input 2: {{Project(w*x AS p, x*y AS q, y*z AS r)}} creates
* Output: {{Project((a+b+c+d) * (b+c+d+e) AS p, (b+c+d+e) * (c+d+e+f) AS q,
(c+d+e+f) * (d+e+f+g) AS r)}}.
The expression "a+b+c+d" has complexity 7 (4 fields and 3 calls). The inputs
have complexity 28 (4 expressions, each of complexity 7) and 9 (3 expressions,
each with complexity 3). The output has complexity 45 (3 expressions, each with
complexity 15 (8 fields and 7 calls)). 45 is greater than 37 (28 + 9), so this
merge would not be allowed.
Example 2:
* Input 1: {{Project(a+b+c+d AS w, b+c+d+e AS x, c+d+e+f AS y, d+e+f+g AS z)}}
(complexity 28)
* Input 2: {{Project(w*x AS p, x*y AS q)}} (complexity 6) creates
* Output: {{Project((a+b+c+d) * (b+c+d+e) AS p, (b+c+d+e) * (c+d+e+f) AS q)}}
(complexity 30).
Output complexity 30 is less than input complexity 34 (28 + 6), and therefore
the merge is allowed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3764) AggregateCaseToFilterRule handles NULL values correctly
Julian Hyde created CALCITE-3764:
Summary: AggregateCaseToFilterRule handles NULL values correctly
Key: CALCITE-3764
URL: https://issues.apache.org/jira/browse/CALCITE-3764
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
{{AggregateCaseToFilterRule}} handles NULL values correctly. It converts
{code:sql}
SELECT COUNT(CASE WHEN b THEN NULL ELSE 1 END) FROM t
{code}
to
{code:sql}
SELECT COUNT(*) FILTER (WHERE b IS FALSE) FROM t
{code}
which fails to count rows where {{b}} is UNKNOWN, so it should convert to
{code:sql}
SELECT COUNT(*) FILTER (WHERE b IS NOT TRUE) FROM t
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3763) RelBuilder.aggregate should prune unused fields from the input, if the input is a Project
Julian Hyde created CALCITE-3763:
Summary: RelBuilder.aggregate should prune unused fields from the
input, if the input is a Project
Key: CALCITE-3763
URL: https://issues.apache.org/jira/browse/CALCITE-3763
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
{{RelBuilder.aggregate}} should prune unused fields from the input, if the
input is a {{Project}}.
Pruning fields during the planning process is desirable, but often cannot do it
- we are applying a {{RelOptRule}} that has to return the same fields, or we
don't want to add an extra Project do so the pruning. But when we are in
{{RelBuilder.aggregate}} and the input is a Project, neither of those
limitations apply. We already have a Project, we are just making it narrower;
and we know what fields the {{Aggregate}} will produce.
For example,
{code:sql}
SELECT deptno, SUM(sal) FILTER (WHERE b)
FROM (
SELECT deptno, empno + 10, sal, job = 'CLERK' AS b
FROM emp)
GROUP BY deptno
{code}
becomes
{code:sql}
SELECT deptno, SUM(sal) FILTER (WHERE b)
FROM (
SELECT deptno, sal, job = 'CLERK' AS b
FROM emp)
GROUP BY deptno
{code}
If there are no fields used, remove the {{Project}}. (A {{RelNode}} with no
fields is not allowed.)
{code:sql}
SELECT COUNT(*) AS C
FROM (
SELECT deptno, empno + 10, sal, job = 'CLERK' AS b
FROM emp)
{code}
becomes
{code:sql}
SELECT COUNT(*) AS c
FROM emp
{code}
Add an option {{RelBuilder.Config.pruneInputOfAggregate}}, default true, so
that people can disable this rewrite if it causes problems.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3752) PIVOT and UNPIVOT
Julian Hyde created CALCITE-3752:
Summary: PIVOT and UNPIVOT
Key: CALCITE-3752
URL: https://issues.apache.org/jira/browse/CALCITE-3752
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Oracle SQL has PIVOT and UNPIVOT operators for cross-tab support.
For
[example|https://oracle-base.com/articles/11g/pivot-and-unpivot-operators-11gr1],
{noformat}
SELECT *
FROM (SELECT customer_id, product_code, quantity
FROM pivot_test)
PIVOT (SUM(quantity) AS sum_quantity FOR (product_code) IN ('A' AS a, 'B' AS
b, 'C' AS c))
ORDER BY customer_id;
CUSTOMER_ID A_SUM_QUANTITY B_SUM_QUANTITY C_SUM_QUANTITY
--- -- -- --
1 10 20 30
2 4050
3 60 70 80
4100
4 rows selected.
{noformat}
In Calcite we could implement this as a prepare-time rewrite, something like
this:
{noformat}
SELECT customer_id,
SUM(DECODE(product_code, 'A', quantity, 0)) AS a_sum_quantity,
SUM(DECODE(product_code, 'B', quantity, 0)) AS b_sum_quantity,
SUM(DECODE(product_code, 'C', quantity, 0)) AS c_sum_quantity
FROM pivot_test
GROUP BY customer_id
ORDER BY customer_id;
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
