[jira] [Created] (CALCITE-2980) Implement the FORMAT clause of the CAST operator (added in SQL:2016)
Julian Hyde created CALCITE-2980:
Summary: Implement the FORMAT clause of the CAST operator (added
in SQL:2016)
Key: CALCITE-2980
URL: https://issues.apache.org/jira/browse/CALCITE-2980
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
SQL:2016 adds an optional {{FORMAT format}} clause to the {{CAST}} operator. It
is a standard way to do what functions like {{TO_DATE}}, {{TO_NUMBER}},
{{TO_CHAR}}, {{TO_TIMESTAMP}} have done in an ad hoc way (and with differing
specifications among databases).
Here is an example:
{code:java}
cast('01-05-2017' as date format 'DD-MM-')
{code}
The following paragraphs are copied from IMPALA-4018, which describes
implementing this in Impala. (That case also describes cases where the
implementations of {{TO_TIMESTAMP}} etc. in Hive, Impala, Oracle and PostgreSQL
are not consistent with each other. We should take note as we implement these
functions in Calcite.)
SQL:2016 defines the following datetime templates
{noformat}
::=
{ }...
::=
|
::=
|
|
|
|
|
|
|
|
|
|
|
|
|
::=
|
|
|
|
|
|
|
::=
| YYY | YY | Y
::=
| RR
::=
MM
::=
DD
::=
DDD
::=
HH | HH12
::=
HH24
::=
MI
::=
SS
::=
S
::=
FF1 | FF2 | FF3 | FF4 | FF5 | FF6 | FF7 | FF8 | FF9
::=
A.M. | P.M.
::=
TZH
::=
TZM
{noformat}
SQL:2016 also introduced the {{FORMAT}} clause for {{CAST}} which is the
standard way to do string <> datetime conversions
{noformat}
::=
CAST
AS
[ FORMAT ]
::=
|
::=
|
::=
{noformat}
For example:
{noformat}
CAST( AS [FORMAT ])
CAST( AS [FORMAT ])
cast(dt as string format 'DD-MM-')
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3022) Babel: Various SQL parsing issues
Julian Hyde created CALCITE-3022: Summary: Babel: Various SQL parsing issues Key: CALCITE-3022 URL: https://issues.apache.org/jira/browse/CALCITE-3022 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Various parsing issues in Babel SQL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-3047) In JDBC adapter, expose multiple schemas of the back-end database
Julian Hyde created CALCITE-3047: Summary: In JDBC adapter, expose multiple schemas of the back-end database Key: CALCITE-3047 URL: https://issues.apache.org/jira/browse/CALCITE-3047 Project: Calcite Issue Type: Bug Reporter: Julian Hyde In JDBC adapter, expose multiple schemas of the back-end database. We do this by adding JdbcCatalogSchema, an implementation of Schema that reads the back-end database catalog and for each schema, creates a sub-schema that is a JdbcSchema. Also add an experimental way to create a Calcite connection with a given schema as its root schema (in this case, a JdbcCatalogSchema) rather than the usual AbstractSchema. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-3048) Improve how JDBC adapter deduces current schema on Redshift
Julian Hyde created CALCITE-3048:
Summary: Improve how JDBC adapter deduces current schema on
Redshift
Key: CALCITE-3048
URL: https://issues.apache.org/jira/browse/CALCITE-3048
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Improve how JDBC adapter deduces current schema on Amazon Redshift or
PostgreSQL. Until now, we either look at the schema connect-string parameter,
or call {{Connection.getSchema()}} (on JDBC 4.1 and above). But in Redshift's
JDBC driver, {{Connection.getSchema()}} always returns null.
Now, if we know we are running against Redshift or PostgreSQL, we will execute
a {{select current_schema()}} query to find the current schema.
Similarly for {{Connection.getCatalog()}} and {{select current_database()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3049) When simplifying expressions, revisit "IS NULL" if its argument has been simplified
Julian Hyde created CALCITE-3049:
Summary: When simplifying expressions, revisit "IS NULL" if its
argument has been simplified
Key: CALCITE-3049
URL: https://issues.apache.org/jira/browse/CALCITE-3049
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
When simplifying expressions, revisit "IS NULL" if its argument has been
simplified. For example, we currently simplify {code}(CASE WHEN FALSE THEN
+(v0) ELSE -1 END) IS UNKNOWN{code} to {code}-1 IS UNKNOWN{code} but we should
further simplify that to {{FALSE}}.
I have a preliminary dev branch, but it needs a little more debugging. I'd be
grateful if someone could finish it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3050) Integrate SqlDialect and SqlParser.Config
Julian Hyde created CALCITE-3050:
Summary: Integrate SqlDialect and SqlParser.Config
Key: CALCITE-3050
URL: https://issues.apache.org/jira/browse/CALCITE-3050
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
{{SqlDialect}} is used by the JDBC adapter to generate SQL in the target
dialect of a data source. {{SqlParser.Config}} is used to set what the parser
should allow for SQL statements sent to Calcite. But they both are a
representation of "dialect". And they come together when we want to use a Babel
parser to understand SQL statements that are meant for a data source.
So it makes sense to integrate them, somehow. We could add a method {code}void
SqlParser.ConfigBuilder.setFrom(SqlDialect dialect){code} or do it from the
other end, {code}SqlDialect.configureParser(SqlParser.ConfigBuilder
configBuilder){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3053) Add a test to ensure that all functions are documented in the SQL reference
Julian Hyde created CALCITE-3053:
Summary: Add a test to ensure that all functions are documented in
the SQL reference
Key: CALCITE-3053
URL: https://issues.apache.org/jira/browse/CALCITE-3053
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Assignee: Julian Hyde
Add a test to ensure that all functions are documented in the [SQL
reference|https://calcite.apache.org/docs/reference.html].
The test would would be called {{DocumentationTest}}, iterate through all
functions and operators in {{SqlStdOperatorTable}} and
{{OracleSqlOperatorTable}} and make sure that there is a line "| functionName
... |" in {{reference.md}}.
There are sure to be exceptions. We can exceptions to a comment inside
{{reference.md}} that is seen by the test but does not appear in the generated
web page.
I'll do this after CALCITE-2846, and make sure that Oracle- and MySQL-specific
functions are flagged as such.
After CALCITE-3016 we will need to refactor the test a little.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3096) In RelBuilder, make alias method idempotent
Julian Hyde created CALCITE-3096:
Summary: In RelBuilder, make alias method idempotent
Key: CALCITE-3096
URL: https://issues.apache.org/jira/browse/CALCITE-3096
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In {{RelBuilder}}, make the {{alias}} method idempotent. If an expression's
alias is already 'foo', then calling {{alias(e, "foo")}} will return the
expression unchanged.
The intent of this change is to reduce the number of {{if}} statements that are
required in client code that is trying to produce a concise {{RelNode}} tree.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3098) Upgrade SQLLine to 1.8.0
Julian Hyde created CALCITE-3098: Summary: Upgrade SQLLine to 1.8.0 Key: CALCITE-3098 URL: https://issues.apache.org/jira/browse/CALCITE-3098 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Upgrade SQLLine to 1.8.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-3102) Deprecation warnings following [CALCITE-2969]
Julian Hyde created CALCITE-3102: Summary: Deprecation warnings following [CALCITE-2969] Key: CALCITE-3102 URL: https://issues.apache.org/jira/browse/CALCITE-3102 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Fix For: 1.20.0 Following CALCITE-2969 there are a couple of dozen deprecation warnings. Our policy is to tolerate no deprecation warnings. So we either need to stop using deprecated APIs or silence the warnings. This is a blocker for 1.20. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-3123) In RelBuilder, eliminate duplicate aggregate calls
Julian Hyde created CALCITE-3123:
Summary: In RelBuilder, eliminate duplicate aggregate calls
Key: CALCITE-3123
URL: https://issues.apache.org/jira/browse/CALCITE-3123
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In {{RelBuilder}}, eliminate duplicate aggregate calls. For example, if you
call {{RelBuilder.aggregate}} with {{sum($1)}}, {{sum($2)}}, {{count($1)}},
{{sum($1)}} then it would create an {{Aggregate}} with 3 aggregate calls
(because {{sum($1)}} occurs twice), then add a {{Project [$1, $2, $3, $1]}}.
The {{Project}} is necessary so that you get the number of output fields that
you are expecting.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3131) In LatticeSuggester, record whether columns are used as "dimensions" or "measures"
Julian Hyde created CALCITE-3131:
Summary: In LatticeSuggester, record whether columns are used as
"dimensions" or "measures"
Key: CALCITE-3131
URL: https://issues.apache.org/jira/browse/CALCITE-3131
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Fix For: 1.21.0
In {{LatticeSuggester}}, record whether columns are used as "dimensions" or
"measures" in the query set. And add a method
{{Lattice.isAlwaysMeasure(Column)}}, populated from these use counts.
To understand the difference between dimensions and measures, consider the SQL
queries
{code}
select deptno,
sum(salary + commission) as remuneration,
sum(salary) as salary
from Emp
group by first_name || ' ' || last_name, deptno;
select min(first_name || ' ' || last_name)
from Emp
where deptno = 10;
{code}
There are two expressions:
* {{salary + commission}} is only used as an argument two an aggregate
function, i.e. a measure.
* {{first_name || ' ' || last_name}} is used as a dimension in the first query
(in {{GROUP BY}}, but in {{WHERE}} or {{SELECT}} or {{ORDER BY}} would also
count), and as a measure in the second.
When applied to a collection of queries, {{LatticeSuggester}} should record
each individual use of a column or derived column (expression), and whether it
is a usage as a measure. Thus {{salary + commission}} has [true], and
{{first_name || ' ' || last_name}} has [false, true].
In future we might store further information about uses. For example, if a
column is used with aliases "expr$1", "profit", "profit" then we could go with
the majority decision and call it "profit". Currently we go with the alias the
first time it is used.
When the {{LatticeSuggester}} builds a {{Lattice}} it passes on the use counts,
and these power a new method {{Lattice.isAlwaysMeasure(Column)}}. A {{Lattice}}
that is made directly, not induced from a set of queries, will have an empty
use map. (It is a minor burden for these lattices, but it is outweighed by the
convenience when an existing lattice needs to be merged with a new set of
queries in an incremental {{LatticeSuggester}} run.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3144) Add rule, CaseFilteredAggregatorRule, that converts "SUM(CASE WHEN b THEN x END)" to "SUM(x) FILTER (WHERE b)"
Julian Hyde created CALCITE-3144:
Summary: Add rule, CaseFilteredAggregatorRule, that converts
"SUM(CASE WHEN b THEN x END)" to "SUM(x) FILTER (WHERE b)"
Key: CALCITE-3144
URL: https://issues.apache.org/jira/browse/CALCITE-3144
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add a rule that converts "SUM(CASE WHEN b THEN x END)" to "SUM(x) FILTER (WHERE
b)".
Druid added {{CaseFilteredAggregatorRule}} in
https://github.com/apache/incubator-druid/pull/4360.
Maybe {{AggregateCaseToFilterRule}} is a slightly better name. Or maybe this
transform could be done in {{RelBuilder.aggregate}}, and we wouldn't need a
rule.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3166) Make RelBuilder configurable
Julian Hyde created CALCITE-3166:
Summary: Make RelBuilder configurable
Key: CALCITE-3166
URL: https://issues.apache.org/jira/browse/CALCITE-3166
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Make {{RelBuilder}} configurable, so that particular optimizations can easily
be turned off.
I propose to add a class {{RelBuilder.Config}}, which is immutable and has a
public final field for each configuration property; also a class
{{RelBuilder.ConfigBuilder}} to create a config.
{{RelBuilder.create(FrameworkConfig frameworkConfig)}} will get a config by
calling {{frameworkConfig.getContext().unwrap(RelBuilder.Config.class)}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3191) In JDBC adapter, generate implement Values by generating SELECT without FROM
Julian Hyde created CALCITE-3191:
Summary: In JDBC adapter, generate implement Values by generating
SELECT without FROM
Key: CALCITE-3191
URL: https://issues.apache.org/jira/browse/CALCITE-3191
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In JDBC adapter, generate implement {{Values}} relational operator by
generating {{SELECT}} without a {{FROM}} clause. (For most dialects we generate
{{VALUES}}, but MySQL does not support VALUES in a query.)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (CALCITE-3196) n Frameworks, add BasePrepareAction (a functional interface) and deprecate PrepareAction
Julian Hyde created CALCITE-3196:
Summary: n Frameworks, add BasePrepareAction (a functional
interface) and deprecate PrepareAction
Key: CALCITE-3196
URL: https://issues.apache.org/jira/browse/CALCITE-3196
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In {{Frameworks}}, add {{interface BasePrepareAction}} (a functional interface)
and deprecate {{abstract class PrepareAction}}. Because {{PrepareAction}} has a
field ({{FrameworkConfig config}}), it cannot be implemented using a lambda. It
is simpler and clearer to pass {{config}} as an argument to all methods where
it is needed.
{{PrepareAction}} was introduced in CALCITE-247.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (CALCITE-3234) For boolean properties, empty string should mean "true"
Julian Hyde created CALCITE-3234: Summary: For boolean properties, empty string should mean "true" Key: CALCITE-3234 URL: https://issues.apache.org/jira/browse/CALCITE-3234 Project: Calcite Issue Type: Bug Reporter: Julian Hyde For boolean properties, empty string should mean "true". For example, if I add -Dcalcite.test.slow to the JVM args, CalciteSystemProperty.TEST_SLOW should have value "true". This was broken by CALCITE-2859 (Util.getBooleanProperty, which was removed, had the desired behavior). -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (CALCITE-3240) Javadoc gives internal error "com.sun.tools.javac.code.ClassFinder$BadClassFile: bad class file: linq4j/target/classes/org/apache/calcite/linq4j/Ord$2.class [ERROR] u
Julian Hyde created CALCITE-3240:
Summary: Javadoc gives internal error
"com.sun.tools.javac.code.ClassFinder$BadClassFile: bad class file:
linq4j/target/classes/org/apache/calcite/linq4j/Ord$2.class [ERROR]
undeclared type variable: E"
Key: CALCITE-3240
URL: https://issues.apache.org/jira/browse/CALCITE-3240
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Javadoc gives an internal error on JDK 9 (version 9.0.4) and JDK 10 (version
10.0.2):
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:test-javadoc (default-cli)
on project calcite-linq4j: An error has occurred in Test Javadoc report
generation:
[ERROR] Exit code: 1 - javadoc: error - An internal exception has occurred.
[ERROR] (com.sun.tools.javac.code.ClassFinder$BadClassFile: bad class
file:
/home/jhyde/open1/calcite.2/linq4j/target/classes/org/apache/calcite/linq4j/Ord$2.class
[ERROR] undeclared type variable: E
[ERROR] Please remove or make sure it appears in the correct subdirectory of
the classpath.)
[ERROR] Please file a bug against the javadoc tool via the Java bug reporting
page
[ERROR] (http://bugreport.java.com) after checking the Bug Database
(http://bugs.java.com)
[ERROR] for duplicates. Include error messages and the following diagnostic in
your report. Thank you.
[ERROR] com.sun.tools.javac.code.ClassFinder$BadClassFile: bad class file:
/home/jhyde/open1/calcite.2/linq4j/target/classes/org/apache/calcite/linq4j/Ord$2.class
[ERROR] undeclared type variable: E
[ERROR] Please remove or make sure it appears in the correct subdirectory of
the classpath.
[ERROR] at
jdk.compiler/com.sun.tools.javac.jvm.ClassReader.badClassFile(ClassReader.java:281)
[ERROR] at
jdk.compiler/com.sun.tools.javac.jvm.ClassReader.findTypeVar(ClassReader.java:966)
[ERROR] at
jdk.compiler/com.sun.tools.javac.jvm.ClassReader.readEnclosingMethodAttr(ClassReader.java:1463)
[ERROR] at
jdk.compiler/com.sun.tools.javac.jvm.ClassReader$10.read(ClassReader.java:1173)
[ERROR] at
jdk.compiler/com.sun.tools.javac.jvm.ClassReader.readAttrs(ClassReader.java:1549)
[ERROR] at
jdk.compiler/com.sun.tools.javac.jvm.ClassReader.readClassAttrs(ClassReader.java:1563)
[ERROR] at
jdk.compiler/com.sun.tools.javac.jvm.ClassReader.readClass(ClassReader.java:2625)
[ERROR] at
jdk.compiler/com.sun.tools.javac.jvm.ClassReader.readClassBuffer(ClassReader.java:2718)
[ERROR] at
jdk.compiler/com.sun.tools.javac.jvm.ClassReader.readClassFile(ClassReader.java:2731)
[ERROR] at
jdk.compiler/com.sun.tools.javac.code.ClassFinder.fillIn(ClassFinder.java:352)
[ERROR] at
jdk.compiler/com.sun.tools.javac.code.ClassFinder.complete(ClassFinder.java:284)
[ERROR] at
jdk.compiler/com.sun.tools.javac.code.Symbol.complete(Symbol.java:633)
[ERROR] at
jdk.compiler/com.sun.tools.javac.code.Symbol$ClassSymbol.complete(Symbol.java:1308)
[ERROR] at
jdk.compiler/com.sun.tools.javac.code.Symbol$ClassSymbol.flags(Symbol.java:1242)
[ERROR] at
jdk.compiler/com.sun.tools.javac.code.Symbol$TypeSymbol.getEnclosedElements(Symbol.java:800)
[ERROR] at
jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.util.Utils.getItems0(Utils.java:2404)
[ERROR] at
jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.util.Utils.getItems0(Utils.java:2399)
[ERROR] at
jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.util.Utils.recursiveGetItems(Utils.java:2387)
[ERROR] at
jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.util.Utils$17.visitPackage(Utils.java:2372)
[ERROR] at
jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.util.Utils$17.visitPackage(Utils.java:2368)
[ERROR] at
jdk.compiler/com.sun.tools.javac.code.Symbol$PackageSymbol.accept(Symbol.java:1160)
[ERROR] at
java.compiler@10.0.2/javax.lang.model.util.AbstractElementVisitor6.visit(AbstractElementVisitor6.java:106)
[ERROR] at
jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.util.Utils.getItems(Utils.java:2381)
[ERROR] at
jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.util.Utils.getClasses(Utils.java:2121)
[ERROR] at
jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.util.Utils.getAllClasses(Utils.java:2294)
[ERROR] at
jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.generateClassFiles(AbstractDoclet.java:277)
[ERROR] at
jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.startGeneration(AbstractDoclet.java:211)
[ERROR] at
jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.run(AbstractDoclet.java:117)
[ERROR] at
jdk.javadoc/jdk.javadoc.doclet.StandardDoclet.run(StandardDoclet.java:72)
[ERROR] at
jdk.javadoc/jdk.javadoc.internal.tool.Start.parseAndExecute(Start.java:581)
[ERROR] at
jdk
[jira] [Created] (CALCITE-3258) Upgrade jackson-databind from 2.9.9 to 2.9.9.2
Julian Hyde created CALCITE-3258: Summary: Upgrade jackson-databind from 2.9.9 to 2.9.9.2 Key: CALCITE-3258 URL: https://issues.apache.org/jira/browse/CALCITE-3258 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Upgrade jackson-databind from 2.9.9 to 2.9.9.2. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (CALCITE-3286) In LatticeSuggester, allow join conditions that use expressions
Julian Hyde created CALCITE-3286:
Summary: In LatticeSuggester, allow join conditions that use
expressions
Key: CALCITE-3286
URL: https://issues.apache.org/jira/browse/CALCITE-3286
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In {{LatticeSuggester}}, allow join conditions that use expressions. For
example, the query
{code}
select *
from emp
join dept on emp.deptno + 1 = dept.deptno
{code}
uses an expression "emp.deptno + 1" on the "emp" side of the join key.
Currently this query throws:
{noformat}
java.lang.ClassCastException: class
org.apache.calcite.materialize.LatticeSuggester$DerivedColRef cannot be cast to
class org.apache.calcite.materialize.LatticeSuggester$BaseColRef
at
org.apache.calcite.materialize.LatticeSuggester.frame(LatticeSuggester.java:478)
at
org.apache.calcite.materialize.LatticeSuggester.frame(LatticeSuggester.java:420)
at
org.apache.calcite.materialize.LatticeSuggester.frame(LatticeSuggester.java:399)
at
org.apache.calcite.materialize.LatticeSuggester.addQuery(LatticeSuggester.java:124)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (CALCITE-3307) PigRelExTest fails on Windows
Julian Hyde created CALCITE-3307:
Summary: PigRelExTest fails on Windows
Key: CALCITE-3307
URL: https://issues.apache.org/jira/browse/CALCITE-3307
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Running {{PigRelExTest}} on Windows, after CALCITE-3122 was merged, gives the
following error:
{noformat}
2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary in
the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the
Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
at org.apache.hadoop.util.Shell.(Shell.java:390)
at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
at
org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
at
org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
at
org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
at
org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
at org.apache.pig.PigServer.(PigServer.java:231)
at org.apache.pig.PigServer.(PigServer.java:219)
at org.apache.pig.PigServer.(PigServer.java:211)
at org.apache.pig.PigServer.(PigServer.java:207)
at org.apache.calcite.piglet.PigConverter.(PigConverter.java:107)
at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
at
org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
{noformat}
This issue has arisen many times; see HADOOP-10775 and SPARK-2356.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (CALCITE-3308) RelBuilder.literal with a string argument should create a character literal with appropriate character set
Julian Hyde created CALCITE-3308:
Summary: RelBuilder.literal with a string argument should create a
character literal with appropriate character set
Key: CALCITE-3308
URL: https://issues.apache.org/jira/browse/CALCITE-3308
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Suppose you call {{RelBuilder.literal}} with a with a string argument that
contains non-ASCII characters:
{code:java}
RexNode e = RelBuilder.literal("☘️ my lucky charms!");{code}
The call should return a character literal with appropriate character set that
can handle the contents of the string. (I'm not sure what character set should
be.)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (CALCITE-3326) SQL with invalid function throws NullPointerException if typeCoercion is disabled
Julian Hyde created CALCITE-3326:
Summary: SQL with invalid function throws NullPointerException if
typeCoercion is disabled
Key: CALCITE-3326
URL: https://issues.apache.org/jira/browse/CALCITE-3326
Project: Calcite
Issue Type: Bug
Affects Versions: 1.21.0
Reporter: Julian Hyde
If you have a SQL query with a function that does not exist, and if
typeCoercion is disabled, the validator throws a NullPointerException; it
should throw a validation exception, same as if typeCoercion is enabled.
Here is a testcase:
{noformat}
diff --git a/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java
b/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java
index 43621a7d6..227cc28e9 100644
--- a/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java
+++ b/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java
@@ -1361,6 +1361,8 @@ public void _testLikeAndSimilarFails() {
@Test public void testInvalidFunction() {
checkWholeExpFails("foo()", "No match found for function signature FOO..");
+checkFails("select foo()", "No match found for function signature FOO..",
+false);
checkWholeExpFails("mod(123)",
"Invalid number of arguments to function 'MOD'. Was expecting 2
arguments");
}
{noformat}
throws
{noformat}
at java.base/java.util.Objects.requireNonNull(Objects.java:221)
at org.apache.calcite.sql.SqlBasicCall.setOperator(SqlBasicCall.java:67)
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:297)
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:216)
at
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5626)
{noformat}
It's clear that the flow at [SqlFunction line
274|https://github.com/apache/calcite/blob/955d4ea7b30b85519529c1c77662fa04141bfa89/core/src/main/java/org/apache/calcite/sql/SqlFunction.java#L274]
will let through {{function = null}} if coercion is disabled. This was caused
by CALCITE-2302.
Are there other possible paths through that block where {{function}} ends up
null at the end? It's not clear from looking at the code.
[~danny0405], Can you please take a look? Cc: [~fib-seq].
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (CALCITE-3327) Simplify SqlValidatorTest by replacing 'boolean typeCoercion' parameter with fluent method 'Sql.withTypeCoercion(boolean)'
Julian Hyde created CALCITE-3327:
Summary: Simplify SqlValidatorTest by replacing 'boolean
typeCoercion' parameter with fluent method 'Sql.withTypeCoercion(boolean)'
Key: CALCITE-3327
URL: https://issues.apache.org/jira/browse/CALCITE-3327
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Simplify SqlValidatorTest by replacing {{boolean typeCoercion}} parameter with
fluent method {{withTypeCoercion(boolean)}} in class
{{SqlValidatorTestCase.Sql}}.
The problem with the {{boolean typeCoercion}} parameter is that it doesn't
compose with all of the other flags we might want to pass - are we testing a
query or an expression?, is it supposed to pass or fail?, does it contain
window functions? - and therefore we have to add it in a combinatorial number
of places.
Fluent interfaces are preferred over "test methods" like {{checkExpFails}} for
big tests like {{SqlValidatorTest}}.
I'm developing a fix, and I'll post the branch URL shortly.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (CALCITE-3328) Immutable beans, powered by reflection
Julian Hyde created CALCITE-3328:
Summary: Immutable beans, powered by reflection
Key: CALCITE-3328
URL: https://issues.apache.org/jira/browse/CALCITE-3328
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In quite a few places we have immutable config classes, and because they are
immutable we have builders. It all gets quite verbose.
I propose to add a utility class, {{ImmutableBeans}}, that is able to generate
an implementation of a given interface. For example,
{code:java}
interface MyBean {
@Property int getAge();
@Property String getName();
}
MyBean b = ImmutableBeans.create(MyBean.class);
assertThat(b.withAge(37).withName("Fred").getAge(), is(37));
{code}
We could replace SqlParser.ConfigBuilder, Frameworks.ConfigBuilder,
RelBuilder.ConfigBuilder, quite a few of the associated Config classes, and
perhaps quite a few fluent classes that we use in tests.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (CALCITE-3344) In JDBC adapter, generate "SELECT TOP(n)" for MSSQL 2008 and earlier
Julian Hyde created CALCITE-3344: Summary: In JDBC adapter, generate "SELECT TOP(n)" for MSSQL 2008 and earlier Key: CALCITE-3344 URL: https://issues.apache.org/jira/browse/CALCITE-3344 Project: Calcite Issue Type: Bug Components: jdbc-adapter Reporter: Julian Hyde In JDBC adapter, generate "SELECT TOP(n)" for MSSQL 2008 and earlier, because it doesn't support "FETCH". For MSSQL 2012 and higher, continue to generate "FETCH NEXT n ROWS". -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (CALCITE-3361) Add a test that parses and validates a SQL statement with every built-in Redshift function
Julian Hyde created CALCITE-3361: Summary: Add a test that parses and validates a SQL statement with every built-in Redshift function Key: CALCITE-3361 URL: https://issues.apache.org/jira/browse/CALCITE-3361 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Add a test that parses and validates a SQL statement with every built-in Redshift function. It would be part of the babel component, but would go beyond parsing, and also validate. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3369) In LatticeSuggester, recommend lattices based on UNION queries
Julian Hyde created CALCITE-3369:
Summary: In LatticeSuggester, recommend lattices based on UNION
queries
Key: CALCITE-3369
URL: https://issues.apache.org/jira/browse/CALCITE-3369
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In LatticeSuggester, recommend lattices based on UNION, EXCEPT and INTERSECT
queries. Currently such queries are ignored.
Given the query
{code:java}
select * from t1 join t2
union
select * from t2 join t3;{code}
suggester should generate the same lattice(s) as if it had been given two
separate queries
{code:java}
select * from t1 join t2;
select * from t2 join t3; {code}
Which may be a single lattice t1 - t2 - t3, or might be two lattices t1 - t2,
t2 - t3.
Same for EXCEPT (MINUS), INTERSECT, UNION ALL, etc.
If the set-op is internal, I'm not sure what to do, e.g.
{code:java}
select *
from sales
join (select * from good_product
union
select * from bad_product) using (product_id){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3383) Allow plural time units in interval literals
Julian Hyde created CALCITE-3383:
Summary: Allow plural time units in interval literals
Key: CALCITE-3383
URL: https://issues.apache.org/jira/browse/CALCITE-3383
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Allow plural time units in interval literals. For example, currently, Calcite
allows {{INTERVAL '2' DAY}} but gives an error for {{INTERVAL '2' DAYS}}.
(Note: DAY vs DAYS.)
Standard SQL only allows the first; PostgreSQL allows both.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3387) Query with GROUP BY and JOIN ... USING wrongly fails with "Column 'DEPTNO' is ambiguous" error
Julian Hyde created CALCITE-3387:
Summary: Query with GROUP BY and JOIN ... USING wrongly fails with
"Column 'DEPTNO' is ambiguous" error
Key: CALCITE-3387
URL: https://issues.apache.org/jira/browse/CALCITE-3387
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Query with GROUP BY and JOIN ... USING wrongly fails with "Column 'DEPTNO' is
ambiguous" error. Here is the query:
{code}
select deptno, count(*)
from emp
join dept using (deptno)
group by deptno;
{code}
Because of USING, the two deptno fields should be merged into one, and
therefore {{deptno}} is not ambiguous. That query works in Oracle. Also note
that
{code}select deptno
from emp
join dept using (deptno)
{code}
works correctly, and has since CALCITE-2227 was fixed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3438) Validator should disallow use of the GROUPING function inside a FILTER clause
Julian Hyde created CALCITE-3438:
Summary: Validator should disallow use of the GROUPING function
inside a FILTER clause
Key: CALCITE-3438
URL: https://issues.apache.org/jira/browse/CALCITE-3438
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Validator should disallow use of GROUPING inside FILTER. For instance, the
following query should be invalid (you can paste it into {{agg.iq}} followed by
'!ok'):
{code}
select deptno, sum(sal) filter (where grouping(deptno) = 0)
from "scott".emp
group by deptno;
{code}
but fails with an internal error:
{noformat}
java.sql.SQLException: Error while executing SQL "select deptno, sum(sal)
filter (where grouping(deptno) = 0)
from "scott".emp
group by deptno": cannot translate call GROUPING($t7)
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
...
Caused by: java.lang.RuntimeException: cannot translate call GROUPING($t7)
at
org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateCall(RexToLixTranslator.java:756)
at
org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate0(RexToLixTranslator.java:730)
...
{noformat}
If you change "grouping" to "sum" the validator correctly gives the error
"FILTER must not contain aggregate expression".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3444) Upgrade SQLLine to 1.9.0, and solve "Class path contains multiple SLF4J bindings" problem
Julian Hyde created CALCITE-3444:
Summary: Upgrade SQLLine to 1.9.0, and solve "Class path contains
multiple SLF4J bindings" problem
Key: CALCITE-3444
URL: https://issues.apache.org/jira/browse/CALCITE-3444
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
The "Class path contains multiple SLF4J bindings" is as follows:
{noformat}
sqlline version 1.9.0
sqlline> !connect jdbc:hsqldb:res:scott SCOTT TIGER
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/Users/julianhyde/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/Users/julianhyde/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
0: jdbc:hsqldb:res:scott> {noformat}
Calcite's {{sqlline}} script generates {{target/fullclasspath.txt}} with both
of the above jar files on it. The solution is to remove one of them. I propose
to remove {{slf4j-log4j12}} from the generated classpath.
Windows uses a different technique (copying dependencies to each module's
target/dependencies directory). This change does not fix Windows.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3445) In web site, automatically redirect http to https
Julian Hyde created CALCITE-3445:
Summary: In web site, automatically redirect http to https
Key: CALCITE-3445
URL: https://issues.apache.org/jira/browse/CALCITE-3445
Project: Calcite
Issue Type: Bug
Components: site
Reporter: Julian Hyde
Fix For: 1.22.0
In web site, automatically redirect http to https.
Per [email thread 'Redirecting to
https'|https://lists.apache.org/thread.html/3a76df309d01eddacd6abcb47cdd036f32718a5c1f28c7226547c872@%3Cdev.calcite.apache.org%3E],
the solution is to add an {{.htaccess}} file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3500) Validator infers incorrect type for "IS NOT DISTINCT FROM"
Julian Hyde created CALCITE-3500:
Summary: Validator infers incorrect type for "IS NOT DISTINCT FROM"
Key: CALCITE-3500
URL: https://issues.apache.org/jira/browse/CALCITE-3500
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Validator infers incorrect type for "IS NOT DISTINCT FROM". I got the following
output from a test:
{noformat}
[INFO] Running org.apache.calcite.test.fuzzer.RexProgramFuzzyTest
2019-11-13 13:30:16,459 [pool-1-thread-3] INFO - Using seed 44 for rex fuzzing
2019-11-13 13:30:19,869 [pool-1-thread-3] INFO - Rex fuzzing results: number
of cases tested=2000, failed cases=0, duplicate failures=0, fuzz rate=586 per
second
Shrinked to 36669 chars, time remaining 19997
Shrinked to 9704 chars, time remaining 19996
Shrinked to 9631 chars, time remaining 19993
Shrinked to 9287 chars, time remaining 19992
Shrinked to 7058 chars, time remaining 19989
Shrinked to 6507 chars, time remaining 19986
Shrinked to 5317 chars, time remaining 19985
Shrinked to 5191 chars, time remaining 19985
Shrinked to 4990 chars, time remaining 19984
Shrinked to 4844 chars, time remaining 19984
Shrinked to 3877 chars, time remaining 19981
Shrinked to 3628 chars, time remaining 19981
Shrinked to 275 chars, time remaining 19980
Shrinked to 258 chars, time remaining 19980
Shrinked to 239 chars, time remaining 19980
Shrinked to 188 chars, time remaining 19979
2019-11-13 13:30:20,616 [pool-1-thread-1] INFO - Rex fuzzing results: number
of cases tested=406, failed cases=1, duplicate failures=0, fuzz rate=100 per
second
2019-11-13 13:30:20,617 [pool-1-thread-1] INFO - Randomized test identified a
potential defect. Feel free to fix that issue
java.lang.IllegalStateException: Unable to simplify unknownAsFalse<(IS NOT
DISTINCT FROM(>(+(CAST(0):INTEGER), +(0)), true), true)
lt(isNotDistinctFrom(gt(unaryPlus(abstractCast(literal(0), tInt(true))),
unaryPlus(literal(0))), trueLiteral), trueLiteral)
at
org.apache.calcite.test.fuzzer.RexProgramFuzzyTest.checkUnknownAs(RexProgramFuzzyTest.java:228)
at
org.apache.calcite.test.fuzzer.RexProgramFuzzyTest.checkUnknownAsAndShrink(RexProgramFuzzyTest.java:209)
at
org.apache.calcite.test.fuzzer.RexProgramFuzzyTest.checkUnknownAs(RexProgramFuzzyTest.java:172)
at
org.apache.calcite.test.fuzzer.RexProgramFuzzyTest.generateRexAndCheckTrueFalse(RexProgramFuzzyTest.java:462)
Suppressed: org.apache.calcite.test.fuzzer.RexProgramFuzzyTest$1: seed
-104518976443406885
Caused by: java.lang.AssertionError
at
org.apache.calcite.rex.RexSimplify.validateStrongPolicy(RexSimplify.java:843)
at org.apache.calcite.rex.RexSimplify.simplifyIs2(RexSimplify.java:682)
at org.apache.calcite.rex.RexSimplify.simplifyIs(RexSimplify.java:653)
at org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:281)
{noformat}
I believe that it means that a particular call to {{IS NOT DISTINCT FROM}} has
a not-null return and has nullable arguments and this contradict's the
Strong.ANY policy. I suspect that the policy for that operator should not be
Strong.ANY.
Discovered by the {{validateStrongPolicy}} method added in CALCITE-3457.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3517) DiffRepository spends too much time writing XML, makes some tests 5x slower
Julian Hyde created CALCITE-3517:
Summary: DiffRepository spends too much time writing XML, makes
some tests 5x slower
Key: CALCITE-3517
URL: https://issues.apache.org/jira/browse/CALCITE-3517
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Tests that use {{DiffRepository}} are spending far too much effort writing XML,
even if the XML matches the reference file. For example, If I comment out [a
call to set(tag,
next)|https://github.com/apache/calcite/blob/ee83efd360793ef4201f4cdfc2af8d837b76ca69/core/src/test/java/org/apache/calcite/test/DiffRepository.java#L267],
{{RelOptRulesTest}} improves from 32s to 6s; {{SqlToRelConverterTest}}
improves from 24s to 4.7s; {{SqlPrettyWriterTest}} remains .8s.
The {{DiffRepository.expand}} method is the cause of the inefficiency. It
causes the entire XML document to be re-generated and written to disk. This is
not just slow but quadratic - if a test has N cases, each test writes the XML
document, an effort proportional to N.
{{DiffRepository}} should remain conservative. If one of the tests fails, and a
later test crashes, the output from the failed test should have been written
out. It is acceptable if the test remains slow if there are test failures.
{{DiffRepository}} is only used in tests; this bug does not affect production
code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3526) SqlPrettyWriter should have options to fold/chop long lines, print leading commas
Julian Hyde created CALCITE-3526:
Summary: SqlPrettyWriter should have options to fold/chop long
lines, print leading commas
Key: CALCITE-3526
URL: https://issues.apache.org/jira/browse/CALCITE-3526
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
SqlPrettyWriter should have options to fold/chop long lines, print leading
commas.
In this change, we change how {{SqlPrettyWriter}} is configured: methods such
as
{{setIndentation}} are now deprecated, there is an immutable config object
{{SqlWriterConfig}}, and the {{SqlDialect}} and {{boolean
alwaysUseParentheses}} constructor arguments are also properties of the config.
So, rather than
{code}
new SqlPrettyWriter(d, p).setIndentation(4)
{code}
you now write
{code}
new SqlPrettyWriter(
SqlPrettyWriter.config()
.withDialect(d)
.withAlwaysUseParentheses(p)
.withIndentation(4))
{code}
This change adds options to fold lists such as the SELECT clause: WIDE always
puts items on the same line, TALL always puts items on separate lines, and FOLD
and CHOP adapt. If the line is too long, FOLD fits as many items on a line as
it can, whereas CHOP puts items onto separate lines. STEP is like WIDE but
observes the {{clauseEndsLine}} property.
You can set folding policy for each clause separately using
{{withSelectFolding}}, {{withOrderByFolding}} etc., or set the default
{{withFolding}}.
A new property {{clauseEndsLine}} causes first SELECT item to be on a separate
line than the SELECT keyword, and similarly other clauses.
For all options, see
[SqlWriterConfig|https://calcite.apache.org/apidocs/org/apache/calcite/sql/SqlWriterConfig.html].
This changes also fixes CALCITE-1585; because config is held in an immutable
object rather than the SqlPrettyWriter, {{reset()}} no longer resets settings
such as {{alwaysUseParentheses}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3595) Test infrastructure overwrites reference log with wrong results
Julian Hyde created CALCITE-3595: Summary: Test infrastructure overwrites reference log with wrong results Key: CALCITE-3595 URL: https://issues.apache.org/jira/browse/CALCITE-3595 Project: Calcite Issue Type: Bug Components: build Reporter: Julian Hyde When I add a test case to a test that is based on DiffRepository (such as SqlPrettyWriterTest), and run it in Intellij, the first time gives an error because the actual output does not match the expected output. This is good. I would expect that if I run it again, it will fail again, but it does not. I'm not sure exactly what is going on, but maybe the test is reading from an output file instead of a reference log. This was probably broken during the migration to gradle, because the paths for reference logs have all changed. This bug prevents me from using the workflow where I add a test case, run it, and when it fails, change the code until the output looks right, then overwrite the reference log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3716) ResultSetMetaData.getTableName should return empty string, not null, when column does not map to a table
Julian Hyde created CALCITE-3716:
Summary: ResultSetMetaData.getTableName should return empty
string, not null, when column does not map to a table
Key: CALCITE-3716
URL: https://issues.apache.org/jira/browse/CALCITE-3716
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Per the [JDBC
spec|https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSetMetaData.html#getTableName-int-],
{{ResultSetMetaData.getTableName}} should return empty string, not null, when
column does not map to a table. Similarly getCatalogName, getSchemaName,
getColumnName.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3723) Following the change to add hints to RelNode, deprecate the old constructors
Julian Hyde created CALCITE-3723:
Summary: Following the change to add hints to RelNode, deprecate
the old constructors
Key: CALCITE-3723
URL: https://issues.apache.org/jira/browse/CALCITE-3723
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In CALCITE-482 and CALCITE-3590 we added constructors for various sub-classes
RelNode ({{LogicalProject}}, {{LogicalAggregate}}, and others) that take a list
of hints. But now those classes have two constructors. Our practice has been to
have only one (public, non-deprecated) constructor in each RelNode class.
(Otherwise we would have dozens.) So, please deprecate the old constructors and
change code that uses them.
Can we do this before 1.22?
Also note that the new and old constructors have exactly the same comment. You
should avoid that. But in this case, just remove the comment of the deprecated
constructor.
The non-hints constructor does {{new ArrayList<>()}}. Please change to use
{{ImmutableList.of()}}, which saves a malloc.
cc [~danny0405] and [~icshuo].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3735) In ImmutableBeans, allow interfaces to have default methods
Julian Hyde created CALCITE-3735: Summary: In ImmutableBeans, allow interfaces to have default methods Key: CALCITE-3735 URL: https://issues.apache.org/jira/browse/CALCITE-3735 Project: Calcite Issue Type: Bug Reporter: Julian Hyde In ImmutableBeans, allow interfaces to have default methods. (These are invoked in the usual way by the client. The mechanics are a bit tricky because they involve method handles, and the APIs are different in JDK 8 than later JDKs.) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3751) JDBC adapter wrongly pushes ORDER BY into sub-query
Julian Hyde created CALCITE-3751:
Summary: JDBC adapter wrongly pushes ORDER BY into sub-query
Key: CALCITE-3751
URL: https://issues.apache.org/jira/browse/CALCITE-3751
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
JDBC adapter wrongly pushes ORDER BY into sub-query. When I run
{code}select deptno, job, sum(sal)
from scott.emp
group by deptno, job
order by 1,2;
{code}
against the hsqldb database via the JDBC adapter I get the error
{noformat}Error: Error while executing SQL "select deptno, job, sum(sal) from
scott.emp group by deptno, job order by 1,2": While executing SQL [SELECT
"DEPTNO", "JOB", SUM("SAL")
FROM (SELECT "JOB", "DEPTNO", SUM("SAL")
FROM "SCOTT"."EMP"
GROUP BY "JOB", "DEPTNO"
ORDER BY "DEPTNO" NULLS LAST, "JOB" NULLS LAST) AS "t0"] on JDBC sub-schema
(state=,code=0)
{noformat}
Note that ORDER BY is inside a sub-query, which I presume is invalid SQL for
hsqldb (and most other DBs).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3752) PIVOT and UNPIVOT
Julian Hyde created CALCITE-3752:
Summary: PIVOT and UNPIVOT
Key: CALCITE-3752
URL: https://issues.apache.org/jira/browse/CALCITE-3752
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Oracle SQL has PIVOT and UNPIVOT operators for cross-tab support.
For
[example|https://oracle-base.com/articles/11g/pivot-and-unpivot-operators-11gr1],
{noformat}
SELECT *
FROM (SELECT customer_id, product_code, quantity
FROM pivot_test)
PIVOT (SUM(quantity) AS sum_quantity FOR (product_code) IN ('A' AS a, 'B' AS
b, 'C' AS c))
ORDER BY customer_id;
CUSTOMER_ID A_SUM_QUANTITY B_SUM_QUANTITY C_SUM_QUANTITY
--- -- -- --
1 10 20 30
2 4050
3 60 70 80
4100
4 rows selected.
{noformat}
In Calcite we could implement this as a prepare-time rewrite, something like
this:
{noformat}
SELECT customer_id,
SUM(DECODE(product_code, 'A', quantity, 0)) AS a_sum_quantity,
SUM(DECODE(product_code, 'B', quantity, 0)) AS b_sum_quantity,
SUM(DECODE(product_code, 'C', quantity, 0)) AS c_sum_quantity
FROM pivot_test
GROUP BY customer_id
ORDER BY customer_id;
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3763) RelBuilder.aggregate should prune unused fields from the input, if the input is a Project
Julian Hyde created CALCITE-3763:
Summary: RelBuilder.aggregate should prune unused fields from the
input, if the input is a Project
Key: CALCITE-3763
URL: https://issues.apache.org/jira/browse/CALCITE-3763
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
{{RelBuilder.aggregate}} should prune unused fields from the input, if the
input is a {{Project}}.
Pruning fields during the planning process is desirable, but often cannot do it
- we are applying a {{RelOptRule}} that has to return the same fields, or we
don't want to add an extra Project do so the pruning. But when we are in
{{RelBuilder.aggregate}} and the input is a Project, neither of those
limitations apply. We already have a Project, we are just making it narrower;
and we know what fields the {{Aggregate}} will produce.
For example,
{code:sql}
SELECT deptno, SUM(sal) FILTER (WHERE b)
FROM (
SELECT deptno, empno + 10, sal, job = 'CLERK' AS b
FROM emp)
GROUP BY deptno
{code}
becomes
{code:sql}
SELECT deptno, SUM(sal) FILTER (WHERE b)
FROM (
SELECT deptno, sal, job = 'CLERK' AS b
FROM emp)
GROUP BY deptno
{code}
If there are no fields used, remove the {{Project}}. (A {{RelNode}} with no
fields is not allowed.)
{code:sql}
SELECT COUNT(*) AS C
FROM (
SELECT deptno, empno + 10, sal, job = 'CLERK' AS b
FROM emp)
{code}
becomes
{code:sql}
SELECT COUNT(*) AS c
FROM emp
{code}
Add an option {{RelBuilder.Config.pruneInputOfAggregate}}, default true, so
that people can disable this rewrite if it causes problems.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3764) AggregateCaseToFilterRule handles NULL values correctly
Julian Hyde created CALCITE-3764:
Summary: AggregateCaseToFilterRule handles NULL values correctly
Key: CALCITE-3764
URL: https://issues.apache.org/jira/browse/CALCITE-3764
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
{{AggregateCaseToFilterRule}} handles NULL values correctly. It converts
{code:sql}
SELECT COUNT(CASE WHEN b THEN NULL ELSE 1 END) FROM t
{code}
to
{code:sql}
SELECT COUNT(*) FILTER (WHERE b IS FALSE) FROM t
{code}
which fails to count rows where {{b}} is UNKNOWN, so it should convert to
{code:sql}
SELECT COUNT(*) FILTER (WHERE b IS NOT TRUE) FROM t
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3774) Add option to RelBuilder to prevent it from merging projects
Julian Hyde created CALCITE-3774:
Summary: Add option to RelBuilder to prevent it from merging
projects
Key: CALCITE-3774
URL: https://issues.apache.org/jira/browse/CALCITE-3774
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add option to {{RelBuilder}} to prevent it from merging projects. Currently if
you call {{RelBuilder.project}} and the input is a {{Project}} it will merge
the expressions. This is usually a good idea, but sometimes it creates very
complex expressions.
There is an existing method {{RelBuilder.shouldMergeProject()}} but by default
it returns true, and in order to change it you have to sub-class
{{RelBuilder}}, which is not easy to do.
I propose to add a property {{RelBuilder.Config.mergeBloat}}, default 0, which
would prevent creating a project that has more complexity than the two projects
that went into it.
Example 1:
* Input 1: {{Project(a+b+c+d AS w, b+c+d+e AS x, c+d+e+f AS y, d+e+f+g AS z)}}
followed by
* Input 2: {{Project(w*x AS p, x*y AS q, y*z AS r)}} creates
* Output: {{Project((a+b+c+d) * (b+c+d+e) AS p, (b+c+d+e) * (c+d+e+f) AS q,
(c+d+e+f) * (d+e+f+g) AS r)}}.
The expression "a+b+c+d" has complexity 7 (4 fields and 3 calls). The inputs
have complexity 28 (4 expressions, each of complexity 7) and 9 (3 expressions,
each with complexity 3). The output has complexity 45 (3 expressions, each with
complexity 15 (8 fields and 7 calls)). 45 is greater than 37 (28 + 9), so this
merge would not be allowed.
Example 2:
* Input 1: {{Project(a+b+c+d AS w, b+c+d+e AS x, c+d+e+f AS y, d+e+f+g AS z)}}
(complexity 28)
* Input 2: {{Project(w*x AS p, x*y AS q)}} (complexity 6) creates
* Output: {{Project((a+b+c+d) * (b+c+d+e) AS p, (b+c+d+e) * (c+d+e+f) AS q)}}
(complexity 30).
Output complexity 30 is less than input complexity 34 (28 + 6), and therefore
the merge is allowed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3811) JDBC adapter generates SQL with invalid field names if Filter's row type is different from its input
Julian Hyde created CALCITE-3811:
Summary: JDBC adapter generates SQL with invalid field names if
Filter's row type is different from its input
Key: CALCITE-3811
URL: https://issues.apache.org/jira/browse/CALCITE-3811
Project: Calcite
Issue Type: Bug
Components: jdbc-adapter
Affects Versions: 1.21.0
Reporter: Julian Hyde
Fix For: 1.22.0
JDBC adapter generates SQL with invalid field names if a {{Filter}} object's
row type is different from its input. Usually a {{Filter}} object's row type is
identical to its input's row type (because that is how it derives its row type
on construction), but if you call {{RelNode.replaceInput}} after construction
it is possible to make the field names inconsistent.
Here is an example of the wrong SQL generated:
{code}
SELECT D AS `emps.deptno`
FROM (
SELECT DEPTNO AS D2, COUNT(*) AS `emps.count`
FROM `scott`.EMP
GROUP BY DEPTNO
HAVING COUNT(*) < 2) AS `t1`
{code}
Note that the sub-query returns a column {{D2}} but the outer {{SELECT}}
expression refers to {{D}}, because that was the name in the row type of the
{{Filter}} (which became the {{HAVING}} clause in the generated SQL).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3814) Support JDK 14
Julian Hyde created CALCITE-3814:
Summary: Support JDK 14
Key: CALCITE-3814
URL: https://issues.apache.org/jira/browse/CALCITE-3814
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
JDK 14 will be released on 2020/03/17.
I tried building Calcite on an early access build and got an error from Gradle:
{noformat}
$ java -version
openjdk version "14" 2020-03-17
OpenJDK Runtime Environment (build 14+36-1461)
OpenJDK 64-Bit Server VM (build 14+36-1461, mixed mode, sharing)
$ ./gradlew build
Starting a Gradle Daemon, 1 busy and 1 incompatible Daemons could not be
reused, use --status for details
java.lang.NoClassDefFoundError: Could not initialize class
org.codehaus.groovy.vmplugin.v7.Java7
at
org.codehaus.groovy.vmplugin.VMPluginFactory.(VMPluginFactory.java:43)
at
org.codehaus.groovy.reflection.GroovyClassValueFactory.(GroovyClassValueFactory.java:35)
at org.codehaus.groovy.reflection.ClassInfo.(ClassInfo.java:107)
...
$ ./gradlew clean --stacktrace
FAILURE: Build failed with an exception.
* What went wrong:
Could not initialize class org.codehaus.groovy.runtime.InvokerHelper
* Try:
Run with --info or --debug option to get more log output. Run with --scan to
get full insights.
* Exception is:
java.lang.NoClassDefFoundError: Could not initialize class
org.codehaus.groovy.runtime.InvokerHelper
at
org.gradle.internal.extensibility.DefaultExtraPropertiesExtension.(DefaultExtraPropertiesExtension.java:29)
at
org.gradle.internal.extensibility.DefaultConvention.(DefaultConvention.java:49)
at
org.gradle.internal.extensibility.ExtensibleDynamicObject.(ExtensibleDynamicObject.java:60)
...
{noformat}
Problem appears to be https://github.com/gradle/gradle/issues/10248, which is
related to (caused by) GROOVY-9211, and alrso related to NETBEANS-3876.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3815) Add missing SQL standard aggregate functions: EVERY, SOME, INTERSECTION
Julian Hyde created CALCITE-3815:
Summary: Add missing SQL standard aggregate functions: EVERY,
SOME, INTERSECTION
Key: CALCITE-3815
URL: https://issues.apache.org/jira/browse/CALCITE-3815
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add missing SQL standard aggregate functions: EVERY, SOME, INTERSECTION.
Examples:
{code}
SELECT deptno,
EVERY(sal < 3000) AS es, SOME(sal < 1000) AS ss,
EVERY(comm < 500) AS ec, SOME(comm < 500) AS sc
FROM emp
GROUP BY deptno;
++---+---+---+--+
| DEPTNO | ES | SS | EC | SC |
++---+---+---+--+
| 10 | FALSE | FALSE | | |
| 20 | FALSE | TRUE | | |
| 30 | TRUE | TRUE | FALSE | TRUE |
++---+---+---+--+
{code}
{{EVERY}} and {{SOME}} can be implemented by translating to {{MIN}}, {{MAX}}:
* {{EVERY(condition)}} is equivalent to {{MIN(condition)}};
* {{SOME(condition)}} is equivalent to {{MAX(condition)}}
where {{condition}} is a {{BOOLEAN}} expression, possibly allowing {{NULL}}
values).
{{INTERSECTION}} computes the intersection of collections (arrays and
multisets). (Compare to {{FUSION}}, which computes the union of collections.)
{{FUSION}} is in the operator table but there is no code-generation for it.
This task should implement {{FUSION}} and {{INTERSECTION}} so that we can run
queries in Enumerable mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3822) Source distribution must not contain fonts under SIL OFL 1.1 license (category B)
Julian Hyde created CALCITE-3822: Summary: Source distribution must not contain fonts under SIL OFL 1.1 license (category B) Key: CALCITE-3822 URL: https://issues.apache.org/jira/browse/CALCITE-3822 Project: Calcite Issue Type: Bug Reporter: Julian Hyde The source distribution contains "font-awesome:fonts" which is licensed as http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License). OFL 1.1 is "category B" (see LEGAL-112). We need to remove "web and desktop font files", probably under site/fonts. Code and icons are OK. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3823) Do not use String.replaceAll
Julian Hyde created CALCITE-3823:
Summary: Do not use String.replaceAll
Key: CALCITE-3823
URL: https://issues.apache.org/jira/browse/CALCITE-3823
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
We use JDK's {{String.replaceAll(String regex, String replacement)}} about 60
times in the code, but most of these should probably use
{{String.replace(CharSequence target, CharSequence replacement)}}.
{{replaceAll}} uses regex semantics, which is slow (it compiles a pattern each
call) and probably not desired behavior.
Consider adding to {{forbidden-apis/signatures.txt}} to prevent it from being
accidentally used in future.
See tweet "[String.replaceAll is the most invisible java perf issue I end up
finding out of
profiles|https://twitter.com/t3rmin4t0r/status/1232353433371336704]";.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3841) Change downloads page to use downloads.apache.org
Julian Hyde created CALCITE-3841:
Summary: Change downloads page to use downloads.apache.org
Key: CALCITE-3841
URL: https://issues.apache.org/jira/browse/CALCITE-3841
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Infra has
[decided|https://lists.apache.org/thread.html/rcd2864e75e417597d342b8eb83080eb2d7a0cafea84fd4521a4d9cfd%40%3Cusers.infra.apache.org%3E]
(login required for that email link) to deprecate
[www.apache.org/dist|https://www.apache.org/dist] and move downloads to
[https://downloads.apache.org|https://downloads.apache.org].
On [Calcite's downloads page|https://calcite.apache.org/downloads/], we need to
change the 'digest' link from (for example)
{{https://www.apache.org/dist/calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz.sha256}}
to
{{https://downloads.apache.org/calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz.sha256}},
and similarly the 'gpg' link.
I believe that the 'tar' link can remain as
{{https://www.apache.org/dyn/closer.lua?filename=calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz&action=download}}
for the latest release and
{{https://archive.apache.org/dist/calcite/apache-calcite-1.20.0/apache-calcite-1.20.0-src.tar.gz}}
for older releases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3892) Make junit test classes and methods non-public
Julian Hyde created CALCITE-3892: Summary: Make junit test classes and methods non-public Key: CALCITE-3892 URL: https://issues.apache.org/jira/browse/CALCITE-3892 Project: Calcite Issue Type: Bug Reporter: Julian Hyde In junit 4 and earlier versions, junit classes and methods must be public. In junit 5 (jupiter) this is no longer the case. This change makes classes and methods package-protected where possible (i.e. removes the "public" modifier). It makes tests a bit more concise, and gives us more freedom to refactor tests in future. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3897) Windowed aggregate with bracket (ROWS or RANGE) without ORDER BY is invalid
Julian Hyde created CALCITE-3897:
Summary: Windowed aggregate with bracket (ROWS or RANGE) without
ORDER BY is invalid
Key: CALCITE-3897
URL: https://issues.apache.org/jira/browse/CALCITE-3897
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
I believe that windowed aggregate with bracket (ROWS or RANGE) without ORDER BY
is invalid. If so, we should throw.
[~danny0405]
[reports|https://github.com/apache/calcite/pull/1896#discussion_r403420946]
that the syntax is invalid on MSSQL 2017.
Someone should check the SQL standard. If it is invalid in the SQL standard
(but only if) the validator should give an error.
I am a little ambivalent. A missing {{ORDER BY}} is basically sorting by zero
columns, an of course it's not deterministic, but even with {{ORDER BY}}, the
ordering may not be deterministic (e.g. apply {{ORDER BY deptno}} to the
{{Emp}} table). So we should go with the SQL standard.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3923) Refactor how planner rules are parameterized
Julian Hyde created CALCITE-3923:
Summary: Refactor how planner rules are parameterized
Key: CALCITE-3923
URL: https://issues.apache.org/jira/browse/CALCITE-3923
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
People often want different variants of planner rules. An example is
{{FilterJoinRule}}, which has a 'boolean smart’ parameter, a predicate (which
returns whether to pull up filter conditions), operands (which determine the
precise sub-classes of {{RelNode}} that the rule should match) and a
{{RelBuilderFactory}} (which controls the type of {{RelNode}} created by this
rule).
Suppose you have an instance of {{FilterJoinRule}} and you want to change
{{smart}} from true to false. The {{smart}} parameter is immutable (good!) but
you can’t easily create a clone of the rule because you don’t know the values
of the other parameters. Your instance might even be (unbeknownst to you) a
sub-class with extra parameters and a private constructor.
So, my proposal is to put all of the config information of a {{RelOptRule}}
into a single {{config}} parameter that contains all relevant properties. Each
sub-class of {{RelOptRule}} would have one constructor with just a ‘config’
parameter. Each config knows which sub-class of {{RelOptRule}} to create.
Therefore it is easy to copy a config, change one or more properties, and
create a new rule instance.
Adding a property to a rule’s config does not require us to add or deprecate
any constructors.
The operands are part of the config, so if you have a rule that matches a
{{EnumerableFilter}} on an {{EnumerableJoin}} and you want to make it match an
{{EnumerableFilter}} on an {{EnumerableNestedLoopJoin}}, you can easily create
one with one changed operand.
The config is immutable and self-describing, so we can use it to automatically
generate a unique description for each rule instance.
(See the email thread [[DISCUSS] Refactor how planner rules are
parameterized|https://lists.apache.org/thread.html/rfdf6f9b7821988bdd92b0377e3d293443a6376f4773c4c658c891cf9%40%3Cdev.calcite.apache.org%3E].)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3967) Publish components "calcite" and "calcite-examples"
Julian Hyde created CALCITE-3967: Summary: Publish components "calcite" and "calcite-examples" Key: CALCITE-3967 URL: https://issues.apache.org/jira/browse/CALCITE-3967 Project: Calcite Issue Type: Bug Components: build Affects Versions: 1.22.0 Reporter: Julian Hyde Until release 1.21 we published components "calcite" and "calcite-examples" as part of the release. In 1.22 these components were not published. I suspect that they were left out when we migrated from Maven to Gradle. One consequence is that if you [search Maven Central for 'apache calcite'|https://search.maven.org/search?q=apache%20calcite] the latest version appears to be 1.21. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4038) Refactor RexVisitor, RexBiVisitor, RelOptUtil.InputFinder
Julian Hyde created CALCITE-4038:
Summary: Refactor RexVisitor, RexBiVisitor, RelOptUtil.InputFinder
Key: CALCITE-4038
URL: https://issues.apache.org/jira/browse/CALCITE-4038
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Various improvements to {{RexVisitor}}, {{RexBiVisitor}},
{{RelOptUtil.InputFinder}} classes.
1. Make mutable field {{RelOptUtil.InputFinder.inputBitSet}} private. The field
still exists, public and deprecated, but it shadows a new private field. Rather
than calling {{InputFinder.inputBitSet.build()}} you should now call
{{InputFinder.build()}}.
2. In {{RexVisitor}} and {{RexBiVisitor}} add methods {{visitList}} and
{{visitEach}} (the former returns a list, and the latter returns void).
3. Deprecate {{RexShuttle.apply}} in favor of {{visitList}}.
4. Add {{RexWindowBound.accept(RexBiVisitor)}}.
5. For {{RexBiVisitor}} add a method {{visitEachIndexed}}, the
index passed as the 'payload' argument.
6. Add abstract implementations of {{RexBiVisitor}}: {{RexUnaryBiVisitor}} and
{{RexBiVisitorImpl}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4061) Build should fail if Calcite code uses deprecated APIs
Julian Hyde created CALCITE-4061: Summary: Build should fail if Calcite code uses deprecated APIs Key: CALCITE-4061 URL: https://issues.apache.org/jira/browse/CALCITE-4061 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Calcite build should fail if Calcite code uses deprecated APIs. This includes external APIs (e.g. Guava methods) and internal APIs (e.g. a deprecated class in the org.apache.calcite.util package). This used to occur when the build was powered by Maven. Compared to a policy where code is allowed to use deprecated APIs for a 'grace period', this policy has a number of advantages. One is that Calcite devs (or users) can upgrade dependencies at short notice, with no code changes. Another is that it forces people who are doing internal refactoring (replacing one internal Calcite API with another) to finish the job. [~vlsi] Can you take this please? I can't figure out how to set javac's '-Xlint:deprecated -Werror' from Gradle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4072) Avatica build fails with various license errors with Guava 22.0 or higher
Julian Hyde created CALCITE-4072:
Summary: Avatica build fails with various license errors with
Guava 22.0 or higher
Key: CALCITE-4072
URL: https://issues.apache.org/jira/browse/CALCITE-4072
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
On Guava 22.0 or higher, Avatica build gives various license errors:
{noformat}
> Task :tck:getLicenses FAILED
Build calcite-avatica FAILURE reason:
Execution failed for task ':tck:getLicenses':
LICENSE-like files are missing
==
MIT
* org.codehaus.mojo:animal-sniffer-annotations:1.14
at
com.github.vlsi.gradle.license.GatherLicenseTask.run(GatherLicenseTask.kt:417)
at
org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
at
org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:49)
{noformat}
JDK does not seem to be a factor: occurs on JDK 8, 11, 14.
The workaround is to disable various license-checking tasks on the command-line:
{noformat}
./gradlew clean build -x :tck:getLicenses -x :tck:license -x
:tck:licenseCopySpec -x :standalone-server:getLicenses -x
:standalone-server:license -x :standalone-server:licenseCopySpec
{noformat}
I had expected to be able to change Guava version using
{{-Dguava.version=22.0}} on the gradle command line. But I had to edit
{{gradle.properties}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4087) Hoist, a utility to replace literals in a SQL string with placeholders
Julian Hyde created CALCITE-4087:
Summary: Hoist, a utility to replace literals in a SQL string with
placeholders
Key: CALCITE-4087
URL: https://issues.apache.org/jira/browse/CALCITE-4087
Project: Calcite
Issue Type: New Feature
Components: babel
Affects Versions: 1.23.0
Reporter: Julian Hyde
Add "Hoist", a utility to replace literals in a SQL string with placeholders.
By default it uses Calcite's parser, but you can configure to use different
quoting, casing, and also to use a different parser such as Babel for other
dialects of SQL.
It generates a data structure, {{Hoisted}}, that contains a map of where the
literals occur within the SQL string. The Hoisted.substitute method generates
an alternative SQL string, replacing the literals with a custom string.
It does not call {{SqlNode.unparse}}, and therefore does not lose any of the
original formatting.
For more examples, see tests added to {{SqlParserTest}} and {{BabelParserTest}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4089) In Babel, allow 'CAST(integer AS DATE)' even though it is illegal in Calcite SQL
Julian Hyde created CALCITE-4089:
Summary: In Babel, allow 'CAST(integer AS DATE)' even though it is
illegal in Calcite SQL
Key: CALCITE-4089
URL: https://issues.apache.org/jira/browse/CALCITE-4089
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In Babel, allow 'CAST(integer AS DATE)', even though it is illegal in Calcite
SQL (and standard SQL).
Map it to valid standard SQL. BigQuery {{CAST(n AS DATE)}} is equivalent to
Calcite {{DATE '1970-01-01' + n * INTERVAL '1' DAY}}.
Also, add functions {{DATE_FROM_UNIX_DATE(INTEGER)}},
{{UNIX_SECONDS(TIMESTAMP)}}, {{UNIX_MILLIS(TIMESTAMP)}},
{{UNIX_MICROS(TIMESTAMP)}}, {{TIMESTAMP_SECONDS(BIGINT)}},
{{TIMESTAMP_MILLIS(BIGINT)}}, {{TIMESTAMP_MICROS(BIGINT)}} in the BigQuery
library. They are well-named and well-specified, so they might be useful to
people working in dialects besides BigQuery. {{DATE_FROM_UNIX_DATE(0)}} should
return {{DATE '1970-01-01'}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4117) Release Calcite 1.24.0
Julian Hyde created CALCITE-4117: Summary: Release Calcite 1.24.0 Key: CALCITE-4117 URL: https://issues.apache.org/jira/browse/CALCITE-4117 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Assignee: Chunwei Lei Fix For: 1.24.0 Release Calcite 1.24.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4128) Remove dependency of File adapter on Demo CSV adapter
Julian Hyde created CALCITE-4128: Summary: Remove dependency of File adapter on Demo CSV adapter Key: CALCITE-4128 URL: https://issues.apache.org/jira/browse/CALCITE-4128 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Remove dependency of File adapter on Demo CSV adapter. The dependency should be in the other direction. We want the File adapter to have lots of capabilities, and the Demo CSV adapter to be easy to understand. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4134) Interval expressions
Julian Hyde created CALCITE-4134:
Summary: Interval expressions
Key: CALCITE-4134
URL: https://issues.apache.org/jira/browse/CALCITE-4134
Project: Calcite
Issue Type: Sub-task
Reporter: Julian Hyde
Support interval expressions.
Standard SQL has only interval literals (e.g. {{INTERVAL '1' HOUR}}, {{INTERVAL
-'1:2' HOUR TO MINUTE}}). Note that the 'value' part is a character literal and
may contain formatting characters like ':'.
Interval expressions are a SQL extension present in
[MySQL|https://dev.mysql.com/doc/refman/8.0/en/expressions.html#temporal-intervals]
and
[BigQuery|https://cloud.google.com/bigquery/docs/reference/standard-sql/datetime_functions#datetime_add].
We support the syntax {{INTERVAL numericExpression timeUnit}}:
* the 'value' part is a numeric expression;
* {{timeUnit}} is a time unit (such as {{HOUR}}), not a time unit range (such
as {{HOUR TO MINUTE}});
* {{timeUnit}} may have precision, e.g. {{HOUR(2)}};
* {{timeUnit}} may be plural, e.g. {{HOURS}}, if
{{SqlConformance.allowPluralTimeUnits}} (see CALCITE-3383);
* numeric literals and identifiers do not require parentheses, but other
expressions require parentheses.
Examples:
* {{SELECT INTERVAL empno HOUR FROM Emp}}
* {{SELECT INTERVAL (empno * 2) MINUTE FROM Emp}}
* {{INTERVAL -3 YEAR}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4137) Checkstyle should ensure that every class has a Javadoc comment
Julian Hyde created CALCITE-4137:
Summary: Checkstyle should ensure that every class has a Javadoc
comment
Key: CALCITE-4137
URL: https://issues.apache.org/jira/browse/CALCITE-4137
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Checkstyle should ensure that every class, interface, enum, package has a
Javadoc comment. And that javadoc types (e.g {{@param}} and {{@return}}) are
not empty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4154) Add a rule to merge a Project onto an Aggregate
Julian Hyde created CALCITE-4154:
Summary: Add a rule to merge a Project onto an Aggregate
Key: CALCITE-4154
URL: https://issues.apache.org/jira/browse/CALCITE-4154
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add a rule to merge a {{Project}} onto an {{Aggregate}}. In practice, it means
that aggregate functions that are not used in the Project are removed from the
Aggregate.
For example,
{code:java}
SELECT deptno, sum_sal
FROM (
SELECT deptno, job, SUM(sal) AS sum_sal, MIN(ename) AS min_ename
FROM emp
GROUP BY deptno, job)
{code}
becomes
{code:java}
SELECT deptno, sum_sal
FROM (
SELECT deptno, job, SUM(sal) AS sum_sal
FROM emp
GROUP BY deptno, job)
{code}
Note that {{min_ename}} is removed from the {{Aggregate}} (because it is not
used by the {{Project}}). {{job}} is not used, but it remains because it is a
{{GROUP BY}} key.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-5208) Reduce size of generated parser by optimizing lookahead
Julian Hyde created CALCITE-5208:
Summary: Reduce size of generated parser by optimizing lookahead
Key: CALCITE-5208
URL: https://issues.apache.org/jira/browse/CALCITE-5208
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
We could reduce the size of generated parser by optimizing lookahead.
The main problem is identifiers and non-reserved keywords. Every point in the
grammar where the parser needs to choose between an identifier and something
else the parser needs to do lookahead, so it generates a {{switch}} expression.
But that {{switch}} contains not just the identifier tokens ({{IDENTIFIER}},
{{UNICODE_QUOTED_IDENTIFIER}} and 5 others) but about 300 non-reserved keywords.
Switches like this, which are typically 300 to 460 lines long, occur 53 times
in the current parser. That's 24,000 lines of code, and the entire parser is
38,000 lines of code. And there are several parsers (core, babel, server, test).
{noformat}
$ find . -name SqlParserImpl.java
./core/build/javacc/javaCCMain/org/apache/calcite/sql/parser/impl/SqlParserImpl.java
$ wc
core/build/javacc/javaCCMain/org/apache/calcite/sql/parser/impl/SqlParserImpl.java
38242 86680 989569
core/build/javacc/javaCCMain/org/apache/calcite/sql/parser/impl/SqlParserImpl.java
$ grep 'case ZONE:'
core/build/javacc/javaCCMain/org/apache/calcite/sql/parser/impl/SqlParserImpl.java
|wc
53 106 865
$ grep 'case [A-Z]*:'
core/build/javacc/javaCCMain/org/apache/calcite/sql/parser/impl/SqlParserImpl.java
|wc
13507 27014 255827
{noformat}
We can optimize by putting common patterns into shared methods. For example,
this pattern occurs several times, and could be a method {{AliasOpt()}}:
{code}
( [ ] alias = SimpleIdentifier() | { alias = null; } )
{code}
It might be possible to remove a choice. Rather than choosing whether or not to
fire a rule, we could fire the rule, call the method, and have the method
return null.
I don't know how many of the 53 switches we can remove. If we can remove just
10 that would be a significant number of lines of code saved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5223) AdjustProjectForCountAggregateRule throws ArrayIndexOutOfBoundsException
Julian Hyde created CALCITE-5223:
Summary: AdjustProjectForCountAggregateRule throws
ArrayIndexOutOfBoundsException
Key: CALCITE-5223
URL: https://issues.apache.org/jira/browse/CALCITE-5223
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
While decorrelating a query with a scalar sub-query,
AdjustProjectForCountAggregateRule throws ArrayIndexOutOfBoundsException. Here
is an example (also in attached PR).
{noformat}
./sqlline
!connect jdbc:calcite:model=core/src/test/resources/hsqldb-model.json sa sa
SELECT deptno, ename, job, sal,
sal / (SELECT SUM(sal) FROM Emp WHERE deptno = e.deptno) AS pct_dept,
sal / (SELECT SUM(sal) FROM Emp) AS pct_total
FROM Emp AS e
WHERE job = 'CLERK'
ORDER BY deptno;
java.lang.ArrayIndexOutOfBoundsException: Index 7 out of bounds for length 6
at
com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:77)
at org.apache.calcite.rex.RexBuilder.makeInputRef(RexBuilder.java:911)
at
org.apache.calcite.sql2rel.RelDecorrelator$AdjustProjectForCountAggregateRule.lambda$onMatch2$4(RelDecorrelator.java:2673)
at
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at
java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at
org.apache.calcite.sql2rel.RelDecorrelator$AdjustProjectForCountAggregateRule.onMatch2(RelDecorrelator.java:2674)
at
org.apache.calcite.sql2rel.RelDecorrelator$AdjustProjectForCountAggregateRule.onMatch(RelDecorrelator.java:2604)
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:337)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:556)
at
org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:420)
at
org.apache.calcite.plan.hep.HepPlanner.executeRuleInstance(HepPlanner.java:243)
at
org.apache.calcite.plan.hep.HepInstruction$RuleInstance$State.execute(HepInstruction.java:178)
at
org.apache.calcite.plan.hep.HepPlanner.lambda$executeProgram$0(HepPlanner.java:211)
at
com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:210)
at
org.apache.calcite.plan.hep.HepProgram$State.execute(HepProgram.java:118)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:205)
at
org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:191)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelate(RelDecorrelator.java:291)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:230)
at
org.apache.calcite.tools.Programs$DecorrelateProgram.run(Programs.java:361)
at
org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:336)
at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:177)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:312)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:220)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:648)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:514)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:484)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:234)
at
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:623)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:677)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5232) Upgrade protobuf-java from 3.17.1 to 3.21.5
Julian Hyde created CALCITE-5232: Summary: Upgrade protobuf-java from 3.17.1 to 3.21.5 Key: CALCITE-5232 URL: https://issues.apache.org/jira/browse/CALCITE-5232 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Upgrade protobuf-java in Calcite and Avatica. Some scanning tools say that the current version, 3.17.1, is affected by [CVE-2021-22569|https://nvd.nist.gov/vuln/detail/CVE-2021-22569]. It is not, but this upgrade makes those warnings go away. As for CALCITE-4626, we plan to upgrade both Avatica and Calcite, to keep them on the same protobuf-java version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5255) gRPC transport for Avatica
Julian Hyde created CALCITE-5255: Summary: gRPC transport for Avatica Key: CALCITE-5255 URL: https://issues.apache.org/jira/browse/CALCITE-5255 Project: Calcite Issue Type: Bug Components: avatica Reporter: Julian Hyde CALCITE-840 added a protobuf transport for Avatica, as an alternative to the original JSON transport. In this change, we propose to add another transport that uses [gRPC|https://en.wikipedia.org/wiki/GRPC]. We should be able to re-use the existing serialization of data and metadata to and from protobuf. Compared to the existing protobuf-over-HTTP transport, the gRPC transport will benefit from gRPC's authentication, flow control, and streaming. This task would add support on the Java server and client. Upgrading the Go client to use gRPC could be a separate task. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5262) Add functions to support WKB (well-known binary) and GeoJSON
Julian Hyde created CALCITE-5262: Summary: Add functions to support WKB (well-known binary) and GeoJSON Key: CALCITE-5262 URL: https://issues.apache.org/jira/browse/CALCITE-5262 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Add functions to support WKB (well-known binary) and GeoJSON. This is made possible by the switch to JTS in CALCITE4294. The functions are: * ST_AsEWKT * ST_AsGeoJSON * ST_AsWKB * ST_GeomFromGeoJSON * ST_GeomFromWKB -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5275) Release Calcite 1.32.0
Julian Hyde created CALCITE-5275: Summary: Release Calcite 1.32.0 Key: CALCITE-5275 URL: https://issues.apache.org/jira/browse/CALCITE-5275 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Fix For: 1.32.0 Release Calcite 1.32.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5278) Upgrade Janino from 3.1.6 to 3.1.8
Julian Hyde created CALCITE-5278: Summary: Upgrade Janino from 3.1.6 to 3.1.8 Key: CALCITE-5278 URL: https://issues.apache.org/jira/browse/CALCITE-5278 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Fix For: 1.32.0 Upgrade Janino from 3.1.6 to 3.1.8. This may or may not solve the issue noted in CALCITE-5227, but the upgrade is unlikely to make things worse and may make things better. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5283) Add ARG_MIN, ARG_MAX (aka MIN_BY, MAX_BY) aggregate functions
Julian Hyde created CALCITE-5283:
Summary: Add ARG_MIN, ARG_MAX (aka MIN_BY, MAX_BY) aggregate
functions
Key: CALCITE-5283
URL: https://issues.apache.org/jira/browse/CALCITE-5283
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add ARG_MIN, ARG_MAX (aka MIN_BY, MAX_BY) aggregate functions.
For example, the following query prints the name of the lowest paid employee in
each department:
{code}
SELECT deptno, ARG_MIN(sal, ename)
FROM emp
GROUP BY deptno;
+++
| deptno | _c1|
+++
| 10 | MILLER |
| 20 | SMITH |
| 30 | JAMES |
+++
{code}
If multiple employees have the same salary, arbitrarily chooses one to return.
See [Yandex description of
ARG_MIN|https://cloud.yandex.com/en/docs/datalens/function-ref/ARG_MIN]. Also
implemented in Kusto, Alibaba MaxCompute, Clickhouse, DuckDB.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5290) Add DATE_TRUNC function consistent with Postgres
Julian Hyde created CALCITE-5290:
Summary: Add DATE_TRUNC function consistent with Postgres
Key: CALCITE-5290
URL: https://issues.apache.org/jira/browse/CALCITE-5290
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add {{DATE_TRUNC}} function consistent with Postgres. Per the
[specification|https://www.postgresql.org/docs/current/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC],
syntax is as follows:
{code}
DATE_TRUNC(, )
{code}
It is a different (and more complex) specification than BigQuery's DATE_TRUNC
(see CALCITE-5180), which has the following syntax:
{code}
DATE_TRUNC(, )
{code}
Postgres DATE_TRUNC would be implemented after CALCITE-5155 and would allow
user-defined time frames in the {{field}} argument.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5292) Add a policy that can prevent SQL statements from running if they contain certain functions / operators
Julian Hyde created CALCITE-5292:
Summary: Add a policy that can prevent SQL statements from running
if they contain certain functions / operators
Key: CALCITE-5292
URL: https://issues.apache.org/jira/browse/CALCITE-5292
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add a policy that can prevent SQL statements from running if they contain
certain operators, including built-in functions and operators.
One use of this policy would be to disable functions that contain serious bugs.
For example, we could have prevented use of {{EXTRACTVALUE}} as a workaround to
CALCITE-5263.
Calcite's predecessor [LucidDB|https://github.com/LucidDB/luciddb] had
policy-driven access to tables, but this code was never fully enabled in
Calcite. Ironically, the last remaining code was deprecated in CALCITE-4953. We
should consider reviving the policy and making it handle tables and
functions/operators.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5295) Read the values of plugins (such as connect string properties) from ThreadLocal fields
Julian Hyde created CALCITE-5295:
Summary: Read the values of plugins (such as connect string
properties) from ThreadLocal fields
Key: CALCITE-5295
URL: https://issues.apache.org/jira/browse/CALCITE-5295
Project: Calcite
Issue Type: Bug
Components: avatica
Reporter: Julian Hyde
This change would allow plugin values to come from a field whose type is
{{ThreadLocal}}. This will be useful in scenarios where the value of the plugin
cannot be statically captured in a class, but depends upon the dynamic state of
the program. This requirement often occurs during testing.
Avatica allows plug-ins in several places, but most notably connection
properties. For example, if I have
{{httpclient_factory=com.example.MyClass#INSTANCE}} in the connect string,
Avatica will look for a static field {{INSTANCE}} in class
{{com.example.MyClass}} and cast it to a {{AvaticaHttpClientFactoryImpl}}.
This change would extend that schema to allow instead such fields to be a
{{ThreadLocal}}, as follows:
{code:java}
public class MyClass {
public static final ThreadLocal INSTANCE =
new ThreadLocal<>();
}
{code}
The code change would be to the {{AvaticaUtilsinstantiatePlugin}} method. Other
code using that utility method, such as Calcite, would inherit that
functionality.
We should evaluate whether this functionality poses a security risk. In
opinion, it does not. To inject a malicious value into the plugin, the attacker
need to already control the JVM instantiating the plugin.
If anything, this reduces security risk, because a {{ThreadLocal}} allows the
value to be set for a shorter duration, and only read/written from the current
thread. Therefore malicious threads in the same JVM, or malicious code
operating earlier or later in the same thread, cannot interfere.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5338) In Avatica's DateTimeUtils, replace floorDiv and floorMod with equivalents in java.lang.Math
Julian Hyde created CALCITE-5338:
Summary: In Avatica's DateTimeUtils, replace floorDiv and floorMod
with equivalents in java.lang.Math
Key: CALCITE-5338
URL: https://issues.apache.org/jira/browse/CALCITE-5338
Project: Calcite
Issue Type: Bug
Components: avatica, core
Reporter: Julian Hyde
In Avatica's {{{}DateTimeUtils{}}}, replace {{floorDiv}} and {{floorMod}} with
equivalents in {{{}java.lang.Math{}}}.
The JDK equivalents were introduced in Java 8 (after Avatica was written) but
now possibly have superior performance (due to intrinsics).
{{{}floorMod(long, int){}}}, which was introduced in JDK 9, is still off-limits.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5347) Add 'SELECT ... BY', a syntax extension to eliminate duplicates and sort
Julian Hyde created CALCITE-5347:
Summary: Add 'SELECT ... BY', a syntax extension to eliminate
duplicates and sort
Key: CALCITE-5347
URL: https://issues.apache.org/jira/browse/CALCITE-5347
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Many people have observed that SQL query syntax is verbose if you want to
eliminate duplicates and sort. Business intelligence tools often have query
languages that eliminate duplicates and sort by default; we cannot do that in
the SQL {{SELECT}} expression, because that would break compatibility. But we
propose the following variation of {{SELECT}} that is distinguishable from
regular {{SELECT}} (i.e. queries that use it would be invalid in ordinary SQL)
but concise and intuitive.
We propose that
{code:java}
SELECT a BY b
FROM t {code}
is syntactic sugar for
{code:java}
SELECT b, a
FROM t
GROUP BY b
ORDER BY b{code}
In the above, {{a}} and {{b}} may be lists of columns, the columns may include
aliases, and the columns in {{b}} may include sort-key modifiers such as
{{DESC}} and {{{}NULLS LAST{}}}.
This feature was inspired by [Shakti|https://shakti.com/], a SQL-like language
by Arthur Whitney, author of the k language.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5363) Move DATEADD, DATEDIFF, DATE_PART from Babel to Core parser
Julian Hyde created CALCITE-5363: Summary: Move DATEADD, DATEDIFF, DATE_PART from Babel to Core parser Key: CALCITE-5363 URL: https://issues.apache.org/jira/browse/CALCITE-5363 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Move the DATEADD, DATEDIFF, DATE_PART functions (maybe also DATEPART) from Babel to Core parser. They require special parsing (maybe not quite as special, following CALCITE-5155) but it would be nice if they could be enabled just by setting 'lib=postgres' without switching parser. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5364) In materialized view substitution, use custom time frames
Julian Hyde created CALCITE-5364:
Summary: In materialized view substitution, use custom time frames
Key: CALCITE-5364
URL: https://issues.apache.org/jira/browse/CALCITE-5364
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Custom time frames were added in CALCITE-5155, and part of the definition of a
custom time frame is the time frames it rolls up to. Materialized view
substitution should use this information, so that you can create materialized
views on custom time frames such as '15 minutes' or '21 days' and have them
roll up.
Also you should be able to use materialized views (on custom and regular time
frames) to answer queries with custom time frames (e.g. {{FLOOR(orderDate TO
month6)}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5368) Automated site build should not bring changes to SQL reference
Julian Hyde created CALCITE-5368:
Summary: Automated site build should not bring changes to SQL
reference
Key: CALCITE-5368
URL: https://issues.apache.org/jira/browse/CALCITE-5368
Project: Calcite
Issue Type: Bug
Components: site
Reporter: Julian Hyde
The automated site build (added in CALCITE-3129) brings over changes to the
[SQL reference|https://calcite.apache.org/docs/reference.html] in the middle of
the release, but these refer to changes that will not be available until the
next release.
Specifically, commit
[f7bf867b|https://github.com/apache/calcite/commit/f7bf867ba96578e7cc40e2aac1adf47015d06913]
on main was cherry-picked as commit
[07421a67https://github.com/apache/calcite/commit/07421a67a021e68d7a6772f9f94b8d994000a54f]
on the [site|https://github.com/apache/calcite/tree/site], and it contains
changes to {{site/doc/reference.md}}. The function {{DATEDIFF}}, which was
added to reference.md in this change, is not available in the current release,
1.32.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5369) In Avatica DateTimeUtils, add support for FLOOR and CEIL to ISOYEAR
Julian Hyde created CALCITE-5369: Summary: In Avatica DateTimeUtils, add support for FLOOR and CEIL to ISOYEAR Key: CALCITE-5369 URL: https://issues.apache.org/jira/browse/CALCITE-5369 Project: Calcite Issue Type: Bug Reporter: Julian Hyde In Avatica DateTimeUtils, add support for computing the FLOOR and CEIL of a date to ISOYEAR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5371) Library grgit 4.0.1 has disappeared from Maven Central
Julian Hyde created CALCITE-5371:
Summary: Library grgit 4.0.1 has disappeared from Maven Central
Key: CALCITE-5371
URL: https://issues.apache.org/jira/browse/CALCITE-5371
Project: Calcite
Issue Type: Bug
Components: avatica
Reporter: Julian Hyde
Avatica depends upon {{org.ajoberstar.grgit:grgit-core:4.0.1}} and
{{org.ajoberstar.grgit:grgit-gradle:4.0.1}} (via
{{com.github.vlsi.gradle:stage-vote-release-plugin:1.78}}) but {{grgit-gradle}}
4.0.1 and {{grgit-core 4.0.1}} are no longer in Maven Central. Builds in GitHub
fail with the following error:
{noformat}
FAILURE: Build failed with an exception.
22
23* What went wrong:
24A problem occurred configuring root project 'calcite-avatica'.
25> Could not resolve all files for configuration ':classpath'.
26 > Could not find org.ajoberstar.grgit:grgit-core:4.0.1.
27 Searched in the following locations:
28 -
https://plugins.gradle.org/m2/org/ajoberstar/grgit/grgit-core/4.0.1/grgit-core-4.0.1.pom
29 If the artifact you are trying to retrieve can be found in the
repository but without metadata in 'Maven POM' format, you need to adjust the
'metadataSources { ... }' of the repository declaration.
30 Required by:
31 project : >
com.github.vlsi.stage-vote-release:com.github.vlsi.stage-vote-release.gradle.plugin:1.82
> com.github.vlsi.gradle:stage-vote-release-plugin:1.82
32 project : >
com.github.vlsi.stage-vote-release:com.github.vlsi.stage-vote-release.gradle.plugin:1.82
> com.github.vlsi.gradle:stage-vote-release-plugin:1.82 >
org.ajoberstar.grgit:grgit-gradle:4.0.1 {noformat}
Presumably builds succeed locally because 4.0.1 is still in my local cache.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5415) In ByteString, add startsWith and endsWith methods
Julian Hyde created CALCITE-5415:
Summary: In ByteString, add startsWith and endsWith methods
Key: CALCITE-5415
URL: https://issues.apache.org/jira/browse/CALCITE-5415
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In {{{}ByteString{}}}, add the following methods:
* {{boolean startsWith(ByteString prefix)}}
* {{boolean startsWith(ByteString prefix, int offset)}}
* {{boolean endsWith(ByteString suffix)}}
Their meaning is analogous to the methods on {{{}String{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5423) Implement TIMESTAMP_DIFF function (compatible with BigQuery)
Julian Hyde created CALCITE-5423: Summary: Implement TIMESTAMP_DIFF function (compatible with BigQuery) Key: CALCITE-5423 URL: https://issues.apache.org/jira/browse/CALCITE-5423 Project: Calcite Issue Type: Sub-task Reporter: Julian Hyde Implement TIMESTAMP_DIFF function (compatible with BigQuery). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5424) Customize handling of literals based on type system
Julian Hyde created CALCITE-5424:
Summary: Customize handling of literals based on type system
Key: CALCITE-5424
URL: https://issues.apache.org/jira/browse/CALCITE-5424
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Currently if you write {{TIMESTAMP '1969-07-20 22:56:00'}} it will be
converted to a value of type {{{}TIMESTAMP{}}}. But if someone has set a custom
type mapping where the name {{TIMESTAMP}} corresponds to the {{TIMESTAMP WITH
LOCAL TIME ZONE}} type then the literal value should be of that type.
Custom type mappings are specified by defining user-defined types in the
schema; see the method
[Schema.getType|https://calcite.apache.org/javadocAggregate/org/apache/calcite/schema/Schema.html#getType(java.lang.String)].
The contents of the string are currently handled by the parser. But in order to
implement this change, the parsing of that string will need to be deferred to
validation time (when the actual type is known, not just its name). This
applies to literals {{TIMESTAMP}}, {{DATE}}, {{TIME}}, {{ARRAY}}, {{INTERVAL}}.
The parser currently emits subclasses of {{SqlLiteral}}, namely
{{SqlTimestampLiteral}}, {{SqlDateLiteral}}, {{SqlTimeLiteral}},
{{SqlArrayLiteral}}, {{SqlIntervalLiteral}}. But after this change, the parser
will emit a new subclass, {{SqlTaggedLiteral}} instead of all of these.
Also as part of this change, add BigQuery-compatible literals:
* {{NUMERIC '0'}}
* {{BIGNUMERIC '0'}}
* {{DATETIME '2014-09-27 12:30:00.45'}}
* {{JSON 'json_formatted_data'}}
And a {{TIMESTAMP WITH LOCAL TIME ZONE}} literal:
* {{TIMESTAMP WITH LOCAL TIME ZONE '1969-07-21 02:56:00'}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5446) Support TIMESTAMP WITH LOCAL TIME ZONE type in JDBC driver
Julian Hyde created CALCITE-5446:
Summary: Support TIMESTAMP WITH LOCAL TIME ZONE type in JDBC driver
Key: CALCITE-5446
URL: https://issues.apache.org/jira/browse/CALCITE-5446
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Support TIMESTAMP WITH LOCAL TIME ZONE type in JDBC driver.
Currently, Calcite types with {{SqlTypeName.TIMESTAMP_WITH_LOCAL_TIME_ZONE}}
are mapped to JDBC type (OTHER). As a result, the ResultSet.toString()
method just converts the (signed) {{long}} value to a string.
Here is what
[Progress|https://docs.progress.com/en-US/bundle/datadirect-connect-jdbc-51/page/TIMESTAMP-WITH-LOCAL-TIME-ZONE-Data-Type.html]
says:
{quote}
The Oracle {{TIMESTAMP WITH LOCAL TIME ZONE}} data type is mapped to the JDBC
{{TIMESTAMP}} data type.
When retrieving {{TIMESTAMP WITH LOCAL TIME ZONE}} columns, the value returned
to the user is converted to the time zone specified by the {{TIME_ZONE}}
session parameter.
When setting {{TIMESTAMP WITH LOCAL TIME ZONE}} columns:
* Using a timestamp (using {{PreparedStatement.setTimestamp}}, for example),
the value set is converted to the time zone specified by the {{TIME_ZONE}}
session parameter.
* Using a string (using {{PreparedStatement.setString()}}, for example), the
string is passed as-is to the server. The supplied string must be in the format
specified by the {{NLS_TIMESTAMP_TZ_FORMAT}} session parameter. If not, the
Oracle server generates an error when it attempts to convert the string to the
{{TIMESTAMP WITH LOCAL TIME ZONE}} type.
{quote}
We should definitely map to {{java.sql.Types.TIMESTAMP}}.
I'm not sure whether Calcite has a {{TIME_ZONE}} session parameter. But values
should be shifted from/to the desired time zone on the way in/out.
And maybe there should also be a Calcite server time zone that is the time zone
in which {{TIMESTAMP WITH LOCAL TIME ZONE}} values are represented. By default
this should be UTC, and using a time zone with a variable offset from UTC (e.g.
"Pacific/Los_Angeles") would be a bad idea.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5457) Upgrade Junit from 4 to 5
Julian Hyde created CALCITE-5457: Summary: Upgrade Junit from 4 to 5 Key: CALCITE-5457 URL: https://issues.apache.org/jira/browse/CALCITE-5457 Project: Calcite Issue Type: Bug Components: avatica Reporter: Julian Hyde Upgrade Junit from 4 to 5. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5480) Release Avatica 1.23.0
Julian Hyde created CALCITE-5480: Summary: Release Avatica 1.23.0 Key: CALCITE-5480 URL: https://issues.apache.org/jira/browse/CALCITE-5480 Project: Calcite Issue Type: Bug Components: avatica Reporter: Julian Hyde Fix For: avatica-1.23.0 Release Avatica 1.23.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5481) Release Calcite 1.33.0
Julian Hyde created CALCITE-5481: Summary: Release Calcite 1.33.0 Key: CALCITE-5481 URL: https://issues.apache.org/jira/browse/CALCITE-5481 Project: Calcite Issue Type: Bug Reporter: Julian Hyde Fix For: 1.33.0 Release Calcite 1.33.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5485) Make type conversion documentation consistent with behavior and tests
Julian Hyde created CALCITE-5485:
Summary: Make type conversion documentation consistent with
behavior and tests
Key: CALCITE-5485
URL: https://issues.apache.org/jira/browse/CALCITE-5485
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Make type conversion documentation consistent with behavior and tests. Use the
following steps:
* Create an internal map that, for each pair of data types, says whether
conversion is implicit, explicit, or disallowed
* Run tests based on that map
* In {{{}DocumentationTest{}}}, generate an HTML table of that mapping, and
check that that table is consistent with the one in the [SQL
reference|https://calcite.apache.org/docs/reference.html#implicit-type-conversion].
I know of two errors in the table:
* The table says that you can explicitly cast from INT to TIMESTAMP (see
CALCITE-5483); this should be disallowed;
* The table says that you can explicitly cast from DECIMAL to BOOLEAN (see
CALCITE-4777); this should be disallowed.
There may be others.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5490) Add a 'naiveMeasures' connection property to allow regular aggregate functions (e.g. SUM, MIN) to be used instead of AGGREGATE
Julian Hyde created CALCITE-5490:
Summary: Add a 'naiveMeasures' connection property to allow
regular aggregate functions (e.g. SUM, MIN) to be used instead of AGGREGATE
Key: CALCITE-5490
URL: https://issues.apache.org/jira/browse/CALCITE-5490
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
When we introduce measures to SQL (see CALCITE-4496) a particular can arise
that users know about measures (have added measure columns to their tables and
views, and want to use those measures in queries) but their querying tools to
not (therefore require an aggregate function when a measure is used in a
{{GROUP BY}} query but do not know about the new {{AGGREGATE}} aggregate
function (added in CALCITE-5105)).
This feature would add a new [connection
property|https://calcite.apache.org/javadocAggregate/org/apache/calcite/config/CalciteConnectionProperty.html],
{{{}naiveMeasures{}}}. Measures and the {{AGGREGATE}} aggregate function are
enabled regardless of its setting. If {{{}naiveMeasures=true{}}}, a call to one
of the core aggregate functions ({{{}MAX{}}}, {{{}MIN{}}}, {{{}SUM{}}},
{{{}AVG{}}}) whose argument is a measure is deemed to be a call to
{{{}AGGREGATE{}}}. If the argument is not a measure, the calls are unchanged.
For example, if {{naiveMeasures=true}} then
{code:java}
SELECT deptno, MAX(avg_sal), MAX(age)
FROM Emp
GROUP BY deptno
{code}
is translated as if the user had written
{code:java}
SELECT deptno, AGGREGATE(avg_sal), MAX(age)
FROM Emp
GROUP BY deptno
{code}
(The {{avg_sal}} column is a measure, {{age}} is a regular column.)
If {{{}naiveMeasures=false{}}}, the default, calls to {{{}MAX{}}}, {{{}MIN{}}},
{{{}SUM{}}}, {{AVG}} are unchanged even if their argument is a measure.
(The semantics of {{MAX(avg_sal)}} are not yet finalized, but one semantics
under consideration is that the {{avg_sal}} measure is evaluated - converted
from a measure to a value - for each row (employee) and the {{MAX}} aggregate
function is then applied to those values.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5495) Allow WEEK and QUARTER in INTERVAL literals
Julian Hyde created CALCITE-5495:
Summary: Allow WEEK and QUARTER in INTERVAL literals
Key: CALCITE-5495
URL: https://issues.apache.org/jira/browse/CALCITE-5495
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Allow interval literals denominated in weeks and quarters, for example:
{code}
INTERVAL '1' WEEK
INTERVAL '2' WEEKS
INTERVAL '3' QUARTERS
INTERVAL '4' QUARTER
{code}
Week intervals have type {{INTERVAL DAY}} (a week is 7 days); quarter intervals
have type {{INTERVAL MONTH}} (a quarter is 3 months).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5508) Add constructor functions for DATE, TIME, TIMESTAMP, DATETIME
Julian Hyde created CALCITE-5508:
Summary: Add constructor functions for DATE, TIME, TIMESTAMP,
DATETIME
Key: CALCITE-5508
URL: https://issues.apache.org/jira/browse/CALCITE-5508
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Add "constructor" functions for DATE, TIME, TIMESTAMP, DATETIME. These are
functions such as "DATE(1969, 7, 20)" that create date-time types. The
functions are based on those in BigQuery, and would be enabled in the BigQuery
library.
Their names follow the terminology of BigQuery's type system, which is
confusing when working in Calcite's type system.
* For example, the function {{DATETIME(1969, 7, 20, 16, 17, 40)}} returns a
Calcite {{TIMESTAMP}}, a type that is called {{DATETIME}} in BigQuery. (Eagle
landed at 4.17pm Houston time.)
* Similarly, {{TIMESTAMP(DATE '1969-07-21', TIME '22:17:40')}} returns a
Calcite {{TIMESTAMP WITH LOCAL TIME ZONE}} representing the same instant in
UTC. That type is called {{TIMESTAMP}} in BigQuery.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5524) JDBC adapter generates LIMIT, OFFSET in wrong order for Presto dialect
Julian Hyde created CALCITE-5524:
Summary: JDBC adapter generates LIMIT, OFFSET in wrong order for
Presto dialect
Key: CALCITE-5524
URL: https://issues.apache.org/jira/browse/CALCITE-5524
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
JDBC adapter generates LIMIT, OFFSET in wrong order for Presto dialect.
The presto paging statement should be:
{code}
SELECT * FROM db1.tb1 offset 100 limit 20;
{code}
The original version defaults to spelling limit first and then offset, which
does not conform to presto syntax.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5529) Improve dialect tests
Julian Hyde created CALCITE-5529:
Summary: Improve dialect tests
Key: CALCITE-5529
URL: https://issues.apache.org/jira/browse/CALCITE-5529
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
The current dialect tests are quick and robust but not very thorough. In
{{RelToSqlConverterTest}} there are about 400 tests (374 SQL statements and 30
relational expressions) that test translation to 538 SQL strings in target
dialects (an average of 1.34 dialects per test). None of those SQL strings are
executed, so we don't know whether they return the correct results or are even
valid.
Improvements:
* Define a list of dialects that we wish to test (the "target dialects") and
provide a means to connect to a database for each of these (typically a driver
on the class path, a URL, user name and password);
* In each test, for each target dialect, generate a SQL string. The generation
should not throw.
* In each test, the query is first executed against Calcite to create a
sample. (For any test, this can be overridden to use a different reference
dialect. Calcite will use a recording, if present.)
* In each test, for each target dialect that has an *enabled* connection,
execute the generated SQL against that connection. If there is a reference
result for the test, check that the result equals the reference result.
* In each test, for each target dialect that has a *recording* connection,
execute the generated SQL against that connection. Record the outcome (error,
column names and types, result) in the file. Proceed as for an enabled
connection. (Check result against reference result, etc.)
* In each test, for each target dialect that has a *replaying* connection,
look up the outcome of the generated SQL. If the outcome is present, proceed as
for an enabled connection. (Check result against reference result, etc.)
* In each test, for each target dialect that has a *replaying+recording*
connection, look up the outcome of the generated SQL. If the outcome is
present, proceed as for an enabled connection. (Check result against reference
result, etc.) If the outcome is not present, proceed as for a recording
connection. (Execute the SQL, record the outcome, check against reference
result, etc.)
With these improvements, the test suite:
* Remains fast and robust (because recordings are used most of the time);
* Can be run offline (using recordings);
* Can be run by contributors who do not necessarily have access to all of the
supported databases;
* Has full coverage (runs all tests against all supported dialects);
* Can re-generate the 'golden' references. (Must be online, have the required
driver and connection, and takes more time.)
Design:
* A new {{class DialectTestConfig}} describes the dialects to be tested, their
names, connections, the connection state (enabled, recording, replaying,
replaying+recording), and recording file name.
* A mock JDBC driver mediates recording/replay.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5540) Wrong plan for query with COUNT(DISTINCT subQuery)
Julian Hyde created CALCITE-5540:
Summary: Wrong plan for query with COUNT(DISTINCT subQuery)
Key: CALCITE-5540
URL: https://issues.apache.org/jira/browse/CALCITE-5540
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
A query with two "COUNT(DISTINCT (SELECT ...))" aggregate functions generates
an incorrect or perhaps inefficient plan when SqlToRelConverter.expand=false.
The query occurs in LatticeSuggesterTest (enabled only with slow tests) and the
error is discussed in CALCITE-5537.
I think output is wrong. And I prefer to set expansion to true.
One edge of LatticeSpace graph is missed when the subquery is not expanded.
The missing case is as follow, the relationship of warehouse#warehouse_class_id
and warehouse_class#warehouse_class_id is missing.
I think it should add to the graph.
foodmart query_id:6739
{code:java}
select
"store"."store_type" as "c0",
count(
distinct (
select
"warehouse_class"."warehouse_class_id" AS "warehouse_class_id"
from
"warehouse_class" AS "warehouse_class"
where
"warehouse_class"."warehouse_class_id" =
"warehouse"."warehouse_class_id"
and "warehouse_class"."description" = 'Large Owned'
)
) as "m0",
count(
distinct (
select
"warehouse_class"."warehouse_class_id" AS "warehouse_class_id"
from
"warehouse_class" AS "warehouse_class"
where
"warehouse_class"."warehouse_class_id" =
"warehouse"."warehouse_class_id"
and "warehouse_class"."description" = 'Large Independent'
)
) as "m1",
count(
(
select
"warehouse_class"."warehouse_class_id" AS "warehouse_class_id"
from
"warehouse_class" AS "warehouse_class"
where
"warehouse_class"."warehouse_class_id" =
"warehouse"."warehouse_class_id"
and "warehouse_class"."description" = 'Large Independent'
)
) as "m2",
count(distinct "store_id" + "warehouse_id") as "m3",
count("store_id" + "warehouse_id") as "m4",
count("warehouse"."stores_id") as "m5"
from
"store" as "store",
"warehouse" as "warehouse"
where
"warehouse"."stores_id" = "store"."store_id"
group by
"store"."store_type"{code}
when set SqlToRelConverter.Config.expand = true
{code:java}
LogicalAggregate(group=[{0}], m0=[COUNT(DISTINCT $1)], m1=[COUNT(DISTINCT $2)],
m2=[COUNT($3)], m3=[COUNT(DISTINCT $4)], m4=[COUNT()], m5=[COUNT($5)])
LogicalProject(c0=[$1], $f0=[$39], $f00=[$40], $f01=[$42], $f4=[+($0, $24)],
stores_id=[$26])
LogicalJoin(condition=[=($25, $41)], joinType=[left])
LogicalProject(store_id=[$0], store_type=[$1], region_id=[$2],
store_name=[$3], store_number=[$4], store_street_address=[$5], store_city=[$6],
store_state=[$7], store_postal_code=[$8], store_country=[$9],
store_manager=[$10], store_phone=[$11], store_fax=[$12],
first_opened_date=[$13], last_remodel_date=[$14], store_sqft=[$15],
grocery_sqft=[$16], frozen_sqft=[$17], meat_sqft=[$18], coffee_bar=[$19],
video_store=[$20], salad_bar=[$21], prepared_food=[$22], florist=[$23],
warehouse_id=[$24], warehouse_class_id=[$25], stores_id=[$26],
warehouse_name=[$27], wa_address1=[$28], wa_address2=[$29], wa_address3=[$30],
wa_address4=[$31], warehouse_city=[$32], warehouse_state_province=[$33],
warehouse_postal_code=[$34], warehouse_country=[$35],
warehouse_owner_name=[$36], warehouse_phone=[$37], warehouse_fax=[$38],
$f0=[$39], $f040=[$41])
LogicalJoin(condition=[=($25, $40)], joinType=[left])
LogicalProject(store_id=[$0], store_type=[$1], region_id=[$2],
store_name=[$3], store_number=[$4], store_street_address=[$5], store_city=[$6],
store_state=[$7], store_postal_code=[$8], store_country=[$9],
store_manager=[$10], store_phone=[$11], store_fax=[$12],
first_opened_date=[$13], last_remodel_date=[$14], store_sqft=[$15],
grocery_sqft=[$16], frozen_sqft=[$17], meat_sqft=[$18], coffee_bar=[$19],
video_store=[$20], salad_bar=[$21], prepared_food=[$22], florist=[$23],
warehouse_id=[$24], warehouse_class_id=[$25], stores_id=[$26],
warehouse_name=[$27], wa_address1=[$28], wa_address2=[$29], wa_address3=[$30],
wa_address4=[$31], warehouse_city=[$32], warehouse_state_province=[$33],
warehouse_postal_code=[$34], warehouse_country=[$35],
warehouse_owner_name=[$36], warehouse_phone=[$37], warehouse_fax=[$38],
$f0=[$40])
LogicalJoin(condition=[=($25, $39)], joinType=[left])
LogicalJoin(condition=[=($26, $0)], joinType=[inner])
JdbcTableScan(table=[[foodmart, store]])
JdbcTableScan(table=[[foodmart, warehouse]])
LogicalAggregate(group=[{0}], agg#0=[SINGLE_VALUE($1)])
LogicalProject(warehouse_class_id1=[$0],
warehouse_class_id=[$0])
LogicalFilter(condition=[=($1, 'Large Owned')])
JdbcTableScan(table=[[foodmart, warehouse_class]])
[jira] [Created] (CALCITE-5546) Code style: Break long assignment expressions after '='
Julian Hyde created CALCITE-5546:
Summary: Code style: Break long assignment expressions after '='
Key: CALCITE-5546
URL: https://issues.apache.org/jira/browse/CALCITE-5546
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
Change the code style to require that long assignment expressions are broken
after '='. After this change,
{code}
final PhysType physType = PhysTypeImpl.of(
implementor.getTypeFactory(),
getRowType(),
pref.prefer(JavaRowFormat.CUSTOM));
{code}
would be changed to
{code}
final PhysType physType =
PhysTypeImpl.of(implementor.getTypeFactory(),
getRowType(),
pref.prefer(JavaRowFormat.CUSTOM));
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5601) Set 'timestamp' and 'timezone' in Quidem test scripts, and have datetime functions observe them
Julian Hyde created CALCITE-5601:
Summary: Set 'timestamp' and 'timezone' in Quidem test scripts,
and have datetime functions observe them
Key: CALCITE-5601
URL: https://issues.apache.org/jira/browse/CALCITE-5601
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
For testing, it is useful to be able to set the session's timezone and the
current time via Quidem variables (and perhaps by other means). Datetime
functions should observe them - e.g. {{CURRENT_TIMESTAMP}} and
{{CURRENT_DATE}} will return a value based on {{timestamp}}, and the
{{TIMESTAMP(string[, timeZone])}} should use the session time zone if the
timeZone argument is omitted.
See [TODO in
big-query.iq|https://github.com/apache/calcite/blob/ba975ba0612768c24677a74fbb9bf7a2922bce36/babel/src/test/resources/sql/big-query.iq#L33-L46].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5629) In TimestampString, allow negative years and years with more than 4 digits
Julian Hyde created CALCITE-5629:
Summary: In TimestampString, allow negative years and years with
more than 4 digits
Key: CALCITE-5629
URL: https://issues.apache.org/jira/browse/CALCITE-5629
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In {{TimestampString}}, allow negative years, and years with more than 4 digits.
Per [ISO 8601|https://en.wikipedia.org/wiki/ISO_8601#Years], means 1 BC,
-0001 means 2 BC, etc.
In the canonical form, years will have at least 4 digits. If they have more
than 4 digits they must not start with zero.
The implementation should be sure to test conversion of negative timestamp
values to and from {{java.util.Calendar}} (which probably represents BC dates
using {{Calendar.ERA}} field). {{TimestampString}} currently assumes that
digits 0..3 are year, 5..6 are month, etc., and the point is at position 20, so
many functions will need to be re-tested.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
