[jira] [Updated] (CALCITE-5921) SqlOperatorFixture.checkFails and checkAggFails don't check runtime failure
[ https://issues.apache.org/jira/browse/CALCITE-5921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CALCITE-5921: Labels: pull-request-available (was: ) > SqlOperatorFixture.checkFails and checkAggFails don't check runtime failure > --- > > Key: CALCITE-5921 > URL: https://issues.apache.org/jira/browse/CALCITE-5921 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Runkang He >Assignee: Runkang He >Priority: Major > Labels: pull-request-available > > SqlOperatorFixture.checkFails and checkAggFails don't check runtime failure. > See more in [code > line|https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/testkit/src/main/java/org/apache/calcite/test/SqlOperatorFixtureImpl.java#L160]. > When the parameter `runtime` of SqlOperatorFixture.checkFails is true, it > should execute the query and check runtime failure, but currently it ignores > this, and only checks the parse and validation failure. > When fix this, there are 4 failed test cases in CalciteSqlOperatorTest. > At last, this issue was found when to implement `BIT_GET` function in > CALCITE-5848. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (CALCITE-5990) Explicit cast to numeric type doesn't check overflow
[
https://issues.apache.org/jira/browse/CALCITE-5990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Runkang He reassigned CALCITE-5990:
---
Assignee: Runkang He
> Explicit cast to numeric type doesn't check overflow
>
>
> Key: CALCITE-5990
> URL: https://issues.apache.org/jira/browse/CALCITE-5990
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Runkang He
>Assignee: Runkang He
>Priority: Major
>
> Explicit cast to numeric type doesn't check overflow, and this issue can be
> reproduced by sqlline:
> {code:sql}
> select cast(empno as tinyint), cast(130 as tinyint) from emps where
> name='Alice'; -- empno is 130
> {code}
> The result is:
> {code:sql}
> -126, -126{code}
> I think it should throw exception when overflow.
> At last, this issue was found when to turn on runtime check for
> CalciteSqlOperatorTest in CALCITE-5921.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5990) Explicit cast to numeric type doesn't check overflow
Runkang He created CALCITE-5990:
---
Summary: Explicit cast to numeric type doesn't check overflow
Key: CALCITE-5990
URL: https://issues.apache.org/jira/browse/CALCITE-5990
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.35.0
Reporter: Runkang He
Explicit cast to numeric type doesn't check overflow, and this issue can be
reproduced by sqlline:
{code:sql}
select cast(empno as tinyint), cast(130 as tinyint) from emps where
name='Alice'; -- empno is 130
{code}
The result is:
{code:sql}
-126, -126{code}
I think it should throw exception when overflow.
At last, this issue was found when to turn on runtime check for
CalciteSqlOperatorTest in CALCITE-5921.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5891) Create a test fixture that would apply PROJECT_REDUCE_EXPRESSIONS to all tests in SqlOperatorTest
[
https://issues.apache.org/jira/browse/CALCITE-5891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763299#comment-17763299
]
Mihai Budiu commented on CALCITE-5891:
--
I have made some progress on this front.
The dataflow is as follows:
* I have a test fixture that extends CalciteSqlOperatorTest
* The fixture overloads the check function
* The check function optimizes the query supplied in 4 steps
** convert to RelNode (reusing a RelOptFixture)
** Use a HepPlanner with a fixed plan; the most important is
PROJECT_REDUCE_EXPRESSIONS
** Convert RelNode to SqlNode using RelToSqlConverter
** Convert SqlNode to a String using toSqlString().
* After optimizing the query, call the super.check(optimizedQuery) function
Ideally this should work, but at this point 70 out of 463 tests fail. There are
various reasons for the failures. Incorrect optimization is only one of them:
* result is not equivalent to expected result (optimizer error)
* type of result is not the same as the type of the expected result
CALCITE-5987
* compilation of query into RelNode throws
* optimizer throws
* conversion of RelNode to SqlNode fails (e.g., CALCITE-5988)
One solution would be for me to submit a PR with this implementation by
overloading all the failing tests and using a JUnit annotation to ignore them.
Then we can assign the bugs to various people responsible for various pieces
and delete the ignored tests as they are fixed.
Happy to hear other proposals.
> Create a test fixture that would apply PROJECT_REDUCE_EXPRESSIONS to all
> tests in SqlOperatorTest
> --
>
> Key: CALCITE-5891
> URL: https://issues.apache.org/jira/browse/CALCITE-5891
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Major
>
> SqlOperatorTest has many tests, including end-to-end tests.
> However, none of these tests exercise the PROJECT_REDUCE_EXPRESSION rules,
> which often produce different results than these tests for constant
> expressions.
> Ideally we should be able to subclass SqlOperatorTest and use a fixture that
> also applies this optimization prior to evaluation.
> I have marked this as a {*}major priority{*}, because I suspect it would
> catch many bugs with minimal effort. (I have found at least 10 so far.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5989) Type inference for RPAD and LPAD functions (BIGQUERY) is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763296#comment-17763296 ] Mihai Budiu commented on CALCITE-5989: -- This was found while trying to make CALCITE-5891 work. > Type inference for RPAD and LPAD functions (BIGQUERY) is incorrect > -- > > Key: CALCITE-5989 > URL: https://issues.apache.org/jira/browse/CALCITE-5989 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > > The type inference uses the type `ReturnTypes.ARG0_NULLABLE_VARYING` for the > output. > This means that the output cannot be longer than arg0. This bug surfaces when > the query is optimized. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5989) Type inference for RPAD and LPAD functions (BIGQUERY) is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763295#comment-17763295 ] Mihai Budiu commented on CALCITE-5989: -- This is very similar to [CALCITE-5813] > Type inference for RPAD and LPAD functions (BIGQUERY) is incorrect > -- > > Key: CALCITE-5989 > URL: https://issues.apache.org/jira/browse/CALCITE-5989 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > > The type inference uses the type `ReturnTypes.ARG0_NULLABLE_VARYING` for the > output. > This means that the output cannot be longer than arg0. This bug surfaces when > the query is optimized. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5989) Type inference for RPAD and LPAD functions (BIGQUERY) is incorrect
Mihai Budiu created CALCITE-5989: Summary: Type inference for RPAD and LPAD functions (BIGQUERY) is incorrect Key: CALCITE-5989 URL: https://issues.apache.org/jira/browse/CALCITE-5989 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.35.0 Reporter: Mihai Budiu The type inference uses the type `ReturnTypes.ARG0_NULLABLE_VARYING` for the output. This means that the output cannot be longer than arg0. This bug surfaces when the query is optimized. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5988) SqlImplementor.toSql cannot emit VARBINARY literals
Mihai Budiu created CALCITE-5988:
Summary: SqlImplementor.toSql cannot emit VARBINARY literals
Key: CALCITE-5988
URL: https://issues.apache.org/jira/browse/CALCITE-5988
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.35.0
Reporter: Mihai Budiu
Given a literal with type VARBINARY the function SqlImplementor.toSql() will
crash with assertion failure error:
{code}
X'41424344':VARBINARY: BINARY
java.lang.AssertionError: X'41424344':VARBINARY: BINARY
at
org.apache.calcite.rel.rel2sql.SqlImplementor.toSql(SqlImplementor.java:1461)
at
org.apache.calcite.rel.rel2sql.SqlImplementor.toSql(SqlImplementor.java:1384)
at
org.apache.calcite.rel.rel2sql.SqlImplementor$Context.toSql(SqlImplementor.java:696)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[
https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763281#comment-17763281
]
Mihai Budiu commented on CALCITE-5986:
--
> The "primary" and "secondary" concepts are very difficult to grasp. I think
> that the primary and secondary family methods in SqlTypeName need more
> documentation.
I didn't invent these concepts, they come directly from SqlTypeFamily. I can
add more documentation, but my understanding doesn't go much deeper than the
JavaDoc on that class, so essentially I can repeat that documentation in the
SqlTypeName class.
> Do the methods both need to be nullable?
Given the existing assignment to families it seems necessary. But I agree that
it would be nice for the primary families to cover all types. There are types
like SYMBOL, or OTHER which didn't have a family previously, and don't have
either a primary or a secondary one now. I don't understand how these are used
to fix them.
> I think that the getFamily method should be deprecated.
I suspect you mean use the Deprecated Java annotation. Some build tools are
upset when you use deprecated methods, but I hope the build won't mind.
> "The SqlTypeFamily for FP types is incorrect" doesn't seem ready for release
> notes
I agree, the original issue name is no longer an apt description. I will think
of a better name. I will rename the commit when I squash at the end.
> Can you add a test that does some sanity checks, e.g. every numeric type has
> a not-null secondary family? Every secondary family is one of (A, B, C).
> Every primary family is one of (X, Y, Z).
I am not sure which objects I would test. The two asserts are a step in this
direction. Perhaps you mean: "if the primary family is NUMERIC, the secondary
one must be APPROXIMATE_NUMERIC or EXACT_NUMERIC"?
> The SqlTypeFamily for FP types is incorrect
> ---
>
> Key: CALCITE-5986
> URL: https://issues.apache.org/jira/browse/CALCITE-5986
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Minor
> Labels: pull-request-available
>
> In SqlTypeFamily we have this code:
> {code:java}
> private static final Map JDBC_TYPE_TO_FAMILY =
> ...
> .put(Types.FLOAT, NUMERIC)
> .put(Types.REAL, NUMERIC)
> .put(Types.DOUBLE, NUMERIC)
> {code}
> But it looks to me like the type family should be APPROXIMATE_NUMERIC.
> This impacts the way RelToSqlConverter works, for instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[
https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763280#comment-17763280
]
Julian Hyde edited comment on CALCITE-5986 at 9/8/23 10:44 PM:
---
I saw your PR.
* The "primary" and "secondary" concepts are very difficult to grasp. I think
that the primary and secondary family methods in SqlTypeName need more
documentation.
* Do the methods both need to be nullable? Non-nullable results are much easier
to work with.
* I think that the getFamily method should be deprecated.
* "The SqlTypeFamily for FP types is incorrect" doesn't seem ready for release
notes
* Convert the {{assert}}s to preconditions calls.
* Can you add a test that does some sanity checks, e.g. every numeric type has
a not-null secondary family? Every secondary family is one of (A, B, C). Every
primary family is one of (X, Y, Z).
* Is "root family" a useful notion? (I.e. don't divide into exactly two strata.
But require each type to have a root type family.)
was (Author: julianhyde):
I saw your PR.
* The "primary" and "secondary" concepts are very difficult to grasp. I think
that the primary and secondary family methods in SqlTypeName need more
documentation.
* Do the methods both need to be nullable? Non-nullable results are much easier
to work with.
* I think that the getFamily method should be deprecated.
* "The SqlTypeFamily for FP types is incorrect" doesn't seem ready for release
notes
* Convert the {{assert}}s to preconditions calls.
* Can you add a test that does some sanity checks, e.g. every numeric type has
a not-null secondary family? Every secondary family is one of (A, B, C). Every
primary family is one of (X, Y, Z).
> The SqlTypeFamily for FP types is incorrect
> ---
>
> Key: CALCITE-5986
> URL: https://issues.apache.org/jira/browse/CALCITE-5986
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Minor
> Labels: pull-request-available
>
> In SqlTypeFamily we have this code:
> {code:java}
> private static final Map JDBC_TYPE_TO_FAMILY =
> ...
> .put(Types.FLOAT, NUMERIC)
> .put(Types.REAL, NUMERIC)
> .put(Types.DOUBLE, NUMERIC)
> {code}
> But it looks to me like the type family should be APPROXIMATE_NUMERIC.
> This impacts the way RelToSqlConverter works, for instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[
https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763280#comment-17763280
]
Julian Hyde edited comment on CALCITE-5986 at 9/8/23 10:43 PM:
---
I saw your PR.
* The "primary" and "secondary" concepts are very difficult to grasp. I think
that the primary and secondary family methods in SqlTypeName need more
documentation.
* Do the methods both need to be nullable? Non-nullable results are much easier
to work with.
* I think that the getFamily method should be deprecated.
* "The SqlTypeFamily for FP types is incorrect" doesn't seem ready for release
notes
* Convert the {{assert}}s to preconditions calls.
* Can you add a test that does some sanity checks, e.g. every numeric type has
a not-null secondary family? Every secondary family is one of (A, B, C). Every
primary family is one of (X, Y, Z).
was (Author: julianhyde):
I saw your PR.
* The "primary" and "secondary" concepts are very difficult to grasp. I think
that the primary and secondary family methods in SqlTypeName need more
documentation.
* Do the methods both need to be nullable?
* I think that the getFamily method should be deprecated.
* "The SqlTypeFamily for FP types is incorrect" doesn't seem ready for release
notes
> The SqlTypeFamily for FP types is incorrect
> ---
>
> Key: CALCITE-5986
> URL: https://issues.apache.org/jira/browse/CALCITE-5986
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Minor
> Labels: pull-request-available
>
> In SqlTypeFamily we have this code:
> {code:java}
> private static final Map JDBC_TYPE_TO_FAMILY =
> ...
> .put(Types.FLOAT, NUMERIC)
> .put(Types.REAL, NUMERIC)
> .put(Types.DOUBLE, NUMERIC)
> {code}
> But it looks to me like the type family should be APPROXIMATE_NUMERIC.
> This impacts the way RelToSqlConverter works, for instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5987) SqlImplementor loses type information for literals
Mihai Budiu created CALCITE-5987:
Summary: SqlImplementor loses type information for literals
Key: CALCITE-5987
URL: https://issues.apache.org/jira/browse/CALCITE-5987
Project: Calcite
Issue Type: Improvement
Components: core
Affects Versions: 1.35.0
Reporter: Mihai Budiu
When converting a SqlNode to a String query, the conversion can produce SQL
that computes different results. This happens because literals do not carry
type information in the result string. For example, this plan:
{code}
rel#7:LogicalValues.(type=RecordType(VARCHAR(3) EXPR$0),tuples=[{ 'A' }])
{code}
will generate a SQL query:
{code}
SELECT 'A'
{code}
While the type of the former result is VARCHAR(3), the latter query produces a
CHAR(1) result.
It would be nice if SqlImplementor had an option to produce a query that
preserves the output type, e.g.:
{code}
SELECT (CAST 'A' as VARCHAR(3))
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[
https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763280#comment-17763280
]
Julian Hyde commented on CALCITE-5986:
--
I saw your PR.
* The "primary" and "secondary" concepts are very difficult to grasp. I think
that the primary and secondary family methods in SqlTypeName need more
documentation.
* Do the methods both need to be nullable?
* I think that the getFamily method should be deprecated.
* "The SqlTypeFamily for FP types is incorrect" doesn't seem ready for release
notes
> The SqlTypeFamily for FP types is incorrect
> ---
>
> Key: CALCITE-5986
> URL: https://issues.apache.org/jira/browse/CALCITE-5986
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Minor
> Labels: pull-request-available
>
> In SqlTypeFamily we have this code:
> {code:java}
> private static final Map JDBC_TYPE_TO_FAMILY =
> ...
> .put(Types.FLOAT, NUMERIC)
> .put(Types.REAL, NUMERIC)
> .put(Types.DOUBLE, NUMERIC)
> {code}
> But it looks to me like the type family should be APPROXIMATE_NUMERIC.
> This impacts the way RelToSqlConverter works, for instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[
https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763275#comment-17763275
]
Jerin John commented on CALCITE-5979:
-
[~julianhyde] [~tanclary], did some further testing with RE2 and it seems like
the the invalid regex cases we identified in CALCITE-5910 are being processed
identical to Java regex, i.e. BQ was able to identify those as incorrect regex
but it might not be getting caught directly by the RE2 library as per our
original assumption, there could be custom error checks they implemented on top
of it.
There might be a possibility that they're using a different/internal version of
RE2 (possibly the C wrapper) while I tested with the [re2j Java
wrapper|https://github.com/google/re2j] which may not have included this
functionality, not sure how we can ascertain that but based on this observation
RE2 backend might not be the reason for these additional supported features.
> Add REGEXP_REPLACE function (enabled in BigQuery library)
> -
>
> Key: CALCITE-5979
> URL: https://issues.apache.org/jira/browse/CALCITE-5979
> Project: Calcite
> Issue Type: Task
>Reporter: Jerin John
>Assignee: Jerin John
>Priority: Major
> Labels: pull-request-available
>
> Add support for [REGEXP_REPLACE
> |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace]
> function from BigQuery.
> *{{REGEXP_REPLACE(value, regexp, replacement)}}*
> Returns a STRING where all substrings of {{value}} that match regular
> expression {{regexp}} are replaced with {{{}replacement{}}}.
> backslashed-escaped digits (\1 to \9) can be used within the {{replacement}}
> argument to insert text matching the corresponding parenthesized group in the
> {{regexp}} pattern.
> Example (added one space between \ \ to override md formatting):
> {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}}
> |result|
> |aXc|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[
https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CALCITE-5986:
Labels: pull-request-available (was: )
> The SqlTypeFamily for FP types is incorrect
> ---
>
> Key: CALCITE-5986
> URL: https://issues.apache.org/jira/browse/CALCITE-5986
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Minor
> Labels: pull-request-available
>
> In SqlTypeFamily we have this code:
> {code:java}
> private static final Map JDBC_TYPE_TO_FAMILY =
> ...
> .put(Types.FLOAT, NUMERIC)
> .put(Types.REAL, NUMERIC)
> .put(Types.DOUBLE, NUMERIC)
> {code}
> But it looks to me like the type family should be APPROXIMATE_NUMERIC.
> This impacts the way RelToSqlConverter works, for instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[
https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763254#comment-17763254
]
Tanner Clary commented on CALCITE-5979:
---
Yeah as long as it is documented I think that sounds like a good idea.
> Add REGEXP_REPLACE function (enabled in BigQuery library)
> -
>
> Key: CALCITE-5979
> URL: https://issues.apache.org/jira/browse/CALCITE-5979
> Project: Calcite
> Issue Type: Task
>Reporter: Jerin John
>Assignee: Jerin John
>Priority: Major
> Labels: pull-request-available
>
> Add support for [REGEXP_REPLACE
> |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace]
> function from BigQuery.
> *{{REGEXP_REPLACE(value, regexp, replacement)}}*
> Returns a STRING where all substrings of {{value}} that match regular
> expression {{regexp}} are replaced with {{{}replacement{}}}.
> backslashed-escaped digits (\1 to \9) can be used within the {{replacement}}
> argument to insert text matching the corresponding parenthesized group in the
> {{regexp}} pattern.
> Example (added one space between \ \ to override md formatting):
> {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}}
> |result|
> |aXc|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[
https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763251#comment-17763251
]
Mihai Budiu commented on CALCITE-5986:
--
There should probably exist a test that checks the consistency of these
mappings.
> The SqlTypeFamily for FP types is incorrect
> ---
>
> Key: CALCITE-5986
> URL: https://issues.apache.org/jira/browse/CALCITE-5986
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> In SqlTypeFamily we have this code:
> {code:java}
> private static final Map JDBC_TYPE_TO_FAMILY =
> ...
> .put(Types.FLOAT, NUMERIC)
> .put(Types.REAL, NUMERIC)
> .put(Types.DOUBLE, NUMERIC)
> {code}
> But it looks to me like the type family should be APPROXIMATE_NUMERIC.
> This impacts the way RelToSqlConverter works, for instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[
https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763249#comment-17763249
]
Mihai Budiu commented on CALCITE-5986:
--
Another problem is that SqlTypeFamily.getTypeNames() DOES map
APPROXIMATE_NUMERIC to DOUBLE, FLOAT, etc. So we have two inconsistent mappings
describing these types.
> The SqlTypeFamily for FP types is incorrect
> ---
>
> Key: CALCITE-5986
> URL: https://issues.apache.org/jira/browse/CALCITE-5986
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> In SqlTypeFamily we have this code:
> {code:java}
> private static final Map JDBC_TYPE_TO_FAMILY =
> ...
> .put(Types.FLOAT, NUMERIC)
> .put(Types.REAL, NUMERIC)
> .put(Types.DOUBLE, NUMERIC)
> {code}
> But it looks to me like the type family should be APPROXIMATE_NUMERIC.
> This impacts the way RelToSqlConverter works, for instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[
https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763250#comment-17763250
]
Julian Hyde commented on CALCITE-5984:
--
I don't know. If there is a sane way to prevent trimming, please contribute it.
And contribute tests to demonstrate that trimming does not occur.
If there are some rules that cannot operate without trimming (I don't know,
just speculating) then I guess those rules would need to be disabled.
> Cannot fully disable trimming unused fields using Calcite configs
> -
>
> Key: CALCITE-5984
> URL: https://issues.apache.org/jira/browse/CALCITE-5984
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.34.0
>Reporter: Wegdan Ghazi
>Priority: Major
>
> In Prepare#prepareSql there are 2 instances of trimming unused fields
> 1.
> [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
> called
> [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297]
>
> {code:java}
> // Trim unused fields.
> root = trimUnusedFields(root); {code}
> 2.
> {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color}
> called
> [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312]
> within
> {code:java}
> root = optimize(root, getMaterializations(), getLattices());
> {code}
> I want to turn of the field trimming due to a personalisation I'm
> implementing to scan a specific API, which I would think is possible using
> [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26]
> I attempted to disable the trimming using hooks [PROGRAM,
> SQL2REL_CONVERTER_CONFIG_BUILDER], which succeeded to remove the trimming
> program, yet on
> [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
> {code:java}
> // code placehprotected RelRoot trimUnusedFields(RelRoot root) {
> final SqlToRelConverter.Config config = SqlToRelConverter.config()
> .withTrimUnusedFields(shouldTrim(root.rel))
> .withExpand(THREAD_EXPAND.get())
>
> .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get()));
> final SqlToRelConverter converter =
> getSqlToRelConverter(getSqlValidator(), catalogReader, config);
> final boolean ordered = !root.collation.getFieldCollations().isEmpty();
> final boolean dml = SqlKind.DML.contains(root.kind);
> return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel));
> }
> {code}
> The config passed (and edited by the hook) gets overwritten and only
> _shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to
> happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}.
> Is this the intended behaviour? Any help is appreciated, thanks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[
https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde updated CALCITE-5984:
-
Fix Version/s: (was: 1.34.0)
> Cannot fully disable trimming unused fields using Calcite configs
> -
>
> Key: CALCITE-5984
> URL: https://issues.apache.org/jira/browse/CALCITE-5984
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.34.0
>Reporter: Wegdan Ghazi
>Priority: Major
>
> In Prepare#prepareSql there are 2 instances of trimming unused fields
> 1.
> [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
> called
> [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297]
>
> {code:java}
> // Trim unused fields.
> root = trimUnusedFields(root); {code}
> 2.
> {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color}
> called
> [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312]
> within
> {code:java}
> root = optimize(root, getMaterializations(), getLattices());
> {code}
> I want to turn of the field trimming due to a personalisation I'm
> implementing to scan a specific API, which I would think is possible using
> [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26]
> I attempted to disable the trimming using hooks [PROGRAM,
> SQL2REL_CONVERTER_CONFIG_BUILDER], which succeeded to remove the trimming
> program, yet on
> [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
> {code:java}
> // code placehprotected RelRoot trimUnusedFields(RelRoot root) {
> final SqlToRelConverter.Config config = SqlToRelConverter.config()
> .withTrimUnusedFields(shouldTrim(root.rel))
> .withExpand(THREAD_EXPAND.get())
>
> .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get()));
> final SqlToRelConverter converter =
> getSqlToRelConverter(getSqlValidator(), catalogReader, config);
> final boolean ordered = !root.collation.getFieldCollations().isEmpty();
> final boolean dml = SqlKind.DML.contains(root.kind);
> return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel));
> }
> {code}
> The config passed (and edited by the hook) gets overwritten and only
> _shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to
> happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}.
> Is this the intended behaviour? Any help is appreciated, thanks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[
https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763246#comment-17763246
]
Mihai Budiu commented on CALCITE-5986:
--
Moreover, if no type belongs to this family, it means that all the code that
checks for this family is dead.
> The SqlTypeFamily for FP types is incorrect
> ---
>
> Key: CALCITE-5986
> URL: https://issues.apache.org/jira/browse/CALCITE-5986
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> In SqlTypeFamily we have this code:
> {code:java}
> private static final Map JDBC_TYPE_TO_FAMILY =
> ...
> .put(Types.FLOAT, NUMERIC)
> .put(Types.REAL, NUMERIC)
> .put(Types.DOUBLE, NUMERIC)
> {code}
> But it looks to me like the type family should be APPROXIMATE_NUMERIC.
> This impacts the way RelToSqlConverter works, for instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[
https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763245#comment-17763245
]
Julian Hyde commented on CALCITE-5986:
--
Probably would have been better if {{SqlTypeName.FLOAT.getFamily()}} (etc.)
returned {{APPROXIMATE_NUMERIC}} rather than {{NUMERIC}}. But changing it now
would probably have huge impact.
The bug here seems to be that the JDBC adapter generates the wrong literal for
{{FLOAT}} etc. Let's work on fixing that bug.
> The SqlTypeFamily for FP types is incorrect
> ---
>
> Key: CALCITE-5986
> URL: https://issues.apache.org/jira/browse/CALCITE-5986
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> In SqlTypeFamily we have this code:
> {code:java}
> private static final Map JDBC_TYPE_TO_FAMILY =
> ...
> .put(Types.FLOAT, NUMERIC)
> .put(Types.REAL, NUMERIC)
> .put(Types.DOUBLE, NUMERIC)
> {code}
> But it looks to me like the type family should be APPROXIMATE_NUMERIC.
> This impacts the way RelToSqlConverter works, for instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[
https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763243#comment-17763243
]
Mihai Budiu commented on CALCITE-5986:
--
But changing the family to APPROXIMATE_NUMERIC seems to break quite a few tests.
I will investigate whether it's worth making the change.
According to the SQL standard use of the term, this is a bug, these are
approximate numeric types, e.g.,
https://www.andrew.cmu.edu/user/shadow/sql/sql1992.txt
> The SqlTypeFamily for FP types is incorrect
> ---
>
> Key: CALCITE-5986
> URL: https://issues.apache.org/jira/browse/CALCITE-5986
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> In SqlTypeFamily we have this code:
> {code:java}
> private static final Map JDBC_TYPE_TO_FAMILY =
> ...
> .put(Types.FLOAT, NUMERIC)
> .put(Types.REAL, NUMERIC)
> .put(Types.DOUBLE, NUMERIC)
> {code}
> But it looks to me like the type family should be APPROXIMATE_NUMERIC.
> This impacts the way RelToSqlConverter works, for instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5974) Elasticsearch adapter throws ClassCastException when index mapping sets dynamic_templates without properties
[
https://issues.apache.org/jira/browse/CALCITE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763241#comment-17763241
]
Julian Hyde commented on CALCITE-5974:
--
Thanks for improving the summary, [~zoovwang]. Much improved. I just tweaked it
a bit more.
> Elasticsearch adapter throws ClassCastException when index mapping sets
> dynamic_templates without properties
>
>
> Key: CALCITE-5974
> URL: https://issues.apache.org/jira/browse/CALCITE-5974
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.35.0
>Reporter: zhaowang
>Assignee: zhaowang
>Priority: Major
>
> When we use the elasticsearch-adapter, as we config an elasticsearch cluster,
> it will fetchMapping during initialization.
> If a index set dynamic_templates bug no mappings like this:
>
> {code:java}
> {
> "test_index" : {
> "mappings" : {
> "dynamic_templates" : [
> {
> "integers" : {
> "match_mapping_type" : "long",
> "mapping" : {
> "type" : "integer"
> }
> }
> }
> ]
> }
> }
> } {code}
> {{org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties}}
> throw exception:
>
> {code:java}
> java.lang.ClassCastException:
> com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to
> com.fasterxml.jackson.databind.node.ObjectNode
> at
>
> org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133)
> {code}
> So ElasticsearchTransport initialize failed, all index of this es cluster
> access failed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5974) Elasticsearch adapter throws ClassCastException when index mapping sets dynamic_templates without properties
[
https://issues.apache.org/jira/browse/CALCITE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde updated CALCITE-5974:
-
Summary: Elasticsearch adapter throws ClassCastException when index mapping
sets dynamic_templates without properties (was: Elasticsearch adapter fetch
mapping error when index mapping set dynamic_templates without properties)
> Elasticsearch adapter throws ClassCastException when index mapping sets
> dynamic_templates without properties
>
>
> Key: CALCITE-5974
> URL: https://issues.apache.org/jira/browse/CALCITE-5974
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.35.0
>Reporter: zhaowang
>Assignee: zhaowang
>Priority: Major
>
> When we use the elasticsearch-adapter, as we config an elasticsearch cluster,
> it will fetchMapping during initialization.
> If a index set dynamic_templates bug no mappings like this:
>
> {code:java}
> {
> "test_index" : {
> "mappings" : {
> "dynamic_templates" : [
> {
> "integers" : {
> "match_mapping_type" : "long",
> "mapping" : {
> "type" : "integer"
> }
> }
> }
> ]
> }
> }
> } {code}
> {{org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties}}
> throw exception:
>
> {code:java}
> java.lang.ClassCastException:
> com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to
> com.fasterxml.jackson.databind.node.ObjectNode
> at
>
> org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133)
> {code}
> So ElasticsearchTransport initialize failed, all index of this es cluster
> access failed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[
https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763238#comment-17763238
]
Mihai Budiu commented on CALCITE-5986:
--
{code:java}
FLOAT(PrecScale.NO_NO, false, Types.FLOAT, SqlTypeFamily.NUMERIC),
REAL(PrecScale.NO_NO, false, Types.REAL, SqlTypeFamily.NUMERIC),
DOUBLE(PrecScale.NO_NO, false, Types.DOUBLE, SqlTypeFamily.NUMERIC),
{code}
This leads to at least one bug, in RelToSqlConverter FP literals are emitted as
DECIMAL literals.
> The SqlTypeFamily for FP types is incorrect
> ---
>
> Key: CALCITE-5986
> URL: https://issues.apache.org/jira/browse/CALCITE-5986
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> In SqlTypeFamily we have this code:
> {code:java}
> private static final Map JDBC_TYPE_TO_FAMILY =
> ...
> .put(Types.FLOAT, NUMERIC)
> .put(Types.REAL, NUMERIC)
> .put(Types.DOUBLE, NUMERIC)
> {code}
> But it looks to me like the type family should be APPROXIMATE_NUMERIC.
> This impacts the way RelToSqlConverter works, for instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[
https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763233#comment-17763233
]
Jerin John edited comment on CALCITE-5979 at 9/8/23 7:44 PM:
-
Totally agree Tanner, we can think about skipping support for this feature and
document as a known bug but not sure of the future impact.
Wrt enabling this feature for other dialects, with some limited testing on
MySQL I can see that the results are altered for scenarios where {{"\ (\d)"}} s
are present in the string along with a capturing group in regex pattern, since
this starts replacing it with the capturing group where originally it might not
be intended, or might throw errors for indices of missing groups (these are
very exceptional scenarios when someone is specifically coding the string with
and pattern to be this way, but still an edge case).
[~julianhyde] makes sense, based on these implementations we have encountered
only two cases where BQ supports additional syntax, one with this above
capturing groups and the other with some invalid regex cases as documented in
[CALCITE-5910|https://issues.apache.org/jira/browse/CALCITE-5910]
was (Author: JIRAUSER301314):
Totally agree Tanner, we can think about skipping support for this feature and
document as a known bug but not sure of the future impact.
Wrt enabling this feature for other dialects, with some limited testing on
MySQL I can see that the results are altered for scenarios where {{"\ (\d)"}} s
are present in the string along with a capturing group in regex pattern, since
this starts replacing it with the capturing group where originally it might not
be intended, or might throw errors for indices of missing groups (these are
very exceptional scenarios when someone is specifically coding the string with
and pattern to be this way, but still an edge case).
> Add REGEXP_REPLACE function (enabled in BigQuery library)
> -
>
> Key: CALCITE-5979
> URL: https://issues.apache.org/jira/browse/CALCITE-5979
> Project: Calcite
> Issue Type: Task
>Reporter: Jerin John
>Assignee: Jerin John
>Priority: Major
> Labels: pull-request-available
>
> Add support for [REGEXP_REPLACE
> |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace]
> function from BigQuery.
> *{{REGEXP_REPLACE(value, regexp, replacement)}}*
> Returns a STRING where all substrings of {{value}} that match regular
> expression {{regexp}} are replaced with {{{}replacement{}}}.
> backslashed-escaped digits (\1 to \9) can be used within the {{replacement}}
> argument to insert text matching the corresponding parenthesized group in the
> {{regexp}} pattern.
> Example (added one space between \ \ to override md formatting):
> {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}}
> |result|
> |aXc|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5974) Elasticsearch adapter fetch mapping error when index mapping set dynamic_templates without properties
[
https://issues.apache.org/jira/browse/CALCITE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde updated CALCITE-5974:
-
Description:
When we use the elasticsearch-adapter, as we config an elasticsearch cluster,
it will fetchMapping during initialization.
If a index set dynamic_templates bug no mappings like this:
{code:java}
{
"test_index" : {
"mappings" : {
"dynamic_templates" : [
{
"integers" : {
"match_mapping_type" : "long",
"mapping" : {
"type" : "integer"
}
}
}
]
}
}
} {code}
{{org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties}}
throw exception:
{code:java}
java.lang.ClassCastException:
com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to
com.fasterxml.jackson.databind.node.ObjectNode
at
org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133)
{code}
So ElasticsearchTransport initialize failed, all index of this es cluster
access failed.
was:
When use es-adapter, we config a es cluster, it will fetchMapping during
initialization.
If a index set dynamic_templates bug no mappings like this:
{code:java}
{
"test_index" : {
"mappings" : {
"dynamic_templates" : [
{
"integers" : {
"match_mapping_type" : "long",
"mapping" : {
"type" : "integer"
}
}
}
]
}
}
} {code}
org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties
throw exception:
{code:java}
java.lang.ClassCastException:
com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to
com.fasterxml.jackson.databind.node.ObjectNode
at
org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133)
{code}
So ElasticsearchTransport initialize failed, all index of this es cluster
access failed.
> Elasticsearch adapter fetch mapping error when index mapping set
> dynamic_templates without properties
> -
>
> Key: CALCITE-5974
> URL: https://issues.apache.org/jira/browse/CALCITE-5974
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.35.0
>Reporter: zhaowang
>Assignee: zhaowang
>Priority: Major
>
> When we use the elasticsearch-adapter, as we config an elasticsearch cluster,
> it will fetchMapping during initialization.
> If a index set dynamic_templates bug no mappings like this:
>
> {code:java}
> {
> "test_index" : {
> "mappings" : {
> "dynamic_templates" : [
> {
> "integers" : {
> "match_mapping_type" : "long",
> "mapping" : {
> "type" : "integer"
> }
> }
> }
> ]
> }
> }
> } {code}
> {{org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties}}
> throw exception:
>
> {code:java}
> java.lang.ClassCastException:
> com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to
> com.fasterxml.jackson.databind.node.ObjectNode
> at
>
> org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133)
> {code}
> So ElasticsearchTransport initialize failed, all index of this es cluster
> access failed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5985) FilterTableFunctionTransposeRule should not use "Logical" RelNodes
[
https://issues.apache.org/jira/browse/CALCITE-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763235#comment-17763235
]
Julian Hyde commented on CALCITE-5985:
--
I agree. Perhaps retain an instance of the rule that continues to match only
{{LogicalFilter}} and {{LogicalTableScan}}, and also add a new instance that is
generic.
Also, use the {{filter}} and {{functionScan}} methods in {{RelBuilder}}.
> FilterTableFunctionTransposeRule should not use "Logical" RelNodes
> --
>
> Key: CALCITE-5985
> URL: https://issues.apache.org/jira/browse/CALCITE-5985
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Steve Carlin
>Priority: Major
>
> The FilterTableFunctionTransposeRule uses LogicalTableFunctionScan and
> LogicalFilter RelNodes in its rule. We should be using the appropriate
> TableFunctionScan and Filter interface RelNodes instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[
https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763233#comment-17763233
]
Jerin John commented on CALCITE-5979:
-
Totally agree Tanner, we can think about skipping support for this feature and
document as a known bug but not sure of the future impact.
Wrt enabling this feature for other dialects, with some limited testing on
MySQL I can see that the results are altered for scenarios where {{"\ (\d)"}} s
are present in the string along with a capturing group in regex pattern, since
this starts replacing it with the capturing group where originally it might not
be intended, or might throw errors for indices of missing groups (these are
very exceptional scenarios when someone is specifically coding the string with
and pattern to be this way, but still an edge case).
> Add REGEXP_REPLACE function (enabled in BigQuery library)
> -
>
> Key: CALCITE-5979
> URL: https://issues.apache.org/jira/browse/CALCITE-5979
> Project: Calcite
> Issue Type: Task
>Reporter: Jerin John
>Assignee: Jerin John
>Priority: Major
> Labels: pull-request-available
>
> Add support for [REGEXP_REPLACE
> |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace]
> function from BigQuery.
> *{{REGEXP_REPLACE(value, regexp, replacement)}}*
> Returns a STRING where all substrings of {{value}} that match regular
> expression {{regexp}} are replaced with {{{}replacement{}}}.
> backslashed-escaped digits (\1 to \9) can be used within the {{replacement}}
> argument to insert text matching the corresponding parenthesized group in the
> {{regexp}} pattern.
> Example (added one space between \ \ to override md formatting):
> {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}}
> |result|
> |aXc|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[
https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763232#comment-17763232
]
Julian Hyde edited comment on CALCITE-5979 at 9/8/23 7:35 PM:
--
The simplest thing is to change the specification to match what is easy to
implement. Specifically, to say that {{REGEXP_REPLACE}} supports Java regexp
syntax.
We can have a follow-up case to allow the {{REGEXP_xxx}} functions support BQ
syntax, and to devise a flag to enable the BQ mode. In that case, list the
cases where the current implementation is wrong. Over time, the demand to fix
that follow-up case will grow, but we can defer it for a while.
As you can see, I'm punting on whether we have one operator or two. I really
think that kicking the can down the road is the best strategy.
was (Author: julianhyde):
The simplest thing is to change the specification to match what is easy to
implement. Specifically, to say that {{REGEXP_REPLACE}} supports Java regexp
syntax.
We can have a follow-up case to allow the {{REGEXP_xxx}} functions support BQ
syntax, and to devise a flag to enable the BQ mode. In that case, list the
cases where the current implementation is wrong. Over time, the demand to fix
that follow-up case will grow, but we can defer it for a while.
> Add REGEXP_REPLACE function (enabled in BigQuery library)
> -
>
> Key: CALCITE-5979
> URL: https://issues.apache.org/jira/browse/CALCITE-5979
> Project: Calcite
> Issue Type: Task
>Reporter: Jerin John
>Assignee: Jerin John
>Priority: Major
> Labels: pull-request-available
>
> Add support for [REGEXP_REPLACE
> |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace]
> function from BigQuery.
> *{{REGEXP_REPLACE(value, regexp, replacement)}}*
> Returns a STRING where all substrings of {{value}} that match regular
> expression {{regexp}} are replaced with {{{}replacement{}}}.
> backslashed-escaped digits (\1 to \9) can be used within the {{replacement}}
> argument to insert text matching the corresponding parenthesized group in the
> {{regexp}} pattern.
> Example (added one space between \ \ to override md formatting):
> {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}}
> |result|
> |aXc|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[
https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763232#comment-17763232
]
Julian Hyde commented on CALCITE-5979:
--
The simplest thing is to change the specification to match what is easy to
implement. Specifically, to say that {{REGEXP_REPLACE}} supports Java regexp
syntax.
We can have a follow-up case to allow the {{REGEXP_xxx}} functions support BQ
syntax, and to devise a flag to enable the BQ mode. In that case, list the
cases where the current implementation is wrong. Over time, the demand to fix
that follow-up case will grow, but we can defer it for a while.
> Add REGEXP_REPLACE function (enabled in BigQuery library)
> -
>
> Key: CALCITE-5979
> URL: https://issues.apache.org/jira/browse/CALCITE-5979
> Project: Calcite
> Issue Type: Task
>Reporter: Jerin John
>Assignee: Jerin John
>Priority: Major
> Labels: pull-request-available
>
> Add support for [REGEXP_REPLACE
> |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace]
> function from BigQuery.
> *{{REGEXP_REPLACE(value, regexp, replacement)}}*
> Returns a STRING where all substrings of {{value}} that match regular
> expression {{regexp}} are replaced with {{{}replacement{}}}.
> backslashed-escaped digits (\1 to \9) can be used within the {{replacement}}
> argument to insert text matching the corresponding parenthesized group in the
> {{regexp}} pattern.
> Example (added one space between \ \ to override md formatting):
> {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}}
> |result|
> |aXc|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[
https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763228#comment-17763228
]
Julian Hyde commented on CALCITE-5986:
--
I don't see any problem in {{enum SqlTypeName}}. This looks perfect:
{code}
public static final List INT_TYPES =
ImmutableList.of(TINYINT, SMALLINT, INTEGER, BIGINT);
public static final List EXACT_TYPES =
combine(INT_TYPES, ImmutableList.of(DECIMAL));
public static final List APPROX_TYPES =
ImmutableList.of(FLOAT, REAL, DOUBLE);
public static final List NUMERIC_TYPES =
combine(EXACT_TYPES, APPROX_TYPES);
{code}
> The SqlTypeFamily for FP types is incorrect
> ---
>
> Key: CALCITE-5986
> URL: https://issues.apache.org/jira/browse/CALCITE-5986
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> In SqlTypeFamily we have this code:
> {code:java}
> private static final Map JDBC_TYPE_TO_FAMILY =
> ...
> .put(Types.FLOAT, NUMERIC)
> .put(Types.REAL, NUMERIC)
> .put(Types.DOUBLE, NUMERIC)
> {code}
> But it looks to me like the type family should be APPROXIMATE_NUMERIC.
> This impacts the way RelToSqlConverter works, for instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[
https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763227#comment-17763227
]
Julian Hyde commented on CALCITE-5986:
--
JDBC_TYPE_TO_FAMILY is private and not used, AFAICT. If so, it's not really a
bug.
I agree that mapping to NUMERIC rather than APPROXIMATE_NUMERIC looks a bit
strange; the data model is a hierarchy. But really, it depends what that field
is used for, which at present is nothing.
> The SqlTypeFamily for FP types is incorrect
> ---
>
> Key: CALCITE-5986
> URL: https://issues.apache.org/jira/browse/CALCITE-5986
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> In SqlTypeFamily we have this code:
> {code:java}
> private static final Map JDBC_TYPE_TO_FAMILY =
> ...
> .put(Types.FLOAT, NUMERIC)
> .put(Types.REAL, NUMERIC)
> .put(Types.DOUBLE, NUMERIC)
> {code}
> But it looks to me like the type family should be APPROXIMATE_NUMERIC.
> This impacts the way RelToSqlConverter works, for instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[
https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763220#comment-17763220
]
Mihai Budiu commented on CALCITE-5986:
--
There's a similar problem in the SqlTypeName enum. Am I misinterpreting the
meaning of APPROXIMATE_NUMERIC?
> The SqlTypeFamily for FP types is incorrect
> ---
>
> Key: CALCITE-5986
> URL: https://issues.apache.org/jira/browse/CALCITE-5986
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> In SqlTypeFamily we have this code:
> {code:java}
> private static final Map JDBC_TYPE_TO_FAMILY =
> ...
> .put(Types.FLOAT, NUMERIC)
> .put(Types.REAL, NUMERIC)
> .put(Types.DOUBLE, NUMERIC)
> {code}
> But it looks to me like the type family should be APPROXIMATE_NUMERIC.
> This impacts the way RelToSqlConverter works, for instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
Mihai Budiu created CALCITE-5986:
Summary: The SqlTypeFamily for FP types is incorrect
Key: CALCITE-5986
URL: https://issues.apache.org/jira/browse/CALCITE-5986
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.35.0
Reporter: Mihai Budiu
In SqlTypeFamily we have this code:
{code:java}
private static final Map JDBC_TYPE_TO_FAMILY =
...
.put(Types.FLOAT, NUMERIC)
.put(Types.REAL, NUMERIC)
.put(Types.DOUBLE, NUMERIC)
{code}
But it looks to me like the type family should be APPROXIMATE_NUMERIC.
This impacts the way RelToSqlConverter works, for instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[
https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763215#comment-17763215
]
Tanner Clary edited comment on CALCITE-5979 at 9/8/23 6:40 PM:
---
I agree that it seems like a bit much to accommodate for what seems to be a
pretty small difference, but I also think it is hard to draw the line on what
warrants a new operator entry or not. For instance, {{FLOOR}} and {{SUBSTR}}
both have dialect-specific operators because the behavior varies across
dialects in one way or another (perhaps in return type, or unparsing syntax,
etc.). I can foresee a situation where we decide not to support this specific
behavior to avoid adding a new operator, and then somewhere down the line a bug
gets filed because someone is expecting this behavior to work because we say we
support BigQuery's functionality, and instead it fails. I only mention that
because I have introduced some of those bugs myself. Maybe that's overthinking
it a bit but it's worth keeping in mind I think.
I am also curious what would happen if this behavior was enabled for the other
dialects? Would it throw an unexpected error? Would it return the wrong
results? Would it change anything?
was (Author: JIRAUSER298151):
I agree that it seems like a bit much to accommodate for what seems to be a
pretty small difference, but I also think it is hard to draw the line on what
warrants a new operator entry or not. For instance, {{FLOOR}} and {{SUBSTR}}
both have dialect-specific operators because the behavior varies across
dialects in one way or another (perhaps in return type, or unparsing syntax,
etc.). I can foresee a situation where we decide not to support this specific
behavior to avoid adding a new operator, and then somewhere down the line a bug
gets filed because someone is expecting this behavior to work because we say we
support BigQuery's functionality, and instead it fails. I only mention that
because I have introduced some of those bugs myself. Maybe that's overthinking
it a bit but it's worth keeping in mind I think.
> Add REGEXP_REPLACE function (enabled in BigQuery library)
> -
>
> Key: CALCITE-5979
> URL: https://issues.apache.org/jira/browse/CALCITE-5979
> Project: Calcite
> Issue Type: Task
>Reporter: Jerin John
>Assignee: Jerin John
>Priority: Major
> Labels: pull-request-available
>
> Add support for [REGEXP_REPLACE
> |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace]
> function from BigQuery.
> *{{REGEXP_REPLACE(value, regexp, replacement)}}*
> Returns a STRING where all substrings of {{value}} that match regular
> expression {{regexp}} are replaced with {{{}replacement{}}}.
> backslashed-escaped digits (\1 to \9) can be used within the {{replacement}}
> argument to insert text matching the corresponding parenthesized group in the
> {{regexp}} pattern.
> Example (added one space between \ \ to override md formatting):
> {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}}
> |result|
> |aXc|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[
https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763215#comment-17763215
]
Tanner Clary edited comment on CALCITE-5979 at 9/8/23 6:39 PM:
---
I agree that it seems like a bit much to accommodate for what seems to be a
pretty small difference, but I also think it is hard to draw the line on what
warrants a new operator entry or not. For instance, {{FLOOR}} and {{SUBSTR}}
both have dialect-specific operators because the behavior varies across
dialects in one way or another (perhaps in return type, or unparsing syntax,
etc.). I can foresee a situation where we decide not to support this specific
behavior to avoid adding a new operator, and then somewhere down the line a bug
gets filed because someone is expecting this behavior to work because we say we
support BigQuery's functionality, and instead it fails. I only mention that
because I have introduced some of those bugs myself. Maybe that's overthinking
it a bit but it's worth keeping in mind I think.
was (Author: JIRAUSER298151):
I agree that it seems like a bit much to accommodate for what seems to be a
pretty small difference, but I also think it is hard to draw the line on what
warrants a new operator entry or not. For instance, {{FLOOR}} and {{SUBSTR}}
both have dialect-specific operators because the behavior varies across
dialects in one way or another (perhaps in return type, or unparsing syntax,
etc.). I can foresee a situation where we decide not to support this specific
behavior to avoid adding a new operator, and then somewhere down the line a bug
gets filed because someone is expecting this behavior to work because we say we
support BigQuery's functionality, and instead it fails. I only mention that
because I have introduced some of those bugs myself (and fixed a couple, too).
Maybe that's overthinking it a bit but it's worth keeping in mind I think.
> Add REGEXP_REPLACE function (enabled in BigQuery library)
> -
>
> Key: CALCITE-5979
> URL: https://issues.apache.org/jira/browse/CALCITE-5979
> Project: Calcite
> Issue Type: Task
>Reporter: Jerin John
>Assignee: Jerin John
>Priority: Major
> Labels: pull-request-available
>
> Add support for [REGEXP_REPLACE
> |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace]
> function from BigQuery.
> *{{REGEXP_REPLACE(value, regexp, replacement)}}*
> Returns a STRING where all substrings of {{value}} that match regular
> expression {{regexp}} are replaced with {{{}replacement{}}}.
> backslashed-escaped digits (\1 to \9) can be used within the {{replacement}}
> argument to insert text matching the corresponding parenthesized group in the
> {{regexp}} pattern.
> Example (added one space between \ \ to override md formatting):
> {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}}
> |result|
> |aXc|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[
https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763215#comment-17763215
]
Tanner Clary commented on CALCITE-5979:
---
I agree that it seems like a bit much to accommodate for what seems to be a
pretty small difference, but I also think it is hard to draw the line on what
warrants a new operator entry or not. For instance, {{FLOOR}} and {{SUBSTR}}
both have dialect-specific operators because the behavior varies across
dialects in one way or another (perhaps in return type, or unparsing syntax,
etc.). I can foresee a situation where we decide not to support this specific
behavior to avoid adding a new operator, and then somewhere down the line a bug
gets filed because someone is expecting this behavior to work because we say we
support BigQuery's functionality, and instead it fails. I only mention that
because I have introduced some of those bugs myself (and fixed a couple, too).
Maybe that's overthinking it a bit but it's worth keeping in mind I think.
> Add REGEXP_REPLACE function (enabled in BigQuery library)
> -
>
> Key: CALCITE-5979
> URL: https://issues.apache.org/jira/browse/CALCITE-5979
> Project: Calcite
> Issue Type: Task
>Reporter: Jerin John
>Assignee: Jerin John
>Priority: Major
> Labels: pull-request-available
>
> Add support for [REGEXP_REPLACE
> |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace]
> function from BigQuery.
> *{{REGEXP_REPLACE(value, regexp, replacement)}}*
> Returns a STRING where all substrings of {{value}} that match regular
> expression {{regexp}} are replaced with {{{}replacement{}}}.
> backslashed-escaped digits (\1 to \9) can be used within the {{replacement}}
> argument to insert text matching the corresponding parenthesized group in the
> {{regexp}} pattern.
> Example (added one space between \ \ to override md formatting):
> {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}}
> |result|
> |aXc|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[
https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763212#comment-17763212
]
Jerin John commented on CALCITE-5979:
-
Hi [~julianhyde] and everyone,
I wanted to get your opinion on this regex implementation for BQ, my attempt
was to reuse the existing implementation of REGEXP_REPLACE from other
libraries. But if you check this [BQ
documentation|https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace],
there's a group replacement syntax supported by BQ involving {{"\"}} s which
is not currently available for the other libraries, so it needs a precursor
step to do a string replacement of this kind:
{{replacement = replacement.replaceAll("\\(\d)", "\$$1");}}
I tested this syntax with {{re2j}} library to check if its something supported
by [RE2|https://github.com/google/re2] and not Java regex but thats not the
case, seems like BQ might be custom processing it first. It's not evident why
they went with this syntax since Java and regex standard annotation to index
capturing groups from an expression is like {{$1, $2..}} and thats how we are
replacing the {{"\"}} s in the above code as well.
I discussed with [~tanclary] and from what i understand, one possible approach
is to modify the parsers to redirect to a {{SqlStdOperatorTable}} method to
check the library conformance, and create a new BQ specific operator
{{REGEXP_REPLACE_BIG_QUERY}} alongside the existing {{REGEXP_REPLACE}}
operator. Op {{REGEXP_REPLACE_BIG_QUERY}} is mapped to its own {{SqlFunction}}
for this custom preprocessing which should then call the original
{{regexReplace}} method used by other dialects.
I get that this design abstracts the dialect specific logic out of rex layer,
but also seems like a lot of steps to identify the conformance. It'd be great
to get your thoughts on if there's a better approach or if this seems like the
right way to go.
> Add REGEXP_REPLACE function (enabled in BigQuery library)
> -
>
> Key: CALCITE-5979
> URL: https://issues.apache.org/jira/browse/CALCITE-5979
> Project: Calcite
> Issue Type: Task
>Reporter: Jerin John
>Assignee: Jerin John
>Priority: Major
> Labels: pull-request-available
>
> Add support for [REGEXP_REPLACE
> |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace]
> function from BigQuery.
> *{{REGEXP_REPLACE(value, regexp, replacement)}}*
> Returns a STRING where all substrings of {{value}} that match regular
> expression {{regexp}} are replaced with {{{}replacement{}}}.
> backslashed-escaped digits (\1 to \9) can be used within the {{replacement}}
> argument to insert text matching the corresponding parenthesized group in the
> {{regexp}} pattern.
> Example (added one space between \ \ to override md formatting):
> {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}}
> |result|
> |aXc|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-5985) FilterTableFunctionTransposeRule should not use "Logical" RelNodes
Steve Carlin created CALCITE-5985: - Summary: FilterTableFunctionTransposeRule should not use "Logical" RelNodes Key: CALCITE-5985 URL: https://issues.apache.org/jira/browse/CALCITE-5985 Project: Calcite Issue Type: Bug Components: core Reporter: Steve Carlin The FilterTableFunctionTransposeRule uses LogicalTableFunctionScan and LogicalFilter RelNodes in its rule. We should be using the appropriate TableFunctionScan and Filter interface RelNodes instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[
https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wegdan Ghazi updated CALCITE-5984:
--
Description:
In Prepare#prepareSql there are 2 instances of trimming unused fields
1.
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297]
{code:java}
// Trim unused fields.
root = trimUnusedFields(root); {code}
2.
{color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color}
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312]
within
{code:java}
root = optimize(root, getMaterializations(), getLattices());
{code}
I want to turn of the field trimming due to a personalisation I'm implementing
to scan a specific API, which I would think is possible using
[SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26]
I attempted to disable the trimming using hooks [PROGRAM,
SQL2REL_CONVERTER_CONFIG_BUILDER], which succeeded to remove the trimming
program, yet on
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
{code:java}
// code placehprotected RelRoot trimUnusedFields(RelRoot root) {
final SqlToRelConverter.Config config = SqlToRelConverter.config()
.withTrimUnusedFields(shouldTrim(root.rel))
.withExpand(THREAD_EXPAND.get())
.withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get()));
final SqlToRelConverter converter =
getSqlToRelConverter(getSqlValidator(), catalogReader, config);
final boolean ordered = !root.collation.getFieldCollations().isEmpty();
final boolean dml = SqlKind.DML.contains(root.kind);
return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel));
}
{code}
The config passed (and edited by the hook) gets overwritten and only
_shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to
happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}.
Is this the intended behaviour? Any help is appreciated, thanks.
was:
In Prepare#prepareSql there are 2 instances of trimming unused fields
1.
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297]
{code:java}
// Trim unused fields.
root = trimUnusedFields(root); {code}
2.
{color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color}
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312]
within
{code:java}
root = optimize(root, getMaterializations(), getLattices());
{code}
I want to turn of the field trimming due to a personalisation I'm implementing
to scan a specific API, which I would think is possible using
[SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26]
I attempted to disable the trimming using hooks [PROGRAM,
SQL2REL_CONVERTER_CONFIG_BUILDER], which succeeded to remove the trimming
program, yet on
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
{code:java}
// code placehprotected RelRoot trimUnusedFields(RelRoot root) {
final SqlToRelConverter.Config config = SqlToRelConverter.config()
.withTrimUnusedFields(shouldTrim(root.rel))
.withExpand(THREAD_EXPAND.get())
.withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get()));
final SqlToRelConverter converter =
getSqlToRelConverter(getSqlValidator(), catalogReader, config);
final boolean ordered = !root.collation.getFieldCollations().isEmpty();
final boolean dml = SqlKind.DML.contains(root.kind);
return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel));
}
{code}
The config passed (and edited by the hook) gets overwritten and
[jira] [Updated] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[
https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wegdan Ghazi updated CALCITE-5984:
--
Description:
In Prepare#prepareSql there are 2 instances of trimming unused fields
1.
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297]
{code:java}
// Trim unused fields.
root = trimUnusedFields(root); {code}
2.
{color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color}
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312]
within
{code:java}
root = optimize(root, getMaterializations(), getLattices());
{code}
I want to turn of the field trimming due to a personalisation I'm implementing
to scan a specific API, which I would think is possible using
[SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26]
I attempted to disable the trimming using hooks [PROGRAM,
SQL2REL_CONVERTER_CONFIG_BUILDER], which succeeded to remove the trimming
program, yet on
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
{code:java}
// code placehprotected RelRoot trimUnusedFields(RelRoot root) {
final SqlToRelConverter.Config config = SqlToRelConverter.config()
.withTrimUnusedFields(shouldTrim(root.rel))
.withExpand(THREAD_EXPAND.get())
.withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get()));
final SqlToRelConverter converter =
getSqlToRelConverter(getSqlValidator(), catalogReader, config);
final boolean ordered = !root.collation.getFieldCollations().isEmpty();
final boolean dml = SqlKind.DML.contains(root.kind);
return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel));
}
{code}
The config passed (and edited by the hook) gets overwritten and only
_shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to
happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}.
was:
In Prepare#prepareSql there are 2 instances of trimming unused fields
1.
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297]
{code:java}
// Trim unused fields.
root = trimUnusedFields(root); {code}
2.
{color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color}
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312]
within
{code:java}
root = optimize(root, getMaterializations(), getLattices());
{code}
I want to turn of the field trimming due to a personalisation I'm implementing
to scan a specific API, which I would think is possible using
[SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26]
I attempted to disable the trimming using hooks [PROGRAM,
SQL2REL_CONVERTER_CONFIG_BUILDER], which succeeded to remove the trimming
program, yet on
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
{code:java}
// code placehprotected RelRoot trimUnusedFields(RelRoot root) {
final SqlToRelConverter.Config config = SqlToRelConverter.config()
.withTrimUnusedFields(shouldTrim(root.rel))
.withExpand(THREAD_EXPAND.get())
.withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get()));
final SqlToRelConverter converter =
getSqlToRelConverter(getSqlValidator(), catalogReader, config);
final boolean ordered = !root.collation.getFieldCollations().isEmpty();
final boolean dml = SqlKind.DML.contains(root.kind);
return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel));
}older
{code}
The config passed (and edited by the hook) gets overwritten and only
_shouldTrim(root.rel)_ can enable/disable trimming, I
[jira] [Updated] (CALCITE-5974) Elasticsearch adapter fetch mapping error when index mapping set dynamic_templates without properties
[
https://issues.apache.org/jira/browse/CALCITE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhaowang updated CALCITE-5974:
--
Summary: Elasticsearch adapter fetch mapping error when index mapping set
dynamic_templates without properties (was: Elasticsearch adapter find mapping
error when index mapping set dynamic_templates without properties)
> Elasticsearch adapter fetch mapping error when index mapping set
> dynamic_templates without properties
> -
>
> Key: CALCITE-5974
> URL: https://issues.apache.org/jira/browse/CALCITE-5974
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.35.0
>Reporter: zhaowang
>Assignee: zhaowang
>Priority: Major
>
> When use es-adapter, we config a es cluster, it will fetchMapping during
> initialization.
> If a index set dynamic_templates bug no mappings like this:
>
> {code:java}
> {
> "test_index" : {
> "mappings" : {
> "dynamic_templates" : [
> {
> "integers" : {
> "match_mapping_type" : "long",
> "mapping" : {
> "type" : "integer"
> }
> }
> }
> ]
> }
> }
> } {code}
>
>
> org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties
> throw exception:
>
> {code:java}
> java.lang.ClassCastException:
> com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to
> com.fasterxml.jackson.databind.node.ObjectNode
> at
>
> org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133)
> {code}
> So ElasticsearchTransport initialize failed, all index of this es cluster
> access failed.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5974) Elasticsearch adapter find mapping error when index mapping set dynamic_templates without properties causing bug
[
https://issues.apache.org/jira/browse/CALCITE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhaowang updated CALCITE-5974:
--
Summary: Elasticsearch adapter find mapping error when index mapping set
dynamic_templates without properties causing bug (was: Es index mapping set
dynamic_templates without properties causing bug)
> Elasticsearch adapter find mapping error when index mapping set
> dynamic_templates without properties causing bug
>
>
> Key: CALCITE-5974
> URL: https://issues.apache.org/jira/browse/CALCITE-5974
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.35.0
>Reporter: zhaowang
>Assignee: zhaowang
>Priority: Major
>
> When use es-adapter, we config a es cluster, it will fetchMapping during
> initialization.
> If a index set dynamic_templates bug no mappings like this:
>
> {code:java}
> {
> "test_index" : {
> "mappings" : {
> "dynamic_templates" : [
> {
> "integers" : {
> "match_mapping_type" : "long",
> "mapping" : {
> "type" : "integer"
> }
> }
> }
> ]
> }
> }
> } {code}
>
>
> org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties
> throw exception:
>
> {code:java}
> java.lang.ClassCastException:
> com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to
> com.fasterxml.jackson.databind.node.ObjectNode
> at
>
> org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133)
> {code}
> So ElasticsearchTransport initialize failed, all index of this es cluster
> access failed.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5974) Elasticsearch adapter find mapping error when index mapping set dynamic_templates without properties
[
https://issues.apache.org/jira/browse/CALCITE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhaowang updated CALCITE-5974:
--
Summary: Elasticsearch adapter find mapping error when index mapping set
dynamic_templates without properties (was: Elasticsearch adapter find mapping
error when index mapping set dynamic_templates without properties causing bug)
> Elasticsearch adapter find mapping error when index mapping set
> dynamic_templates without properties
>
>
> Key: CALCITE-5974
> URL: https://issues.apache.org/jira/browse/CALCITE-5974
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.35.0
>Reporter: zhaowang
>Assignee: zhaowang
>Priority: Major
>
> When use es-adapter, we config a es cluster, it will fetchMapping during
> initialization.
> If a index set dynamic_templates bug no mappings like this:
>
> {code:java}
> {
> "test_index" : {
> "mappings" : {
> "dynamic_templates" : [
> {
> "integers" : {
> "match_mapping_type" : "long",
> "mapping" : {
> "type" : "integer"
> }
> }
> }
> ]
> }
> }
> } {code}
>
>
> org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties
> throw exception:
>
> {code:java}
> java.lang.ClassCastException:
> com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to
> com.fasterxml.jackson.databind.node.ObjectNode
> at
>
> org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133)
> {code}
> So ElasticsearchTransport initialize failed, all index of this es cluster
> access failed.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5732) EnumerableHashJoin and EnumerableMergeJoin on composite key return rows matching condition 'null = null'
[
https://issues.apache.org/jira/browse/CALCITE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763005#comment-17763005
]
Ruben Q L edited comment on CALCITE-5732 at 9/8/23 9:43 AM:
Fixed via
[{{3aee0b8}}|https://github.com/apache/calcite/commit/3aee0b86aa23476cbdecc75ad5d43b936a6fff7b]
Thansk [~viggoc] for reporting this issue, and [~libenchao] for the review!
was (Author: rubenql):
Fixed via
[{{3aee0b8}}|https://github.com/apache/calcite/commit/3aee0b86aa23476cbdecc75ad5d43b936a6fff7b]
Thansk [~viggoc] for reporting this issue, and [~libenchao] for the review!
> EnumerableHashJoin and EnumerableMergeJoin on composite key return rows
> matching condition 'null = null'
>
>
> Key: CALCITE-5732
> URL: https://issues.apache.org/jira/browse/CALCITE-5732
> Project: Calcite
> Issue Type: Bug
> Components: core, linq4j
>Reporter: Viggo Chen
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.36.0
>
>
> In `EnumerableJoinTest#testMergeJoinWithCompositeKeyAndNullValues`, the query
> is like
> {code:java}
> select
> emps.empid
> from
> emps a join emps b
> on a.deptno = b.deptno
> and a.commission = b.commission;{code}
> and the data is like
> {code:java}
> INSERT INTO "emps" VALUES (100, 10, 'Bill', 1, 1000);
> INSERT INTO "emps" VALUES (200, 20, 'Eric', 8000, 500);
> INSERT INTO "emps" VALUES (150, 10, 'Sebastian', 7000, null);
> INSERT INTO "emps" VALUES (110, 10, 'Theodore', 11500, 250); {code}
> And row with empid = 150 is in expected result. Is this the expected result
> of join with null condition.
> Whats more hash join result with condition a.deptno = b.deptno and
> a.commission = b.commission is same as merge join. And if there is just one
> condition a.commission = b.commission, the result do not include empid = 150.
>
> Here is a unit test for it
> {code:java}
> @Test void testHashJoinWithCompositeKeyAndNullValues() {
> // Both join side 'commission' a limited to null, so a.commission =
> b.commission should always be false.
> // So all columns in right table b are expected to be null, this sql should
> result in 0 rows.
> final String sql = "select * from\n"
> + " (select empid, salary, commission from emps where commission is
> null) as a\n"
> + " left join\n"
> + " (select empid, salary, commission from emps where commission is
> null) as b\n"
> + " on a.salary = b.salary and a.commission = b.commission\n"
> + " where b.empid is not null";
> CalciteAssert.that()
> .with(CalciteConnectionProperty.LEX, Lex.JAVA)
> .with(CalciteConnectionProperty.FORCE_DECORRELATE, false)
> .withSchema("s", new ReflectiveSchema(new HrSchemaBig()))
> .query(sql)
> .withHook(Hook.PLANNER, (Consumer) planner -> {
> planner.removeRule(EnumerableRules.ENUMERABLE_JOIN_RULE);
> planner.addRule(EnumerableRules.ENUMERABLE_MERGE_JOIN_RULE);
> })
> .explainContains("EnumerableHashJoin")
> .returnsCount(0)
> ;
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CALCITE-5732) EnumerableHashJoin and EnumerableMergeJoin on composite key return rows matching condition 'null = null'
[
https://issues.apache.org/jira/browse/CALCITE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L resolved CALCITE-5732.
Resolution: Fixed
Fixed via
[{{3aee0b8}}|https://github.com/apache/calcite/commit/3aee0b86aa23476cbdecc75ad5d43b936a6fff7b]
Thansk [~viggoc] for reporting this issue, and [~libenchao] for the review!
> EnumerableHashJoin and EnumerableMergeJoin on composite key return rows
> matching condition 'null = null'
>
>
> Key: CALCITE-5732
> URL: https://issues.apache.org/jira/browse/CALCITE-5732
> Project: Calcite
> Issue Type: Bug
> Components: core, linq4j
>Reporter: Viggo Chen
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.36.0
>
>
> In `EnumerableJoinTest#testMergeJoinWithCompositeKeyAndNullValues`, the query
> is like
> {code:java}
> select
> emps.empid
> from
> emps a join emps b
> on a.deptno = b.deptno
> and a.commission = b.commission;{code}
> and the data is like
> {code:java}
> INSERT INTO "emps" VALUES (100, 10, 'Bill', 1, 1000);
> INSERT INTO "emps" VALUES (200, 20, 'Eric', 8000, 500);
> INSERT INTO "emps" VALUES (150, 10, 'Sebastian', 7000, null);
> INSERT INTO "emps" VALUES (110, 10, 'Theodore', 11500, 250); {code}
> And row with empid = 150 is in expected result. Is this the expected result
> of join with null condition.
> Whats more hash join result with condition a.deptno = b.deptno and
> a.commission = b.commission is same as merge join. And if there is just one
> condition a.commission = b.commission, the result do not include empid = 150.
>
> Here is a unit test for it
> {code:java}
> @Test void testHashJoinWithCompositeKeyAndNullValues() {
> // Both join side 'commission' a limited to null, so a.commission =
> b.commission should always be false.
> // So all columns in right table b are expected to be null, this sql should
> result in 0 rows.
> final String sql = "select * from\n"
> + " (select empid, salary, commission from emps where commission is
> null) as a\n"
> + " left join\n"
> + " (select empid, salary, commission from emps where commission is
> null) as b\n"
> + " on a.salary = b.salary and a.commission = b.commission\n"
> + " where b.empid is not null";
> CalciteAssert.that()
> .with(CalciteConnectionProperty.LEX, Lex.JAVA)
> .with(CalciteConnectionProperty.FORCE_DECORRELATE, false)
> .withSchema("s", new ReflectiveSchema(new HrSchemaBig()))
> .query(sql)
> .withHook(Hook.PLANNER, (Consumer) planner -> {
> planner.removeRule(EnumerableRules.ENUMERABLE_JOIN_RULE);
> planner.addRule(EnumerableRules.ENUMERABLE_MERGE_JOIN_RULE);
> })
> .explainContains("EnumerableHashJoin")
> .returnsCount(0)
> ;
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5953) AggregateCaseToFilterRule may make inaccurate SUM transformation
[
https://issues.apache.org/jira/browse/CALCITE-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated CALCITE-5953:
--
Description:
consider: {{sum(case when x = 1 then 2 else 0 end) as b}}
notice that this expression may only be null if there are no rows in the table
{{AggregateCaseToFilterRule}} rewrites the above expression to {{sum(2) filter
(where x=1)}} which broadens when it could be `null` to when there are no
matches to the filter
* *A* is *0* correctly in this case; but I think it will be still *0* in case
there are 0 input rows
* The result for *B* supposed to be *0* but since there are no matches by the
filter it becomes *null*
* *C* is not touched
{code}
# Convert CASE to FILTER without matches
select sum(case when x = 1 then 1 else 0 end) as a,
sum(case when x = 1 then 2 else 0 end) as b,
sum(case when x = 1 then 3 else -1 end) as c
from (values 0, null, 0, 2) as t(x);
+---+---++
| A | B | C |
+---+---++
| 0 | | -4 | # incorrect result for B
| 0 | 0 | -4 | # correct
+---+---++
(1 row)
!ok
EnumerableCalc(expr#0..2=[{inputs}], expr#3=[CAST($t0):INTEGER], A=[$t3],
B=[$t1], C=[$t2])
EnumerableAggregate(group=[{}], A=[COUNT() FILTER $1], B=[SUM($2) FILTER $3],
C=[SUM($0)])
EnumerableCalc(expr#0=[{inputs}], expr#1=[1], expr#2=[=($t0, $t1)],
expr#3=[3], expr#4=[-1], expr#5=[CASE($t2, $t3, $t4)], expr#6=[IS TRUE($t2)],
expr#7=[2], $f2=[$t5], $f3=[$t6], $f4=[$t7], $f5=[$t6])
EnumerableValues(tuples=[[{ 0 }, { null }, { 0 }, { 2 }]])
!plan
# Convert CASE to FILTER with no input rows
select sum(case when x = 1 then 1 else 0 end) as a,
sum(case when x = 1 then 2 else 0 end) as b,
sum(case when x = 1 then 3 else -1 end) as c
from (values 0, null, 0, 2) as t(x) where x*x=1;
+---+---+---+
| A | B | C |
+---+---+---+
| 0 | | | # incorrect results - for 0 input rows sum supposed to be null
| | | | # correct result
+---+---+---+
(1 row)
!ok
{code}
was:
consider: {{sum(case when x = 1 then 2 else 0 end) as b}}
notice that this expression may only be null if there are no rows in the table
{{AggregateCaseToFilterRule}} rewrites the above expression to {{sum(2) filter
(where x=1)}} which broadens when it could be `null` to when there are no
matches to the filter
* *A* is *0* correctly in this case; but I think it will be still *0* in case
there are 0 input rows
* The result for *B* supposed to be *0* but since there are no matches by the
filter it becomes *null*
* *C* is not touched
{code}
# Convert CASE to FILTER without matches
select sum(case when x = 1 then 1 else 0 end) as a,
sum(case when x = 1 then 2 else 0 end) as b,
sum(case when x = 1 then 3 else -1 end) as c
from (values 0, null, 0, 2) as t(x);
+---+---++
| A | B | C |
+---+---++
| 0 | 0 | -4 |
+---+---++
(1 row)
!ok
EnumerableCalc(expr#0..2=[{inputs}], expr#3=[CAST($t0):INTEGER], A=[$t3],
B=[$t1], C=[$t2])
EnumerableAggregate(group=[{}], A=[COUNT() FILTER $1], B=[SUM($2) FILTER $3],
C=[SUM($0)])
EnumerableCalc(expr#0=[{inputs}], expr#1=[1], expr#2=[=($t0, $t1)],
expr#3=[3], expr#4=[-1], expr#5=[CASE($t2, $t3, $t4)], expr#6=[IS TRUE($t2)],
expr#7=[2], $f2=[$t5], $f3=[$t6], $f4=[$t7], $f5=[$t6])
EnumerableValues(tuples=[[{ 0 }, { null }, { 0 }, { 2 }]])
!plan
{code}
> AggregateCaseToFilterRule may make inaccurate SUM transformation
>
>
> Key: CALCITE-5953
> URL: https://issues.apache.org/jira/browse/CALCITE-5953
> Project: Calcite
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.36.0
>
>
> consider: {{sum(case when x = 1 then 2 else 0 end) as b}}
> notice that this expression may only be null if there are no rows in the table
> {{AggregateCaseToFilterRule}} rewrites the above expression to {{sum(2)
> filter (where x=1)}} which broadens when it could be `null` to when there are
> no matches to the filter
> * *A* is *0* correctly in this case; but I think it will be still *0* in case
> there are 0 input rows
> * The result for *B* supposed to be *0* but since there are no matches by the
> filter it becomes *null*
> * *C* is not touched
> {code}
> # Convert CASE to FILTER without matches
> select sum(case when x = 1 then 1 else 0 end) as a,
> sum(case when x = 1 then 2 else 0 end) as b,
> sum(case when x = 1 then 3 else -1 end) as c
> from (values 0, null, 0, 2) as t(x);
> +---+---++
> | A | B | C |
> +---+---++
> | 0 | | -4 | # incorrect result for B
> | 0 | 0 | -4 | # correct
> +---+---++
> (1 row)
> !ok
> EnumerableCalc(expr#0..2=[{inputs}], expr#3=[CAST($t0):INTEGER], A=[$t3],
> B=[$t1], C=[$t2])
>
[jira] [Updated] (CALCITE-5953) AggregateCaseToFilterRule may make inaccurate SUM transformation
[
https://issues.apache.org/jira/browse/CALCITE-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated CALCITE-5953:
--
Description:
consider: {{sum(case when x = 1 then 2 else 0 end) as b}}
notice that this expression may only be null if there are no rows in the table
{{AggregateCaseToFilterRule}} rewrites the above expression to {{sum(2) filter
(where x=1)}} which broadens when it could be `null` to when there are no
matches to the filter
* *A* is *0* correctly in this case; but I think it will be still *0* in case
there are 0 input rows
* The result for *B* supposed to be *0* but since there are no matches by the
filter it becomes *null*
* *C* is not touched
{code}
# Convert CASE to FILTER without matches
select sum(case when x = 1 then 1 else 0 end) as a,
sum(case when x = 1 then 2 else 0 end) as b,
sum(case when x = 1 then 3 else -1 end) as c
from (values 0, null, 0, 2) as t(x);
+---+---++
| A | B | C |
+---+---++
| 0 | 0 | -4 |
+---+---++
(1 row)
!ok
EnumerableCalc(expr#0..2=[{inputs}], expr#3=[CAST($t0):INTEGER], A=[$t3],
B=[$t1], C=[$t2])
EnumerableAggregate(group=[{}], A=[COUNT() FILTER $1], B=[SUM($2) FILTER $3],
C=[SUM($0)])
EnumerableCalc(expr#0=[{inputs}], expr#1=[1], expr#2=[=($t0, $t1)],
expr#3=[3], expr#4=[-1], expr#5=[CASE($t2, $t3, $t4)], expr#6=[IS TRUE($t2)],
expr#7=[2], $f2=[$t5], $f3=[$t6], $f4=[$t7], $f5=[$t6])
EnumerableValues(tuples=[[{ 0 }, { null }, { 0 }, { 2 }]])
!plan
{code}
was:
consider: {{sum(case when x = 1 then 2 else 0 end) as b}}
notice that this expression may only be null if there are no rows in the table
{{AggregateCaseToFilterRule}} rewrites the above expression to {{sum(1) filter
(where x=2)}} which broadens when it could be `null` to when there are no
matches to the filter
* *A* is *0* correctly in this case; but I think it will be still *0* in case
there are 0 input rows
* The result for *B* supposed to be *0* but since there are no matches by the
filter it becomes *null*
* *C* is not touched
{code}
# Convert CASE to FILTER without matches
select sum(case when x = 1 then 1 else 0 end) as a,
sum(case when x = 1 then 2 else 0 end) as b,
sum(case when x = 1 then 3 else -1 end) as c
from (values 0, null, 0, 2) as t(x);
+---+---++
| A | B | C |
+---+---++
| 0 | 0 | -4 |
+---+---++
(1 row)
!ok
EnumerableCalc(expr#0..2=[{inputs}], expr#3=[CAST($t0):INTEGER], A=[$t3],
B=[$t1], C=[$t2])
EnumerableAggregate(group=[{}], A=[COUNT() FILTER $1], B=[SUM($2) FILTER $3],
C=[SUM($0)])
EnumerableCalc(expr#0=[{inputs}], expr#1=[1], expr#2=[=($t0, $t1)],
expr#3=[3], expr#4=[-1], expr#5=[CASE($t2, $t3, $t4)], expr#6=[IS TRUE($t2)],
expr#7=[2], $f2=[$t5], $f3=[$t6], $f4=[$t7], $f5=[$t6])
EnumerableValues(tuples=[[{ 0 }, { null }, { 0 }, { 2 }]])
!plan
{code}
> AggregateCaseToFilterRule may make inaccurate SUM transformation
>
>
> Key: CALCITE-5953
> URL: https://issues.apache.org/jira/browse/CALCITE-5953
> Project: Calcite
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.36.0
>
>
> consider: {{sum(case when x = 1 then 2 else 0 end) as b}}
> notice that this expression may only be null if there are no rows in the table
> {{AggregateCaseToFilterRule}} rewrites the above expression to {{sum(2)
> filter (where x=1)}} which broadens when it could be `null` to when there are
> no matches to the filter
> * *A* is *0* correctly in this case; but I think it will be still *0* in case
> there are 0 input rows
> * The result for *B* supposed to be *0* but since there are no matches by the
> filter it becomes *null*
> * *C* is not touched
> {code}
> # Convert CASE to FILTER without matches
> select sum(case when x = 1 then 1 else 0 end) as a,
> sum(case when x = 1 then 2 else 0 end) as b,
> sum(case when x = 1 then 3 else -1 end) as c
> from (values 0, null, 0, 2) as t(x);
> +---+---++
> | A | B | C |
> +---+---++
> | 0 | 0 | -4 |
> +---+---++
> (1 row)
> !ok
> EnumerableCalc(expr#0..2=[{inputs}], expr#3=[CAST($t0):INTEGER], A=[$t3],
> B=[$t1], C=[$t2])
> EnumerableAggregate(group=[{}], A=[COUNT() FILTER $1], B=[SUM($2) FILTER
> $3], C=[SUM($0)])
> EnumerableCalc(expr#0=[{inputs}], expr#1=[1], expr#2=[=($t0, $t1)],
> expr#3=[3], expr#4=[-1], expr#5=[CASE($t2, $t3, $t4)], expr#6=[IS TRUE($t2)],
> expr#7=[2], $f2=[$t5], $f3=[$t6], $f4=[$t7], $f5=[$t6])
> EnumerableValues(tuples=[[{ 0 }, { null }, { 0 }, { 2 }]])
> !plan
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[
https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wegdan Ghazi updated CALCITE-5984:
--
Description:
In Prepare#prepareSql there are 2 instances of trimming unused fields
1.
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297]
{code:java}
// Trim unused fields.
root = trimUnusedFields(root); {code}
2.
{color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color}
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312]
within
{code:java}
root = optimize(root, getMaterializations(), getLattices());
{code}
I want to turn of the field trimming due to a personalisation I'm implementing
to scan a specific API, which I would think is possible using
[SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26]
I attempted to disable the trimming using hooks [PROGRAM,
SQL2REL_CONVERTER_CONFIG_BUILDER], which succeeded to remove the trimming
program, yet on
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
{code:java}
// code placehprotected RelRoot trimUnusedFields(RelRoot root) {
final SqlToRelConverter.Config config = SqlToRelConverter.config()
.withTrimUnusedFields(shouldTrim(root.rel))
.withExpand(THREAD_EXPAND.get())
.withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get()));
final SqlToRelConverter converter =
getSqlToRelConverter(getSqlValidator(), catalogReader, config);
final boolean ordered = !root.collation.getFieldCollations().isEmpty();
final boolean dml = SqlKind.DML.contains(root.kind);
return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel));
}older
{code}
The config passed (and edited by the hook) gets overwritten and only
_shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to
happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}.
was:
In Prepare#prepareSql there are 2 instances of trimming unused fields
1.
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297]
{code:java}
// Trim unused fields.
root = trimUnusedFields(root); {code}
2.
{color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color}
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312]
within
{code:java}
root = optimize(root, getMaterializations(), getLattices());
{code}
I want to turn of the field trimming due to a personalisation I'm implementing
to scan a specific API, which I would think is possible using
[SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26]
I attempted to disable the trimming using hooks [{_}PROGRAM,
SQL2REL_CONVERTER_CONFIG_BUILDER{_}], which succeeded to remove the trimming
program, yet on
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
{code:java}
// code placehprotected RelRoot trimUnusedFields(RelRoot root) {
final SqlToRelConverter.Config config = SqlToRelConverter.config()
.withTrimUnusedFields(shouldTrim(root.rel))
.withExpand(THREAD_EXPAND.get())
.withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get()));
final SqlToRelConverter converter =
getSqlToRelConverter(getSqlValidator(), catalogReader, config);
final boolean ordered = !root.collation.getFieldCollations().isEmpty();
final boolean dml = SqlKind.DML.contains(root.kind);
return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel));
}older
{code}
The config passed (and edited by the hook) gets overwritten and only
_shouldTrim(root.rel)_ can enable/disable
[jira] [Updated] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[
https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wegdan Ghazi updated CALCITE-5984:
--
Description:
In Prepare#prepareSql there are 2 instances of trimming unused fields
1.
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297]
{code:java}
// Trim unused fields.
root = trimUnusedFields(root); {code}
2.
{color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color}
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312]
within
{code:java}
root = optimize(root, getMaterializations(), getLattices());
{code}
I want to turn of the field trimming due to a personalisation I'm implementing
to scan a specific API, which I would think is possible using
[SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26]
I attempted to disable the trimming using hooks [{_}PROGRAM,
SQL2REL_CONVERTER_CONFIG_BUILDER{_}], which succeeded to remove the trimming
program, yet on
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
{code:java}
// code placehprotected RelRoot trimUnusedFields(RelRoot root) {
final SqlToRelConverter.Config config = SqlToRelConverter.config()
.withTrimUnusedFields(shouldTrim(root.rel))
.withExpand(THREAD_EXPAND.get())
.withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get()));
final SqlToRelConverter converter =
getSqlToRelConverter(getSqlValidator(), catalogReader, config);
final boolean ordered = !root.collation.getFieldCollations().isEmpty();
final boolean dml = SqlKind.DML.contains(root.kind);
return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel));
}older
{code}
The config passed (and edited by the hook) gets overwritten and only
_shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to
happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}.
was:
In Prepare#prepareSql there are 2 instances of trimming unused fields
1.
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297]
{code:java}
// Trim unused fields.
root = trimUnusedFields(root); {code}
2.
{color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color}
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312]
within
{code:java}
root = optimize(root, getMaterializations(), getLattices());
{code}
I want to turn of the field trimming due to a personalisation I'm implementing
to scan a specific API, which I would think is possible using
[SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26]
I attempted to disable the trimming using hooks [\{_}PROGRAM,
SQL2REL_CONVERTER_CONFIG_BUILDER\{_}], which succeeded to remove the trimming
program, yet on
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
{code:java}
// code placehprotected RelRoot trimUnusedFields(RelRoot root) {
final SqlToRelConverter.Config config = SqlToRelConverter.config()
.withTrimUnusedFields(shouldTrim(root.rel))
.withExpand(THREAD_EXPAND.get())
.withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get()));
final SqlToRelConverter converter =
getSqlToRelConverter(getSqlValidator(), catalogReader, config);
final boolean ordered = !root.collation.getFieldCollations().isEmpty();
final boolean dml = SqlKind.DML.contains(root.kind);
return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel));
}older
{code}
The config passed (and edited by the hook) gets overwritten and only
_shouldTrim(root.rel)_ can
[jira] [Updated] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[
https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wegdan Ghazi updated CALCITE-5984:
--
Description:
In Prepare#prepareSql there are 2 instances of trimming unused fields
1.
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297]
{code:java}
// Trim unused fields.
root = trimUnusedFields(root); {code}
2.
{color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color}
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312]
within
{code:java}
root = optimize(root, getMaterializations(), getLattices());
{code}
I want to turn of the field trimming due to a personalisation I'm implementing
to scan a specific API, which I would think is possible using
[SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26]
I attempted to disable the trimming using hooks [\{_}PROGRAM,
SQL2REL_CONVERTER_CONFIG_BUILDER\{_}], which succeeded to remove the trimming
program, yet on
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
{code:java}
// code placehprotected RelRoot trimUnusedFields(RelRoot root) {
final SqlToRelConverter.Config config = SqlToRelConverter.config()
.withTrimUnusedFields(shouldTrim(root.rel))
.withExpand(THREAD_EXPAND.get())
.withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get()));
final SqlToRelConverter converter =
getSqlToRelConverter(getSqlValidator(), catalogReader, config);
final boolean ordered = !root.collation.getFieldCollations().isEmpty();
final boolean dml = SqlKind.DML.contains(root.kind);
return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel));
}older
{code}
The config passed (and edited by the hook) gets overwritten and only
_shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to
happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}.
Is this the intended behaviour? Any help is appreciated, thanks.
was:
In Prepare#prepareSql there are 2 instances of trimming unused fields
1.
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297]
{code:java}
// Trim unused fields.
root = trimUnusedFields(root); {code}
2.
{color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color}
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312]
within
{code:java}
root = optimize(root, getMaterializations(), getLattices());
{code}
I want to turn of the field trimming due to a personalisation I'm implementing
to scan a specific API, which I would think is possible using
[SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26]
I attempted to disable the trimming using hooks [{_}PROGRAM,
SQL2REL_CONVERTER_CONFIG_BUILDER{_}], which succeeded to remove the trimming
program, yet on
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
{code:java}
// code placehprotected RelRoot trimUnusedFields(RelRoot root) {
final SqlToRelConverter.Config config = SqlToRelConverter.config()
.withTrimUnusedFields(shouldTrim(root.rel))
.withExpand(THREAD_EXPAND.get())
.withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get()));
final SqlToRelConverter converter =
getSqlToRelConverter(getSqlValidator(), catalogReader, config);
final boolean ordered = !root.collation.getFieldCollations().isEmpty();
final boolean dml = SqlKind.DML.contains(root.kind);
return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel));
}older
{code}
The config passed (and edited by
[jira] [Created] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
Wegdan Ghazi created CALCITE-5984:
-
Summary: Cannot fully disable trimming unused fields using Calcite
configs
Key: CALCITE-5984
URL: https://issues.apache.org/jira/browse/CALCITE-5984
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.34.0
Environment: Spring boot application integrating _org.apache.calcite_
Reporter: Wegdan Ghazi
Fix For: 1.34.0
In Prepare#prepareSql there are 2 instances of trimming unused fields
1.
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297]
{code:java}
// Trim unused fields.
root = trimUnusedFields(root); {code}
2.
{color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color}
called
[here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312]
within
{code:java}
root = optimize(root, getMaterializations(), getLattices());
{code}
I want to turn of the field trimming due to a personalisation I'm implementing
to scan a specific API, which I would think is possible using
[SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26]
I attempted to disable the trimming using hooks [{_}PROGRAM,
SQL2REL_CONVERTER_CONFIG_BUILDER{_}], which succeeded to remove the trimming
program, yet on
[Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
{code:java}
// code placehprotected RelRoot trimUnusedFields(RelRoot root) {
final SqlToRelConverter.Config config = SqlToRelConverter.config()
.withTrimUnusedFields(shouldTrim(root.rel))
.withExpand(THREAD_EXPAND.get())
.withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get()));
final SqlToRelConverter converter =
getSqlToRelConverter(getSqlValidator(), catalogReader, config);
final boolean ordered = !root.collation.getFieldCollations().isEmpty();
final boolean dml = SqlKind.DML.contains(root.kind);
return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel));
}older
{code}
The config passed (and edited by the hook) gets overwritten and only
_shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to
happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[
https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wegdan Ghazi updated CALCITE-5984:
--
Environment: (was: Spring boot application integrating
_org.apache.calcite_)
> Cannot fully disable trimming unused fields using Calcite configs
> -
>
> Key: CALCITE-5984
> URL: https://issues.apache.org/jira/browse/CALCITE-5984
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.34.0
>Reporter: Wegdan Ghazi
>Priority: Major
> Fix For: 1.34.0
>
>
> In Prepare#prepareSql there are 2 instances of trimming unused fields
> 1.
> [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
> called
> [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297]
>
> {code:java}
> // Trim unused fields.
> root = trimUnusedFields(root); {code}
> 2.
> {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color}
> called
> [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312]
> within
> {code:java}
> root = optimize(root, getMaterializations(), getLattices());
> {code}
> I want to turn of the field trimming due to a personalisation I'm
> implementing to scan a specific API, which I would think is possible using
> [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26]
> I attempted to disable the trimming using hooks [{_}PROGRAM,
> SQL2REL_CONVERTER_CONFIG_BUILDER{_}], which succeeded to remove the trimming
> program, yet on
> [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375]
> {code:java}
> // code placehprotected RelRoot trimUnusedFields(RelRoot root) {
> final SqlToRelConverter.Config config = SqlToRelConverter.config()
> .withTrimUnusedFields(shouldTrim(root.rel))
> .withExpand(THREAD_EXPAND.get())
>
> .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get()));
> final SqlToRelConverter converter =
> getSqlToRelConverter(getSqlValidator(), catalogReader, config);
> final boolean ordered = !root.collation.getFieldCollations().isEmpty();
> final boolean dml = SqlKind.DML.contains(root.kind);
> return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel));
> }older
> {code}
> The config passed (and edited by the hook) gets overwritten and only
> _shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to
> happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
