[jira] [Commented] (CALCITE-5931) Allow integers like 1.00 in window frame
[ https://issues.apache.org/jira/browse/CALCITE-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763309#comment-17763309 ] Jiajun Xie commented on CALCITE-5931: - [~cbrisson] , Your PR is already dirty because force-pushed main. I cherry-pick your commit and add test in `winagg.iq`, welcome to review. [https://github.com/apache/calcite/pull/3413] Let's merge it before the release of version 1.36. > Allow integers like 1.00 in window frame > > > Key: CALCITE-5931 > URL: https://issues.apache.org/jira/browse/CALCITE-5931 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Claude Brisson >Priority: Minor > Labels: pull-request-available > Fix For: 1.36.0 > > > When a number of rows is specified for a window, the code checks that the > provided literal is an integral number. > But the provided check leaves aside numbers like "1.00" (with a scale of 2, > but with trailing zeros). > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5980) QuidemTests are not effectively executed on Windows
[ https://issues.apache.org/jira/browse/CALCITE-5980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CALCITE-5980: Labels: pull-request-available (was: ) > QuidemTests are not effectively executed on Windows > --- > > Key: CALCITE-5980 > URL: https://issues.apache.org/jira/browse/CALCITE-5980 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Ruben Q L >Assignee: Ruben Q L >Priority: Major > Labels: pull-request-available > Fix For: 1.36.0 > > > Discovered by accident on my Windows+IntelliJ environment while I was trying > to add new tests on a *.iq file. My new tests did not seem to be executed. I > even tried adding syntax errors on purpose into different iq files to force > them to fail, but the tests were still successful. The reason seems to be > that, at least on my environment (Windows), the test files do not execute any > of their statements. This seems caused by the changes introduced in > CALCITE-5786. > While debugging, I found this line in QuidemTest.java (that aims to create > the inFile and outFile): > {code:java} > protected void checkRun(String path) throws Exception { > ... > // e.g. path = "sql/agg.iq" > // inUrl = "file:/home/fred/calcite/core/build/resources/test/sql/agg.iq" > // inFile = "/home/fred/calcite/core/build/resources/test/sql/agg.iq" > // outDir = "/home/fred/calcite/core/build/quidem/test/sql" > // outFile = "/home/fred/calcite/core/build/quidem/test/sql/agg.iq" > inFile = Sources.of(requireNonNull(inUrl, "inUrl")).file(); > outFile = replaceDir(inFile, "resources", "quidem"); > ... > } > {code} > But in my case it results on {*}both files being the same{*}, thus when the > outFile is created, it actually erases all the tests that were contained > inside the inFile, so the test runs nothing. > The reason for that is that the auxiliary method {{{}replaceDir{}}}: > {code:java} > private static File replaceDir(File file, String target, String > replacement) { > return new File( > file.getAbsolutePath().replace(n2u('/' + target + '/'), > n2u('/' + replacement + '/'))); > } > {code} > is trying to replace "/resources/" with "/quidem/" from the inFile absolute > path, but in my case this path does not contain the pattern to be replaced, > since it contains backslashes: > "C:\...\calcite\core\build\resources\test\sql\agg.iq"; so the replacement > operation does nothing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5990) Explicit cast to numeric type doesn't check overflow
[ https://issues.apache.org/jira/browse/CALCITE-5990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Runkang He updated CALCITE-5990: Description: Explicit cast to numeric type doesn't check overflow, and this issue can be reproduced by sqlline: {code:sql} select cast(empno as tinyint), cast(130 as tinyint) from emps where name='Alice'; -- empno is 130 {code} The empno is INT type. The result is: {code:sql} -126, -126{code} I think it should throw exception when overflow. At last, this issue was found when to turn on runtime check for CalciteSqlOperatorTest in CALCITE-5921. was: Explicit cast to numeric type doesn't check overflow, and this issue can be reproduced by sqlline: {code:sql} select cast(empno as tinyint), cast(130 as tinyint) from emps where name='Alice'; -- empno is 130 {code} The result is: {code:sql} -126, -126{code} I think it should throw exception when overflow. At last, this issue was found when to turn on runtime check for CalciteSqlOperatorTest in CALCITE-5921. > Explicit cast to numeric type doesn't check overflow > > > Key: CALCITE-5990 > URL: https://issues.apache.org/jira/browse/CALCITE-5990 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Runkang He >Assignee: Runkang He >Priority: Major > > Explicit cast to numeric type doesn't check overflow, and this issue can be > reproduced by sqlline: > {code:sql} > select cast(empno as tinyint), cast(130 as tinyint) from emps where > name='Alice'; -- empno is 130 > {code} > The empno is INT type. The result is: > {code:sql} > -126, -126{code} > I think it should throw exception when overflow. > At last, this issue was found when to turn on runtime check for > CalciteSqlOperatorTest in CALCITE-5921. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5921) SqlOperatorFixture.checkFails and checkAggFails don't check runtime failure
[ https://issues.apache.org/jira/browse/CALCITE-5921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CALCITE-5921: Labels: pull-request-available (was: ) > SqlOperatorFixture.checkFails and checkAggFails don't check runtime failure > --- > > Key: CALCITE-5921 > URL: https://issues.apache.org/jira/browse/CALCITE-5921 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Runkang He >Assignee: Runkang He >Priority: Major > Labels: pull-request-available > > SqlOperatorFixture.checkFails and checkAggFails don't check runtime failure. > See more in [code > line|https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/testkit/src/main/java/org/apache/calcite/test/SqlOperatorFixtureImpl.java#L160]. > When the parameter `runtime` of SqlOperatorFixture.checkFails is true, it > should execute the query and check runtime failure, but currently it ignores > this, and only checks the parse and validation failure. > When fix this, there are 4 failed test cases in CalciteSqlOperatorTest. > At last, this issue was found when to implement `BIT_GET` function in > CALCITE-5848. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (CALCITE-5990) Explicit cast to numeric type doesn't check overflow
[ https://issues.apache.org/jira/browse/CALCITE-5990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Runkang He reassigned CALCITE-5990: --- Assignee: Runkang He > Explicit cast to numeric type doesn't check overflow > > > Key: CALCITE-5990 > URL: https://issues.apache.org/jira/browse/CALCITE-5990 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Runkang He >Assignee: Runkang He >Priority: Major > > Explicit cast to numeric type doesn't check overflow, and this issue can be > reproduced by sqlline: > {code:sql} > select cast(empno as tinyint), cast(130 as tinyint) from emps where > name='Alice'; -- empno is 130 > {code} > The result is: > {code:sql} > -126, -126{code} > I think it should throw exception when overflow. > At last, this issue was found when to turn on runtime check for > CalciteSqlOperatorTest in CALCITE-5921. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5990) Explicit cast to numeric type doesn't check overflow
Runkang He created CALCITE-5990: --- Summary: Explicit cast to numeric type doesn't check overflow Key: CALCITE-5990 URL: https://issues.apache.org/jira/browse/CALCITE-5990 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.35.0 Reporter: Runkang He Explicit cast to numeric type doesn't check overflow, and this issue can be reproduced by sqlline: {code:sql} select cast(empno as tinyint), cast(130 as tinyint) from emps where name='Alice'; -- empno is 130 {code} The result is: {code:sql} -126, -126{code} I think it should throw exception when overflow. At last, this issue was found when to turn on runtime check for CalciteSqlOperatorTest in CALCITE-5921. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5891) Create a test fixture that would apply PROJECT_REDUCE_EXPRESSIONS to all tests in SqlOperatorTest
[ https://issues.apache.org/jira/browse/CALCITE-5891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763299#comment-17763299 ] Mihai Budiu commented on CALCITE-5891: -- I have made some progress on this front. The dataflow is as follows: * I have a test fixture that extends CalciteSqlOperatorTest * The fixture overloads the check function * The check function optimizes the query supplied in 4 steps ** convert to RelNode (reusing a RelOptFixture) ** Use a HepPlanner with a fixed plan; the most important is PROJECT_REDUCE_EXPRESSIONS ** Convert RelNode to SqlNode using RelToSqlConverter ** Convert SqlNode to a String using toSqlString(). * After optimizing the query, call the super.check(optimizedQuery) function Ideally this should work, but at this point 70 out of 463 tests fail. There are various reasons for the failures. Incorrect optimization is only one of them: * result is not equivalent to expected result (optimizer error) * type of result is not the same as the type of the expected result CALCITE-5987 * compilation of query into RelNode throws * optimizer throws * conversion of RelNode to SqlNode fails (e.g., CALCITE-5988) One solution would be for me to submit a PR with this implementation by overloading all the failing tests and using a JUnit annotation to ignore them. Then we can assign the bugs to various people responsible for various pieces and delete the ignored tests as they are fixed. Happy to hear other proposals. > Create a test fixture that would apply PROJECT_REDUCE_EXPRESSIONS to all > tests in SqlOperatorTest > -- > > Key: CALCITE-5891 > URL: https://issues.apache.org/jira/browse/CALCITE-5891 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Major > > SqlOperatorTest has many tests, including end-to-end tests. > However, none of these tests exercise the PROJECT_REDUCE_EXPRESSION rules, > which often produce different results than these tests for constant > expressions. > Ideally we should be able to subclass SqlOperatorTest and use a fixture that > also applies this optimization prior to evaluation. > I have marked this as a {*}major priority{*}, because I suspect it would > catch many bugs with minimal effort. (I have found at least 10 so far.) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5989) Type inference for RPAD and LPAD functions (BIGQUERY) is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763296#comment-17763296 ] Mihai Budiu commented on CALCITE-5989: -- This was found while trying to make CALCITE-5891 work. > Type inference for RPAD and LPAD functions (BIGQUERY) is incorrect > -- > > Key: CALCITE-5989 > URL: https://issues.apache.org/jira/browse/CALCITE-5989 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > > The type inference uses the type `ReturnTypes.ARG0_NULLABLE_VARYING` for the > output. > This means that the output cannot be longer than arg0. This bug surfaces when > the query is optimized. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5989) Type inference for RPAD and LPAD functions (BIGQUERY) is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763295#comment-17763295 ] Mihai Budiu commented on CALCITE-5989: -- This is very similar to [CALCITE-5813] > Type inference for RPAD and LPAD functions (BIGQUERY) is incorrect > -- > > Key: CALCITE-5989 > URL: https://issues.apache.org/jira/browse/CALCITE-5989 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > > The type inference uses the type `ReturnTypes.ARG0_NULLABLE_VARYING` for the > output. > This means that the output cannot be longer than arg0. This bug surfaces when > the query is optimized. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5989) Type inference for RPAD and LPAD functions (BIGQUERY) is incorrect
Mihai Budiu created CALCITE-5989: Summary: Type inference for RPAD and LPAD functions (BIGQUERY) is incorrect Key: CALCITE-5989 URL: https://issues.apache.org/jira/browse/CALCITE-5989 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.35.0 Reporter: Mihai Budiu The type inference uses the type `ReturnTypes.ARG0_NULLABLE_VARYING` for the output. This means that the output cannot be longer than arg0. This bug surfaces when the query is optimized. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5988) SqlImplementor.toSql cannot emit VARBINARY literals
Mihai Budiu created CALCITE-5988: Summary: SqlImplementor.toSql cannot emit VARBINARY literals Key: CALCITE-5988 URL: https://issues.apache.org/jira/browse/CALCITE-5988 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.35.0 Reporter: Mihai Budiu Given a literal with type VARBINARY the function SqlImplementor.toSql() will crash with assertion failure error: {code} X'41424344':VARBINARY: BINARY java.lang.AssertionError: X'41424344':VARBINARY: BINARY at org.apache.calcite.rel.rel2sql.SqlImplementor.toSql(SqlImplementor.java:1461) at org.apache.calcite.rel.rel2sql.SqlImplementor.toSql(SqlImplementor.java:1384) at org.apache.calcite.rel.rel2sql.SqlImplementor$Context.toSql(SqlImplementor.java:696) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763281#comment-17763281 ] Mihai Budiu commented on CALCITE-5986: -- > The "primary" and "secondary" concepts are very difficult to grasp. I think > that the primary and secondary family methods in SqlTypeName need more > documentation. I didn't invent these concepts, they come directly from SqlTypeFamily. I can add more documentation, but my understanding doesn't go much deeper than the JavaDoc on that class, so essentially I can repeat that documentation in the SqlTypeName class. > Do the methods both need to be nullable? Given the existing assignment to families it seems necessary. But I agree that it would be nice for the primary families to cover all types. There are types like SYMBOL, or OTHER which didn't have a family previously, and don't have either a primary or a secondary one now. I don't understand how these are used to fix them. > I think that the getFamily method should be deprecated. I suspect you mean use the Deprecated Java annotation. Some build tools are upset when you use deprecated methods, but I hope the build won't mind. > "The SqlTypeFamily for FP types is incorrect" doesn't seem ready for release > notes I agree, the original issue name is no longer an apt description. I will think of a better name. I will rename the commit when I squash at the end. > Can you add a test that does some sanity checks, e.g. every numeric type has > a not-null secondary family? Every secondary family is one of (A, B, C). > Every primary family is one of (X, Y, Z). I am not sure which objects I would test. The two asserts are a step in this direction. Perhaps you mean: "if the primary family is NUMERIC, the secondary one must be APPROXIMATE_NUMERIC or EXACT_NUMERIC"? > The SqlTypeFamily for FP types is incorrect > --- > > Key: CALCITE-5986 > URL: https://issues.apache.org/jira/browse/CALCITE-5986 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > Labels: pull-request-available > > In SqlTypeFamily we have this code: > {code:java} > private static final Map JDBC_TYPE_TO_FAMILY = > ... > .put(Types.FLOAT, NUMERIC) > .put(Types.REAL, NUMERIC) > .put(Types.DOUBLE, NUMERIC) > {code} > But it looks to me like the type family should be APPROXIMATE_NUMERIC. > This impacts the way RelToSqlConverter works, for instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763280#comment-17763280 ] Julian Hyde edited comment on CALCITE-5986 at 9/8/23 10:44 PM: --- I saw your PR. * The "primary" and "secondary" concepts are very difficult to grasp. I think that the primary and secondary family methods in SqlTypeName need more documentation. * Do the methods both need to be nullable? Non-nullable results are much easier to work with. * I think that the getFamily method should be deprecated. * "The SqlTypeFamily for FP types is incorrect" doesn't seem ready for release notes * Convert the {{assert}}s to preconditions calls. * Can you add a test that does some sanity checks, e.g. every numeric type has a not-null secondary family? Every secondary family is one of (A, B, C). Every primary family is one of (X, Y, Z). * Is "root family" a useful notion? (I.e. don't divide into exactly two strata. But require each type to have a root type family.) was (Author: julianhyde): I saw your PR. * The "primary" and "secondary" concepts are very difficult to grasp. I think that the primary and secondary family methods in SqlTypeName need more documentation. * Do the methods both need to be nullable? Non-nullable results are much easier to work with. * I think that the getFamily method should be deprecated. * "The SqlTypeFamily for FP types is incorrect" doesn't seem ready for release notes * Convert the {{assert}}s to preconditions calls. * Can you add a test that does some sanity checks, e.g. every numeric type has a not-null secondary family? Every secondary family is one of (A, B, C). Every primary family is one of (X, Y, Z). > The SqlTypeFamily for FP types is incorrect > --- > > Key: CALCITE-5986 > URL: https://issues.apache.org/jira/browse/CALCITE-5986 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > Labels: pull-request-available > > In SqlTypeFamily we have this code: > {code:java} > private static final Map JDBC_TYPE_TO_FAMILY = > ... > .put(Types.FLOAT, NUMERIC) > .put(Types.REAL, NUMERIC) > .put(Types.DOUBLE, NUMERIC) > {code} > But it looks to me like the type family should be APPROXIMATE_NUMERIC. > This impacts the way RelToSqlConverter works, for instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763280#comment-17763280 ] Julian Hyde edited comment on CALCITE-5986 at 9/8/23 10:43 PM: --- I saw your PR. * The "primary" and "secondary" concepts are very difficult to grasp. I think that the primary and secondary family methods in SqlTypeName need more documentation. * Do the methods both need to be nullable? Non-nullable results are much easier to work with. * I think that the getFamily method should be deprecated. * "The SqlTypeFamily for FP types is incorrect" doesn't seem ready for release notes * Convert the {{assert}}s to preconditions calls. * Can you add a test that does some sanity checks, e.g. every numeric type has a not-null secondary family? Every secondary family is one of (A, B, C). Every primary family is one of (X, Y, Z). was (Author: julianhyde): I saw your PR. * The "primary" and "secondary" concepts are very difficult to grasp. I think that the primary and secondary family methods in SqlTypeName need more documentation. * Do the methods both need to be nullable? * I think that the getFamily method should be deprecated. * "The SqlTypeFamily for FP types is incorrect" doesn't seem ready for release notes > The SqlTypeFamily for FP types is incorrect > --- > > Key: CALCITE-5986 > URL: https://issues.apache.org/jira/browse/CALCITE-5986 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > Labels: pull-request-available > > In SqlTypeFamily we have this code: > {code:java} > private static final Map JDBC_TYPE_TO_FAMILY = > ... > .put(Types.FLOAT, NUMERIC) > .put(Types.REAL, NUMERIC) > .put(Types.DOUBLE, NUMERIC) > {code} > But it looks to me like the type family should be APPROXIMATE_NUMERIC. > This impacts the way RelToSqlConverter works, for instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5987) SqlImplementor loses type information for literals
Mihai Budiu created CALCITE-5987: Summary: SqlImplementor loses type information for literals Key: CALCITE-5987 URL: https://issues.apache.org/jira/browse/CALCITE-5987 Project: Calcite Issue Type: Improvement Components: core Affects Versions: 1.35.0 Reporter: Mihai Budiu When converting a SqlNode to a String query, the conversion can produce SQL that computes different results. This happens because literals do not carry type information in the result string. For example, this plan: {code} rel#7:LogicalValues.(type=RecordType(VARCHAR(3) EXPR$0),tuples=[{ 'A' }]) {code} will generate a SQL query: {code} SELECT 'A' {code} While the type of the former result is VARCHAR(3), the latter query produces a CHAR(1) result. It would be nice if SqlImplementor had an option to produce a query that preserves the output type, e.g.: {code} SELECT (CAST 'A' as VARCHAR(3)) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763280#comment-17763280 ] Julian Hyde commented on CALCITE-5986: -- I saw your PR. * The "primary" and "secondary" concepts are very difficult to grasp. I think that the primary and secondary family methods in SqlTypeName need more documentation. * Do the methods both need to be nullable? * I think that the getFamily method should be deprecated. * "The SqlTypeFamily for FP types is incorrect" doesn't seem ready for release notes > The SqlTypeFamily for FP types is incorrect > --- > > Key: CALCITE-5986 > URL: https://issues.apache.org/jira/browse/CALCITE-5986 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > Labels: pull-request-available > > In SqlTypeFamily we have this code: > {code:java} > private static final Map JDBC_TYPE_TO_FAMILY = > ... > .put(Types.FLOAT, NUMERIC) > .put(Types.REAL, NUMERIC) > .put(Types.DOUBLE, NUMERIC) > {code} > But it looks to me like the type family should be APPROXIMATE_NUMERIC. > This impacts the way RelToSqlConverter works, for instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[ https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763275#comment-17763275 ] Jerin John commented on CALCITE-5979: - [~julianhyde] [~tanclary], did some further testing with RE2 and it seems like the the invalid regex cases we identified in CALCITE-5910 are being processed identical to Java regex, i.e. BQ was able to identify those as incorrect regex but it might not be getting caught directly by the RE2 library as per our original assumption, there could be custom error checks they implemented on top of it. There might be a possibility that they're using a different/internal version of RE2 (possibly the C wrapper) while I tested with the [re2j Java wrapper|https://github.com/google/re2j] which may not have included this functionality, not sure how we can ascertain that but based on this observation RE2 backend might not be the reason for these additional supported features. > Add REGEXP_REPLACE function (enabled in BigQuery library) > - > > Key: CALCITE-5979 > URL: https://issues.apache.org/jira/browse/CALCITE-5979 > Project: Calcite > Issue Type: Task >Reporter: Jerin John >Assignee: Jerin John >Priority: Major > Labels: pull-request-available > > Add support for [REGEXP_REPLACE > |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace] > function from BigQuery. > *{{REGEXP_REPLACE(value, regexp, replacement)}}* > Returns a STRING where all substrings of {{value}} that match regular > expression {{regexp}} are replaced with {{{}replacement{}}}. > backslashed-escaped digits (\1 to \9) can be used within the {{replacement}} > argument to insert text matching the corresponding parenthesized group in the > {{regexp}} pattern. > Example (added one space between \ \ to override md formatting): > {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}} > |result| > |aXc| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CALCITE-5986: Labels: pull-request-available (was: ) > The SqlTypeFamily for FP types is incorrect > --- > > Key: CALCITE-5986 > URL: https://issues.apache.org/jira/browse/CALCITE-5986 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > Labels: pull-request-available > > In SqlTypeFamily we have this code: > {code:java} > private static final Map JDBC_TYPE_TO_FAMILY = > ... > .put(Types.FLOAT, NUMERIC) > .put(Types.REAL, NUMERIC) > .put(Types.DOUBLE, NUMERIC) > {code} > But it looks to me like the type family should be APPROXIMATE_NUMERIC. > This impacts the way RelToSqlConverter works, for instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[ https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763254#comment-17763254 ] Tanner Clary commented on CALCITE-5979: --- Yeah as long as it is documented I think that sounds like a good idea. > Add REGEXP_REPLACE function (enabled in BigQuery library) > - > > Key: CALCITE-5979 > URL: https://issues.apache.org/jira/browse/CALCITE-5979 > Project: Calcite > Issue Type: Task >Reporter: Jerin John >Assignee: Jerin John >Priority: Major > Labels: pull-request-available > > Add support for [REGEXP_REPLACE > |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace] > function from BigQuery. > *{{REGEXP_REPLACE(value, regexp, replacement)}}* > Returns a STRING where all substrings of {{value}} that match regular > expression {{regexp}} are replaced with {{{}replacement{}}}. > backslashed-escaped digits (\1 to \9) can be used within the {{replacement}} > argument to insert text matching the corresponding parenthesized group in the > {{regexp}} pattern. > Example (added one space between \ \ to override md formatting): > {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}} > |result| > |aXc| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763251#comment-17763251 ] Mihai Budiu commented on CALCITE-5986: -- There should probably exist a test that checks the consistency of these mappings. > The SqlTypeFamily for FP types is incorrect > --- > > Key: CALCITE-5986 > URL: https://issues.apache.org/jira/browse/CALCITE-5986 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > > In SqlTypeFamily we have this code: > {code:java} > private static final Map JDBC_TYPE_TO_FAMILY = > ... > .put(Types.FLOAT, NUMERIC) > .put(Types.REAL, NUMERIC) > .put(Types.DOUBLE, NUMERIC) > {code} > But it looks to me like the type family should be APPROXIMATE_NUMERIC. > This impacts the way RelToSqlConverter works, for instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763249#comment-17763249 ] Mihai Budiu commented on CALCITE-5986: -- Another problem is that SqlTypeFamily.getTypeNames() DOES map APPROXIMATE_NUMERIC to DOUBLE, FLOAT, etc. So we have two inconsistent mappings describing these types. > The SqlTypeFamily for FP types is incorrect > --- > > Key: CALCITE-5986 > URL: https://issues.apache.org/jira/browse/CALCITE-5986 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > > In SqlTypeFamily we have this code: > {code:java} > private static final Map JDBC_TYPE_TO_FAMILY = > ... > .put(Types.FLOAT, NUMERIC) > .put(Types.REAL, NUMERIC) > .put(Types.DOUBLE, NUMERIC) > {code} > But it looks to me like the type family should be APPROXIMATE_NUMERIC. > This impacts the way RelToSqlConverter works, for instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[ https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763250#comment-17763250 ] Julian Hyde commented on CALCITE-5984: -- I don't know. If there is a sane way to prevent trimming, please contribute it. And contribute tests to demonstrate that trimming does not occur. If there are some rules that cannot operate without trimming (I don't know, just speculating) then I guess those rules would need to be disabled. > Cannot fully disable trimming unused fields using Calcite configs > - > > Key: CALCITE-5984 > URL: https://issues.apache.org/jira/browse/CALCITE-5984 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.34.0 >Reporter: Wegdan Ghazi >Priority: Major > > In Prepare#prepareSql there are 2 instances of trimming unused fields > 1. > [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] > called > [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297] > > {code:java} > // Trim unused fields. > root = trimUnusedFields(root); {code} > 2. > {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color} > called > [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312] > within > {code:java} > root = optimize(root, getMaterializations(), getLattices()); > {code} > I want to turn of the field trimming due to a personalisation I'm > implementing to scan a specific API, which I would think is possible using > [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26] > I attempted to disable the trimming using hooks [PROGRAM, > SQL2REL_CONVERTER_CONFIG_BUILDER], which succeeded to remove the trimming > program, yet on > [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] > {code:java} > // code placehprotected RelRoot trimUnusedFields(RelRoot root) { > final SqlToRelConverter.Config config = SqlToRelConverter.config() > .withTrimUnusedFields(shouldTrim(root.rel)) > .withExpand(THREAD_EXPAND.get()) > > .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get())); > final SqlToRelConverter converter = > getSqlToRelConverter(getSqlValidator(), catalogReader, config); > final boolean ordered = !root.collation.getFieldCollations().isEmpty(); > final boolean dml = SqlKind.DML.contains(root.kind); > return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel)); > } > {code} > The config passed (and edited by the hook) gets overwritten and only > _shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to > happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}. > Is this the intended behaviour? Any help is appreciated, thanks. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[ https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde updated CALCITE-5984: - Fix Version/s: (was: 1.34.0) > Cannot fully disable trimming unused fields using Calcite configs > - > > Key: CALCITE-5984 > URL: https://issues.apache.org/jira/browse/CALCITE-5984 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.34.0 >Reporter: Wegdan Ghazi >Priority: Major > > In Prepare#prepareSql there are 2 instances of trimming unused fields > 1. > [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] > called > [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297] > > {code:java} > // Trim unused fields. > root = trimUnusedFields(root); {code} > 2. > {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color} > called > [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312] > within > {code:java} > root = optimize(root, getMaterializations(), getLattices()); > {code} > I want to turn of the field trimming due to a personalisation I'm > implementing to scan a specific API, which I would think is possible using > [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26] > I attempted to disable the trimming using hooks [PROGRAM, > SQL2REL_CONVERTER_CONFIG_BUILDER], which succeeded to remove the trimming > program, yet on > [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] > {code:java} > // code placehprotected RelRoot trimUnusedFields(RelRoot root) { > final SqlToRelConverter.Config config = SqlToRelConverter.config() > .withTrimUnusedFields(shouldTrim(root.rel)) > .withExpand(THREAD_EXPAND.get()) > > .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get())); > final SqlToRelConverter converter = > getSqlToRelConverter(getSqlValidator(), catalogReader, config); > final boolean ordered = !root.collation.getFieldCollations().isEmpty(); > final boolean dml = SqlKind.DML.contains(root.kind); > return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel)); > } > {code} > The config passed (and edited by the hook) gets overwritten and only > _shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to > happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}. > Is this the intended behaviour? Any help is appreciated, thanks. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763246#comment-17763246 ] Mihai Budiu commented on CALCITE-5986: -- Moreover, if no type belongs to this family, it means that all the code that checks for this family is dead. > The SqlTypeFamily for FP types is incorrect > --- > > Key: CALCITE-5986 > URL: https://issues.apache.org/jira/browse/CALCITE-5986 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > > In SqlTypeFamily we have this code: > {code:java} > private static final Map JDBC_TYPE_TO_FAMILY = > ... > .put(Types.FLOAT, NUMERIC) > .put(Types.REAL, NUMERIC) > .put(Types.DOUBLE, NUMERIC) > {code} > But it looks to me like the type family should be APPROXIMATE_NUMERIC. > This impacts the way RelToSqlConverter works, for instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763245#comment-17763245 ] Julian Hyde commented on CALCITE-5986: -- Probably would have been better if {{SqlTypeName.FLOAT.getFamily()}} (etc.) returned {{APPROXIMATE_NUMERIC}} rather than {{NUMERIC}}. But changing it now would probably have huge impact. The bug here seems to be that the JDBC adapter generates the wrong literal for {{FLOAT}} etc. Let's work on fixing that bug. > The SqlTypeFamily for FP types is incorrect > --- > > Key: CALCITE-5986 > URL: https://issues.apache.org/jira/browse/CALCITE-5986 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > > In SqlTypeFamily we have this code: > {code:java} > private static final Map JDBC_TYPE_TO_FAMILY = > ... > .put(Types.FLOAT, NUMERIC) > .put(Types.REAL, NUMERIC) > .put(Types.DOUBLE, NUMERIC) > {code} > But it looks to me like the type family should be APPROXIMATE_NUMERIC. > This impacts the way RelToSqlConverter works, for instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763243#comment-17763243 ] Mihai Budiu commented on CALCITE-5986: -- But changing the family to APPROXIMATE_NUMERIC seems to break quite a few tests. I will investigate whether it's worth making the change. According to the SQL standard use of the term, this is a bug, these are approximate numeric types, e.g., https://www.andrew.cmu.edu/user/shadow/sql/sql1992.txt > The SqlTypeFamily for FP types is incorrect > --- > > Key: CALCITE-5986 > URL: https://issues.apache.org/jira/browse/CALCITE-5986 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > > In SqlTypeFamily we have this code: > {code:java} > private static final Map JDBC_TYPE_TO_FAMILY = > ... > .put(Types.FLOAT, NUMERIC) > .put(Types.REAL, NUMERIC) > .put(Types.DOUBLE, NUMERIC) > {code} > But it looks to me like the type family should be APPROXIMATE_NUMERIC. > This impacts the way RelToSqlConverter works, for instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5974) Elasticsearch adapter throws ClassCastException when index mapping sets dynamic_templates without properties
[ https://issues.apache.org/jira/browse/CALCITE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763241#comment-17763241 ] Julian Hyde commented on CALCITE-5974: -- Thanks for improving the summary, [~zoovwang]. Much improved. I just tweaked it a bit more. > Elasticsearch adapter throws ClassCastException when index mapping sets > dynamic_templates without properties > > > Key: CALCITE-5974 > URL: https://issues.apache.org/jira/browse/CALCITE-5974 > Project: Calcite > Issue Type: Bug >Affects Versions: 1.35.0 >Reporter: zhaowang >Assignee: zhaowang >Priority: Major > > When we use the elasticsearch-adapter, as we config an elasticsearch cluster, > it will fetchMapping during initialization. > If a index set dynamic_templates bug no mappings like this: > > {code:java} > { > "test_index" : { > "mappings" : { > "dynamic_templates" : [ > { > "integers" : { > "match_mapping_type" : "long", > "mapping" : { > "type" : "integer" > } > } > } > ] > } > } > } {code} > {{org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties}} > throw exception: > > {code:java} > java.lang.ClassCastException: > com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to > com.fasterxml.jackson.databind.node.ObjectNode > at > > org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133) > {code} > So ElasticsearchTransport initialize failed, all index of this es cluster > access failed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5974) Elasticsearch adapter throws ClassCastException when index mapping sets dynamic_templates without properties
[ https://issues.apache.org/jira/browse/CALCITE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde updated CALCITE-5974: - Summary: Elasticsearch adapter throws ClassCastException when index mapping sets dynamic_templates without properties (was: Elasticsearch adapter fetch mapping error when index mapping set dynamic_templates without properties) > Elasticsearch adapter throws ClassCastException when index mapping sets > dynamic_templates without properties > > > Key: CALCITE-5974 > URL: https://issues.apache.org/jira/browse/CALCITE-5974 > Project: Calcite > Issue Type: Bug >Affects Versions: 1.35.0 >Reporter: zhaowang >Assignee: zhaowang >Priority: Major > > When we use the elasticsearch-adapter, as we config an elasticsearch cluster, > it will fetchMapping during initialization. > If a index set dynamic_templates bug no mappings like this: > > {code:java} > { > "test_index" : { > "mappings" : { > "dynamic_templates" : [ > { > "integers" : { > "match_mapping_type" : "long", > "mapping" : { > "type" : "integer" > } > } > } > ] > } > } > } {code} > {{org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties}} > throw exception: > > {code:java} > java.lang.ClassCastException: > com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to > com.fasterxml.jackson.databind.node.ObjectNode > at > > org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133) > {code} > So ElasticsearchTransport initialize failed, all index of this es cluster > access failed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763238#comment-17763238 ] Mihai Budiu commented on CALCITE-5986: -- {code:java} FLOAT(PrecScale.NO_NO, false, Types.FLOAT, SqlTypeFamily.NUMERIC), REAL(PrecScale.NO_NO, false, Types.REAL, SqlTypeFamily.NUMERIC), DOUBLE(PrecScale.NO_NO, false, Types.DOUBLE, SqlTypeFamily.NUMERIC), {code} This leads to at least one bug, in RelToSqlConverter FP literals are emitted as DECIMAL literals. > The SqlTypeFamily for FP types is incorrect > --- > > Key: CALCITE-5986 > URL: https://issues.apache.org/jira/browse/CALCITE-5986 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > > In SqlTypeFamily we have this code: > {code:java} > private static final Map JDBC_TYPE_TO_FAMILY = > ... > .put(Types.FLOAT, NUMERIC) > .put(Types.REAL, NUMERIC) > .put(Types.DOUBLE, NUMERIC) > {code} > But it looks to me like the type family should be APPROXIMATE_NUMERIC. > This impacts the way RelToSqlConverter works, for instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[ https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763233#comment-17763233 ] Jerin John edited comment on CALCITE-5979 at 9/8/23 7:44 PM: - Totally agree Tanner, we can think about skipping support for this feature and document as a known bug but not sure of the future impact. Wrt enabling this feature for other dialects, with some limited testing on MySQL I can see that the results are altered for scenarios where {{"\ (\d)"}} s are present in the string along with a capturing group in regex pattern, since this starts replacing it with the capturing group where originally it might not be intended, or might throw errors for indices of missing groups (these are very exceptional scenarios when someone is specifically coding the string with and pattern to be this way, but still an edge case). [~julianhyde] makes sense, based on these implementations we have encountered only two cases where BQ supports additional syntax, one with this above capturing groups and the other with some invalid regex cases as documented in [CALCITE-5910|https://issues.apache.org/jira/browse/CALCITE-5910] was (Author: JIRAUSER301314): Totally agree Tanner, we can think about skipping support for this feature and document as a known bug but not sure of the future impact. Wrt enabling this feature for other dialects, with some limited testing on MySQL I can see that the results are altered for scenarios where {{"\ (\d)"}} s are present in the string along with a capturing group in regex pattern, since this starts replacing it with the capturing group where originally it might not be intended, or might throw errors for indices of missing groups (these are very exceptional scenarios when someone is specifically coding the string with and pattern to be this way, but still an edge case). > Add REGEXP_REPLACE function (enabled in BigQuery library) > - > > Key: CALCITE-5979 > URL: https://issues.apache.org/jira/browse/CALCITE-5979 > Project: Calcite > Issue Type: Task >Reporter: Jerin John >Assignee: Jerin John >Priority: Major > Labels: pull-request-available > > Add support for [REGEXP_REPLACE > |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace] > function from BigQuery. > *{{REGEXP_REPLACE(value, regexp, replacement)}}* > Returns a STRING where all substrings of {{value}} that match regular > expression {{regexp}} are replaced with {{{}replacement{}}}. > backslashed-escaped digits (\1 to \9) can be used within the {{replacement}} > argument to insert text matching the corresponding parenthesized group in the > {{regexp}} pattern. > Example (added one space between \ \ to override md formatting): > {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}} > |result| > |aXc| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5974) Elasticsearch adapter fetch mapping error when index mapping set dynamic_templates without properties
[ https://issues.apache.org/jira/browse/CALCITE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde updated CALCITE-5974: - Description: When we use the elasticsearch-adapter, as we config an elasticsearch cluster, it will fetchMapping during initialization. If a index set dynamic_templates bug no mappings like this: {code:java} { "test_index" : { "mappings" : { "dynamic_templates" : [ { "integers" : { "match_mapping_type" : "long", "mapping" : { "type" : "integer" } } } ] } } } {code} {{org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties}} throw exception: {code:java} java.lang.ClassCastException: com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to com.fasterxml.jackson.databind.node.ObjectNode at org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133) {code} So ElasticsearchTransport initialize failed, all index of this es cluster access failed. was: When use es-adapter, we config a es cluster, it will fetchMapping during initialization. If a index set dynamic_templates bug no mappings like this: {code:java} { "test_index" : { "mappings" : { "dynamic_templates" : [ { "integers" : { "match_mapping_type" : "long", "mapping" : { "type" : "integer" } } } ] } } } {code} org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties throw exception: {code:java} java.lang.ClassCastException: com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to com.fasterxml.jackson.databind.node.ObjectNode at org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133) {code} So ElasticsearchTransport initialize failed, all index of this es cluster access failed. > Elasticsearch adapter fetch mapping error when index mapping set > dynamic_templates without properties > - > > Key: CALCITE-5974 > URL: https://issues.apache.org/jira/browse/CALCITE-5974 > Project: Calcite > Issue Type: Bug >Affects Versions: 1.35.0 >Reporter: zhaowang >Assignee: zhaowang >Priority: Major > > When we use the elasticsearch-adapter, as we config an elasticsearch cluster, > it will fetchMapping during initialization. > If a index set dynamic_templates bug no mappings like this: > > {code:java} > { > "test_index" : { > "mappings" : { > "dynamic_templates" : [ > { > "integers" : { > "match_mapping_type" : "long", > "mapping" : { > "type" : "integer" > } > } > } > ] > } > } > } {code} > {{org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties}} > throw exception: > > {code:java} > java.lang.ClassCastException: > com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to > com.fasterxml.jackson.databind.node.ObjectNode > at > > org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133) > {code} > So ElasticsearchTransport initialize failed, all index of this es cluster > access failed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5985) FilterTableFunctionTransposeRule should not use "Logical" RelNodes
[ https://issues.apache.org/jira/browse/CALCITE-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763235#comment-17763235 ] Julian Hyde commented on CALCITE-5985: -- I agree. Perhaps retain an instance of the rule that continues to match only {{LogicalFilter}} and {{LogicalTableScan}}, and also add a new instance that is generic. Also, use the {{filter}} and {{functionScan}} methods in {{RelBuilder}}. > FilterTableFunctionTransposeRule should not use "Logical" RelNodes > -- > > Key: CALCITE-5985 > URL: https://issues.apache.org/jira/browse/CALCITE-5985 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Steve Carlin >Priority: Major > > The FilterTableFunctionTransposeRule uses LogicalTableFunctionScan and > LogicalFilter RelNodes in its rule. We should be using the appropriate > TableFunctionScan and Filter interface RelNodes instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[ https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763233#comment-17763233 ] Jerin John commented on CALCITE-5979: - Totally agree Tanner, we can think about skipping support for this feature and document as a known bug but not sure of the future impact. Wrt enabling this feature for other dialects, with some limited testing on MySQL I can see that the results are altered for scenarios where {{"\ (\d)"}} s are present in the string along with a capturing group in regex pattern, since this starts replacing it with the capturing group where originally it might not be intended, or might throw errors for indices of missing groups (these are very exceptional scenarios when someone is specifically coding the string with and pattern to be this way, but still an edge case). > Add REGEXP_REPLACE function (enabled in BigQuery library) > - > > Key: CALCITE-5979 > URL: https://issues.apache.org/jira/browse/CALCITE-5979 > Project: Calcite > Issue Type: Task >Reporter: Jerin John >Assignee: Jerin John >Priority: Major > Labels: pull-request-available > > Add support for [REGEXP_REPLACE > |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace] > function from BigQuery. > *{{REGEXP_REPLACE(value, regexp, replacement)}}* > Returns a STRING where all substrings of {{value}} that match regular > expression {{regexp}} are replaced with {{{}replacement{}}}. > backslashed-escaped digits (\1 to \9) can be used within the {{replacement}} > argument to insert text matching the corresponding parenthesized group in the > {{regexp}} pattern. > Example (added one space between \ \ to override md formatting): > {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}} > |result| > |aXc| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[ https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763232#comment-17763232 ] Julian Hyde edited comment on CALCITE-5979 at 9/8/23 7:35 PM: -- The simplest thing is to change the specification to match what is easy to implement. Specifically, to say that {{REGEXP_REPLACE}} supports Java regexp syntax. We can have a follow-up case to allow the {{REGEXP_xxx}} functions support BQ syntax, and to devise a flag to enable the BQ mode. In that case, list the cases where the current implementation is wrong. Over time, the demand to fix that follow-up case will grow, but we can defer it for a while. As you can see, I'm punting on whether we have one operator or two. I really think that kicking the can down the road is the best strategy. was (Author: julianhyde): The simplest thing is to change the specification to match what is easy to implement. Specifically, to say that {{REGEXP_REPLACE}} supports Java regexp syntax. We can have a follow-up case to allow the {{REGEXP_xxx}} functions support BQ syntax, and to devise a flag to enable the BQ mode. In that case, list the cases where the current implementation is wrong. Over time, the demand to fix that follow-up case will grow, but we can defer it for a while. > Add REGEXP_REPLACE function (enabled in BigQuery library) > - > > Key: CALCITE-5979 > URL: https://issues.apache.org/jira/browse/CALCITE-5979 > Project: Calcite > Issue Type: Task >Reporter: Jerin John >Assignee: Jerin John >Priority: Major > Labels: pull-request-available > > Add support for [REGEXP_REPLACE > |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace] > function from BigQuery. > *{{REGEXP_REPLACE(value, regexp, replacement)}}* > Returns a STRING where all substrings of {{value}} that match regular > expression {{regexp}} are replaced with {{{}replacement{}}}. > backslashed-escaped digits (\1 to \9) can be used within the {{replacement}} > argument to insert text matching the corresponding parenthesized group in the > {{regexp}} pattern. > Example (added one space between \ \ to override md formatting): > {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}} > |result| > |aXc| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[ https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763232#comment-17763232 ] Julian Hyde commented on CALCITE-5979: -- The simplest thing is to change the specification to match what is easy to implement. Specifically, to say that {{REGEXP_REPLACE}} supports Java regexp syntax. We can have a follow-up case to allow the {{REGEXP_xxx}} functions support BQ syntax, and to devise a flag to enable the BQ mode. In that case, list the cases where the current implementation is wrong. Over time, the demand to fix that follow-up case will grow, but we can defer it for a while. > Add REGEXP_REPLACE function (enabled in BigQuery library) > - > > Key: CALCITE-5979 > URL: https://issues.apache.org/jira/browse/CALCITE-5979 > Project: Calcite > Issue Type: Task >Reporter: Jerin John >Assignee: Jerin John >Priority: Major > Labels: pull-request-available > > Add support for [REGEXP_REPLACE > |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace] > function from BigQuery. > *{{REGEXP_REPLACE(value, regexp, replacement)}}* > Returns a STRING where all substrings of {{value}} that match regular > expression {{regexp}} are replaced with {{{}replacement{}}}. > backslashed-escaped digits (\1 to \9) can be used within the {{replacement}} > argument to insert text matching the corresponding parenthesized group in the > {{regexp}} pattern. > Example (added one space between \ \ to override md formatting): > {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}} > |result| > |aXc| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763228#comment-17763228 ] Julian Hyde commented on CALCITE-5986: -- I don't see any problem in {{enum SqlTypeName}}. This looks perfect: {code} public static final List INT_TYPES = ImmutableList.of(TINYINT, SMALLINT, INTEGER, BIGINT); public static final List EXACT_TYPES = combine(INT_TYPES, ImmutableList.of(DECIMAL)); public static final List APPROX_TYPES = ImmutableList.of(FLOAT, REAL, DOUBLE); public static final List NUMERIC_TYPES = combine(EXACT_TYPES, APPROX_TYPES); {code} > The SqlTypeFamily for FP types is incorrect > --- > > Key: CALCITE-5986 > URL: https://issues.apache.org/jira/browse/CALCITE-5986 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > > In SqlTypeFamily we have this code: > {code:java} > private static final Map JDBC_TYPE_TO_FAMILY = > ... > .put(Types.FLOAT, NUMERIC) > .put(Types.REAL, NUMERIC) > .put(Types.DOUBLE, NUMERIC) > {code} > But it looks to me like the type family should be APPROXIMATE_NUMERIC. > This impacts the way RelToSqlConverter works, for instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763227#comment-17763227 ] Julian Hyde commented on CALCITE-5986: -- JDBC_TYPE_TO_FAMILY is private and not used, AFAICT. If so, it's not really a bug. I agree that mapping to NUMERIC rather than APPROXIMATE_NUMERIC looks a bit strange; the data model is a hierarchy. But really, it depends what that field is used for, which at present is nothing. > The SqlTypeFamily for FP types is incorrect > --- > > Key: CALCITE-5986 > URL: https://issues.apache.org/jira/browse/CALCITE-5986 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > > In SqlTypeFamily we have this code: > {code:java} > private static final Map JDBC_TYPE_TO_FAMILY = > ... > .put(Types.FLOAT, NUMERIC) > .put(Types.REAL, NUMERIC) > .put(Types.DOUBLE, NUMERIC) > {code} > But it looks to me like the type family should be APPROXIMATE_NUMERIC. > This impacts the way RelToSqlConverter works, for instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
[ https://issues.apache.org/jira/browse/CALCITE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763220#comment-17763220 ] Mihai Budiu commented on CALCITE-5986: -- There's a similar problem in the SqlTypeName enum. Am I misinterpreting the meaning of APPROXIMATE_NUMERIC? > The SqlTypeFamily for FP types is incorrect > --- > > Key: CALCITE-5986 > URL: https://issues.apache.org/jira/browse/CALCITE-5986 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.35.0 >Reporter: Mihai Budiu >Priority: Minor > > In SqlTypeFamily we have this code: > {code:java} > private static final Map JDBC_TYPE_TO_FAMILY = > ... > .put(Types.FLOAT, NUMERIC) > .put(Types.REAL, NUMERIC) > .put(Types.DOUBLE, NUMERIC) > {code} > But it looks to me like the type family should be APPROXIMATE_NUMERIC. > This impacts the way RelToSqlConverter works, for instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5986) The SqlTypeFamily for FP types is incorrect
Mihai Budiu created CALCITE-5986: Summary: The SqlTypeFamily for FP types is incorrect Key: CALCITE-5986 URL: https://issues.apache.org/jira/browse/CALCITE-5986 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.35.0 Reporter: Mihai Budiu In SqlTypeFamily we have this code: {code:java} private static final Map JDBC_TYPE_TO_FAMILY = ... .put(Types.FLOAT, NUMERIC) .put(Types.REAL, NUMERIC) .put(Types.DOUBLE, NUMERIC) {code} But it looks to me like the type family should be APPROXIMATE_NUMERIC. This impacts the way RelToSqlConverter works, for instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[ https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763215#comment-17763215 ] Tanner Clary edited comment on CALCITE-5979 at 9/8/23 6:40 PM: --- I agree that it seems like a bit much to accommodate for what seems to be a pretty small difference, but I also think it is hard to draw the line on what warrants a new operator entry or not. For instance, {{FLOOR}} and {{SUBSTR}} both have dialect-specific operators because the behavior varies across dialects in one way or another (perhaps in return type, or unparsing syntax, etc.). I can foresee a situation where we decide not to support this specific behavior to avoid adding a new operator, and then somewhere down the line a bug gets filed because someone is expecting this behavior to work because we say we support BigQuery's functionality, and instead it fails. I only mention that because I have introduced some of those bugs myself. Maybe that's overthinking it a bit but it's worth keeping in mind I think. I am also curious what would happen if this behavior was enabled for the other dialects? Would it throw an unexpected error? Would it return the wrong results? Would it change anything? was (Author: JIRAUSER298151): I agree that it seems like a bit much to accommodate for what seems to be a pretty small difference, but I also think it is hard to draw the line on what warrants a new operator entry or not. For instance, {{FLOOR}} and {{SUBSTR}} both have dialect-specific operators because the behavior varies across dialects in one way or another (perhaps in return type, or unparsing syntax, etc.). I can foresee a situation where we decide not to support this specific behavior to avoid adding a new operator, and then somewhere down the line a bug gets filed because someone is expecting this behavior to work because we say we support BigQuery's functionality, and instead it fails. I only mention that because I have introduced some of those bugs myself. Maybe that's overthinking it a bit but it's worth keeping in mind I think. > Add REGEXP_REPLACE function (enabled in BigQuery library) > - > > Key: CALCITE-5979 > URL: https://issues.apache.org/jira/browse/CALCITE-5979 > Project: Calcite > Issue Type: Task >Reporter: Jerin John >Assignee: Jerin John >Priority: Major > Labels: pull-request-available > > Add support for [REGEXP_REPLACE > |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace] > function from BigQuery. > *{{REGEXP_REPLACE(value, regexp, replacement)}}* > Returns a STRING where all substrings of {{value}} that match regular > expression {{regexp}} are replaced with {{{}replacement{}}}. > backslashed-escaped digits (\1 to \9) can be used within the {{replacement}} > argument to insert text matching the corresponding parenthesized group in the > {{regexp}} pattern. > Example (added one space between \ \ to override md formatting): > {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}} > |result| > |aXc| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[ https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763215#comment-17763215 ] Tanner Clary edited comment on CALCITE-5979 at 9/8/23 6:39 PM: --- I agree that it seems like a bit much to accommodate for what seems to be a pretty small difference, but I also think it is hard to draw the line on what warrants a new operator entry or not. For instance, {{FLOOR}} and {{SUBSTR}} both have dialect-specific operators because the behavior varies across dialects in one way or another (perhaps in return type, or unparsing syntax, etc.). I can foresee a situation where we decide not to support this specific behavior to avoid adding a new operator, and then somewhere down the line a bug gets filed because someone is expecting this behavior to work because we say we support BigQuery's functionality, and instead it fails. I only mention that because I have introduced some of those bugs myself. Maybe that's overthinking it a bit but it's worth keeping in mind I think. was (Author: JIRAUSER298151): I agree that it seems like a bit much to accommodate for what seems to be a pretty small difference, but I also think it is hard to draw the line on what warrants a new operator entry or not. For instance, {{FLOOR}} and {{SUBSTR}} both have dialect-specific operators because the behavior varies across dialects in one way or another (perhaps in return type, or unparsing syntax, etc.). I can foresee a situation where we decide not to support this specific behavior to avoid adding a new operator, and then somewhere down the line a bug gets filed because someone is expecting this behavior to work because we say we support BigQuery's functionality, and instead it fails. I only mention that because I have introduced some of those bugs myself (and fixed a couple, too). Maybe that's overthinking it a bit but it's worth keeping in mind I think. > Add REGEXP_REPLACE function (enabled in BigQuery library) > - > > Key: CALCITE-5979 > URL: https://issues.apache.org/jira/browse/CALCITE-5979 > Project: Calcite > Issue Type: Task >Reporter: Jerin John >Assignee: Jerin John >Priority: Major > Labels: pull-request-available > > Add support for [REGEXP_REPLACE > |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace] > function from BigQuery. > *{{REGEXP_REPLACE(value, regexp, replacement)}}* > Returns a STRING where all substrings of {{value}} that match regular > expression {{regexp}} are replaced with {{{}replacement{}}}. > backslashed-escaped digits (\1 to \9) can be used within the {{replacement}} > argument to insert text matching the corresponding parenthesized group in the > {{regexp}} pattern. > Example (added one space between \ \ to override md formatting): > {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}} > |result| > |aXc| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[ https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763215#comment-17763215 ] Tanner Clary commented on CALCITE-5979: --- I agree that it seems like a bit much to accommodate for what seems to be a pretty small difference, but I also think it is hard to draw the line on what warrants a new operator entry or not. For instance, {{FLOOR}} and {{SUBSTR}} both have dialect-specific operators because the behavior varies across dialects in one way or another (perhaps in return type, or unparsing syntax, etc.). I can foresee a situation where we decide not to support this specific behavior to avoid adding a new operator, and then somewhere down the line a bug gets filed because someone is expecting this behavior to work because we say we support BigQuery's functionality, and instead it fails. I only mention that because I have introduced some of those bugs myself (and fixed a couple, too). Maybe that's overthinking it a bit but it's worth keeping in mind I think. > Add REGEXP_REPLACE function (enabled in BigQuery library) > - > > Key: CALCITE-5979 > URL: https://issues.apache.org/jira/browse/CALCITE-5979 > Project: Calcite > Issue Type: Task >Reporter: Jerin John >Assignee: Jerin John >Priority: Major > Labels: pull-request-available > > Add support for [REGEXP_REPLACE > |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace] > function from BigQuery. > *{{REGEXP_REPLACE(value, regexp, replacement)}}* > Returns a STRING where all substrings of {{value}} that match regular > expression {{regexp}} are replaced with {{{}replacement{}}}. > backslashed-escaped digits (\1 to \9) can be used within the {{replacement}} > argument to insert text matching the corresponding parenthesized group in the > {{regexp}} pattern. > Example (added one space between \ \ to override md formatting): > {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}} > |result| > |aXc| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5979) Add REGEXP_REPLACE function (enabled in BigQuery library)
[ https://issues.apache.org/jira/browse/CALCITE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763212#comment-17763212 ] Jerin John commented on CALCITE-5979: - Hi [~julianhyde] and everyone, I wanted to get your opinion on this regex implementation for BQ, my attempt was to reuse the existing implementation of REGEXP_REPLACE from other libraries. But if you check this [BQ documentation|https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace], there's a group replacement syntax supported by BQ involving {{"\"}} s which is not currently available for the other libraries, so it needs a precursor step to do a string replacement of this kind: {{replacement = replacement.replaceAll("\\(\d)", "\$$1");}} I tested this syntax with {{re2j}} library to check if its something supported by [RE2|https://github.com/google/re2] and not Java regex but thats not the case, seems like BQ might be custom processing it first. It's not evident why they went with this syntax since Java and regex standard annotation to index capturing groups from an expression is like {{$1, $2..}} and thats how we are replacing the {{"\"}} s in the above code as well. I discussed with [~tanclary] and from what i understand, one possible approach is to modify the parsers to redirect to a {{SqlStdOperatorTable}} method to check the library conformance, and create a new BQ specific operator {{REGEXP_REPLACE_BIG_QUERY}} alongside the existing {{REGEXP_REPLACE}} operator. Op {{REGEXP_REPLACE_BIG_QUERY}} is mapped to its own {{SqlFunction}} for this custom preprocessing which should then call the original {{regexReplace}} method used by other dialects. I get that this design abstracts the dialect specific logic out of rex layer, but also seems like a lot of steps to identify the conformance. It'd be great to get your thoughts on if there's a better approach or if this seems like the right way to go. > Add REGEXP_REPLACE function (enabled in BigQuery library) > - > > Key: CALCITE-5979 > URL: https://issues.apache.org/jira/browse/CALCITE-5979 > Project: Calcite > Issue Type: Task >Reporter: Jerin John >Assignee: Jerin John >Priority: Major > Labels: pull-request-available > > Add support for [REGEXP_REPLACE > |https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace] > function from BigQuery. > *{{REGEXP_REPLACE(value, regexp, replacement)}}* > Returns a STRING where all substrings of {{value}} that match regular > expression {{regexp}} are replaced with {{{}replacement{}}}. > backslashed-escaped digits (\1 to \9) can be used within the {{replacement}} > argument to insert text matching the corresponding parenthesized group in the > {{regexp}} pattern. > Example (added one space between \ \ to override md formatting): > {{SELECT REGEXP_REPLACE("abc'", "b(.)", "X\ \1") as result;}} > |result| > |aXc| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5985) FilterTableFunctionTransposeRule should not use "Logical" RelNodes
Steve Carlin created CALCITE-5985: - Summary: FilterTableFunctionTransposeRule should not use "Logical" RelNodes Key: CALCITE-5985 URL: https://issues.apache.org/jira/browse/CALCITE-5985 Project: Calcite Issue Type: Bug Components: core Reporter: Steve Carlin The FilterTableFunctionTransposeRule uses LogicalTableFunctionScan and LogicalFilter RelNodes in its rule. We should be using the appropriate TableFunctionScan and Filter interface RelNodes instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[ https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wegdan Ghazi updated CALCITE-5984: -- Description: In Prepare#prepareSql there are 2 instances of trimming unused fields 1. [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297] {code:java} // Trim unused fields. root = trimUnusedFields(root); {code} 2. {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color} called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312] within {code:java} root = optimize(root, getMaterializations(), getLattices()); {code} I want to turn of the field trimming due to a personalisation I'm implementing to scan a specific API, which I would think is possible using [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26] I attempted to disable the trimming using hooks [PROGRAM, SQL2REL_CONVERTER_CONFIG_BUILDER], which succeeded to remove the trimming program, yet on [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] {code:java} // code placehprotected RelRoot trimUnusedFields(RelRoot root) { final SqlToRelConverter.Config config = SqlToRelConverter.config() .withTrimUnusedFields(shouldTrim(root.rel)) .withExpand(THREAD_EXPAND.get()) .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get())); final SqlToRelConverter converter = getSqlToRelConverter(getSqlValidator(), catalogReader, config); final boolean ordered = !root.collation.getFieldCollations().isEmpty(); final boolean dml = SqlKind.DML.contains(root.kind); return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel)); } {code} The config passed (and edited by the hook) gets overwritten and only _shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}. Is this the intended behaviour? Any help is appreciated, thanks. was: In Prepare#prepareSql there are 2 instances of trimming unused fields 1. [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297] {code:java} // Trim unused fields. root = trimUnusedFields(root); {code} 2. {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color} called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312] within {code:java} root = optimize(root, getMaterializations(), getLattices()); {code} I want to turn of the field trimming due to a personalisation I'm implementing to scan a specific API, which I would think is possible using [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26] I attempted to disable the trimming using hooks [PROGRAM, SQL2REL_CONVERTER_CONFIG_BUILDER], which succeeded to remove the trimming program, yet on [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] {code:java} // code placehprotected RelRoot trimUnusedFields(RelRoot root) { final SqlToRelConverter.Config config = SqlToRelConverter.config() .withTrimUnusedFields(shouldTrim(root.rel)) .withExpand(THREAD_EXPAND.get()) .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get())); final SqlToRelConverter converter = getSqlToRelConverter(getSqlValidator(), catalogReader, config); final boolean ordered = !root.collation.getFieldCollations().isEmpty(); final boolean dml = SqlKind.DML.contains(root.kind); return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel)); } {code} The config passed (and edited by the hook) gets overwritten and o
[jira] [Updated] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[ https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wegdan Ghazi updated CALCITE-5984: -- Description: In Prepare#prepareSql there are 2 instances of trimming unused fields 1. [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297] {code:java} // Trim unused fields. root = trimUnusedFields(root); {code} 2. {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color} called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312] within {code:java} root = optimize(root, getMaterializations(), getLattices()); {code} I want to turn of the field trimming due to a personalisation I'm implementing to scan a specific API, which I would think is possible using [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26] I attempted to disable the trimming using hooks [PROGRAM, SQL2REL_CONVERTER_CONFIG_BUILDER], which succeeded to remove the trimming program, yet on [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] {code:java} // code placehprotected RelRoot trimUnusedFields(RelRoot root) { final SqlToRelConverter.Config config = SqlToRelConverter.config() .withTrimUnusedFields(shouldTrim(root.rel)) .withExpand(THREAD_EXPAND.get()) .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get())); final SqlToRelConverter converter = getSqlToRelConverter(getSqlValidator(), catalogReader, config); final boolean ordered = !root.collation.getFieldCollations().isEmpty(); final boolean dml = SqlKind.DML.contains(root.kind); return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel)); } {code} The config passed (and edited by the hook) gets overwritten and only _shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}. was: In Prepare#prepareSql there are 2 instances of trimming unused fields 1. [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297] {code:java} // Trim unused fields. root = trimUnusedFields(root); {code} 2. {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color} called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312] within {code:java} root = optimize(root, getMaterializations(), getLattices()); {code} I want to turn of the field trimming due to a personalisation I'm implementing to scan a specific API, which I would think is possible using [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26] I attempted to disable the trimming using hooks [PROGRAM, SQL2REL_CONVERTER_CONFIG_BUILDER], which succeeded to remove the trimming program, yet on [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] {code:java} // code placehprotected RelRoot trimUnusedFields(RelRoot root) { final SqlToRelConverter.Config config = SqlToRelConverter.config() .withTrimUnusedFields(shouldTrim(root.rel)) .withExpand(THREAD_EXPAND.get()) .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get())); final SqlToRelConverter converter = getSqlToRelConverter(getSqlValidator(), catalogReader, config); final boolean ordered = !root.collation.getFieldCollations().isEmpty(); final boolean dml = SqlKind.DML.contains(root.kind); return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel)); }older {code} The config passed (and edited by the hook) gets overwritten and only _shouldTrim(root.rel)_ can enable/disable trimming, I
[jira] [Updated] (CALCITE-5974) Elasticsearch adapter fetch mapping error when index mapping set dynamic_templates without properties
[ https://issues.apache.org/jira/browse/CALCITE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaowang updated CALCITE-5974: -- Summary: Elasticsearch adapter fetch mapping error when index mapping set dynamic_templates without properties (was: Elasticsearch adapter find mapping error when index mapping set dynamic_templates without properties) > Elasticsearch adapter fetch mapping error when index mapping set > dynamic_templates without properties > - > > Key: CALCITE-5974 > URL: https://issues.apache.org/jira/browse/CALCITE-5974 > Project: Calcite > Issue Type: Bug >Affects Versions: 1.35.0 >Reporter: zhaowang >Assignee: zhaowang >Priority: Major > > When use es-adapter, we config a es cluster, it will fetchMapping during > initialization. > If a index set dynamic_templates bug no mappings like this: > > {code:java} > { > "test_index" : { > "mappings" : { > "dynamic_templates" : [ > { > "integers" : { > "match_mapping_type" : "long", > "mapping" : { > "type" : "integer" > } > } > } > ] > } > } > } {code} > > > org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties > throw exception: > > {code:java} > java.lang.ClassCastException: > com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to > com.fasterxml.jackson.databind.node.ObjectNode > at > > org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133) > {code} > So ElasticsearchTransport initialize failed, all index of this es cluster > access failed. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5974) Elasticsearch adapter find mapping error when index mapping set dynamic_templates without properties causing bug
[ https://issues.apache.org/jira/browse/CALCITE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaowang updated CALCITE-5974: -- Summary: Elasticsearch adapter find mapping error when index mapping set dynamic_templates without properties causing bug (was: Es index mapping set dynamic_templates without properties causing bug) > Elasticsearch adapter find mapping error when index mapping set > dynamic_templates without properties causing bug > > > Key: CALCITE-5974 > URL: https://issues.apache.org/jira/browse/CALCITE-5974 > Project: Calcite > Issue Type: Bug >Affects Versions: 1.35.0 >Reporter: zhaowang >Assignee: zhaowang >Priority: Major > > When use es-adapter, we config a es cluster, it will fetchMapping during > initialization. > If a index set dynamic_templates bug no mappings like this: > > {code:java} > { > "test_index" : { > "mappings" : { > "dynamic_templates" : [ > { > "integers" : { > "match_mapping_type" : "long", > "mapping" : { > "type" : "integer" > } > } > } > ] > } > } > } {code} > > > org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties > throw exception: > > {code:java} > java.lang.ClassCastException: > com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to > com.fasterxml.jackson.databind.node.ObjectNode > at > > org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133) > {code} > So ElasticsearchTransport initialize failed, all index of this es cluster > access failed. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5974) Elasticsearch adapter find mapping error when index mapping set dynamic_templates without properties
[ https://issues.apache.org/jira/browse/CALCITE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaowang updated CALCITE-5974: -- Summary: Elasticsearch adapter find mapping error when index mapping set dynamic_templates without properties (was: Elasticsearch adapter find mapping error when index mapping set dynamic_templates without properties causing bug) > Elasticsearch adapter find mapping error when index mapping set > dynamic_templates without properties > > > Key: CALCITE-5974 > URL: https://issues.apache.org/jira/browse/CALCITE-5974 > Project: Calcite > Issue Type: Bug >Affects Versions: 1.35.0 >Reporter: zhaowang >Assignee: zhaowang >Priority: Major > > When use es-adapter, we config a es cluster, it will fetchMapping during > initialization. > If a index set dynamic_templates bug no mappings like this: > > {code:java} > { > "test_index" : { > "mappings" : { > "dynamic_templates" : [ > { > "integers" : { > "match_mapping_type" : "long", > "mapping" : { > "type" : "integer" > } > } > } > ] > } > } > } {code} > > > org.apache.calcite.adapter.elasticsearch.ElasticsearchJson#visitMappingProperties > throw exception: > > {code:java} > java.lang.ClassCastException: > com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to > com.fasterxml.jackson.databind.node.ObjectNode > at > > org.apache.calcite.adapter.elasticsearch.ElasticsearchJson.visitMappingProperties(ElasticsearchJson.java:133) > {code} > So ElasticsearchTransport initialize failed, all index of this es cluster > access failed. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5732) EnumerableHashJoin and EnumerableMergeJoin on composite key return rows matching condition 'null = null'
[ https://issues.apache.org/jira/browse/CALCITE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763005#comment-17763005 ] Ruben Q L edited comment on CALCITE-5732 at 9/8/23 9:43 AM: Fixed via [{{3aee0b8}}|https://github.com/apache/calcite/commit/3aee0b86aa23476cbdecc75ad5d43b936a6fff7b] Thansk [~viggoc] for reporting this issue, and [~libenchao] for the review! was (Author: rubenql): Fixed via [{{3aee0b8}}|https://github.com/apache/calcite/commit/3aee0b86aa23476cbdecc75ad5d43b936a6fff7b] Thansk [~viggoc] for reporting this issue, and [~libenchao] for the review! > EnumerableHashJoin and EnumerableMergeJoin on composite key return rows > matching condition 'null = null' > > > Key: CALCITE-5732 > URL: https://issues.apache.org/jira/browse/CALCITE-5732 > Project: Calcite > Issue Type: Bug > Components: core, linq4j >Reporter: Viggo Chen >Assignee: Ruben Q L >Priority: Major > Labels: pull-request-available > Fix For: 1.36.0 > > > In `EnumerableJoinTest#testMergeJoinWithCompositeKeyAndNullValues`, the query > is like > {code:java} > select > emps.empid > from > emps a join emps b > on a.deptno = b.deptno > and a.commission = b.commission;{code} > and the data is like > {code:java} > INSERT INTO "emps" VALUES (100, 10, 'Bill', 1, 1000); > INSERT INTO "emps" VALUES (200, 20, 'Eric', 8000, 500); > INSERT INTO "emps" VALUES (150, 10, 'Sebastian', 7000, null); > INSERT INTO "emps" VALUES (110, 10, 'Theodore', 11500, 250); {code} > And row with empid = 150 is in expected result. Is this the expected result > of join with null condition. > Whats more hash join result with condition a.deptno = b.deptno and > a.commission = b.commission is same as merge join. And if there is just one > condition a.commission = b.commission, the result do not include empid = 150. > > Here is a unit test for it > {code:java} > @Test void testHashJoinWithCompositeKeyAndNullValues() { > // Both join side 'commission' a limited to null, so a.commission = > b.commission should always be false. > // So all columns in right table b are expected to be null, this sql should > result in 0 rows. > final String sql = "select * from\n" > + " (select empid, salary, commission from emps where commission is > null) as a\n" > + " left join\n" > + " (select empid, salary, commission from emps where commission is > null) as b\n" > + " on a.salary = b.salary and a.commission = b.commission\n" > + " where b.empid is not null"; > CalciteAssert.that() > .with(CalciteConnectionProperty.LEX, Lex.JAVA) > .with(CalciteConnectionProperty.FORCE_DECORRELATE, false) > .withSchema("s", new ReflectiveSchema(new HrSchemaBig())) > .query(sql) > .withHook(Hook.PLANNER, (Consumer) planner -> { > planner.removeRule(EnumerableRules.ENUMERABLE_JOIN_RULE); > planner.addRule(EnumerableRules.ENUMERABLE_MERGE_JOIN_RULE); > }) > .explainContains("EnumerableHashJoin") > .returnsCount(0) > ; > } {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CALCITE-5732) EnumerableHashJoin and EnumerableMergeJoin on composite key return rows matching condition 'null = null'
[ https://issues.apache.org/jira/browse/CALCITE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Q L resolved CALCITE-5732. Resolution: Fixed Fixed via [{{3aee0b8}}|https://github.com/apache/calcite/commit/3aee0b86aa23476cbdecc75ad5d43b936a6fff7b] Thansk [~viggoc] for reporting this issue, and [~libenchao] for the review! > EnumerableHashJoin and EnumerableMergeJoin on composite key return rows > matching condition 'null = null' > > > Key: CALCITE-5732 > URL: https://issues.apache.org/jira/browse/CALCITE-5732 > Project: Calcite > Issue Type: Bug > Components: core, linq4j >Reporter: Viggo Chen >Assignee: Ruben Q L >Priority: Major > Labels: pull-request-available > Fix For: 1.36.0 > > > In `EnumerableJoinTest#testMergeJoinWithCompositeKeyAndNullValues`, the query > is like > {code:java} > select > emps.empid > from > emps a join emps b > on a.deptno = b.deptno > and a.commission = b.commission;{code} > and the data is like > {code:java} > INSERT INTO "emps" VALUES (100, 10, 'Bill', 1, 1000); > INSERT INTO "emps" VALUES (200, 20, 'Eric', 8000, 500); > INSERT INTO "emps" VALUES (150, 10, 'Sebastian', 7000, null); > INSERT INTO "emps" VALUES (110, 10, 'Theodore', 11500, 250); {code} > And row with empid = 150 is in expected result. Is this the expected result > of join with null condition. > Whats more hash join result with condition a.deptno = b.deptno and > a.commission = b.commission is same as merge join. And if there is just one > condition a.commission = b.commission, the result do not include empid = 150. > > Here is a unit test for it > {code:java} > @Test void testHashJoinWithCompositeKeyAndNullValues() { > // Both join side 'commission' a limited to null, so a.commission = > b.commission should always be false. > // So all columns in right table b are expected to be null, this sql should > result in 0 rows. > final String sql = "select * from\n" > + " (select empid, salary, commission from emps where commission is > null) as a\n" > + " left join\n" > + " (select empid, salary, commission from emps where commission is > null) as b\n" > + " on a.salary = b.salary and a.commission = b.commission\n" > + " where b.empid is not null"; > CalciteAssert.that() > .with(CalciteConnectionProperty.LEX, Lex.JAVA) > .with(CalciteConnectionProperty.FORCE_DECORRELATE, false) > .withSchema("s", new ReflectiveSchema(new HrSchemaBig())) > .query(sql) > .withHook(Hook.PLANNER, (Consumer) planner -> { > planner.removeRule(EnumerableRules.ENUMERABLE_JOIN_RULE); > planner.addRule(EnumerableRules.ENUMERABLE_MERGE_JOIN_RULE); > }) > .explainContains("EnumerableHashJoin") > .returnsCount(0) > ; > } {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5953) AggregateCaseToFilterRule may make inaccurate SUM transformation
[ https://issues.apache.org/jira/browse/CALCITE-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated CALCITE-5953: -- Description: consider: {{sum(case when x = 1 then 2 else 0 end) as b}} notice that this expression may only be null if there are no rows in the table {{AggregateCaseToFilterRule}} rewrites the above expression to {{sum(2) filter (where x=1)}} which broadens when it could be `null` to when there are no matches to the filter * *A* is *0* correctly in this case; but I think it will be still *0* in case there are 0 input rows * The result for *B* supposed to be *0* but since there are no matches by the filter it becomes *null* * *C* is not touched {code} # Convert CASE to FILTER without matches select sum(case when x = 1 then 1 else 0 end) as a, sum(case when x = 1 then 2 else 0 end) as b, sum(case when x = 1 then 3 else -1 end) as c from (values 0, null, 0, 2) as t(x); +---+---++ | A | B | C | +---+---++ | 0 | | -4 | # incorrect result for B | 0 | 0 | -4 | # correct +---+---++ (1 row) !ok EnumerableCalc(expr#0..2=[{inputs}], expr#3=[CAST($t0):INTEGER], A=[$t3], B=[$t1], C=[$t2]) EnumerableAggregate(group=[{}], A=[COUNT() FILTER $1], B=[SUM($2) FILTER $3], C=[SUM($0)]) EnumerableCalc(expr#0=[{inputs}], expr#1=[1], expr#2=[=($t0, $t1)], expr#3=[3], expr#4=[-1], expr#5=[CASE($t2, $t3, $t4)], expr#6=[IS TRUE($t2)], expr#7=[2], $f2=[$t5], $f3=[$t6], $f4=[$t7], $f5=[$t6]) EnumerableValues(tuples=[[{ 0 }, { null }, { 0 }, { 2 }]]) !plan # Convert CASE to FILTER with no input rows select sum(case when x = 1 then 1 else 0 end) as a, sum(case when x = 1 then 2 else 0 end) as b, sum(case when x = 1 then 3 else -1 end) as c from (values 0, null, 0, 2) as t(x) where x*x=1; +---+---+---+ | A | B | C | +---+---+---+ | 0 | | | # incorrect results - for 0 input rows sum supposed to be null | | | | # correct result +---+---+---+ (1 row) !ok {code} was: consider: {{sum(case when x = 1 then 2 else 0 end) as b}} notice that this expression may only be null if there are no rows in the table {{AggregateCaseToFilterRule}} rewrites the above expression to {{sum(2) filter (where x=1)}} which broadens when it could be `null` to when there are no matches to the filter * *A* is *0* correctly in this case; but I think it will be still *0* in case there are 0 input rows * The result for *B* supposed to be *0* but since there are no matches by the filter it becomes *null* * *C* is not touched {code} # Convert CASE to FILTER without matches select sum(case when x = 1 then 1 else 0 end) as a, sum(case when x = 1 then 2 else 0 end) as b, sum(case when x = 1 then 3 else -1 end) as c from (values 0, null, 0, 2) as t(x); +---+---++ | A | B | C | +---+---++ | 0 | 0 | -4 | +---+---++ (1 row) !ok EnumerableCalc(expr#0..2=[{inputs}], expr#3=[CAST($t0):INTEGER], A=[$t3], B=[$t1], C=[$t2]) EnumerableAggregate(group=[{}], A=[COUNT() FILTER $1], B=[SUM($2) FILTER $3], C=[SUM($0)]) EnumerableCalc(expr#0=[{inputs}], expr#1=[1], expr#2=[=($t0, $t1)], expr#3=[3], expr#4=[-1], expr#5=[CASE($t2, $t3, $t4)], expr#6=[IS TRUE($t2)], expr#7=[2], $f2=[$t5], $f3=[$t6], $f4=[$t7], $f5=[$t6]) EnumerableValues(tuples=[[{ 0 }, { null }, { 0 }, { 2 }]]) !plan {code} > AggregateCaseToFilterRule may make inaccurate SUM transformation > > > Key: CALCITE-5953 > URL: https://issues.apache.org/jira/browse/CALCITE-5953 > Project: Calcite > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Fix For: 1.36.0 > > > consider: {{sum(case when x = 1 then 2 else 0 end) as b}} > notice that this expression may only be null if there are no rows in the table > {{AggregateCaseToFilterRule}} rewrites the above expression to {{sum(2) > filter (where x=1)}} which broadens when it could be `null` to when there are > no matches to the filter > * *A* is *0* correctly in this case; but I think it will be still *0* in case > there are 0 input rows > * The result for *B* supposed to be *0* but since there are no matches by the > filter it becomes *null* > * *C* is not touched > {code} > # Convert CASE to FILTER without matches > select sum(case when x = 1 then 1 else 0 end) as a, > sum(case when x = 1 then 2 else 0 end) as b, > sum(case when x = 1 then 3 else -1 end) as c > from (values 0, null, 0, 2) as t(x); > +---+---++ > | A | B | C | > +---+---++ > | 0 | | -4 | # incorrect result for B > | 0 | 0 | -4 | # correct > +---+---++ > (1 row) > !ok > EnumerableCalc(expr#0..2=[{inputs}], expr#3=[CAST($t0):INTEGER], A=[$t3], > B=[$t1], C=[$t2]) > EnumerableAggregate
[jira] [Updated] (CALCITE-5953) AggregateCaseToFilterRule may make inaccurate SUM transformation
[ https://issues.apache.org/jira/browse/CALCITE-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated CALCITE-5953: -- Description: consider: {{sum(case when x = 1 then 2 else 0 end) as b}} notice that this expression may only be null if there are no rows in the table {{AggregateCaseToFilterRule}} rewrites the above expression to {{sum(2) filter (where x=1)}} which broadens when it could be `null` to when there are no matches to the filter * *A* is *0* correctly in this case; but I think it will be still *0* in case there are 0 input rows * The result for *B* supposed to be *0* but since there are no matches by the filter it becomes *null* * *C* is not touched {code} # Convert CASE to FILTER without matches select sum(case when x = 1 then 1 else 0 end) as a, sum(case when x = 1 then 2 else 0 end) as b, sum(case when x = 1 then 3 else -1 end) as c from (values 0, null, 0, 2) as t(x); +---+---++ | A | B | C | +---+---++ | 0 | 0 | -4 | +---+---++ (1 row) !ok EnumerableCalc(expr#0..2=[{inputs}], expr#3=[CAST($t0):INTEGER], A=[$t3], B=[$t1], C=[$t2]) EnumerableAggregate(group=[{}], A=[COUNT() FILTER $1], B=[SUM($2) FILTER $3], C=[SUM($0)]) EnumerableCalc(expr#0=[{inputs}], expr#1=[1], expr#2=[=($t0, $t1)], expr#3=[3], expr#4=[-1], expr#5=[CASE($t2, $t3, $t4)], expr#6=[IS TRUE($t2)], expr#7=[2], $f2=[$t5], $f3=[$t6], $f4=[$t7], $f5=[$t6]) EnumerableValues(tuples=[[{ 0 }, { null }, { 0 }, { 2 }]]) !plan {code} was: consider: {{sum(case when x = 1 then 2 else 0 end) as b}} notice that this expression may only be null if there are no rows in the table {{AggregateCaseToFilterRule}} rewrites the above expression to {{sum(1) filter (where x=2)}} which broadens when it could be `null` to when there are no matches to the filter * *A* is *0* correctly in this case; but I think it will be still *0* in case there are 0 input rows * The result for *B* supposed to be *0* but since there are no matches by the filter it becomes *null* * *C* is not touched {code} # Convert CASE to FILTER without matches select sum(case when x = 1 then 1 else 0 end) as a, sum(case when x = 1 then 2 else 0 end) as b, sum(case when x = 1 then 3 else -1 end) as c from (values 0, null, 0, 2) as t(x); +---+---++ | A | B | C | +---+---++ | 0 | 0 | -4 | +---+---++ (1 row) !ok EnumerableCalc(expr#0..2=[{inputs}], expr#3=[CAST($t0):INTEGER], A=[$t3], B=[$t1], C=[$t2]) EnumerableAggregate(group=[{}], A=[COUNT() FILTER $1], B=[SUM($2) FILTER $3], C=[SUM($0)]) EnumerableCalc(expr#0=[{inputs}], expr#1=[1], expr#2=[=($t0, $t1)], expr#3=[3], expr#4=[-1], expr#5=[CASE($t2, $t3, $t4)], expr#6=[IS TRUE($t2)], expr#7=[2], $f2=[$t5], $f3=[$t6], $f4=[$t7], $f5=[$t6]) EnumerableValues(tuples=[[{ 0 }, { null }, { 0 }, { 2 }]]) !plan {code} > AggregateCaseToFilterRule may make inaccurate SUM transformation > > > Key: CALCITE-5953 > URL: https://issues.apache.org/jira/browse/CALCITE-5953 > Project: Calcite > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Fix For: 1.36.0 > > > consider: {{sum(case when x = 1 then 2 else 0 end) as b}} > notice that this expression may only be null if there are no rows in the table > {{AggregateCaseToFilterRule}} rewrites the above expression to {{sum(2) > filter (where x=1)}} which broadens when it could be `null` to when there are > no matches to the filter > * *A* is *0* correctly in this case; but I think it will be still *0* in case > there are 0 input rows > * The result for *B* supposed to be *0* but since there are no matches by the > filter it becomes *null* > * *C* is not touched > {code} > # Convert CASE to FILTER without matches > select sum(case when x = 1 then 1 else 0 end) as a, > sum(case when x = 1 then 2 else 0 end) as b, > sum(case when x = 1 then 3 else -1 end) as c > from (values 0, null, 0, 2) as t(x); > +---+---++ > | A | B | C | > +---+---++ > | 0 | 0 | -4 | > +---+---++ > (1 row) > !ok > EnumerableCalc(expr#0..2=[{inputs}], expr#3=[CAST($t0):INTEGER], A=[$t3], > B=[$t1], C=[$t2]) > EnumerableAggregate(group=[{}], A=[COUNT() FILTER $1], B=[SUM($2) FILTER > $3], C=[SUM($0)]) > EnumerableCalc(expr#0=[{inputs}], expr#1=[1], expr#2=[=($t0, $t1)], > expr#3=[3], expr#4=[-1], expr#5=[CASE($t2, $t3, $t4)], expr#6=[IS TRUE($t2)], > expr#7=[2], $f2=[$t5], $f3=[$t6], $f4=[$t7], $f5=[$t6]) > EnumerableValues(tuples=[[{ 0 }, { null }, { 0 }, { 2 }]]) > !plan > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[ https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wegdan Ghazi updated CALCITE-5984: -- Description: In Prepare#prepareSql there are 2 instances of trimming unused fields 1. [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297] {code:java} // Trim unused fields. root = trimUnusedFields(root); {code} 2. {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color} called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312] within {code:java} root = optimize(root, getMaterializations(), getLattices()); {code} I want to turn of the field trimming due to a personalisation I'm implementing to scan a specific API, which I would think is possible using [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26] I attempted to disable the trimming using hooks [PROGRAM, SQL2REL_CONVERTER_CONFIG_BUILDER], which succeeded to remove the trimming program, yet on [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] {code:java} // code placehprotected RelRoot trimUnusedFields(RelRoot root) { final SqlToRelConverter.Config config = SqlToRelConverter.config() .withTrimUnusedFields(shouldTrim(root.rel)) .withExpand(THREAD_EXPAND.get()) .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get())); final SqlToRelConverter converter = getSqlToRelConverter(getSqlValidator(), catalogReader, config); final boolean ordered = !root.collation.getFieldCollations().isEmpty(); final boolean dml = SqlKind.DML.contains(root.kind); return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel)); }older {code} The config passed (and edited by the hook) gets overwritten and only _shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}. was: In Prepare#prepareSql there are 2 instances of trimming unused fields 1. [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297] {code:java} // Trim unused fields. root = trimUnusedFields(root); {code} 2. {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color} called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312] within {code:java} root = optimize(root, getMaterializations(), getLattices()); {code} I want to turn of the field trimming due to a personalisation I'm implementing to scan a specific API, which I would think is possible using [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26] I attempted to disable the trimming using hooks [{_}PROGRAM, SQL2REL_CONVERTER_CONFIG_BUILDER{_}], which succeeded to remove the trimming program, yet on [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] {code:java} // code placehprotected RelRoot trimUnusedFields(RelRoot root) { final SqlToRelConverter.Config config = SqlToRelConverter.config() .withTrimUnusedFields(shouldTrim(root.rel)) .withExpand(THREAD_EXPAND.get()) .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get())); final SqlToRelConverter converter = getSqlToRelConverter(getSqlValidator(), catalogReader, config); final boolean ordered = !root.collation.getFieldCollations().isEmpty(); final boolean dml = SqlKind.DML.contains(root.kind); return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel)); }older {code} The config passed (and edited by the hook) gets overwritten and only _shouldTrim(root.rel)_ can enable/disable t
[jira] [Updated] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[ https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wegdan Ghazi updated CALCITE-5984: -- Description: In Prepare#prepareSql there are 2 instances of trimming unused fields 1. [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297] {code:java} // Trim unused fields. root = trimUnusedFields(root); {code} 2. {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color} called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312] within {code:java} root = optimize(root, getMaterializations(), getLattices()); {code} I want to turn of the field trimming due to a personalisation I'm implementing to scan a specific API, which I would think is possible using [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26] I attempted to disable the trimming using hooks [{_}PROGRAM, SQL2REL_CONVERTER_CONFIG_BUILDER{_}], which succeeded to remove the trimming program, yet on [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] {code:java} // code placehprotected RelRoot trimUnusedFields(RelRoot root) { final SqlToRelConverter.Config config = SqlToRelConverter.config() .withTrimUnusedFields(shouldTrim(root.rel)) .withExpand(THREAD_EXPAND.get()) .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get())); final SqlToRelConverter converter = getSqlToRelConverter(getSqlValidator(), catalogReader, config); final boolean ordered = !root.collation.getFieldCollations().isEmpty(); final boolean dml = SqlKind.DML.contains(root.kind); return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel)); }older {code} The config passed (and edited by the hook) gets overwritten and only _shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}. was: In Prepare#prepareSql there are 2 instances of trimming unused fields 1. [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297] {code:java} // Trim unused fields. root = trimUnusedFields(root); {code} 2. {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color} called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312] within {code:java} root = optimize(root, getMaterializations(), getLattices()); {code} I want to turn of the field trimming due to a personalisation I'm implementing to scan a specific API, which I would think is possible using [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26] I attempted to disable the trimming using hooks [\{_}PROGRAM, SQL2REL_CONVERTER_CONFIG_BUILDER\{_}], which succeeded to remove the trimming program, yet on [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] {code:java} // code placehprotected RelRoot trimUnusedFields(RelRoot root) { final SqlToRelConverter.Config config = SqlToRelConverter.config() .withTrimUnusedFields(shouldTrim(root.rel)) .withExpand(THREAD_EXPAND.get()) .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get())); final SqlToRelConverter converter = getSqlToRelConverter(getSqlValidator(), catalogReader, config); final boolean ordered = !root.collation.getFieldCollations().isEmpty(); final boolean dml = SqlKind.DML.contains(root.kind); return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel)); }older {code} The config passed (and edited by the hook) gets overwritten and only _shouldTrim(root.rel)_ can enable/d
[jira] [Updated] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[ https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wegdan Ghazi updated CALCITE-5984: -- Description: In Prepare#prepareSql there are 2 instances of trimming unused fields 1. [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297] {code:java} // Trim unused fields. root = trimUnusedFields(root); {code} 2. {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color} called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312] within {code:java} root = optimize(root, getMaterializations(), getLattices()); {code} I want to turn of the field trimming due to a personalisation I'm implementing to scan a specific API, which I would think is possible using [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26] I attempted to disable the trimming using hooks [\{_}PROGRAM, SQL2REL_CONVERTER_CONFIG_BUILDER\{_}], which succeeded to remove the trimming program, yet on [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] {code:java} // code placehprotected RelRoot trimUnusedFields(RelRoot root) { final SqlToRelConverter.Config config = SqlToRelConverter.config() .withTrimUnusedFields(shouldTrim(root.rel)) .withExpand(THREAD_EXPAND.get()) .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get())); final SqlToRelConverter converter = getSqlToRelConverter(getSqlValidator(), catalogReader, config); final boolean ordered = !root.collation.getFieldCollations().isEmpty(); final boolean dml = SqlKind.DML.contains(root.kind); return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel)); }older {code} The config passed (and edited by the hook) gets overwritten and only _shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}. Is this the intended behaviour? Any help is appreciated, thanks. was: In Prepare#prepareSql there are 2 instances of trimming unused fields 1. [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297] {code:java} // Trim unused fields. root = trimUnusedFields(root); {code} 2. {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color} called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312] within {code:java} root = optimize(root, getMaterializations(), getLattices()); {code} I want to turn of the field trimming due to a personalisation I'm implementing to scan a specific API, which I would think is possible using [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26] I attempted to disable the trimming using hooks [{_}PROGRAM, SQL2REL_CONVERTER_CONFIG_BUILDER{_}], which succeeded to remove the trimming program, yet on [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] {code:java} // code placehprotected RelRoot trimUnusedFields(RelRoot root) { final SqlToRelConverter.Config config = SqlToRelConverter.config() .withTrimUnusedFields(shouldTrim(root.rel)) .withExpand(THREAD_EXPAND.get()) .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get())); final SqlToRelConverter converter = getSqlToRelConverter(getSqlValidator(), catalogReader, config); final boolean ordered = !root.collation.getFieldCollations().isEmpty(); final boolean dml = SqlKind.DML.contains(root.kind); return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel)); }older {code} The config passed (and edited by th
[jira] [Created] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
Wegdan Ghazi created CALCITE-5984: - Summary: Cannot fully disable trimming unused fields using Calcite configs Key: CALCITE-5984 URL: https://issues.apache.org/jira/browse/CALCITE-5984 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.34.0 Environment: Spring boot application integrating _org.apache.calcite_ Reporter: Wegdan Ghazi Fix For: 1.34.0 In Prepare#prepareSql there are 2 instances of trimming unused fields 1. [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297] {code:java} // Trim unused fields. root = trimUnusedFields(root); {code} 2. {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color} called [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312] within {code:java} root = optimize(root, getMaterializations(), getLattices()); {code} I want to turn of the field trimming due to a personalisation I'm implementing to scan a specific API, which I would think is possible using [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26] I attempted to disable the trimming using hooks [{_}PROGRAM, SQL2REL_CONVERTER_CONFIG_BUILDER{_}], which succeeded to remove the trimming program, yet on [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] {code:java} // code placehprotected RelRoot trimUnusedFields(RelRoot root) { final SqlToRelConverter.Config config = SqlToRelConverter.config() .withTrimUnusedFields(shouldTrim(root.rel)) .withExpand(THREAD_EXPAND.get()) .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get())); final SqlToRelConverter converter = getSqlToRelConverter(getSqlValidator(), catalogReader, config); final boolean ordered = !root.collation.getFieldCollations().isEmpty(); final boolean dml = SqlKind.DML.contains(root.kind); return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel)); }older {code} The config passed (and edited by the hook) gets overwritten and only _shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5984) Cannot fully disable trimming unused fields using Calcite configs
[ https://issues.apache.org/jira/browse/CALCITE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wegdan Ghazi updated CALCITE-5984: -- Environment: (was: Spring boot application integrating _org.apache.calcite_) > Cannot fully disable trimming unused fields using Calcite configs > - > > Key: CALCITE-5984 > URL: https://issues.apache.org/jira/browse/CALCITE-5984 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.34.0 >Reporter: Wegdan Ghazi >Priority: Major > Fix For: 1.34.0 > > > In Prepare#prepareSql there are 2 instances of trimming unused fields > 1. > [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] > called > [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L297] > > {code:java} > // Trim unused fields. > root = trimUnusedFields(root); {code} > 2. > {color:#172b4d}[TrimFieldsProgram|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/tools/Programs.java#L369]{color} > called > [here|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L312] > within > {code:java} > root = optimize(root, getMaterializations(), getLattices()); > {code} > I want to turn of the field trimming due to a personalisation I'm > implementing to scan a specific API, which I would think is possible using > [SqlConverter.Config|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L6085C20-L6085C26] > I attempted to disable the trimming using hooks [{_}PROGRAM, > SQL2REL_CONVERTER_CONFIG_BUILDER{_}], which succeeded to remove the trimming > program, yet on > [Prepare#trimUnusedFields|https://github.com/apache/calcite/blob/64268b9dd70bcdc15a3421ab120b8e5ecba17339/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L375] > {code:java} > // code placehprotected RelRoot trimUnusedFields(RelRoot root) { > final SqlToRelConverter.Config config = SqlToRelConverter.config() > .withTrimUnusedFields(shouldTrim(root.rel)) > .withExpand(THREAD_EXPAND.get()) > > .withInSubQueryThreshold(castNonNull(THREAD_INSUBQUERY_THRESHOLD.get())); > final SqlToRelConverter converter = > getSqlToRelConverter(getSqlValidator(), catalogReader, config); > final boolean ordered = !root.collation.getFieldCollations().isEmpty(); > final boolean dml = SqlKind.DML.contains(root.kind); > return root.withRel(converter.trimUnusedFields(dml || ordered, root.rel)); > }older > {code} > The config passed (and edited by the hook) gets overwritten and only > _shouldTrim(root.rel)_ can enable/disable trimming, I would expect this to > happen iff the config _withTrimUnusedFields_ is only set to {_}true{_}. > -- This message was sent by Atlassian Jira (v8.20.10#820010)