[jira] [Commented] (CALCITE-1581) UDTF like in hive
[
https://issues.apache.org/jira/browse/CALCITE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917387#comment-16917387
]
pengzhiwei commented on CALCITE-1581:
-
Thank [~julianhyde] and [~zabetak], I have fix the choice conflict.Please have
a review again. And I agree with you on removing minor comments.
> UDTF like in hive
> -
>
> Key: CALCITE-1581
> URL: https://issues.apache.org/jira/browse/CALCITE-1581
> Project: Calcite
> Issue Type: New Feature
>Reporter: Xiaoyong Deng
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available, udtf
> Fix For: 1.21.0
>
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> Support one row in and multi-column/multi-row out(one-to-many mapping), just
> like udtf in hive.
> The query would like this:
> {code}
> select
> func(c0, c1) as (f0, f1, f2)
> from table_name;
> {code}
> c0 and c1 are 'table_name' columns. f0, f1 and f2 are new generated columns.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (CALCITE-3244) RelDecorrelator unable to decorrelate expression with filter and aggregate on top
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917367#comment-16917367
]
Danny Chan commented on CALCITE-3244:
-
I haven't digged into the failing test case in misc.iq#L709, [~vvysotskyi] can
you help to take a look ?
> RelDecorrelator unable to decorrelate expression with filter and aggregate on
> top
> -
>
> Key: CALCITE-3244
> URL: https://issues.apache.org/jira/browse/CALCITE-3244
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: benj
>Assignee: Danny Chan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Some very useful type of requests currently failed with:
> {code:java}
> SYSTEM ERROR: UnsupportedOperationException: Adding Implicit RowID column is
> not supported for ValuesPrel operator
> {code}
> Examples from DRILL-7050:
> {code:sql}
> select t1.id,
> (select count(t2.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t2 where t2.id = t1.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t1
> {code}
> {code:sql}
> SELECT t,
> (SELECT count(*) FROM
> (SELECT split(r,' ') AS r FROM
> (SELECT sub.t AS r)) AS x
> ,LATERAL(SELECT $unnest AS u FROM unnest(x.r))
> /* WHERE ... */) t2
> FROM
> (SELECT 'unnest is useful' AS t) sub
> {code}
>
> _Please note that in 1.18 the error for these requests was:_
> {code:java}
> Error: PLAN ERROR: Cannot convert RexNode to equivalent Drill expression.
> RexNode Class: org.apache.calcite.rex.RexCorrelVariable, RexNode Digest: $cor0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (CALCITE-2302) Implicit type cast support
[
https://issues.apache.org/jira/browse/CALCITE-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908903#comment-16908903
]
Danny Chan edited comment on CALCITE-2302 at 8/28/19 2:58 AM:
--
Thanks so much for your review [~hyuan] !
I have add a new SqlConformance for PostgreSql and SqlServer which seem the
only engine that returns integer for 2 integers division. I checked again for
the behavior:
* Hive and Spark all return double, Hive also have a "A DIV B" function to
return integer
* PostgreSql and SQL_SERVER returns integer.
* Mysql returns double while it has a DIV() function to return integer
* Oracle returns double, but have no function to return integer
was (Author: danny0405):
Thanks so much for your review [~hyuan] !
I have add a new SqlConformance for PostgreSql and SqlServer which seem the
only engine that returns integer for 2 integers division. I checked again for
the behavior:
* Hive and Spark all return double, Hive also have a "A DIV B" function to
return integer
* PostgreSql and SQL_SERVER returns integer.
* Mysql returns double while it has a DIV() function to return integer
* Oracle returns double, it also has a DIV() function to return integer
> Implicit type cast support
> --
>
> Key: CALCITE-2302
> URL: https://issues.apache.org/jira/browse/CALCITE-2302
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.17.0
>Reporter: Danny Chan
>Assignee: Danny Chan
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> Now many DBs have support implicit type cast, eg: SqlServer, Oracle, Hive.
> Implicit type cast is an useful function for many cases, So we should support
> this.
> I checkout Calcite code and found that:
> # Now we use a validator to validate our operands types[ through kinds of
> namespaces and scopes ]
> # Most of the validations will finally goes to
> {code:java}
> SqlOperator.validateOperands
> {code}
> # which will use validation logic defined in corresponding
> SqlOperandTypeChecker
> What i'm confused about is where should i put the implicit type cast logic
> in? I figured out 2 ways:
> # Supply a tool class/rules to add casts into a parsed SqlNode tree which
> will then go through the validation logic later on.
> # Unleash the validation logic in kinds of SqlOperandTypeChecker, then
> modify the RelNode/RexNodes tree converted from a validated SqlNode tree to
> add in casts through custom RelOptRules.
> So guys, which of the 2 ways should i go, or if there are better way to do
> this?
> I need your help.
>
> Updated 18-05-30:
> Hi guys, i have made a PR in
> [CALCITE-2302|https://github.com/apache/calcite/pull/706]
> This is design doc: [Calcite Implicit Type Cast
> Design|https://docs.google.com/document/d/1g2RUnLXyp_LjUlO-wbblKuP5hqEu3a_2Mt2k4dh6RwU/edit?usp=sharing].
> This is the conversion types mapping: [Conversion Types
> Mapping|https://docs.google.com/spreadsheets/d/1GhleX5h5W8-kJKh7NMJ4vtoE78pwfaZRJl88ULX_MgU/edit?usp=sharing].
> I really appreciate your suggestions, thx.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (CALCITE-3244) RelDecorrelator unable to decorrelate expression with filter and aggregate on top
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917355#comment-16917355
]
Danny Chan commented on CALCITE-3244:
-
Thanks [~vvysotskyi], i have fixed the test case you pasted.
> RelDecorrelator unable to decorrelate expression with filter and aggregate on
> top
> -
>
> Key: CALCITE-3244
> URL: https://issues.apache.org/jira/browse/CALCITE-3244
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: benj
>Assignee: Danny Chan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Some very useful type of requests currently failed with:
> {code:java}
> SYSTEM ERROR: UnsupportedOperationException: Adding Implicit RowID column is
> not supported for ValuesPrel operator
> {code}
> Examples from DRILL-7050:
> {code:sql}
> select t1.id,
> (select count(t2.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t2 where t2.id = t1.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t1
> {code}
> {code:sql}
> SELECT t,
> (SELECT count(*) FROM
> (SELECT split(r,' ') AS r FROM
> (SELECT sub.t AS r)) AS x
> ,LATERAL(SELECT $unnest AS u FROM unnest(x.r))
> /* WHERE ... */) t2
> FROM
> (SELECT 'unnest is useful' AS t) sub
> {code}
>
> _Please note that in 1.18 the error for these requests was:_
> {code:java}
> Error: PLAN ERROR: Cannot convert RexNode to equivalent Drill expression.
> RexNode Class: org.apache.calcite.rex.RexCorrelVariable, RexNode Digest: $cor0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Issue Comment Deleted] (CALCITE-2302) Implicit type cast support
[
https://issues.apache.org/jira/browse/CALCITE-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chan updated CALCITE-2302:
Comment: was deleted
(was: [~julianhyde] Does the latest commit solve your review objections ? I'm
planning to merge this PR if there are no more comments in 24 hours.)
> Implicit type cast support
> --
>
> Key: CALCITE-2302
> URL: https://issues.apache.org/jira/browse/CALCITE-2302
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.17.0
>Reporter: Danny Chan
>Assignee: Danny Chan
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> Now many DBs have support implicit type cast, eg: SqlServer, Oracle, Hive.
> Implicit type cast is an useful function for many cases, So we should support
> this.
> I checkout Calcite code and found that:
> # Now we use a validator to validate our operands types[ through kinds of
> namespaces and scopes ]
> # Most of the validations will finally goes to
> {code:java}
> SqlOperator.validateOperands
> {code}
> # which will use validation logic defined in corresponding
> SqlOperandTypeChecker
> What i'm confused about is where should i put the implicit type cast logic
> in? I figured out 2 ways:
> # Supply a tool class/rules to add casts into a parsed SqlNode tree which
> will then go through the validation logic later on.
> # Unleash the validation logic in kinds of SqlOperandTypeChecker, then
> modify the RelNode/RexNodes tree converted from a validated SqlNode tree to
> add in casts through custom RelOptRules.
> So guys, which of the 2 ways should i go, or if there are better way to do
> this?
> I need your help.
>
> Updated 18-05-30:
> Hi guys, i have made a PR in
> [CALCITE-2302|https://github.com/apache/calcite/pull/706]
> This is design doc: [Calcite Implicit Type Cast
> Design|https://docs.google.com/document/d/1g2RUnLXyp_LjUlO-wbblKuP5hqEu3a_2Mt2k4dh6RwU/edit?usp=sharing].
> This is the conversion types mapping: [Conversion Types
> Mapping|https://docs.google.com/spreadsheets/d/1GhleX5h5W8-kJKh7NMJ4vtoE78pwfaZRJl88ULX_MgU/edit?usp=sharing].
> I really appreciate your suggestions, thx.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (CALCITE-2302) Implicit type cast support
[
https://issues.apache.org/jira/browse/CALCITE-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914070#comment-16914070
]
Danny Chan edited comment on CALCITE-2302 at 8/28/19 1:32 AM:
--
I have made the latest change in commit
[94134d5|https://github.com/apache/calcite/pull/706/commits/94134d5dd6a9c38cc22ff74bdd05a31d76e41b29]
!
Instead of leveraging the existing RelDataTypeSystem.deriveDecimalDivideType
method, i add a new SqlReturnTypeInference named NUMERIC_QUOTIENT chained into
the exists return type inference of "DIVIDE" operator.
Also i modified the expression to make them returns double(2 QuidemTest changes
because of this change, their results become more accurate).
was (Author: danny0405):
I have made the latest change in commit
[7b03ac7|https://github.com/apache/calcite/pull/706/commits/7b03ac7f466c248a0765447ce4a18c9d9393aa95]
!
Instead of leveraging the existing RelDataTypeSystem.deriveDecimalDivideType
method, i add a new SqlReturnTypeInference named NUMERIC_QUOTIENT chained into
the exists return type inference of "DIVIDE" operator.
Also i modified the expression to make them returns double(2 QuidemTest changes
because of this change, their results become more accurate).
> Implicit type cast support
> --
>
> Key: CALCITE-2302
> URL: https://issues.apache.org/jira/browse/CALCITE-2302
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.17.0
>Reporter: Danny Chan
>Assignee: Danny Chan
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> Now many DBs have support implicit type cast, eg: SqlServer, Oracle, Hive.
> Implicit type cast is an useful function for many cases, So we should support
> this.
> I checkout Calcite code and found that:
> # Now we use a validator to validate our operands types[ through kinds of
> namespaces and scopes ]
> # Most of the validations will finally goes to
> {code:java}
> SqlOperator.validateOperands
> {code}
> # which will use validation logic defined in corresponding
> SqlOperandTypeChecker
> What i'm confused about is where should i put the implicit type cast logic
> in? I figured out 2 ways:
> # Supply a tool class/rules to add casts into a parsed SqlNode tree which
> will then go through the validation logic later on.
> # Unleash the validation logic in kinds of SqlOperandTypeChecker, then
> modify the RelNode/RexNodes tree converted from a validated SqlNode tree to
> add in casts through custom RelOptRules.
> So guys, which of the 2 ways should i go, or if there are better way to do
> this?
> I need your help.
>
> Updated 18-05-30:
> Hi guys, i have made a PR in
> [CALCITE-2302|https://github.com/apache/calcite/pull/706]
> This is design doc: [Calcite Implicit Type Cast
> Design|https://docs.google.com/document/d/1g2RUnLXyp_LjUlO-wbblKuP5hqEu3a_2Mt2k4dh6RwU/edit?usp=sharing].
> This is the conversion types mapping: [Conversion Types
> Mapping|https://docs.google.com/spreadsheets/d/1GhleX5h5W8-kJKh7NMJ4vtoE78pwfaZRJl88ULX_MgU/edit?usp=sharing].
> I really appreciate your suggestions, thx.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (CALCITE-3295) Aggregate call name lost in serialized json string.
[
https://issues.apache.org/jira/browse/CALCITE-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-3295.
Fix Version/s: 1.21.0
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/97d68952d467540cbe8350f706028001dfbf,
thanks for the PR, [~yanlin-Lynn]!
> Aggregate call name lost in serialized json string.
> ---
>
> Key: CALCITE-3295
> URL: https://issues.apache.org/jira/browse/CALCITE-3295
> Project: Calcite
> Issue Type: Bug
>Reporter: Wang Yanlin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When serialize a relnode to json string, the name of aggregate call was lost.
> For a sql string like this
> {code:java}
> SELECT max(SAL) as max_sal from EMP group by JOB;
> {code}
> The serialized json string is like this
> {code:java}
> {
> "rels": [
> {
> "id": "0",
> "relOp": "LogicalTableScan",
> "table": [
> "scott",
> "EMP"
> ],
> "inputs": []
> },
> {
> "id": "1",
> "relOp": "LogicalProject",
> "fields": [
> "JOB",
> "SAL"
> ],
> "exprs": [
> {
> "input": 2,
> "name": "$2"
> },
> {
> "input": 5,
> "name": "$5"
> }
> ]
> },
> {
> "id": "2",
> "relOp": "LogicalAggregate",
> "group": [
> 0
> ],
> "aggs": [
> {
> "agg": {
> "name": "MAX",
> "kind": "MAX",
> "syntax": "FUNCTION"
> },
> "type": {
> "type": "DECIMAL",
> "nullable": true,
> "precision": 7,
> "scale": 2
> },
> "distinct": false,
> "operands": [
> 1
> ],
> }
> ]
> },
> {
> "id": "3",
> "relOp": "LogicalProject",
> "fields": [
> "max_sal"
> ],
> "exprs": [
> {
> "input": 1,
> "name": "$1"
> }
> ]
> }
> ]
> }
> {code}
> Lost the name of max call.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (CALCITE-3296) Decorrelator gives empty result after decorrelating sort rel with null offset and fetch
[ https://issues.apache.org/jira/browse/CALCITE-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3296. Fix Version/s: 1.21.0 Resolution: Fixed Fixed in https://github.com/apache/calcite/commit/70f1453086d67bd6d431d63b93a27baf0cdba06d, thanks for the PR, [~Juhwan]! > Decorrelator gives empty result after decorrelating sort rel with null offset > and fetch > --- > > Key: CALCITE-3296 > URL: https://issues.apache.org/jira/browse/CALCITE-3296 > Project: Calcite > Issue Type: Bug >Reporter: Juhwan Kim >Assignee: Juhwan Kim >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > It seems like I introduced a bug with the recent decorrelator patch. Sort rel > with null offset and fetch would end up becoming an empty value since > decorrelator would call sortLimit builder with 0 limit and 0 offset. It > should pass negative values to builder when values are null. > https://github.com/apache/calcite/blob/e863294ccfbed9dd520c999f75ed0bbe03f9fb1d/core/src/main/java/org/apache/calcite/sql2rel/RelDecorrelator.java#L423 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-3122) Convert Pig Latin scripts into Calcite logical plans
[ https://issues.apache.org/jira/browse/CALCITE-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916979#comment-16916979 ] Stamatis Zampetakis commented on CALCITE-3122: -- [~khaitran] Yes I noticed that it is related with the environment but I don't have a Window machine either to test it. Normally, AppVeyor runs also for the PR if it didn't fail after the submit then I guess it is an intermittent failure. > Convert Pig Latin scripts into Calcite logical plans > - > > Key: CALCITE-3122 > URL: https://issues.apache.org/jira/browse/CALCITE-3122 > Project: Calcite > Issue Type: New Feature > Components: core, piglet >Reporter: Khai Tran >Assignee: Julian Hyde >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > We create an internal Calcite repo at LinkedIn and develop APIs to parse any > Pig Latin scripts into Calcite logical plan. The code was tested in nearly > ~1000 Pig scripts written at LinkedIn. > Changes: > 1. piglet: main conversion code live there, include: > * APIs to convert any Pig scripts into RelNode plans or SQL statements > * Use Pig Grunt parser to parse Pig Latin scripts into Pig logical plan > (DAGs) > * Convert Pig schemas into RelDatatype > * Traverse through Pig expression plan and convert Pig expressions into > RexNodes > * Map some basic Pig UDFs to Calcite SQL operators > * Build Calcite UDFs for any other Pig UDFs, including UDFs written in both > Java and Python > * Traverse (DFS) through Pig logical plans to convert each Pig logical nodes > to RelNodes > * Have an optimizer rule to optimize Pig group/cogroup into Aggregate > operators > 2. core: > * Implement other RelNode in Rel2Sql so that Pig can be translated into SQL > * Other minor changes in a few other classes to make Pig to Calcite works -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-1581) UDTF like in hive
[
https://issues.apache.org/jira/browse/CALCITE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916892#comment-16916892
]
Julian Hyde commented on CALCITE-1581:
--
I looked into the causes of this choice conflict. With this extension, the
parser allows {code}select myFunc (x, y) from t{code}
That's a call to a function with arguments x and y, right? No, that's a call to
a nilary function, assigning names to the result columns. Identical effect to
this:
{code}select myFunc() as (x, y) from t{code}
Can we change this PR so that {{AS}} is mandatory if followed by "(col, ...)"?
Is it mandatory in Hive.
By the way, I agree with [~zabetak] on removing minor comments like "fix
format" when you squash. It's not a complete log of everything you did. It is
whatever will be most helpful to the person reading the commit in a week or a
year.
> UDTF like in hive
> -
>
> Key: CALCITE-1581
> URL: https://issues.apache.org/jira/browse/CALCITE-1581
> Project: Calcite
> Issue Type: New Feature
>Reporter: Xiaoyong Deng
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available, udtf
> Fix For: 1.21.0
>
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> Support one row in and multi-column/multi-row out(one-to-many mapping), just
> like udtf in hive.
> The query would like this:
> {code}
> select
> func(c0, c1) as (f0, f1, f2)
> from table_name;
> {code}
> c0 and c1 are 'table_name' columns. f0, f1 and f2 are new generated columns.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (CALCITE-3122) Convert Pig Latin scripts into Calcite logical plans
[ https://issues.apache.org/jira/browse/CALCITE-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916782#comment-16916782 ] Khai Tran commented on CALCITE-3122: [~julianhyde] Do you know who can help on this? This seems Hadoop authentication issue on Window as we need to create a local Pig Server. > Convert Pig Latin scripts into Calcite logical plans > - > > Key: CALCITE-3122 > URL: https://issues.apache.org/jira/browse/CALCITE-3122 > Project: Calcite > Issue Type: New Feature > Components: core, piglet >Reporter: Khai Tran >Assignee: Julian Hyde >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > We create an internal Calcite repo at LinkedIn and develop APIs to parse any > Pig Latin scripts into Calcite logical plan. The code was tested in nearly > ~1000 Pig scripts written at LinkedIn. > Changes: > 1. piglet: main conversion code live there, include: > * APIs to convert any Pig scripts into RelNode plans or SQL statements > * Use Pig Grunt parser to parse Pig Latin scripts into Pig logical plan > (DAGs) > * Convert Pig schemas into RelDatatype > * Traverse through Pig expression plan and convert Pig expressions into > RexNodes > * Map some basic Pig UDFs to Calcite SQL operators > * Build Calcite UDFs for any other Pig UDFs, including UDFs written in both > Java and Python > * Traverse (DFS) through Pig logical plans to convert each Pig logical nodes > to RelNodes > * Have an optimizer rule to optimize Pig group/cogroup into Aggregate > operators > 2. core: > * Implement other RelNode in Rel2Sql so that Pig can be translated into SQL > * Other minor changes in a few other classes to make Pig to Calcite works -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-3122) Convert Pig Latin scripts into Calcite logical plans
[ https://issues.apache.org/jira/browse/CALCITE-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916775#comment-16916775 ] Khai Tran commented on CALCITE-3122: [~zabetak]: It's environment issue, not the test itself. The test failed on Window machine and seems some settings for Hadoop was missing or not correctly configured. I don't have a Window machine and is not able to reproduce it locally. > Convert Pig Latin scripts into Calcite logical plans > - > > Key: CALCITE-3122 > URL: https://issues.apache.org/jira/browse/CALCITE-3122 > Project: Calcite > Issue Type: New Feature > Components: core, piglet >Reporter: Khai Tran >Assignee: Julian Hyde >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > We create an internal Calcite repo at LinkedIn and develop APIs to parse any > Pig Latin scripts into Calcite logical plan. The code was tested in nearly > ~1000 Pig scripts written at LinkedIn. > Changes: > 1. piglet: main conversion code live there, include: > * APIs to convert any Pig scripts into RelNode plans or SQL statements > * Use Pig Grunt parser to parse Pig Latin scripts into Pig logical plan > (DAGs) > * Convert Pig schemas into RelDatatype > * Traverse through Pig expression plan and convert Pig expressions into > RexNodes > * Map some basic Pig UDFs to Calcite SQL operators > * Build Calcite UDFs for any other Pig UDFs, including UDFs written in both > Java and Python > * Traverse (DFS) through Pig logical plans to convert each Pig logical nodes > to RelNodes > * Have an optimizer rule to optimize Pig group/cogroup into Aggregate > operators > 2. core: > * Implement other RelNode in Rel2Sql so that Pig can be translated into SQL > * Other minor changes in a few other classes to make Pig to Calcite works -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (CALCITE-3244) RelDecorrelator unable to decorrelate expression with filter and aggregate on top
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916763#comment-16916763
]
Volodymyr Vysotskyi edited comment on CALCITE-3244 at 8/27/19 2:45 PM:
---
I have checked your branch, and problem with types wasn't fixed fully, so for
example, the following test fails:
{code:java}
@Test public void testDecorrelateExists3() throws Exception {
final String sql = "select \n"
+ " exists (select min(t2.id) \n"
+ "from (values(1), (2)) t2(id) where t2.id = t1.id)\n"
+ "from (values(3), (4)) t1(id)";
checkSubQuery(sql).withLateDecorrelation(true).check();
}
{code}
{noformat}
java.lang.AssertionError: Cannot add expression of different type to set:
set type is RecordType(INTEGER NOT NULL ID, BOOLEAN i) NOT NULL
expression type is RecordType(INTEGER NOT NULL ID, BOOLEAN NOT NULL i) NOT NULL
set is
rel#44:LogicalCorrelate(left=HepRelVertex#36,right=HepRelVertex#43,correlation=$cor0,joinType=left,requiredColumns={0})
expression is LogicalProject(ID=[$0], i=[true])
LogicalCorrelate(correlation=[$cor0], joinType=[left], requiredColumns=[{0}])
LogicalValues(tuples=[[{ 3 }, { 4 }]])
LogicalAggregate(group=[{}], EXPR$0=[MIN($0)])
LogicalFilter(condition=[=($0, $cor0.ID)])
LogicalValues(tuples=[[{ 1 }, { 2 }]])
at
org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:383)
at
org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:57)
at
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
at
org.apache.calcite.sql2rel.RelDecorrelator$AdjustProjectForCountAggregateRule.onMatch2(RelDecorrelator.java:2528)
at
org.apache.calcite.sql2rel.RelDecorrelator$AdjustProjectForCountAggregateRule.onMatch(RelDecorrelator.java:2436)
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:319)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:560)
at
org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:419)
at
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:256)
at
org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:215)
at
org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:202)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelate(RelDecorrelator.java:250)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:215)
at
org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:213)
at
org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:338)
at
org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:321)
at
org.apache.calcite.test.RelOptRulesTest.testDecorrelateExists2(RelOptRulesTest.java:5626)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
[jira] [Commented] (CALCITE-3122) Convert Pig Latin scripts into Calcite logical plans
[ https://issues.apache.org/jira/browse/CALCITE-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916764#comment-16916764 ] Stamatis Zampetakis commented on CALCITE-3122: -- I just noticed that after this commit AppVeyor [fails|https://ci.appveyor.com/project/ApacheSoftwareFoundation/calcite/builds/26980850] due to PigRelExTest. Do you have an idea why [~khaitran]? Is it really related with this JIRA? > Convert Pig Latin scripts into Calcite logical plans > - > > Key: CALCITE-3122 > URL: https://issues.apache.org/jira/browse/CALCITE-3122 > Project: Calcite > Issue Type: New Feature > Components: core, piglet >Reporter: Khai Tran >Assignee: Julian Hyde >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > We create an internal Calcite repo at LinkedIn and develop APIs to parse any > Pig Latin scripts into Calcite logical plan. The code was tested in nearly > ~1000 Pig scripts written at LinkedIn. > Changes: > 1. piglet: main conversion code live there, include: > * APIs to convert any Pig scripts into RelNode plans or SQL statements > * Use Pig Grunt parser to parse Pig Latin scripts into Pig logical plan > (DAGs) > * Convert Pig schemas into RelDatatype > * Traverse through Pig expression plan and convert Pig expressions into > RexNodes > * Map some basic Pig UDFs to Calcite SQL operators > * Build Calcite UDFs for any other Pig UDFs, including UDFs written in both > Java and Python > * Traverse (DFS) through Pig logical plans to convert each Pig logical nodes > to RelNodes > * Have an optimizer rule to optimize Pig group/cogroup into Aggregate > operators > 2. core: > * Implement other RelNode in Rel2Sql so that Pig can be translated into SQL > * Other minor changes in a few other classes to make Pig to Calcite works -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-3244) RelDecorrelator unable to decorrelate expression with filter and aggregate on top
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916763#comment-16916763
]
Volodymyr Vysotskyi commented on CALCITE-3244:
--
I have checked your branch, and the following test fails:
{code:java}
@Test public void testDecorrelateExists3() throws Exception {
final String sql = "select \n"
+ " exists (select min(t2.id) \n"
+ "from (values(1), (2)) t2(id) where t2.id = t1.id)\n"
+ "from (values(3), (4)) t1(id)";
checkSubQuery(sql).withLateDecorrelation(true).check();
}
{code}
{noformat}
java.lang.AssertionError: Cannot add expression of different type to set:
set type is RecordType(INTEGER NOT NULL ID, BOOLEAN i) NOT NULL
expression type is RecordType(INTEGER NOT NULL ID, BOOLEAN NOT NULL i) NOT NULL
set is
rel#44:LogicalCorrelate(left=HepRelVertex#36,right=HepRelVertex#43,correlation=$cor0,joinType=left,requiredColumns={0})
expression is LogicalProject(ID=[$0], i=[true])
LogicalCorrelate(correlation=[$cor0], joinType=[left], requiredColumns=[{0}])
LogicalValues(tuples=[[{ 3 }, { 4 }]])
LogicalAggregate(group=[{}], EXPR$0=[MIN($0)])
LogicalFilter(condition=[=($0, $cor0.ID)])
LogicalValues(tuples=[[{ 1 }, { 2 }]])
at
org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:383)
at
org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:57)
at
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
at
org.apache.calcite.sql2rel.RelDecorrelator$AdjustProjectForCountAggregateRule.onMatch2(RelDecorrelator.java:2528)
at
org.apache.calcite.sql2rel.RelDecorrelator$AdjustProjectForCountAggregateRule.onMatch(RelDecorrelator.java:2436)
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:319)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:560)
at
org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:419)
at
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:256)
at
org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:215)
at
org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:202)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelate(RelDecorrelator.java:250)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:215)
at
org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:213)
at
org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:338)
at
org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:321)
at
org.apache.calcite.test.RelOptRulesTest.testDecorrelateExists2(RelOptRulesTest.java:5626)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{noformat}
> RelDecorrelator unable to decorrelate expression with filter and aggregate on
> top
>
[jira] [Updated] (CALCITE-3122) Convert Pig Latin scripts into Calcite logical plans
[ https://issues.apache.org/jira/browse/CALCITE-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khai Tran updated CALCITE-3122: --- Summary: Convert Pig Latin scripts into Calcite logical plans (was: Convert Pig Latin scripts into Calcite logical plan ) > Convert Pig Latin scripts into Calcite logical plans > - > > Key: CALCITE-3122 > URL: https://issues.apache.org/jira/browse/CALCITE-3122 > Project: Calcite > Issue Type: New Feature > Components: core, piglet >Reporter: Khai Tran >Assignee: Julian Hyde >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > We create an internal Calcite repo at LinkedIn and develop APIs to parse any > Pig Latin scripts into Calcite logical plan. The code was tested in nearly > ~1000 Pig scripts written at LinkedIn. > Changes: > 1. piglet: main conversion code live there, include: > * APIs to convert any Pig scripts into RelNode plans or SQL statements > * Use Pig Grunt parser to parse Pig Latin scripts into Pig logical plan > (DAGs) > * Convert Pig schemas into RelDatatype > * Traverse through Pig expression plan and convert Pig expressions into > RexNodes > * Map some basic Pig UDFs to Calcite SQL operators > * Build Calcite UDFs for any other Pig UDFs, including UDFs written in both > Java and Python > * Traverse (DFS) through Pig logical plans to convert each Pig logical nodes > to RelNodes > * Have an optimizer rule to optimize Pig group/cogroup into Aggregate > operators > 2. core: > * Implement other RelNode in Rel2Sql so that Pig can be translated into SQL > * Other minor changes in a few other classes to make Pig to Calcite works -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-3244) RelDecorrelator unable to decorrelate expression with filter and aggregate on top
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916743#comment-16916743
]
Volodymyr Vysotskyi commented on CALCITE-3244:
--
I have also worked on this issue, here is my branch with the changes
[https://github.com/vvysotskyi/calcite/tree/CALCITE-3244] (without modifying
some tests baselines and there are some similar things to things you did, but
there are some additional changes connected with attempts to fix another issue).
Regarding disabled unit test, fix for this issue will cause wrong results for
queries which currently work, though without decorrelation. Looks like
[~julianhyde] predicted this issue in his comment above and we should fix it
with current issue.
> RelDecorrelator unable to decorrelate expression with filter and aggregate on
> top
> -
>
> Key: CALCITE-3244
> URL: https://issues.apache.org/jira/browse/CALCITE-3244
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: benj
>Assignee: Danny Chan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Some very useful type of requests currently failed with:
> {code:java}
> SYSTEM ERROR: UnsupportedOperationException: Adding Implicit RowID column is
> not supported for ValuesPrel operator
> {code}
> Examples from DRILL-7050:
> {code:sql}
> select t1.id,
> (select count(t2.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t2 where t2.id = t1.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t1
> {code}
> {code:sql}
> SELECT t,
> (SELECT count(*) FROM
> (SELECT split(r,' ') AS r FROM
> (SELECT sub.t AS r)) AS x
> ,LATERAL(SELECT $unnest AS u FROM unnest(x.r))
> /* WHERE ... */) t2
> FROM
> (SELECT 'unnest is useful' AS t) sub
> {code}
>
> _Please note that in 1.18 the error for these requests was:_
> {code:java}
> Error: PLAN ERROR: Cannot convert RexNode to equivalent Drill expression.
> RexNode Class: org.apache.calcite.rex.RexCorrelVariable, RexNode Digest: $cor0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (CALCITE-3122) Convert Pig Latin scripts into Calcite logical plan
[ https://issues.apache.org/jira/browse/CALCITE-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916719#comment-16916719 ] Khai Tran commented on CALCITE-3122: Thanks so much for refactoring the tests, [~julianhyde]. They look much nicer now. > Convert Pig Latin scripts into Calcite logical plan > > > Key: CALCITE-3122 > URL: https://issues.apache.org/jira/browse/CALCITE-3122 > Project: Calcite > Issue Type: New Feature > Components: core, piglet >Reporter: Khai Tran >Assignee: Julian Hyde >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > We create an internal Calcite repo at LinkedIn and develop APIs to parse any > Pig Latin scripts into Calcite logical plan. The code was tested in nearly > ~1000 Pig scripts written at LinkedIn. > Changes: > 1. piglet: main conversion code live there, include: > * APIs to convert any Pig scripts into RelNode plans or SQL statements > * Use Pig Grunt parser to parse Pig Latin scripts into Pig logical plan > (DAGs) > * Convert Pig schemas into RelDatatype > * Traverse through Pig expression plan and convert Pig expressions into > RexNodes > * Map some basic Pig UDFs to Calcite SQL operators > * Build Calcite UDFs for any other Pig UDFs, including UDFs written in both > Java and Python > * Traverse (DFS) through Pig logical plans to convert each Pig logical nodes > to RelNodes > * Have an optimizer rule to optimize Pig group/cogroup into Aggregate > operators > 2. core: > * Implement other RelNode in Rel2Sql so that Pig can be translated into SQL > * Other minor changes in a few other classes to make Pig to Calcite works -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-3244) RelDecorrelator unable to decorrelate expression with filter and aggregate on top
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916709#comment-16916709
]
Danny Chan commented on CALCITE-3244:
-
[~vvysotskyi] I have applied a patch to fix the decorrelation.
But with the fix, there is a breaking test in misc.iq#L709, the original test
passes because just because it can not decorrelate values expression, if the
change the values to project, the test still fails, so i think this is another
fix and should fire the fix in another issue.
> RelDecorrelator unable to decorrelate expression with filter and aggregate on
> top
> -
>
> Key: CALCITE-3244
> URL: https://issues.apache.org/jira/browse/CALCITE-3244
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: benj
>Assignee: Danny Chan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Some very useful type of requests currently failed with:
> {code:java}
> SYSTEM ERROR: UnsupportedOperationException: Adding Implicit RowID column is
> not supported for ValuesPrel operator
> {code}
> Examples from DRILL-7050:
> {code:sql}
> select t1.id,
> (select count(t2.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t2 where t2.id = t1.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t1
> {code}
> {code:sql}
> SELECT t,
> (SELECT count(*) FROM
> (SELECT split(r,' ') AS r FROM
> (SELECT sub.t AS r)) AS x
> ,LATERAL(SELECT $unnest AS u FROM unnest(x.r))
> /* WHERE ... */) t2
> FROM
> (SELECT 'unnest is useful' AS t) sub
> {code}
>
> _Please note that in 1.18 the error for these requests was:_
> {code:java}
> Error: PLAN ERROR: Cannot convert RexNode to equivalent Drill expression.
> RexNode Class: org.apache.calcite.rex.RexCorrelVariable, RexNode Digest: $cor0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (CALCITE-3244) RelDecorrelator unable to decorrelate expression with filter and aggregate on top
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CALCITE-3244:
Labels: pull-request-available (was: )
> RelDecorrelator unable to decorrelate expression with filter and aggregate on
> top
> -
>
> Key: CALCITE-3244
> URL: https://issues.apache.org/jira/browse/CALCITE-3244
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: benj
>Assignee: Danny Chan
>Priority: Major
> Labels: pull-request-available
>
> Some very useful type of requests currently failed with:
> {code:java}
> SYSTEM ERROR: UnsupportedOperationException: Adding Implicit RowID column is
> not supported for ValuesPrel operator
> {code}
> Examples from DRILL-7050:
> {code:sql}
> select t1.id,
> (select count(t2.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t2 where t2.id = t1.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t1
> {code}
> {code:sql}
> SELECT t,
> (SELECT count(*) FROM
> (SELECT split(r,' ') AS r FROM
> (SELECT sub.t AS r)) AS x
> ,LATERAL(SELECT $unnest AS u FROM unnest(x.r))
> /* WHERE ... */) t2
> FROM
> (SELECT 'unnest is useful' AS t) sub
> {code}
>
> _Please note that in 1.18 the error for these requests was:_
> {code:java}
> Error: PLAN ERROR: Cannot convert RexNode to equivalent Drill expression.
> RexNode Class: org.apache.calcite.rex.RexCorrelVariable, RexNode Digest: $cor0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (CALCITE-3298) Add getSelectivity for TableScan in RelMdSelectivity
Hong Shen created CALCITE-3298: -- Summary: Add getSelectivity for TableScan in RelMdSelectivity Key: CALCITE-3298 URL: https://issues.apache.org/jira/browse/CALCITE-3298 Project: Calcite Issue Type: Improvement Components: core Affects Versions: 1.20.0 Reporter: Hong Shen Now there is no getSelectivity for TableScan in RelMdSelectivity, I can add this function if needed. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-2973) Allow theta joins that have equi conditions to be executed using a hash join algorithm
[ https://issues.apache.org/jira/browse/CALCITE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916597#comment-16916597 ] Stamatis Zampetakis commented on CALCITE-2973: -- I think one +1 is enough for this PR; go ahead and merge it [~rubenql]! > Allow theta joins that have equi conditions to be executed using a hash join > algorithm > -- > > Key: CALCITE-2973 > URL: https://issues.apache.org/jira/browse/CALCITE-2973 > Project: Calcite > Issue Type: New Feature > Components: core >Affects Versions: 1.19.0 >Reporter: Lai Zhou >Priority: Minor > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 8h 20m > Remaining Estimate: 0h > > Now the EnumerableMergeJoinRule only supports an inner and equi join. > If users make a theta-join query for a large dataset (such as 1*1), > the nested-loop join process will take dozens of time than the sort-merge > join process . > So if we can apply merge-join or hash-join rule for a theta join, it will > improve the performance greatly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-1581) UDTF like in hive
[
https://issues.apache.org/jira/browse/CALCITE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916569#comment-16916569
]
Stamatis Zampetakis commented on CALCITE-1581:
--
I was ready to merge this to master but I noticed that it creates various
javacc warnings (see below). What do you think [~pzw2018], is it easy to
resolve them?
{noformat}
Java Compiler Compiler Version 4.0 (Parser Generator)
(type "javacc" with no arguments for help)
Reading from file
/home/zabetak/Workspaces/Apache/Calcite/core/target/generated-sources/fmpp/javacc/Parser.jj
. . .
Note: UNICODE_INPUT option is specified. Please make sure you create the
parser/lexer using a Reader with the correct character encoding.
Warning: Choice conflict in [...] construct at line 4063, column 16.
Expansion nested within construct and expansion following construct
have common prefixes, one of which is: "("
Consider using a lookahead of 2 or more for nested expansion.
Warning: Choice conflict in [...] construct at line 4072, column 17.
Expansion nested within construct and expansion following construct
have common prefixes, one of which is: "("
Consider using a lookahead of 2 or more for nested expansion.
Warning: Choice conflict in [...] construct at line 4075, column 15.
Expansion nested within construct and expansion following construct
have common prefixes, one of which is: "("
Consider using a lookahead of 2 or more for nested expansion.
Warning: Choice conflict in [...] construct at line 4083, column 17.
Expansion nested within construct and expansion following construct
have common prefixes, one of which is: "("
Consider using a lookahead of 2 or more for nested expansion.
Warning: Choice conflict in [...] construct at line 4088, column 16.
Expansion nested within construct and expansion following construct
have common prefixes, one of which is: "("
Consider using a lookahead of 2 or more for nested expansion.
Warning: Choice conflict in [...] construct at line 4094, column 17.
Expansion nested within construct and expansion following construct
have common prefixes, one of which is: "("
Consider using a lookahead of 2 or more for nested expansion.
Warning: Choice conflict in [...] construct at line 4099, column 18.
Expansion nested within construct and expansion following construct
have common prefixes, one of which is: "("
Consider using a lookahead of 2 or more for nested expansion.
Warning: Choice conflict in [...] construct at line 4103, column 17.
Expansion nested within construct and expansion following construct
have common prefixes, one of which is: "("
Consider using a lookahead of 2 or more for nested expansion.
Warning: Choice conflict in [...] construct at line 4109, column 9.
Expansion nested within construct and expansion following construct
have common prefixes, one of which is: "("
Consider using a lookahead of 2 or more for nested expansion.
{noformat}
I have also a minor comment regarding the commit message. The details are not
very informative especially since the commit is squashed so I was ready to
remove most of them. In the future if the commit is squashed try to include
only those details that are significant (for instance 'fix format' etc., does
not need to be present). If the commit is not squashed try to have a commit
message that is self contained to facilitate the person who is doing the
squash.
> UDTF like in hive
> -
>
> Key: CALCITE-1581
> URL: https://issues.apache.org/jira/browse/CALCITE-1581
> Project: Calcite
> Issue Type: New Feature
>Reporter: Xiaoyong Deng
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available, udtf
> Fix For: 1.21.0
>
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> Support one row in and multi-column/multi-row out(one-to-many mapping), just
> like udtf in hive.
> The query would like this:
> {code}
> select
> func(c0, c1) as (f0, f1, f2)
> from table_name;
> {code}
> c0 and c1 are 'table_name' columns. f0, f1 and f2 are new generated columns.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (CALCITE-2973) Allow theta joins that have equi conditions to be executed using a hash join algorithm
[ https://issues.apache.org/jira/browse/CALCITE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez updated CALCITE-2973: - Fix Version/s: 1.21.0 > Allow theta joins that have equi conditions to be executed using a hash join > algorithm > -- > > Key: CALCITE-2973 > URL: https://issues.apache.org/jira/browse/CALCITE-2973 > Project: Calcite > Issue Type: New Feature > Components: core >Affects Versions: 1.19.0 >Reporter: Lai Zhou >Priority: Minor > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 8h 20m > Remaining Estimate: 0h > > Now the EnumerableMergeJoinRule only supports an inner and equi join. > If users make a theta-join query for a large dataset (such as 1*1), > the nested-loop join process will take dozens of time than the sort-merge > join process . > So if we can apply merge-join or hash-join rule for a theta join, it will > improve the performance greatly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (CALCITE-3280) Cannot parse query REGEXP_REPLACE in Redshift
[
https://issues.apache.org/jira/browse/CALCITE-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde resolved CALCITE-3280.
--
Resolution: Fixed
Fixed in
[b039b152|https://github.com/apache/calcite/commit/b039b152d8d2e91ab2ca0232ebddfc998f3bb331];
thanks for the PR, [~lishuming]!
> Cannot parse query REGEXP_REPLACE in Redshift
> -
>
> Key: CALCITE-3280
> URL: https://issues.apache.org/jira/browse/CALCITE-3280
> Project: Calcite
> Issue Type: Improvement
>Reporter: Ryan Fu
>Assignee: Julian Hyde
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> REGEXP_REPLACE error:
> {code:}
> No match found for function signature REGEXP_REPLACE(,
> , ){code}
>
> Example query:
> {code:sql}
> SELECT * , MD5(TRIM(BOTH ' ' FROM REGEXP_REPLACE(LOWER(name),
> '([[:space:]]|,)+([iInNcC]|[lLcC]).*$', ''))) AS company_id FROM
> public.account {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (CALCITE-3260) Add a public API for evaluating linq4j Expression nodes
[ https://issues.apache.org/jira/browse/CALCITE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde resolved CALCITE-3260. -- Resolution: Fixed Fixed in [be678e3d|https://github.com/apache/calcite/commit/be678e3dc65f5933f85927e53b2811738d124f11]; thanks for the PR, [~yanlin-Lynn]! > Add a public API for evaluating linq4j Expression nodes > --- > > Key: CALCITE-3260 > URL: https://issues.apache.org/jira/browse/CALCITE-3260 > Project: Calcite > Issue Type: New Feature >Reporter: Wang Yanlin >Assignee: Julian Hyde >Priority: Minor > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently, the public method *evaluate* of AbstractNode need a Evaluator > object as parameter, but Evaluator class has default access control. This > limit the access of *evaluate* method. So may be we can add an overload > *evaluate* method with default Evaluator, thus, allow to evaluate Expression > directly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (CALCITE-3122) Convert Pig Latin scripts into Calcite logical plan
[ https://issues.apache.org/jira/browse/CALCITE-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde resolved CALCITE-3122. -- Resolution: Fixed Fixed in [e5ae1796|https://github.com/apache/calcite/commit/e5ae1796e8016b0a5e943f458bf31a7e7e8aa75a]; thanks for the PR, [~khaitran]! > Convert Pig Latin scripts into Calcite logical plan > > > Key: CALCITE-3122 > URL: https://issues.apache.org/jira/browse/CALCITE-3122 > Project: Calcite > Issue Type: New Feature > Components: core, piglet >Reporter: Khai Tran >Assignee: Julian Hyde >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > We create an internal Calcite repo at LinkedIn and develop APIs to parse any > Pig Latin scripts into Calcite logical plan. The code was tested in nearly > ~1000 Pig scripts written at LinkedIn. > Changes: > 1. piglet: main conversion code live there, include: > * APIs to convert any Pig scripts into RelNode plans or SQL statements > * Use Pig Grunt parser to parse Pig Latin scripts into Pig logical plan > (DAGs) > * Convert Pig schemas into RelDatatype > * Traverse through Pig expression plan and convert Pig expressions into > RexNodes > * Map some basic Pig UDFs to Calcite SQL operators > * Build Calcite UDFs for any other Pig UDFs, including UDFs written in both > Java and Python > * Traverse (DFS) through Pig logical plans to convert each Pig logical nodes > to RelNodes > * Have an optimizer rule to optimize Pig group/cogroup into Aggregate > operators > 2. core: > * Implement other RelNode in Rel2Sql so that Pig can be translated into SQL > * Other minor changes in a few other classes to make Pig to Calcite works -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Closed] (CALCITE-3291) SqlFunctionsTest failed
[
https://issues.apache.org/jira/browse/CALCITE-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Shen closed CALCITE-3291.
--
Resolution: Won't Fix
When run testcase SqlFunctionsTest in IDEA, it failed with
java.lang.IllegalArgumentException. but when run mvn clean test again, the
testcase SqlFunctionsTest successed. Maybe it's not a bug.
> SqlFunctionsTest failed
> ---
>
> Key: CALCITE-3291
> URL: https://issues.apache.org/jira/browse/CALCITE-3291
> Project: Calcite
> Issue Type: Test
> Components: core
>Reporter: Hong Shen
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When I run mvn test, I find some testcase failed.
> {code}
> [ERROR] Tests run: 39, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 0.017 s <<< FAILURE! - in org.apache.calcite.test.SqlFunctionsTest
> [ERROR] testMd5(org.apache.calcite.test.SqlFunctionsTest) Time elapsed:
> 0.009 s <<< ERROR!
> java.lang.IllegalArgumentException: Argument for @Nonnull parameter 'string'
> of org/apache/calcite/runtime/SqlFunctions.md5 must not be null
> at
> org.apache.calcite.test.SqlFunctionsTest.testMd5(SqlFunctionsTest.java:882)
>
> [ERROR] testSha1(org.apache.calcite.test.SqlFunctionsTest) Time elapsed:
> 0.007 s <<< ERROR!
> java.lang.IllegalArgumentException: Argument for @Nonnull parameter 'string'
> of org/apache/calcite/runtime/SqlFunctions.sha1 must not be null
> at
> org.apache.calcite.test.SqlFunctionsTest.testSha1(SqlFunctionsTest.java:896)
> {code}
> The code is
> {code}
> @Test public void testMd5() {
> assertThat("d41d8cd98f00b204e9800998ecf8427e", is(md5("")));
> assertThat("d41d8cd98f00b204e9800998ecf8427e", is(md5(ByteString.of("",
> 16;
> assertThat("902fbdd2b1df0c4f70b4a5d23525e932", is(md5("ABC")));
> assertThat("902fbdd2b1df0c4f70b4a5d23525e932",
> is(md5(new ByteString("ABC".getBytes(UTF_8);
> try {
> String o = md5((String) null);
> fail("Expected NPE, got " + o);
> } catch (NullPointerException e) {
> // ok
> }
> }
> {code}
> It should catch java.lang.IllegalArgumentException, but not
> NullPointerException. I will add a patch to fix it.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (CALCITE-2973) Allow theta joins that have equi conditions to be executed using a hash join algorithm
[ https://issues.apache.org/jira/browse/CALCITE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916465#comment-16916465 ] Danny Chan commented on CALCITE-2973: - CALCITE-2302 and CALCITE-1581 are almost ready to be merged, i think it's okey to push this patch into release-1.21 if you think it is in enough good shape. > Allow theta joins that have equi conditions to be executed using a hash join > algorithm > -- > > Key: CALCITE-2973 > URL: https://issues.apache.org/jira/browse/CALCITE-2973 > Project: Calcite > Issue Type: New Feature > Components: core >Affects Versions: 1.19.0 >Reporter: Lai Zhou >Priority: Minor > Labels: pull-request-available > Time Spent: 8h 20m > Remaining Estimate: 0h > > Now the EnumerableMergeJoinRule only supports an inner and equi join. > If users make a theta-join query for a large dataset (such as 1*1), > the nested-loop join process will take dozens of time than the sort-merge > join process . > So if we can apply merge-join or hash-join rule for a theta join, it will > improve the performance greatly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-2973) Allow theta joins that have equi conditions to be executed using a hash join algorithm
[ https://issues.apache.org/jira/browse/CALCITE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916457#comment-16916457 ] Ruben Quesada Lopez commented on CALCITE-2973: -- [~hyuan], [~danny0405], [~julianhyde], I think we could try to push this into 1.21, what do you think? > Allow theta joins that have equi conditions to be executed using a hash join > algorithm > -- > > Key: CALCITE-2973 > URL: https://issues.apache.org/jira/browse/CALCITE-2973 > Project: Calcite > Issue Type: New Feature > Components: core >Affects Versions: 1.19.0 >Reporter: Lai Zhou >Priority: Minor > Labels: pull-request-available > Time Spent: 8h 20m > Remaining Estimate: 0h > > Now the EnumerableMergeJoinRule only supports an inner and equi join. > If users make a theta-join query for a large dataset (such as 1*1), > the nested-loop join process will take dozens of time than the sort-merge > join process . > So if we can apply merge-join or hash-join rule for a theta join, it will > improve the performance greatly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
