[jira] [Created] (CALCITE-1681) Provide a way to copy RelNode trees between planners

Remus Rusanu created CALCITE-1681: - Summary: Provide a way to copy RelNode trees between planners Key: CALCITE-1681 URL: https://issues.apache.org/jira/browse/CALCITE-1681 Project: Calcite Is

Re: Towards Calcite release 1.12

We have only one issue left to resolve, the materialized views RelNode cloning into a new planner. I expect to have a resolution by end of this week, so the all-done ETA from me would be mid-next week, let say March 15. Thanks, ~Remus On 3/7/17, 5:08 PM, "Julian Hyde" wrote: Remus,

[jira] [Created] (CALCITE-1682) New metadata providers for expression column origin and all predicates in plan

Jesus Camacho Rodriguez created CALCITE-1682: Summary: New metadata providers for expression column origin and all predicates in plan Key: CALCITE-1682 URL: https://issues.apache.org/jira/browse/CALCIT

Question on `in or not in` subquery plan

Hi, I got a question on `in (non-scalar-subquery1) or not in (non-scalar-subquery2)` pattern, for SQL: ``` select count(*) from db.foo where (int_test in (select int_test from db.foo where id<=20 )) or (long_test not in (select long_test from db.foo where id <=1)); ``` plan generated: ```

Re: Cluster mismatch between RelNodes of a query and a materialized view

I created CALCITE-1681 https://issues.apache.org/jira/browse/CALCITE-1681 and I intent to work on it for finishing HIVE-15708. My current thinking is to create a RelCopier based on RelShuttle and add a new abstract RelNode.copyTo(RelOptPlanner) that each concrete Rel type must override. The Rex

Re: Cluster mismatch between RelNodes of a query and a materialized view

The argument should be a RelOptCluster, not a RelOptPlanner. The link from RelNode to planner is indirect currently (via cluster) and will be non-existent after CALCITE-1536. I question whether we need a new method. Putting an abstract method on RelNode is a huge burden because every RelNode su

Re: Cluster mismatch between RelNodes of a query and a materialized view

Agree on the RelOptCluster. I noticed how adding new method to RelNode yields a gargantuan task (from the list of compile errors I got…). But I’m not sure a RelShuttle can handle this. For my test case the 3 nodes that need to be cloned are HiveScanTable, HiveFilter and HiveProject, all declare

Re: Towards Calcite release 1.12

Thanks. That last change should be fairly low-risk, so we should be able to move to an RC and vote soon afterwards. Committers, Let’s try to stabilize. I propose we get any “risky” changes in by the end of the weekend. By the way, I volunteer to be release manager. Julian > On Mar 8, 2017,

Re: Cluster mismatch between RelNodes of a query and a materialized view

We had created MaterializedCluster by extending RelOptCluster for solving this issue : https://github.com/qubole/quark/blob/master/optimizer/src/main/java/org/apache/calcite/plan/QuarkMaterializeCluster.java New cluster was used in RelNodes of Materialised Views and it allowed changing it's RelOpt

Re: Towards Calcite release 1.12

First draft of release notes: https://github.com/julianhyde/calcite/blob/branch-1.12/site/_docs/history.md (contains commits up to 12 days ago). > On Mar 8, 2017, at 9:16 AM, Julian Hyde wrote: > > Thanks. That la

Re: Cluster mismatch between RelNodes of a query and a materialized view

Could you work on LogicalXxx rather than HiveXxx? I know the Hive team likes to do everything in terms of HiveXxx RelNodes but I’m not sure it has to be that way. Julian > On Mar 8, 2017, at 9:37 AM, Remus Rusanu wrote: > > Agree on the RelOptCluster. > > I noticed how adding new method to R

[jira] [Created] (CALCITE-1683) Implement ProjectFilterTranspose, FilterProjectTranspose, AggregateFilterTranspose, FilterAggregateTranspose rules for Druid

Nishant Bangarwa created CALCITE-1683: - Summary: Implement ProjectFilterTranspose, FilterProjectTranspose, AggregateFilterTranspose, FilterAggregateTranspose rules for Druid Key: CALCITE-1683 URL: https://issu

Re: Towards Calcite release 1.12

Hi Julian, I think CALCITE-1615 is not on the list. Can you please double check? Haohui On Wed, Mar 8, 2017 at 9:49 AM Julian Hyde wrote: > First draft of release notes: > https://github.com/julianhyde/calcite/blob/branch-1.12/site/_docs/history.md > < > https://github.com/julianhyde/calcite/bl

Re: Cluster mismatch between RelNodes of a query and a materialized view

Amogh, Thanks for sharing. A hack, but an elegant one! Julian > On Mar 8, 2017, at 9:24 AM, Amogh Margoor wrote: > > We had created MaterializedCluster by extending RelOptCluster for solving > this issue : > > https://github.com/qubole/quark/blob/master/optimizer/src/main/java/org/apache/calc

Re: Cluster mismatch between RelNodes of a query and a materialized view

Thanks Amogh, I will consider this, but replacing the planner inside cached tree (or cluster) will require adding locking around access to the ‘one’ cached MV plan, and in the concurrent HiveServer2 it process may lead to more problems (my biggest fear being deadlocks). Thanks, ~Remus On 3/8/

Re: Towards Calcite release 1.12

Commit 49888a6c581f6243a5b3f1a48ea2c9edd6aac9d5, which fixed CALCITE-1615, had not made it to https://github.com/julianhyde/calcite/tree/branch-1.12 at the time I wrote that draft of the release notes. There are about a dozen other cases f

Re: Towards Calcite release 1.12

https://issues.apache.org/jira/browse/CALCITE-1598 (Pig adapter) does not seem to be on the list. Thanks, Eli On Wed, Mar 8, 2017 at 10:40 AM, Julian Hyde wrote: > Commit 49888a6c581f6243a5b3f1a48ea2c9edd6aac9d5, which fixed > CALCITE-1615, had not made it to https://github.com/julianhyde/ >

Re: Cluster mismatch between RelNodes of a query and a materialized view

Using a ThreadLocal might fix some of those concerns. > On Mar 8, 2017, at 10:38 AM, Remus Rusanu wrote: > > Thanks Amogh, I will consider this, but replacing the planner inside cached > tree (or cluster) will require adding locking around access to the ‘one’ > cached MV plan, and in the conc

Re: Cluster mismatch between RelNodes of a query and a materialized view

To make sure I understand, you’re saying that the cloning would take a tree of various Xxx operators (eg. HiveProject) and generate a tree of equivalent LogicalXxx operators (HiveProject -> LogicalProject, HiveFilter->LogicalFilter, HiveAggregate -> LogicalAggregate and so on). Is my understa

Re: Towards Calcite release 1.12

Same comments as for CALCITE-1615. The list contains commits up to 2/24. I will fix before the release. > On Mar 8, 2017, at 10:43 AM, Eli Levine wrote: > > https://issues.apache.org/jira/browse/CALCITE-1598 (Pig adapter) does not > seem to be on the list. > > Thanks, > > Eli > > > On Wed,

Re: Cluster mismatch between RelNodes of a query and a materialized view

Working on LogicalRelNodes doesnt solve problem for Hive (or any other project which use RelFactories to generate custom rel node). Fact that RelFactories exist is evidence enough that LogicalRelNodes are not sufficient outside of Calcite. We have to solve this problem in that context. On Wed, Mar

Re: Question on `in or not in` subquery plan

It’s just possible that Calcite is being smart. I see a filter LogicalFilter(condition=[<=($0, 20)]) deep in your plan, so maybe Calcite is exploiting the constraint derived from that, and the fact that it’s the same table inside and outside the sub-query (work we did in CALCITE-1494). I also

Re: Cluster mismatch between RelNodes of a query and a materialized view

> On Mar 8, 2017, at 11:27 AM, Ashutosh Chauhan wrote: > > Fact that > RelFactories exist is evidence enough that LogicalRelNodes are not > sufficient outside of Calcite. Not true. Hive uses its own RelNodes far more than other projects do. The other projects do a significant part of their pla

Re: Towards Calcite release 1.12

Looks like CALCITE-1681 ‘Provide a way to copy RelNode trees between clusters’ is also needed. I hope I’ll have it ready this week. On 3/8/17, 10:49 AM, "Julian Hyde" wrote: Same comments as for CALCITE-1615. The list contains commits up to 2/24. I will fix before the release. > O

Re: Towards Calcite release 1.12

We have that dependency already, albeit transitively: CALCITE-1681 blocks HIVE-15708, which is depended upon by CALCITE-1657 “Release Calcite 1.12.0”. > On Mar 8, 2017, at 12:12 PM, Remus Rusanu wrote: > > Looks like CALCITE-1681 ‘Provide a way to copy RelNode trees between > clusters’ is also

Go Avatica/Phoenix driver updated to support new Go 1.8 features

Hi all, Go 1.8 was released recently and the database/sql package saw a lot of new features. I just tagged the v1.3.0 release for the Go Avatica driver[0] which ships all of these new features. The full list of changes in the database/sql package is available here: https://docs.google.com/do

Re: Go Avatica/Phoenix driver updated to support new Go 1.8 features

Great work, Francis!! F21 wrote: Hi all, Go 1.8 was released recently and the database/sql package saw a lot of new features. I just tagged the v1.3.0 release for the Go Avatica driver[0] which ships all of these new features. The full list of changes in the database/sql package is available h

[jira] [Created] (CALCITE-1684) Default precision for VARCHAR should be the maximum value

Kevin Liew created CALCITE-1684: --- Summary: Default precision for VARCHAR should be the maximum value Key: CALCITE-1684 URL: https://issues.apache.org/jira/browse/CALCITE-1684 Project: Calcite I

[jira] [Created] (CALCITE-1685) defaultNullCollation key listed as materializationsEnabled

Kevin Liew created CALCITE-1685: --- Summary: defaultNullCollation key listed as materializationsEnabled Key: CALCITE-1685 URL: https://issues.apache.org/jira/browse/CALCITE-1685 Project: Calcite