[
https://issues.apache.org/jira/browse/PHOENIX-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15547766#comment-15547766
]
Maryann Xue commented on PHOENIX-2679:
--------------------------------------
I'm implementing it using shadow tables, which means each structured type
Phoenix table (referenced in the query) will have a corresponding flattened
type shadow table. At a point right before real query planning starts, we'll
have a Hep program that replaces a "flattening" Project on top of a TableScan
of the structured type table with a TableScan of the flattened type table.
After this point, everything will work exactly the same as they did before.
Secondary indexes will be defined based on these shadow tables, which I assume
would just work the same as before too.
I've also managed to avoid the side effects of shadow tables by adding them
through a Hook that's triggered after sql-to-rel conversion, so that they won't
be visible at validation time.
Part of it is already working, but with some changes on the Calcite side, as
shown below:
{code}
diff --git a/core/src/main/java/org/apache/calcite/prepare/Prepare.java
b/core/src/main/java/org/apache/calcite/prepare/Prepare.java
index 70cddf6..8d9b5f4 100644
--- a/core/src/main/java/org/apache/calcite/prepare/Prepare.java
+++ b/core/src/main/java/org/apache/calcite/prepare/Prepare.java
@@ -258,7 +258,7 @@ public PreparedResult prepareSql(
// Structured type flattening, view expansion, and plugging in physical
// storage.
- root = root.withRel(flattenTypes(root.rel, true));
+ root = root.withRel(sqlToRelConverter.flattenTypes(root.rel, true));
if (this.context.config().forceDecorrelate()) {
// Subquery decorrelation.
@@ -363,7 +363,7 @@ private boolean shouldTrim(RelNode rootRel) {
// For now, don't trim if there are more than 3 joins. The projects
// near the leaves created by trim migrate past joins and seem to
// prevent join-reordering.
- return THREAD_TRIM.get() || RelOptUtil.countJoins(rootRel) < 2;
+ return THREAD_TRIM.get() && RelOptUtil.countJoins(rootRel) < 2;
}
public RelRoot expandView(RelDataType rowType, String queryString,
{code}
The reasons are:
1. CalcitePrepareStmt.flattenTypes() doesn't really do anything to flatten
types, but SqlToRelConverter.flattenTypes() does.
2. trimUnusedFields() doesn't seem to work right for structured types after
flattening. The OR logic seems to be wrong by itself, or was it intended that
way?
{code}
diff --git
a/core/src/main/java/org/apache/calcite/sql2rel/RelStructuredTypeFlattener.java
b/core/src/main/java/org/apache/calcite/sql2rel/RelStructuredTypeFlattener.java
index 82b1f4e..db508d6 100644
---
a/core/src/main/java/org/apache/calcite/sql2rel/RelStructuredTypeFlattener.java
+++
b/core/src/main/java/org/apache/calcite/sql2rel/RelStructuredTypeFlattener.java
@@ -27,6 +27,7 @@
import org.apache.calcite.rel.core.CorrelationId;
import org.apache.calcite.rel.core.Sample;
import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.core.TableScan;
import org.apache.calcite.rel.core.Uncollect;
import org.apache.calcite.rel.logical.LogicalAggregate;
import org.apache.calcite.rel.logical.LogicalCalc;
@@ -39,7 +40,6 @@
import org.apache.calcite.rel.logical.LogicalSort;
import org.apache.calcite.rel.logical.LogicalTableFunctionScan;
import org.apache.calcite.rel.logical.LogicalTableModify;
-import org.apache.calcite.rel.logical.LogicalTableScan;
import org.apache.calcite.rel.logical.LogicalUnion;
import org.apache.calcite.rel.logical.LogicalValues;
import org.apache.calcite.rel.stream.LogicalChi;
@@ -648,7 +648,7 @@ private boolean isConstructor(RexNode rexNode) {
|| (call.isA(SqlKind.NEW_SPECIFICATION));
}
- public void rewriteRel(LogicalTableScan rel) {
+ public void rewriteRel(TableScan rel) {
RelNode newRel = rel.getTable().toRel(toRelContext);
if (!SqlTypeUtil.isFlat(rel.getRowType())) {
final List<Pair<RexNode, String>> flattenedExpList =
Lists.newArrayList();
{code}
3. Apparently it wouldn't work for PhoenixTableScan, which is part of rel.
[~julianhyde], for (2) and (3), do you think it would be reasonable to make the
changes into Calcite? for (1) can we make it an option?
Aside from the points listed above, one big blocker right now is that the
ProjectFilterTransposeRule could not push the "flattening" Project through the
filter, coz the "flattening" Project was thought to be a trivial Project. So
right now any query with a filter cannot work. I'll try to see if I can work
this out.
> Implement column family schema structure in Calcite-Phoenix
> -----------------------------------------------------------
>
> Key: PHOENIX-2679
> URL: https://issues.apache.org/jira/browse/PHOENIX-2679
> Project: Phoenix
> Issue Type: Task
> Reporter: Maryann Xue
> Assignee: Maryann Xue
> Labels: calcite
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
