[
https://issues.apache.org/jira/browse/DRILL-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15212258#comment-15212258
]
ASF GitHub Bot commented on DRILL-4531:
---------------------------------------
Github user jinfengni commented on the pull request:
https://github.com/apache/drill/pull/444#issuecomment-201430413
I agree that your expectation for RelSubset makes sense to me. However, for
now it does not happen that way. The following is the trace for the query which
went through planning with this customized rule (I removed some rels).
Set#1, type: RecordType(BIGINT custkey, ANY custAddress)
...
Set#2, type: (DrillRecordRow[*, l_orderkey, l_partkey, l_linenumber])
rel#273:Subset#2.LOGICAL.ANY([]).[], best=rel#442,
importance=0.31381059609000006
rel#275:AbstractConverter.LOGICAL.ANY([]).[](input=rel#80:Subset#2.ENUMERABLE.ANY([]).[],convention=LOGICAL,DrillDistributionTraitDef=ANY([]),sort=[]),
rowcount=100.0, cumulative cost={inf}
rel#442:DrillScanRel.LOGICAL.ANY([]).[](table=[cp,
tpch/lineitem.parquet],groupscan=ParquetGroupScan [entries=[ReadEntryWithPath
[path=classpath:/tpch/lineitem.parquet]],
selectionRoot=classpath:/tpch/lineitem.parquet, numFiles=1,
usedMetadataFile=false, columns=[`*`]]), rowcount=60175.0, cumulative
cost={60175.0 rows, 6.0175E8 cpu, 0.0 io, 0.0 network, 0.0 memory}
Set#3, type: (DrillRecordRow[*, l_orderkey, l_partkey, l_linenumber])
rel#82:Subset#3.NONE.ANY([]).[], best=null, importance=0.3874204890000001
rel#345:LogicalFilter.NONE.ANY([]).[[]](input=rel#273:Subset#2.LOGICAL.ANY([]).[],condition=AND(>=($1,
20160101), <=($2, 20160301), OR(=($2, 1), =($2, 2), =($2, 5), =($2, 6)))),
rowcount=6.25, cumulative cost={inf}
rel#345:LogicalFilter has a child rel#273 with LOGICAL convention.
As another example, for the following query:
Select n_name, n_nationkey from cp.`tpch/nation.parquet` where
n_nationkey > 5
The trace:
Set#0, type: (DrillRecordRow[*, n_nationkey, n_name])
...
Set#1, type: (DrillRecordRow[*, n_nationkey, n_name])
rel#21:Subset#1.NONE.ANY([]).[], best=null, importance=0.81
rel#20:LogicalFilter.NONE.ANY([]).[](input=rel#19:Subset#0.ENUMERABLE.ANY([]).[],condition=>($1,
5)), rowcount=50.0, cumulative cost={inf}
rel#37:LogicalFilter.NONE.ANY([]).[[]](input=rel#19:Subset#0.ENUMERABLE.ANY([]).[],condition=>($1,
5)), rowcount=50.0, cumulative cost={inf}
rel#35:Subset#1.LOGICAL.ANY([]).[], best=rel#60, importance=0.81
rel#60:DrillFilterRel.LOGICAL.ANY([]).[](input=rel#57:Subset#0.LOGICAL.ANY([]).[],condition=>($1,
5)), rowcount=50.0, cumulative cost={125.0 rows, 250600.0 cpu, 0.0 io, 0.0
network, 0.0 memory}
Set#2, type: RecordType(ANY n_name, ANY n_nationkey)
rel#23:Subset#2.NONE.ANY([]).[], best=null, importance=0.9
rel#53:LogicalProject.NONE.ANY([]).[](input=rel#35:Subset#1.LOGICAL.ANY([]).[],n_name=$2,n_nationkey=$1),
rowcount=50.0, cumulative cost={inf}
Again, rel#53:LogicalProject has a child rel#35 whose convention is LOGICAL.
I think the reason that we have such mixed rels is we have different kinds
of rules, used in a single Volcano planning phase.
1) Rule matchs base class Filter/Project, etc only.
2) Rule matches LogicalFilter/LogicalProject, etc
3) Rule uses copy() method to generate a new Rel
4) Rule uses RelFactory to generate a new Rel.
5) convent rule, which convert from Calcite logical (NONE/Enumerable) to
Drill logical (LOGICAL)
For instance, ProjectMergeRule, which matches base Project, yet uses
default RelFactory, will match both LogicalProject and DrillProject, but
produce LogicalProject as outcome. That will cause the mixed rels.
2 things we may consider to fix this:
1) Separate the convent rules from the other transformation rules. Apply
convert rule first, then transformation rule match DrillLogical only. That's
similar to what other system (hive) is doing.
2) go through every rule we use, and we need make sure the convention of
input and ouptput of a transformation rule should be same, except for the
convert rule.
The above 2 things would take some considerably effort, though.
> Query with filter and aggregate hangs in planning phase
> -------------------------------------------------------
>
> Key: DRILL-4531
> URL: https://issues.apache.org/jira/browse/DRILL-4531
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
> Reporter: Jinfeng Ni
> Assignee: Jinfeng Ni
> Fix For: 1.7.0
>
>
> For the following query,
> {code}
> SELECT cust.custAddress,
> lineitem.provider
> FROM (
> SELECT cast(c_custkey AS bigint) AS custkey,
> c_address AS custAddress
> FROM cp.`tpch/customer.parquet` ) cust
> LEFT JOIN
> (
> SELECT DISTINCT l_linenumber,
> CASE
> WHEN l_partkey IN (1, 2) THEN 'Store1'
> WHEN l_partkey IN (5, 6) THEN 'Store2'
> END AS provider
> FROM cp.`tpch/lineitem.parquet`
> WHERE ( l_orderkey >=20160101 AND l_partkey <=20160301)
> AND l_partkey IN (1,2, 5, 6) ) lineitem
> ON cust.custkey = lineitem.l_linenumber
> WHERE provider IS NOT NULL
> GROUP BY cust.custAddress,
> lineitem.provider
> ORDER BY cust.custAddress,
> lineitem.provider;
> {code}
> When run on today's master branch commit:
> 79a3c164c1df7a5d7a0b82574316b4a0b1c7593e, query just hangs there in the
> planning phase.
> Log shows that it stuck in Drill_Logical planning phase.
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
