[
https://issues.apache.org/jira/browse/DRILL-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rahul Challapalli updated DRILL-4860:
-------------------------------------
Attachment: l_3level.tgz
Contents of the log file
{code}
2016-08-22 18:43:10,348 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
2844b840-84f0-eeb8-e8a9-a7697f9f0043: explain plan for select count (*) from (
select t1.l_orderkey, t1.l_linenumber from
dfs.`/drill/testdata/partition_pruning/nested/l_3level` t1
where
t1.dir0 = 1 and t1.dir1='one' and t1.dir2 = '2015-7-12' and t1.l_orderkey
in (
select t2.l_orderkey from
dfs.`/drill/testdata/partition_pruning/nested/l_3level` t2 where t2.dir0 = 1
and t2.dir1='two' and t2.dir2 = '2015-8-12' and t2.l_discount = t1.l_discount
)
) data
2016-08-22 18:43:10,544 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning
class:
org.apache.drill.exec.planner.logical.partition.PruneScanRule$DirPruneScanFilterOnProjectRule
2016-08-22 18:43:10,545 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze
filter tree: 0 ms
2016-08-22 18:43:10,545 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.e.p.l.partition.PruneScanRule - Elapsed time to populate partitioning
column vectors: 0 ms within batchIndex: 0
2016-08-22 18:43:10,547 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.e.p.l.partition.PruneScanRule - Elapsed time in interpreter evaluation: 1
ms within batchIndex: 0 with # of partitions : 11
2016-08-22 18:43:10,547 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.e.p.l.partition.PruneScanRule - Pruned 11 partitions down to 1
2016-08-22 18:43:10,547 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.e.p.l.partition.PruneScanRule - Total pruning elapsed time: 2 ms
2016-08-22 18:43:10,547 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning
class:
org.apache.drill.exec.planner.logical.partition.PruneScanRule$DirPruneScanFilterOnScanRule
2016-08-22 18:43:10,547 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze
filter tree: 0 ms
2016-08-22 18:43:10,548 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.e.p.l.partition.PruneScanRule - Elapsed time to populate partitioning
column vectors: 0 ms within batchIndex: 0
2016-08-22 18:43:10,549 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.e.p.l.partition.PruneScanRule - Elapsed time in interpreter evaluation: 1
ms within batchIndex: 0 with # of partitions : 11
2016-08-22 18:43:10,549 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.e.p.l.partition.PruneScanRule - Pruned 11 partitions down to 1
2016-08-22 18:43:10,549 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.e.p.l.partition.PruneScanRule - Total pruning elapsed time: 1 ms
2016-08-22 18:43:10,561 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Took 0 ms to get file statuses
2016-08-22 18:43:10,563 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1
using 1 threads. Time: 1ms total, 1.914413ms avg, 1ms max.
2016-08-22 18:43:10,563 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1
using 1 threads. Earliest start: 0.326000 μs, Latest start: 0.326000 μs,
Average start: 0.326000 μs .
2016-08-22 18:43:10,563 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Took 2 ms to read file metadata
2016-08-22 18:43:10,577 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 11 out of
11 using 11 threads. Time: 4ms total, 2.500455ms avg, 2ms max.
2016-08-22 18:43:10,577 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 11 out of
11 using 11 threads. Earliest start: 208.722000 μs, Latest start: 1811.004000
μs, Average start: 1085.901091 μs .
2016-08-22 18:43:10,580 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Took 0 ms to get file statuses
2016-08-22 18:43:10,582 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1
using 1 threads. Time: 1ms total, 1.589404ms avg, 1ms max.
2016-08-22 18:43:10,582 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1
using 1 threads. Earliest start: 0.205000 μs, Latest start: 0.205000 μs,
Average start: 0.205000 μs .
2016-08-22 18:43:10,582 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Took 1 ms to read file metadata
2016-08-22 18:43:10,993 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 2844b840-84f0-eeb8-e8a9-a7697f9f0043:0:0:
State change requested AWAITING_ALLOCATION --> RUNNING
2016-08-22 18:43:10,993 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 2844b840-84f0-eeb8-e8a9-a7697f9f0043:0:0:
State to report: RUNNING
2016-08-22 18:43:11,000 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 2844b840-84f0-eeb8-e8a9-a7697f9f0043:0:0:
State change requested RUNNING --> FINISHED
2016-08-22 18:43:11,001 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 2844b840-84f0-eeb8-e8a9-a7697f9f0043:0:0:
State to report: FINISHED
{code}
> Partition pruning does not work when we have a correlated sub-query
> -------------------------------------------------------------------
>
> Key: DRILL-4860
> URL: https://issues.apache.org/jira/browse/DRILL-4860
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
> Affects Versions: 1.8.0
> Reporter: Rahul Challapalli
> Attachments: l_3level.tgz
>
>
> git commit # : 0a4c21cc15329c063f256f6fbf2c4c69a90d9fa1
> The below query indicates that partition pruning did not occur for the
> sub-query
> {code}
> explain plan for select count (*) from (
> select t1.l_orderkey, t1.l_linenumber from
> dfs.`/drill/testdata/partition_pruning/nested/l_3level` t1
> where
> t1.dir0 = 1 and t1.dir1='one' and t1.dir2 = '2015-7-12' and t1.l_orderkey
> in (
> select t2.l_orderkey from
> dfs.`/drill/testdata/partition_pruning/nested/l_3level` t2 where t2.dir0 = 1
> and t2.dir1='two' and t2.dir2 = '2015-8-12' and t2.l_discount > t1.l_discount
> )
> ) data;
> +------+------+
> | text | json |
> +------+------+
> | 00-00 Screen
> 00-01 Project(EXPR$0=[$0])
> 00-02 StreamAgg(group=[{}], EXPR$0=[COUNT()])
> 00-03 Project($f0=[0])
> 00-04 Project(l_discount=[$2], l_orderkey=[$3], l_orderkey0=[$0],
> l_discount0=[$1])
> 00-05 HashJoin(condition=[AND(=($2, $1), =($3, $0))],
> joinType=[inner])
> 00-07 HashAgg(group=[{0, 1}])
> 00-09 Project(l_orderkey=[$0], l_discount0=[$2])
> 00-10 Project(l_orderkey=[$1], l_discount=[$2],
> l_discount0=[$0])
> 00-11 HashJoin(condition=[=($2, $0)], joinType=[inner])
> 00-13 HashAgg(group=[{0}])
> 00-15 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath
> [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/two/2015-8-12/30.parquet],
> ReadEntryWithPath
> [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/three/2015-7-12/40.parquet],
> ReadEntryWithPath
> [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/one/2015-7-13/20.parquet],
> ReadEntryWithPath
> [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/one/2015-7-12/10.parquet],
> ReadEntryWithPath
> [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/two/2015-9-12/50.parquet],
> ReadEntryWithPath
> [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/two/2015-7-12/30.parquet],
> ReadEntryWithPath
> [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/two/2015-8-12/40.parquet],
> ReadEntryWithPath
> [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/three/2015-8-12/10.parquet],
> ReadEntryWithPath
> [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/one/2015-9-12/20.parquet],
> ReadEntryWithPath
> [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/one/2015-7-12/50.parquet],
> ReadEntryWithPath
> [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/one/2015-8-12/10.parquet]],
> selectionRoot=maprfs:/drill/testdata/partition_pruning/nested/l_3level,
> numFiles=11, usedMetadataFile=false, columns=[`l_discount`]]])
> 00-12 Project(l_orderkey=[$0], l_discount0=[$1])
> 00-14 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath
> [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/two/2015-8-12/30.parquet]],
> selectionRoot=maprfs:/drill/testdata/partition_pruning/nested/l_3level,
> numFiles=1, usedMetadataFile=false, columns=[`l_orderkey`, `l_discount`]]])
> 00-06 Project(l_discount=[$0], l_orderkey0=[$1])
> 00-08 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath
> [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/one/2015-7-12/10.parquet]],
> selectionRoot=maprfs:/drill/testdata/partition_pruning/nested/l_3level,
> numFiles=1, usedMetadataFile=false, columns=[`l_discount`, `l_orderkey`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
