Chun Chang created DRILL-4665:
---------------------------------
Summary: Partition pruning not working for hive partitioned table
with 'LIKE' and '=' filter
Key: DRILL-4665
URL: https://issues.apache.org/jira/browse/DRILL-4665
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization, Storage - Hive
Affects Versions: 1.6.0
Reporter: Chun Chang
This problem was initially reported by Shankar Mane <shankar.m...@games24x7.com>
> Problem:
>
> 1. In drill, we are using hive partition table. But explain plan (same
> query) for like and = operator differs and used all partitions in case of
> like operator.
> 2. If you see below drill explain plans: Like operator uses *all*
> partitions where
> = operator uses *only* partition filtered by log_date condition.
I reproduced the reported issue. I have a partitioned hive external table with
the following schema:
{noformat}
create external table if not exists lineitem_partitioned (
l_orderkey bigint,
l_partkey bigint,
l_suppkey bigint,
l_linenumber bigint,
l_quantity double,
l_extendedprice double,
l_discount double,
l_tax double,
l_returnflag string,
l_linestatus string,
l_shipdate string,
l_commitdate string,
l_receiptdate string,
l_shipinstruct string,
l_shipmode string,
l_comment string
)
partitioned by (year int, month int, day int)
STORED AS PARQUET
LOCATION '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem';
alter table lineitem_partitioned add partition(year=2015, month=1, day=1)
location
'/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/1';
alter table lineitem_partitioned add partition(year=2015, month=1, day=2)
location
'/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/2';
alter table lineitem_partitioned add partition(year=2015, month=1, day=3)
location
'/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/3';
alter table lineitem_partitioned add partition(year=2015, month=1, day=4)
location
'/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/4';
alter table lineitem_partitioned add partition(year=2015, month=1, day=5)
location
'/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/5';
alter table lineitem_partitioned add partition(year=2015, month=1, day=6)
location
'/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/6';
alter table lineitem_partitioned add partition(year=2015, month=1, day=7)
location
'/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/7';
alter table lineitem_partitioned add partition(year=2015, month=1, day=8)
location
'/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/8';
alter table lineitem_partitioned add partition(year=2015, month=1, day=9)
location
'/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/9';
alter table lineitem_partitioned add partition(year=2015, month=1, day=10)
location
'/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/10';
{noformat}
Without 'LIKE', drill plans right:
{noformat}
0: jdbc:drill:schema=dfs.drillTestDir> explain plan for select l_shipdate,
l_linestatus, l_shipinstruct, `day` from lineitem_partitioned_db2k_2_999 where
`day` = 2 limit 10;
+------+------+
| text | json |
+------+------+
| 00-00 Screen
00-01 Project(l_shipdate=[$0], l_linestatus=[$1], l_shipinstruct=[$2],
day=[$3])
00-02 SelectionVectorRemover
00-03 Limit(fetch=[10])
00-04 Limit(fetch=[10])
00-05 Project(l_shipdate=[$1], l_linestatus=[$0],
l_shipinstruct=[$2], day=[$3])
00-06 Scan(groupscan=[HiveScan [table=Table(dbName:md815_db2k,
tableName:lineitem_partitioned_db2k_2_999), columns=[`l_linestatus`,
`l_shipdate`, `l_shipinstruct`, `day`], numPartitions=1, partitions=
[Partition(values:[2015, 1, 2])],
inputDirectories=[maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/2]]])
{noformat}
With 'LIKE', pruning is not happening:
{noformat}
0: jdbc:drill:schema=dfs.drillTestDir> explain plan for select l_shipdate,
l_linestatus, l_shipinstruct, `day` from lineitem_partitioned_db2k_2_999 where
`day` = 2 and l_shipinstruct like '%BACK%' limit 10;
+------+------+
| text | json |
+------+------+
| 00-00 Screen
00-01 Project(l_shipdate=[$0], l_linestatus=[$1], l_shipinstruct=[$2],
day=[$3])
00-02 SelectionVectorRemover
00-03 Limit(fetch=[10])
00-04 Limit(fetch=[10])
00-05 Project(l_shipdate=[$1], l_linestatus=[$0],
l_shipinstruct=[$2], day=[$3])
00-06 SelectionVectorRemover
00-07 Filter(condition=[AND(=($3, 2), LIKE($2, '%BACK%'))])
00-08 Scan(groupscan=[HiveScan
[table=Table(dbName:md815_db2k, tableName:lineitem_partitioned_db2k_2_999),
columns=[`l_linestatus`, `l_shipdate`, `l_shipinstruct`, `day`],
numPartitions=10, partitions= [Partition(values:[2015, 1, 1]),
Partition(values:[2015, 1, 10]), Partition(values:[2015, 1, 2]),
Partition(values:[2015, 1, 3]), Partition(values:[2015, 1, 4]),
Partition(values:[2015, 1, 5]), Partition(values:[2015, 1, 6]),
Partition(values:[2015, 1, 7]), Partition(values:[2015, 1, 8]),
Partition(values:[2015, 1, 9])],
inputDirectories=[maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/1,
maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/10,
maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/2,
maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/3,
maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/4,
maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/5,
maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/6,
maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/7,
maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/8,
maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/9]]])
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
