[ https://issues.apache.org/jira/browse/DRILL-4665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15279011#comment-15279011 ]
Rahul Challapalli commented on DRILL-4665: ------------------------------------------ I can confirm that this does not work with DFS storage plugin as well. Also this seems to happen when we use like operator on the non-partitioning column > Partition pruning not working for hive partitioned table with 'LIKE' and '=' > filter > ----------------------------------------------------------------------------------- > > Key: DRILL-4665 > URL: https://issues.apache.org/jira/browse/DRILL-4665 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization, Storage - Hive > Affects Versions: 1.6.0 > Reporter: Chun Chang > > This problem was initially reported by Shankar Mane > <shankar.m...@games24x7.com> > > Problem: > > > > 1. In drill, we are using hive partition table. But explain plan (same > > query) for like and = operator differs and used all partitions in case of > > like operator. > > 2. If you see below drill explain plans: Like operator uses *all* > > partitions where > > = operator uses *only* partition filtered by log_date condition. > I reproduced the reported issue. I have a partitioned hive external table > with the following schema: > {noformat} > create external table if not exists lineitem_partitioned ( > l_orderkey bigint, > l_partkey bigint, > l_suppkey bigint, > l_linenumber bigint, > l_quantity double, > l_extendedprice double, > l_discount double, > l_tax double, > l_returnflag string, > l_linestatus string, > l_shipdate string, > l_commitdate string, > l_receiptdate string, > l_shipinstruct string, > l_shipmode string, > l_comment string > ) > partitioned by (year int, month int, day int) > STORED AS PARQUET > LOCATION '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem'; > alter table lineitem_partitioned add partition(year=2015, month=1, day=1) > location > '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/1'; > alter table lineitem_partitioned add partition(year=2015, month=1, day=2) > location > '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/2'; > alter table lineitem_partitioned add partition(year=2015, month=1, day=3) > location > '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/3'; > alter table lineitem_partitioned add partition(year=2015, month=1, day=4) > location > '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/4'; > alter table lineitem_partitioned add partition(year=2015, month=1, day=5) > location > '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/5'; > alter table lineitem_partitioned add partition(year=2015, month=1, day=6) > location > '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/6'; > alter table lineitem_partitioned add partition(year=2015, month=1, day=7) > location > '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/7'; > alter table lineitem_partitioned add partition(year=2015, month=1, day=8) > location > '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/8'; > alter table lineitem_partitioned add partition(year=2015, month=1, day=9) > location > '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/9'; > alter table lineitem_partitioned add partition(year=2015, month=1, day=10) > location > '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/10'; > {noformat} > Without 'LIKE', drill plans right: > {noformat} > 0: jdbc:drill:schema=dfs.drillTestDir> explain plan for select l_shipdate, > l_linestatus, l_shipinstruct, `day` from lineitem_partitioned_db2k_2_999 > where `day` = 2 limit 10; > +------+------+ > | text | json | > +------+------+ > | 00-00 Screen > 00-01 Project(l_shipdate=[$0], l_linestatus=[$1], l_shipinstruct=[$2], > day=[$3]) > 00-02 SelectionVectorRemover > 00-03 Limit(fetch=[10]) > 00-04 Limit(fetch=[10]) > 00-05 Project(l_shipdate=[$1], l_linestatus=[$0], > l_shipinstruct=[$2], day=[$3]) > 00-06 Scan(groupscan=[HiveScan [table=Table(dbName:md815_db2k, > tableName:lineitem_partitioned_db2k_2_999), columns=[`l_linestatus`, > `l_shipdate`, `l_shipinstruct`, `day`], numPartitions=1, partitions= > [Partition(values:[2015, 1, 2])], > inputDirectories=[maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/2]]]) > {noformat} > With 'LIKE', pruning is not happening: > {noformat} > 0: jdbc:drill:schema=dfs.drillTestDir> explain plan for select l_shipdate, > l_linestatus, l_shipinstruct, `day` from lineitem_partitioned_db2k_2_999 > where `day` = 2 and l_shipinstruct like '%BACK%' limit 10; > +------+------+ > | text | json | > +------+------+ > | 00-00 Screen > 00-01 Project(l_shipdate=[$0], l_linestatus=[$1], l_shipinstruct=[$2], > day=[$3]) > 00-02 SelectionVectorRemover > 00-03 Limit(fetch=[10]) > 00-04 Limit(fetch=[10]) > 00-05 Project(l_shipdate=[$1], l_linestatus=[$0], > l_shipinstruct=[$2], day=[$3]) > 00-06 SelectionVectorRemover > 00-07 Filter(condition=[AND(=($3, 2), LIKE($2, '%BACK%'))]) > 00-08 Scan(groupscan=[HiveScan > [table=Table(dbName:md815_db2k, tableName:lineitem_partitioned_db2k_2_999), > columns=[`l_linestatus`, `l_shipdate`, `l_shipinstruct`, `day`], > numPartitions=10, partitions= [Partition(values:[2015, 1, 1]), > Partition(values:[2015, 1, 10]), Partition(values:[2015, 1, 2]), > Partition(values:[2015, 1, 3]), Partition(values:[2015, 1, 4]), > Partition(values:[2015, 1, 5]), Partition(values:[2015, 1, 6]), > Partition(values:[2015, 1, 7]), Partition(values:[2015, 1, 8]), > Partition(values:[2015, 1, 9])], > inputDirectories=[maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/1, > > maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/10, > maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/2, > maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/3, > maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/4, > maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/5, > maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/6, > maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/7, > maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/8, > maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/9]]]) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)