[ https://issues.apache.org/jira/browse/DRILL-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216117#comment-16216117 ]
ASF GitHub Bot commented on DRILL-5795: --------------------------------------- Github user cchang738 commented on the issue: https://github.com/apache/drill/pull/949 There is a plan verification failure due to plan change. The plan baseline needs to be changed after this PR is merged. Plan Verification Failures: /root/drillAutomation/mapr/framework/resources/Functional/int96/q28.q Query: explain plan for select voter_id, name from `hive1_parquet_part` where date_part('year', create_timestamp1)=2018 Expected and actual text plans are different. Expected: .*numFiles=2, usedMetadataFile=true.* Actual: 00-00 Screen 00-01 Project(voter_id=[$0], name=[$1]) 00-02 Project(voter_id=[$1], name=[$2]) 00-03 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=/drill/testdata/subqueries/hive1_parquet_part/0_0_10.parquet], ReadEntryWithPath [path=/drill/testdata/subqueries/hive1_parquet_part/0_0_9.parquet]], selectionRoot=/drill/testdata/subqueries/hive1_parquet_part, numFiles=2, numRowGroups=2, usedMetadataFile=true, cacheFileRoot=/drill/testdata/subqueries/hive1_parquet_part, columns=[`create_timestamp1`, `voter_id`, `name`]]]) > Filter pushdown for parquet handles multi rowgroup file > ------------------------------------------------------- > > Key: DRILL-5795 > URL: https://issues.apache.org/jira/browse/DRILL-5795 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet > Affects Versions: 1.11.0 > Reporter: Damien Profeta > Assignee: Damien Profeta > Labels: doc-impacting > Fix For: 1.12.0 > > Attachments: multirowgroup_overlap.parquet > > > DRILL-1950 implemented the filter pushdown for parquet file but only in the > case of one rowgroup per parquet file. In the case of multiple rowgroups per > files, it detects that the rowgroup can be pruned but then tell to the > drillbit to read the whole file which leads to performance issue. > Having multiple rowgroup per file helps to handle partitioned dataset and > still read only the relevant subset of data without ending with more file > than really needed. > Let's say for instance you have a Parquet file composed of RG1 and RG2 with > only one column a. Min/max in RG1 are 1-2 and min/max in RG2 are 2-3. > If I do "select a from file where a=3", today it will read the whole file, > with the patch it will only read RG2. > *For documentation* > Support / Other section in > https://drill.apache.org/docs/parquet-filter-pushdown/ should be updated. > After the fix files with multiple row groups will be supported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)