[ https://issues.apache.org/jira/browse/DRILL-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16737743#comment-16737743 ]
Robert Hou commented on DRILL-5796: ----------------------------------- Found our documentation on this, and the default limit is 10K rowgroups. Which means we are limited to 10K files. > Filter pruning for multi rowgroup parquet file > ---------------------------------------------- > > Key: DRILL-5796 > URL: https://issues.apache.org/jira/browse/DRILL-5796 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet > Reporter: Damien Profeta > Assignee: Jean-Blas IMBERT > Priority: Major > Labels: ready-to-commit > Fix For: 1.15.0 > > > Today, filter pruning use the file name as the partitioning key. This means > you can remove a partition only if the whole file is for the same partition. > With parquet, you can prune the filter if the rowgroup make a partition of > your dataset as the unit of work if the rowgroup not the file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)