[ https://issues.apache.org/jira/browse/DRILL-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142243#comment-15142243 ]
ASF GitHub Bot commented on DRILL-4363: --------------------------------------- Github user jinfengni commented on a diff in the pull request: https://github.com/apache/drill/pull/371#discussion_r52563951 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractGroupScan.java --- @@ -128,4 +128,12 @@ public int getOperatorType() { public List<SchemaPath> getPartitionColumns() { return Lists.newArrayList(); } + + /** + * By default, return null to indicate rowcount based prune is not supported. Each groupscan subclass should override, if it supports rowcount based prune. + */ + public GroupScan applyLimit(long maxRecords) { --- End diff -- Will do. > Apply row count based pruning for parquet table in LIMIT n query > ---------------------------------------------------------------- > > Key: DRILL-4363 > URL: https://issues.apache.org/jira/browse/DRILL-4363 > Project: Apache Drill > Issue Type: Improvement > Reporter: Jinfeng Ni > Assignee: Aman Sinha > Fix For: 1.6.0 > > > In interactive data exploration use case, one common and probably first query > that users would use is " SELECT * from table LIMIT n", where n is a small > number. Such query will give user idea about the columns in the table. > Normally, user would expect such query should be completed in very short > time, since it's just asking for small amount of rows, without any > sort/aggregation. > When table is small, there is no big problem for Drill. However, when the > table is extremely large, Drill's response time is not as fast as what user > would expect. > In case of parquet table, it seems that query planner could do a bit better > job : by applying row count based pruning for such LIMIT n query. The > pruning is kind of similar to what partition pruning will do, except that it > uses row count, in stead of partition column values. Since row count is > available in parquet table, it's possible to do such pruning. > The benefit of doing such pruning is clear: 1) for small "n", such pruning > would end up with a few parquet files, in stead of thousands, or millions of > files to scan. 2) execution probably does not have to put scan into multiple > minor fragments and start reading the files concurrently, which will cause > big IO overhead. 3) the physical plan itself is much smaller, since it does > not include the long list of parquet files, reduce rpc cost of sending the > fragment plans to multiple drillbits, and the overhead to > serialize/deserialize the fragment plans. > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)