[ https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004845#comment-15004845 ]
ASF GitHub Bot commented on DRILL-3765: --------------------------------------- GitHub user jinfengni opened a pull request: https://github.com/apache/drill/pull/255 DRILL-3765: Move partitioning pruning to HepPlanner to avoid the perf… …ormance overhead for redundant rule execution. Add fall back option in planner. Run the regression/unit with this feature turned on /off. Please see DRILL-3765 for some performance number. Regarding the planner options: 1. planner.enable_join_hep_opt is replaced with planner.enable_hep_opt. This controls whether HepPlanner would be used in drill logical planning. If this option is off, it would fall back to Volcano planner. 2. Add new planning option : planner.enable_hep_partition_pruning, which will turn on/off this feature. If this option is off, it will fall back to the code prior to this patch. @amansinha100 , could you please review this patch? You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinfengni/incubator-drill DRILL-3765 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/255.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #255 ---- commit 739ac9482605cd2593b569ce2ec22fd176f6afbd Author: Jinfeng Ni <j...@apache.org> Date: 2015-11-03T23:51:10Z DRILL-3765: Move partitioning pruning to HepPlanner to avoid the performance overhead for redundant rule execution. Add fall back option in planner. ---- > Partition prune rule is unnecessary fired multiple times. > ---------------------------------------------------------- > > Key: DRILL-3765 > URL: https://issues.apache.org/jira/browse/DRILL-3765 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization > Reporter: Jinfeng Ni > Assignee: Jinfeng Ni > > It seems that the partition prune rule may be fired multiple times, even > after the first rule execution has pushed the filter into the scan operator. > Since partition prune has to build the vectors to contain the partition /file > / directory information, to invoke the partition prune rule unnecessary may > lead to big memory overhead. > Drill planner should avoid the un-necessary partition prune rule, in order to > reduce the chance of hitting OOM exception, while the partition prune rule is > executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)