[jira] [Commented] (DRILL-8290) Short cut recursive file listings for LIMIT 0 queries

[ASF GitHub Bot (Jira)]

    [ 
https://issues.apache.org/jira/browse/DRILL-8290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685716#comment-17685716
 ] 

ASF GitHub Bot commented on DRILL-8290:
---------------------------------------

jnturton commented on code in PR #2636:
URL: https://github.com/apache/drill/pull/2636#discussion_r1099737668


##########
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java:
##########
@@ -645,10 +646,18 @@ public Void visitOp(PhysicalOperator op, 
Collection<PhysicalOperator> collection
       }
       return null;
     }
-
   }
 
   protected Pair<SqlNode, RelDataType> validateNode(SqlNode sqlNode) throws 
ValidationException, RelConversionException, ForemanSetupException {
+    if (context.getOptions().getOption(ExecConstants.FILE_LISTING_LIMIT0_OPT)) 
{
+      // Check for a LIMIT 0 in the root portion of the query before validation
+      // because validation of the query's FROM clauses will already trigger
+      // the recursive listing files to which FILE_LISTING_LIMIT0_OPT is meant
+      // to apply.
+      boolean rootSelectLimit0 = FindLimit0SqlVisitor.containsLimit0(sqlNode);
+      context.getPlannerSettings().setRootSelectLimit0(rootSelectLimit0);

Review Comment:
   @vvysotskyi I've made use of a query option, how do things look now?





> Short cut recursive file listings for LIMIT 0 queries
> -----------------------------------------------------
>
>                 Key: DRILL-8290
>                 URL: https://issues.apache.org/jira/browse/DRILL-8290
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Query Planning &amp; Optimization
>    Affects Versions: 1.20.2
>            Reporter: James Turton
>            Assignee: James Turton
>            Priority: Minor
>             Fix For: 1.21.0
>
>
> The existing LIMIT 0 query optimisations do not prevent a query run against 
> the top of a deep DFS directory tree from recursively listing FileStatuses 
> for everything within it using a pool of worker threads. This Issue proposes 
> a new optimisation whereby such queries will recurse into the directory tree 
> on a single thread that returns as soon as any single FileStatus has been 
> obtained.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)