xuanyuanking commented on a change in pull request #23327: [SPARK-26222][SQL] Track file listing time URL: https://github.com/apache/spark/pull/23327#discussion_r242952170
########## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileIndex.scala ########## @@ -74,12 +74,11 @@ trait FileIndex { def partitionSchema: StructType /** - * Returns an optional metadata operation time, in nanoseconds, for listing files. + * Returns an optional file listing phase summary. * - * We do file listing in query optimization (in order to get the proper statistics) and we want - * to account for file listing time in physical execution (as metrics). To do that, we save the - * file listing time in some implementations and physical execution calls it in this method - * to update the metrics. + * We call this in physical execution while we want to account for the file listing time as + * metrics. If partition pruning happened in query planning, the phase also contains this + * part of the cost, otherwise, it only contains file listing time of FileIndex initialize. */ - def metadataOpsTimeNs: Option[Long] = None + def fileListingPhase: Option[QueryPlanningTracker.PhaseSummary] = None Review comment: Copy that, done in 1ed45b1. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org