shahidki31 commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation URL: https://github.com/apache/spark/pull/24715#discussion_r315120002
########## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala ########## @@ -619,3 +619,34 @@ object DataSourceStrategy { (nonconvertiblePredicates ++ unhandledPredicates, pushedFilters, handledFilters) } } + + +/** + * Defines default table statistics if table statistics are not available. + */ +class DetermineTableStats(session: SparkSession) extends Rule[LogicalPlan] { + + private val conf = session.sessionState.conf + + override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators { + // For the data source table, we only recalculate the table statistics Review comment: Yes. Both the PR checks, if using `CatalogFileIndex` and table stats is empty, then fallback to HDFS to get the statistics. And both the PRs never return statistics from `CatalogFileIndex` irrespective of the stats is empty or not, if enable fallback. That is why I said the behavior is same. The pr looks good. only concern is, if the catalogFileIndex already has statistics, then in the rules, we shouldn't fallback to HDFS, because that was the issue with #22502. Thanks. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org