Github user kmanamcheri commented on the issue: https://github.com/apache/spark/pull/22614 @gatorsmile, @tejasapatil was reviewing the code before I added the new config option. I have asked him to review the new code. Lets see what his thoughts are on that. I have also asked him clarification on what he means by exponential backoff with retries. I want to take a step back and revisit [SPARK-17992](https://issues.apache.org/jira/browse/SPARK-17992) and in particular [one of the comments](https://github.com/apache/spark/pull/15673#issuecomment-257120666) from @ericl > For large tables, the degraded performance should be considered a bug as well. > > How about this. > >If direct sql is disabled, log a warning about degraded performance with this flag and fall back to >fetching all partitions. >If direct sql is enabled, crash with a message suggesting to disable filesource partition management >and report a bug. >That way, we will know if there are cases where metastore pruning fails with direct sql enabled. It looks like a compromise was reached where we don't support fetching all the time (and only for a subset of cases). My suggested fix is a cleaner way of approaching it through a SQLConf instead of looking at the Hive config. Thoughts @mallman @ericl
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org