Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21618#discussion_r216118629 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -724,4 +726,37 @@ object DataSource extends Logging { """.stripMargin) } } + + /** + * Return all paths represented by the wildcard string. + * This will be done in main thread by default while the value of config + * `spark.sql.sources.parallelGetGlobbedPath.numThreads` > 0, a local thread + * pool will expand the globbed paths. + */ + private def getGlobbedPaths( + sparkSession: SparkSession, --- End diff -- Can you drop `sparkSession`, then add `parallelGetGlobbedPathThreshold`?
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org