[
https://issues.apache.org/jira/browse/SPARK-20408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520976#comment-16520976
]
Apache Spark commented on SPARK-20408:
--------------------------------------
User 'xuanyuanking' has created a pull request for this issue:
https://github.com/apache/spark/pull/21618
> Get glob path in parallel to reduce resolve relation time
> ---------------------------------------------------------
>
> Key: SPARK-20408
> URL: https://issues.apache.org/jira/browse/SPARK-20408
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Li Yuanjian
> Priority: Major
>
> The datasource read from a path with wildcard like below will cause a long
> time waiting(each star may represent 100~1000 file or dir), especially in a
> cross region env, driver and hdfs in different region, the drawback will
> enlarge.
> bq. spark.read.text("/log/product/201704/\*/\*/\*/\*")
> Optimize strategy is same with bulkListLeafFiles in InMemoryFileIndex, get
> the wildcard path in parallel.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
