[ https://issues.apache.org/jira/browse/SPARK-24200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16466700#comment-16466700 ]
Hyukjin Kwon commented on SPARK-24200: -------------------------------------- If it's a question for now, I would suggest to ask it to mailing list first before filing a JIRA as an issue. > Read subdirectories with out asterisks > -------------------------------------- > > Key: SPARK-24200 > URL: https://issues.apache.org/jira/browse/SPARK-24200 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.3.0 > Reporter: kumar > Priority: Major > > String folder = "/Users/test/data/* /* "; > sparkContext.textFile(folder, 1).toJavaRDD() > Is asterisks mandatory to read a folder -Yes, otherwise it does not read > files under subdirectories. > What if I get a folder which is having more subdirectories than the number of > asterisks mentioned ? How to handle this scenario ? > For example: > 1) {{/Users/test/data/}} This would work ONLY if I get data as > /Users/test/data/folder1/file.txt > 2)How to make this expression as *generic* ? It should still work if I get a > folder as: {{/Users/test/data/folder1/folder2/folder3/folder4}} > My input folder structure is not same all the time. > Is there anything exists in Spark to handle this kind of scenario ? I know > you might have thought about this, but i am wondering why this has not been > implemented ? -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org