[
https://issues.apache.org/jira/browse/HDFS-10327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257170#comment-15257170
]
Chris Nauroth commented on HDFS-10327:
--------------------------------------
Hello [~SonGuko]. I'm not familiar with how this feature of Spark works. Does
Spark use some kind of special serialization format that persists its own
directory information as part of the file contents? If so, then I think it's
unlikely that we would incorporate understanding of that specific file format
into HDFS itself. Currently there is a separation of concerns, such that HDFS
is responsible for storage of bytes, and individual applications are
responsible for the interpretation of the meaning of those bytes. There is no
precedent for violating that separation of concerns. I don't think there is a
compelling reason to start doing so.
> Open files in WEBHDFS which are stored in folders by Spark
> ----------------------------------------------------------
>
> Key: HDFS-10327
> URL: https://issues.apache.org/jira/browse/HDFS-10327
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Reporter: Thomas Hille
> Labels: features
>
> When Spark saves a file in HDFS it creates a directory which includes many
> parts of the file. When you read it with spark programmatically, you can read
> this directory as it is a normal file.
> If you try to read this directory-style file in webhdfs, it returns
> {"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException","message":"Path
> is not a file: [...]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
