Github user ehiggs commented on the issue:
https://github.com/apache/spark/pull/19030
This will be fixed upstream in HDFS-12344.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user ehiggs closed the pull request at:
https://github.com/apache/spark/pull/19030
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
GitHub user ehiggs opened a pull request:
https://github.com/apache/spark/pull/19030
[SPARK-21817][SQL] Pass FSPermission to LocatedFileStatus from
InMemoryFileIndex
## What changes were proposed in this pull request?
Pass `FSPermission` to `LocatedFileStatus` as `Hadoop
Github user ehiggs commented on the pull request:
https://github.com/apache/spark/pull/4204#issuecomment-119210146
Closed as it's not the correct approach.
Thanks @srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on G
Github user ehiggs closed the pull request at:
https://github.com/apache/spark/pull/4204
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user ehiggs commented on the pull request:
https://github.com/apache/spark/pull/4204#issuecomment-71740260
Thanks for your feedback.
So the `FileInputFormat` is responsible for sorting the file pieces. I
think this means any file format that one expects `sortByKey` to
Github user ehiggs commented on the pull request:
https://github.com/apache/spark/pull/4204#issuecomment-71448742
Well, I could submit a patch to hadoop-common to do the sort in
`o.a.h.fs.FileSystem`, and that would fix it basically everywhere. However, I
had assumed that hadoop
Github user ehiggs commented on the pull request:
https://github.com/apache/spark/pull/4204#issuecomment-71438376
The recommendation on the mailing list was to provide a `FileSystem` that
could be used to from the config `spark.hadoop.fs.file.impl`.
`Path` doesn't ha
GitHub user ehiggs opened a pull request:
https://github.com/apache/spark/pull/4204
SPARK-5300 Add LocalFileSystem which will return file parts in the corre...
...ct order.
We override listLocatedStatus and slurp up the iterator from the Hadoop
LocalFileSystem version
