Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/164#issuecomment-38008356
@mengxr There are 3 reasons from me:
* The file length could be larger than `Int.maxValue`, and usually the
buffer size of `FSDataInputStream ` is smaller than `Int.maxValue`.
* `innerBuffer ` I used is an array of byte, we cannot use a length larger
than `Int.maxValue` to initialize the array.
* `byte[] innerBuffer = new byte[maxBufferLength];` could cause OOM if the
`maxBufferLength` is too large to fit in your current usable memory.
Indeed, the scenario that a file length is larger than `Int.maxValue` is
rare, but it could happen. Mahout also have encountered the problem, but they
just use `(int) fileLength` to cast a `long` to an `int`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
