Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/43#discussion_r10188872
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager,
diskManager: DiskBlockManage
override def getBytes(blockId: BlockId): Option[ByteBuffer] = {
val segment = diskManager.getBlockLocation(blockId)
val channel = new RandomAccessFile(segment.file, "r").getChannel()
- val buffer = try {
- channel.map(MapMode.READ_ONLY, segment.offset, segment.length)
- } finally {
- channel.close()
+
+ val buffer =
+ // For small files, directly read rather than memory map
+ if (segment.length < 2 * 4096) {
--- End diff --
Another way to think about it is this. This code path is really just to
protect users who are doing something that already leads to very poor
performance - shuffles where they use way too many partitions and the files are
really tiny. The best thing a user can do there is to coalesce the number of
mappers or reducers to get much better peformance. If they can figure that out,
there is no reason to ever tune this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
