Hi Ling, Yes! It is because of performance concerns. We want to keep and merge map outputs in memory as much as we can. The amount of memory reserved for this purpose is configurable. Obviously storing fetched map outputs on disk, then reading them back from disk to merge them and then write out back to disk, is a lot more expensive than if it were done in memory.
Please let us know if you find there was an opportunity to keep the map output in memory but we did not, and instead shuffled to disk. Thanks Ravi ________________________________ From: Ling Kun <lkun.e...@gmail.com> To: mapreduce-dev@hadoop.apache.org Sent: Monday, March 11, 2013 5:27 AM Subject: Why In-memory Mapoutput is necessary in ReduceCopier Dear all, I am focusing on the Mapoutput copier implementation. This part of code will try to get mapoutputs, and merge them into a file that can feed to reduce functions. I have the following questions. 1. All the local file mapoutput data will be merged together by the LocalFSMerge, and the in-memory mapout will be merged by InMemFSMergeThread. For the InMemFSMergeThread, there is also a writer object which write the result to outputPath ( ReduceTask.java Line 2843). It seems after merging, in-memory mapoutput and local file mapoutput data will all be stored in local file system. Why not just using the local file for all mapoutput data. 2. After using http to get some fragment of a map output file, some of the mapoutput data will be selected and keep in memory, while others are directly write to local disk of reducers. Which mapoutput wil be kept in memory is determined in MapOutputCopier.getMapOutput(), this method will call ramManager.canFitInMemory(). why not store all the data to disk? 3. According to the comment, Hadoop will put a file in memory if it meets: a, the size of the (decompressed) file should be less than 25% of the total inmem fs; b, there is space available in the inmem fs. Why ? Is it because of the performance? Thanks yours, Ling Kun -- http://www.lingcc.com
