[ https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111794#comment-16111794 ]
Muhammad Samir Khan commented on TEZ-3809: ------------------------------------------ Took a heap dump on ordered word count before final merge. In the after case, one of the outputs was written to disk instead of kept in memory and that is why it has 37 entries. Before: Class Name | Shallow Heap | Retained Heap | Percentage ----------------------------------------------------------------------------------------------------------------------------------------------------------- java.lang.Thread @ 0x5d2c473f8 ShuffleAndMergeRunner {Tokenizer} Thread | 120 | 2,229,207,992 | 96.48% |- java.util.ArrayList @ 0x73f978f10 | 24 | 2,229,206,760 | 96.48% | '- java.lang.Object[38] @ 0x73f979130 | 168 | 2,229,206,736 | 96.48% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e4a88898| 32 | 68,078,192 | 2.95% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0b260| 32 | 67,839,520 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e4a888b8| 32 | 67,700,608 | 2.93% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x73f9db168| 32 | 67,500,816 | 2.92% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60ab36218| 32 | 67,408,704 | 2.92% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631deed28| 32 | 67,367,424 | 2.92% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x743b86ee0| 32 | 67,337,936 | 2.91% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60af3a698| 32 | 67,300,896 | 2.91% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0c5b8| 32 | 67,282,464 | 2.91% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60ab33140| 32 | 67,264,304 | 2.91% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e4a88878| 32 | 67,127,368 | 2.91% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0b218| 32 | 67,098,216 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0c6c8| 32 | 67,064,504 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d239a6c8| 32 | 67,003,776 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d23b7e10| 32 | 66,965,296 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631def2b8| 32 | 66,928,032 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60ab351d0| 32 | 66,916,896 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x74805dfb8| 32 | 66,886,272 | 2.89% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60af39598| 32 | 66,718,800 | 2.89% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x73fb0fb78| 32 | 66,688,296 | 2.89% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0c4b0| 32 | 66,656,312 | 2.88% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60af39578| 32 | 66,629,936 | 2.88% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631deec30| 32 | 66,584,576 | 2.88% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0c680| 32 | 66,537,624 | 2.88% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60af3a620| 32 | 66,529,416 | 2.88% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x631e0c570| 32 | 66,229,848 | 2.87% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x73f9d8f08| 32 | 65,616,336 | 2.84% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60af39500| 32 | 63,794,840 | 2.76% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e4a7ed98| 32 | 60,026,520 | 2.60% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60ab36280| 32 | 50,731,856 | 2.20% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d23c0da8| 32 | 48,164,288 | 2.08% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d23b85b8| 32 | 46,712,064 | 2.02% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d23b8310| 32 | 46,564,520 | 2.02% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e4a7e528| 32 | 45,990,432 | 1.99% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60aec1cf0| 32 | 45,642,312 | 1.98% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5d23c0a48| 32 | 8,122,976 | 0.35% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60ab362a0| 32 | 3,712,608 | 0.16% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x60aec1ca8| 32 | 481,784 | 0.02% | '- Total: 38 entries | | | ----------------------------------------------------------------------------------------------------------------------------------------------------------- After: Class Name | Shallow Heap | Retained Heap | Percentage ----------------------------------------------------------------------------------------------------------------------------------------------------------- java.lang.Thread @ 0x5ce7a24e8 ShuffleAndMergeRunner {Tokenizer} Thread | 120 | 2,182,643,840 | 96.38% |- java.util.ArrayList @ 0x7343b6ae8 | 24 | 2,182,642,112 | 96.38% | '- java.lang.Object[37] @ 0x7343b6c18 | 168 | 2,182,642,088 | 96.38% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767263f70| 32 | 68,078,192 | 3.01% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x719c6dbe8| 32 | 67,839,512 | 3.00% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x7672080b0| 32 | 67,700,608 | 2.99% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767204a18| 32 | 67,500,816 | 2.98% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e3138258| 32 | 67,408,704 | 2.98% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x73cc9bb50| 32 | 67,367,416 | 2.97% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e3140588| 32 | 67,337,928 | 2.97% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767262eb8| 32 | 67,300,888 | 2.97% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x7155fa180| 32 | 67,282,464 | 2.97% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767232980| 32 | 67,264,296 | 2.97% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x7632001d8| 32 | 67,127,360 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x738be08c8| 32 | 67,098,208 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767204c50| 32 | 67,064,496 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x76725fb38| 32 | 67,003,776 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e31303e8| 32 | 66,965,296 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x763200058| 32 | 66,928,024 | 2.96% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e3140160| 32 | 66,916,888 | 2.95% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x715866b18| 32 | 66,886,272 | 2.95% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x721d7d950| 32 | 66,718,792 | 2.95% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x719c6b988| 32 | 66,688,296 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x734290ec8| 32 | 66,656,312 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5ce73f150| 32 | 66,629,936 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767261d48| 32 | 66,584,576 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x721d7b6f0| 32 | 66,537,624 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x7632000d0| 32 | 66,529,416 | 2.94% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767204dc0| 32 | 66,229,848 | 2.92% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x7349ccf30| 32 | 65,616,336 | 2.90% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x767260ca8| 32 | 63,794,832 | 2.82% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x76725f9f0| 32 | 60,026,512 | 2.65% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e31405c8| 32 | 50,731,848 | 2.24% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5ce73a870| 32 | 48,164,288 | 2.13% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5ce7525d8| 32 | 46,712,056 | 2.06% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e053c7d0| 32 | 45,990,432 | 2.03% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5ce74ae60| 32 | 45,642,312 | 2.02% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e317aba0| 32 | 8,122,976 | 0.36% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e316e3f8| 32 | 3,712,600 | 0.16% | |- org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput @ 0x5e3177a48| 32 | 481,784 | 0.02% | '- Total: 37 entries | | | ----------------------------------------------------------------------------------------------------------------------------------------------------------- > The buffer size allocated for InMemoryMapOutput can be optimized > ---------------------------------------------------------------- > > Key: TEZ-3809 > URL: https://issues.apache.org/jira/browse/TEZ-3809 > Project: Apache Tez > Issue Type: Bug > Reporter: Muhammad Samir Khan > Assignee: Muhammad Samir Khan > Attachments: TEZ-3809.001.patch > > > Related jiras: TEZ-3752 and TEZ-3732. > -When shuffling input to memory, the decompressed length is used to create > the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 > bytes less (the IFile header). These 4 bytes can optimized and, in an extreme > case of 10,000,000 fetches, can save ~38 MB (TEZ-3732). > -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers > to allocate the new InMemoryMapOutput. However, each input has two > EOF_MARKERs while only two are needed at the end. > -InMemoryWriter wraps the output BoundedByteArrayOutputStream in > IFileOutputStream which will write checksum at close. This creates an > inconsistency between the primary input buffers which don't have checksum and > the merged buffers which do. IFileOutputStream wrap can be removed to save 4 > bytes per merged buffers. > -InMemoryWriter does not account for two EOF_MARKERs written at close() in > its accounting so that the getRawLength() method is off by two bytes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)