Muhammad Samir Khan created TEZ-3809: ----------------------------------------
Summary: The buffer size allocated for InMemoryMapOutput can be optimized Key: TEZ-3809 URL: https://issues.apache.org/jira/browse/TEZ-3809 Project: Apache Tez Issue Type: Bug Reporter: Muhammad Samir Khan Assignee: Muhammad Samir Khan Related jiras: TEZ-3752 and TEZ-3732. -When shuffling input to memory, the decompressed length is used to create the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 bytes less (the IFile header). These 4 bytes can optimized and, in an extreme case of 10,000,000 fetches, can save ~38 MB (TEZ-3732). -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers to allocate the new InMemoryMapOutput. However, each input has two EOF_MARKERs while only two are needed at the end. -InMemoryWriter wraps the output BoundedByteArrayOutputStream in IFileOutputStream which will write checksum at close. This can create a mismatch between the primary input buffers which don't have checksum and the merged buffers which do. IFileOutputStream wrap can be removed to save 4 bytes per merged buffers. -- This message was sent by Atlassian JIRA (v6.4.14#64029)