[
https://issues.apache.org/jira/browse/TEZ-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-1228:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.5.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks [~sseth] and [~gopalv] for the review.
Review details: https://reviews.apache.org/r/23227/
commit 2316c1dc951996765cbd887f6c3ab80a8e784fef
Author: Rajesh Balamohan <[email protected]>
Date: Fri Jul 11 11:18:27 2014 +0530
TEZ-1228. Define a memory & merge optimized vertex-intermediate file format for
Tez
> Prototype IFile : Define a memory & merge optimized vertex-intermediate file
> format for Tez
> -------------------------------------------------------------------------------------------
>
> Key: TEZ-1228
> URL: https://issues.apache.org/jira/browse/TEZ-1228
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Labels: perfomance
> Fix For: 0.5.0
>
> Attachments: TEZ-1228-IFile.pdf, TEZ-1228.1.patch, TEZ-1228.2.patch,
> TEZ-1228.3.patch, TEZ-1228.4.patch, TEZ-1228.5.patch, TEZ-1228.WIP.1.patch,
> TEZ-1228.WIP.2.patch
>
>
> The current vertex-intermediate format used all across Tez is a flat file of
> variable length k,v pairs. For a significant number of use-cases, in
> particular the sorted output phase, a large number of consecutive identical
> keys are found within the same stream. The IFile format ends up writing each
> key out fully into the stream to generate (K,V) pairs instead of ordering it
> into a more efficient K, {V1, .. Vn} list.
> This duplication of key data needs larger buffers to hold in memory and
> requires comparison between keys known to be identical while doing a merge
> sort.
> This bug tracks the building of a prototype IFile format which is optimized
> for lower uncompressed sizes within memory buffers and less compute intensive
> to perform merge sorts during the reducer phase.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
