[
https://issues.apache.org/jira/browse/TEZ-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saikat updated TEZ-2575:
------------------------
Attachment: TEZ-2575.patch
Summary of Change:
1. Added a new method spillSingleRecord()
2. In PipelinedSorter::sort() method added check for empty span, and return a
status to indicate this condition.
3. Check the status returned from sort() function to determine if the KV pair
didnot fit into the span, is so call spillSingleRecord().
4. relevant junit testcases added
> Handle KeyValue pairs size which do not fit in a single block
> -------------------------------------------------------------
>
> Key: TEZ-2575
> URL: https://issues.apache.org/jira/browse/TEZ-2575
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Saikat
> Assignee: Saikat
> Attachments: TEZ-2575.patch
>
>
> In the present implementation, the available buffer is divided into blocks
> (specified in the constructor for pipeline sort). and a linked list of these
> block byte buffers is maintained.
> A span is created out of the buffers.
> The present logic, doesnot handle scenario where a single key-value pair size
> doesnot fit into any of the blocks.
> example if 1mb total memory is divided into 4 blocks, (256 kb each),
> if a single KV pair is greater than the blocksize(~ignoring meta data size),
> then it fails with buffer exceptions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
