Cyrille Chépélov created TEZ-2256: ------------------------------------- Summary: Avoid use of BufferTooSmallException to signal end of buffer in UnorderedPartitionedKVWriter Key: TEZ-2256 URL: https://issues.apache.org/jira/browse/TEZ-2256 Project: Apache Tez Issue Type: Improvement Affects Versions: 0.6.0, 0.7.0 Reporter: Cyrille Chépélov Priority: Minor
UnorderedPartitionedKVWriter delegates serialization to the application, passing it a private ByteArrayOutputStream. In case the buffer is exhausted, ByteArrayOutputStream signals that with a private BufferTooSmallException, which can be seen but not dealt with by the application. As [~cwensel] pointed out, when the application is in fact a complex framework, there is no way to distinguish this exception from a real failure, which compels logging the full stack even for reasonable events such as "buffer complete". Suggested approach: set a "complete" flag in ByteArrayOutputStream that disables any further output, and replace BufferTooSmallException (BTSE) handling by checking that flag. [~sseth] suggested checking out SortedOutput as well, as the mechanisms there should be similar. I'll give this a go this week. -- This message was sent by Atlassian JIRA (v6.3.4#6332)