[ https://issues.apache.org/jira/browse/HIVE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16950277#comment-16950277 ]
Attila Zsolt Piros commented on HIVE-22330: ------------------------------------------- I think the intention to use *smallbuffers* was to avoid fragmented allocation. As you see when the *nextElemLength* is over the *MAX_SIZE_FOR_SMALL_BUFFER* (1MB) then there is a new buffer allocated for the data and in that case the only possible exception is OOM. So I think when the *smallbuffer* is chosen the same should be kept: no other exception should be thrown (beside the implicit OOM). > Maximize smallBuffer usage in BytesColumnVector > ----------------------------------------------- > > Key: HIVE-22330 > URL: https://issues.apache.org/jira/browse/HIVE-22330 > Project: Hive > Issue Type: Improvement > Affects Versions: 4.0.0 > Reporter: Karen Coppage > Assignee: Karen Coppage > Priority: Major > Attachments: HIVE-22330.01.patch > > > When BytesColumnVector is populated with values, it either creates a new > (byte[]) buffer object to help take the values, but if the values array is > <=1MB, then instead of creating a new buffer it reuses a single > "smallBuffer". Every time the smallBuffer is too small for the data we want > to store there, the size is doubled; when the size ends up larger than 1 GB > (or Integer.MAX_VALUE / 2) then the next time we try to double the size, > overflow occurs and an error is thrown. > A quick fix here is to set the smallBuffer size to Integer.MAX_VALUE in this > case. -- This message was sent by Atlassian Jira (v8.3.4#803005)