[
https://issues.apache.org/jira/browse/ARROW-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17662409#comment-17662409
]
Rok Mihevc commented on ARROW-5386:
-----------------------------------
This issue has been migrated to [issue
#21844|https://github.com/apache/arrow/issues/21844] on GitHub. Please see the
[migration documentation|https://github.com/apache/arrow/issues/14542] for
further details.
> Making the rounding behavior of the buffer capacity configurable
> ----------------------------------------------------------------
>
> Key: ARROW-5386
> URL: https://issues.apache.org/jira/browse/ARROW-5386
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Java
> Reporter: Liya Fan
> Assignee: Liya Fan
> Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> In our scenario, the following code snippet is frequent in our code base:
> int requestSize = ...;
> if (requestSize <= allocator.getLimit() - allocator.getAllocatedMemory())
> { ArrowBuf buffer = allocator.buffer(requestSize); }
> However, it often causes OutOfMemoryException, due to Arrow's rounding
> behavior.
> For example, we have only 12 MB memory left, and we request a buffer with
> size 10 MB. Appearantly, there is sufficient memory to meet the request.
> However, the rounding behavior rounds the request size from 10 MB to 16 MB,
> and there is no 16 MB memory, so an OutOfMemoryException will be thrown.
> We propose two ways to solve this problem:
> 1. We provide a rounding option as an argument to the BaseAllocator#buffer
> method. There are two possible values for the rounding option: rounding up
> and rounding down. In the above scenario, the rounding down option can solve
> the problem.
> 2. We add a method to the allocator:
> int getRoundedSize(final int size, BaseAllocator.AllocationRoundingOption
> roundingOption)
> This method will give the rounded buffer size, given the initial request
> size. With this method, the user can freely adjust their request size to
> avoid OOM.
> To make it more convenient to use Arrow, we think both solutions should be
> implemented.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
