[ https://issues.apache.org/jira/browse/ARROW-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rok Mihevc updated ARROW-5386: ------------------------------ External issue URL: https://github.com/apache/arrow/issues/21844 > Making the rounding behavior of the buffer capacity configurable > ---------------------------------------------------------------- > > Key: ARROW-5386 > URL: https://issues.apache.org/jira/browse/ARROW-5386 > Project: Apache Arrow > Issue Type: Improvement > Components: Java > Reporter: Liya Fan > Assignee: Liya Fan > Priority: Major > Labels: pull-request-available > Time Spent: 7h 40m > Remaining Estimate: 0h > > In our scenario, the following code snippet is frequent in our code base: > int requestSize = ...; > if (requestSize <= allocator.getLimit() - allocator.getAllocatedMemory()) > { ArrowBuf buffer = allocator.buffer(requestSize); } > However, it often causes OutOfMemoryException, due to Arrow's rounding > behavior. > For example, we have only 12 MB memory left, and we request a buffer with > size 10 MB. Appearantly, there is sufficient memory to meet the request. > However, the rounding behavior rounds the request size from 10 MB to 16 MB, > and there is no 16 MB memory, so an OutOfMemoryException will be thrown. > We propose two ways to solve this problem: > 1. We provide a rounding option as an argument to the BaseAllocator#buffer > method. There are two possible values for the rounding option: rounding up > and rounding down. In the above scenario, the rounding down option can solve > the problem. > 2. We add a method to the allocator: > int getRoundedSize(final int size, BaseAllocator.AllocationRoundingOption > roundingOption) > This method will give the rounded buffer size, given the initial request > size. With this method, the user can freely adjust their request size to > avoid OOM. > To make it more convenient to use Arrow, we think both solutions should be > implemented. > -- This message was sent by Atlassian Jira (v8.20.10#820010)