[ https://issues.apache.org/jira/browse/ARROW-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Teotia updated ARROW-1533: ------------------------------------ Summary: [JAVA] realloc should consider the existing buffer capacity for computing target memory requirement (was: As part of buffer transfer (transferTo function), we should transfer the state for realloc) > [JAVA] realloc should consider the existing buffer capacity for computing > target memory requirement > --------------------------------------------------------------------------------------------------- > > Key: ARROW-1533 > URL: https://issues.apache.org/jira/browse/ARROW-1533 > Project: Apache Arrow > Issue Type: Bug > Components: Java - Vectors > Reporter: Siddharth Teotia > Assignee: Siddharth Teotia > > We recently encountered a problem when we were trying to add JSON files with > complex schema as datasets. > Initially we started with a Float8Vector with default memory allocation of > (4096 * 8) 32KB. > Went through several iterations of setSafe() to trigger a realloc() from 32KB > to 64KB. > Another round of setSafe() calls to trigger a realloc() from 64KB to 128KB > After that we encountered a BigInt and promoted our vector to UnionVector. > This required us to create a UnionVector with BigIntVector and Float8Vector. > The latter required us to transfer the Float8Vector we were earlier working > with to the Float8Vector inside the Union. > As part of transferTo(), the target Float8Vector got all the ArrowBuf state > (capacity, buffer contents) etc transferred from the source vector. > Later, a realloc was triggered on the Float8Vector inside the UnionVector. > The computation inside realloc() to determine the amount of memory to be > reallocated goes wrong since it makes the decision based on > allocateSizeInBytes -- although this vector was created as part of transfer() > from 128KB source vector, allocateSizeInBytes is still at the initial/default > value of 32KB > We end up allocating a 64KB buffer and attempt to copy 128KB over 64KB and > seg fault when invoking setBytes(). > There is a wrong assumption in realloc() that allocateSizeInBytes is always > equal to data.capacity(). The particular scenario described above exposes > where this assumption could go wrong. -- This message was sent by Atlassian JIRA (v6.4.14#64029)