[ https://issues.apache.org/jira/browse/ARROW-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wes McKinney resolved ARROW-4503. --------------------------------- Resolution: Fixed Fix Version/s: 0.14.0 Issue resolved by pull request 3925 [https://github.com/apache/arrow/pull/3925] > [C#] ArrowStreamReader allocates and copies data excessively > ------------------------------------------------------------ > > Key: ARROW-4503 > URL: https://issues.apache.org/jira/browse/ARROW-4503 > Project: Apache Arrow > Issue Type: Improvement > Components: C# > Reporter: Eric Erhardt > Assignee: Eric Erhardt > Priority: Major > Labels: performance, pull-request-available > Fix For: 0.14.0 > > Original Estimate: 48h > Time Spent: 11.5h > Remaining Estimate: 36.5h > > When reading `RecordBatch` instances using the `ArrowStreamReader` class, it > is currently allocating and copying memory 3 times for the data. > # It is allocating memory in order to [read the data from the > Stream|https://github.com/apache/arrow/blob/044b418fa108a57f0b4e2e887546cc3e68271397/csharp/src/Apache.Arrow/Ipc/ArrowStreamReader.cs#L72-L74], > and then reading from the Stream. (This should be the only allocation that > is necessary.) > # It then [creates a new > `ArrowBuffer.Builder`|https://github.com/apache/arrow/blob/044b418fa108a57f0b4e2e887546cc3e68271397/csharp/src/Apache.Arrow/Ipc/ArrowStreamReader.cs#L227-L228], > which allocates another `byte[]`, and calls `Append` on it, which copies the > values to the new `byte[]`. > # Finally, it then calls `.Build()` on the `ArrowBuffer.Builder`, which > [allocates memory from the MemoryPool, and then copies the intermediate > buffer|https://github.com/apache/arrow/blob/044b418fa108a57f0b4e2e887546cc3e68271397/csharp/src/Apache.Arrow/ArrowBuffer.Builder.cs#L112-L121] > into it. > > We should reduce this overhead to only allocating a single time (from the > MemoryPool), and not copying the data more times than necessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)