Eric Erhardt created ARROW-4503:
-----------------------------------
Summary: [C#] ArrowStreamReader allocates and copies data
excessively
Key: ARROW-4503
URL: https://issues.apache.org/jira/browse/ARROW-4503
Project: Apache Arrow
Issue Type: Improvement
Components: C#
Reporter: Eric Erhardt
When reading `RecordBatch` instances using the `ArrowStreamReader` class, it is
currently allocating and copying memory 3 times for the data.
# It is allocating memory in order to [read the data from the
Stream|https://github.com/apache/arrow/blob/044b418fa108a57f0b4e2e887546cc3e68271397/csharp/src/Apache.Arrow/Ipc/ArrowStreamReader.cs#L72-L74],
and then reading from the Stream. (This should be the only allocation that is
necessary.)
# It then [creates a new
`ArrowBuffer.Builder`|https://github.com/apache/arrow/blob/044b418fa108a57f0b4e2e887546cc3e68271397/csharp/src/Apache.Arrow/Ipc/ArrowStreamReader.cs#L227-L228],
which allocates another `byte[]`, and calls `Append` on it, which copies the
values to the new `byte[]`.
# Finally, it then calls `.Build()` on the `ArrowBuffer.Builder`, which
[allocates memory from the MemoryPool, and then copies the intermediate
buffer|https://github.com/apache/arrow/blob/044b418fa108a57f0b4e2e887546cc3e68271397/csharp/src/Apache.Arrow/ArrowBuffer.Builder.cs#L112-L121]
into it.
We should reduce this overhead to only allocating a single time (from the
MemoryPool), and not copying the data more times than necessary.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
