alamb opened a new pull request, #7650: URL: https://github.com/apache/arrow-rs/pull/7650
# Which issue does this PR close? We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. Closes #NNN. # Rationale for this change Currently coalesce buffers views / data until there are enough rows and then concat's the results together This is wasteful because it 1. Buffers memory (has 2x the peak usage) 2. Copies the data twice We can make it faster and more memory efficient by directly creating the output array # What changes are included in this PR? Note this PR does NOT yet add specialized filtering -- instead it focuses on reducing the overhead of appending views. # Open questions: 1. There is substantial overlap / duplication with StringViewBuilder What is different: 1. Block size calculation management (aka look at the buffer sizes of the incoming buffers) 2. Finishing array allocates sufficient space for views # Are there any user-facing changes? If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
