[ https://issues.apache.org/jira/browse/ARROW-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266688#comment-17266688 ]
R J commented on ARROW-11279: ----------------------------- I think this can be solved by incrementally calculating array levels per column as definition levels appear distinct per column. First time contributing but I have prepared a pull request > [Rust][Parquet] ArrowWriter Definition Levels Memory Usage > ---------------------------------------------------------- > > Key: ARROW-11279 > URL: https://issues.apache.org/jira/browse/ARROW-11279 > Project: Apache Arrow > Issue Type: Improvement > Components: Rust > Reporter: R J > Priority: Minor > Original Estimate: 0.25h > Remaining Estimate: 0.25h > > In the rust implementation of an Arrow RecordBatch writer to parquet > (3.0.0-SNAPSHOT), the ArrowWriter::write call potentially allocates more > memory than required. > For a RecordBatch with m rows and n columns, ArrowWriter::write allocates m*n > definition levels, leading to m times the required memory usage. -- This message was sent by Atlassian Jira (v8.3.4#803005)