thisisnic commented on issue #18944: URL: https://github.com/apache/arrow/issues/18944#issuecomment-5990499547
Closing this as superseded. The cause Weston identified (backpressure measured in batches rather than bytes) was addressed in the dataset writer in 16.0.0, and the remaining memory growth people see with large parquet inputs is tracked in #39808. If re-partitioning still runs out of memory on a current release, please open a new issue with a reprex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
