crepererum opened a new issue, #8010: URL: https://github.com/apache/arrow-rs/issues/8010
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Add a way to automatically chunk parquet data based on content to allow the usage of content-addressable storage (CAS). This shall reduce upload throughput for subsequent, partial changes of parquet files (e.g. just updating a few rows). This is basically https://github.com/apache/arrow/issues/45750 but for arrow-rs. **Describe the solution you'd like** Add some kind of wrapper or parameter to the parquet writer that automatically chunks the data. **Describe alternatives you've considered** \- **Additional context** - https://huggingface.co/blog/parquet-cdc - https://github.com/apache/arrow/pull/45360 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
