crepererum opened a new issue, #8917: URL: https://github.com/apache/arrow-rs/issues/8917
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Record batches sent over IPC can be compressed (currently via LZ4 or ZSTD). Support for compressed can be enabled via [Cargo feature flags](https://github.com/apache/arrow-rs/blob/88b135a2fadfce80d7962ca0d91473f9c62337fb/arrow-ipc/Cargo.toml#L45-L46), but NOT at runtime. Compression however is somewhat tricky, since decompressing untrusted data may easily spend many CPU cycles and result in large outputs (casually called "ZIP bombs"). Hence it would be nice to control -- from a readers PoV -- if compression is accepted or not. **Describe the solution you'd like** Add options to [`StreamReader`](https://docs.rs/arrow-ipc/latest/arrow_ipc/reader/struct.StreamReader.html) and [`FileReader`](https://docs.rs/arrow-ipc/latest/arrow_ipc/reader/struct.FileReader.html) to control if compression is accepted or not. **Describe alternatives you've considered** - **feature flags only:** Just rely on the feature flags currently exposed. This however is two downsides: a) feature-unification in a larger Cargo project may accidentally opt-in these flags and b) one single software artifact (e.g. a server communicating with an untrusted client and a trusted internal party in the same security domain) may have different requirements for different connections - **expose decompressed size:** let the reader judge the impact of the compression by exposing the "size after decompression" / "uncompressed size". While this somewhat circumvents the unknown memory size after decompression, it only partly addresses the CPU overhead. **Additional context** - #7816 is somewhat related -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
