V0ldek opened a new issue, #6099: URL: https://github.com/apache/arrow-rs/issues/6099
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** I am designing an API boundary that uses Arrow IPC to pass messages around. My requirement is that I need control at granularity of individual encapsulated messages, so for example I want to be able to force that at a given moment the initial Schema has definitely been written to the stream, or that all RecordBatches written thus far have been flushed. I am using a custom `Writer` implementation with `StreamWriter`. Currently this is impossible to achieve because the `StreamWriter` [internally wraps any passed `Writer` into a `BufWriter`](https://github.com/apache/arrow-rs/blob/ee5694078c86c8201549654246900a4232d531a9/arrow-ipc/src/writer.rs#L1035) and exposes no way to flush. **Describe the solution you'd like** The simplest solution would be to expose an `fn flush(&mut self)` on the `StreamWriter` that would force the internal `BufWriter` to flush to the user-provided `Writer`. **Describe alternatives you've considered** As an alternative and a bigger (IMO) improvement, an option could be given to `IpcWriteOptions` to disable internal buffering. In my case the custom `Writer` implementation is quite complex and well-optimised already, and the `BufWriter` on top does not actually give any benefit, rather creates an additional overhead. A simple flag `use_buf_writer` set to `true` by default but possible to override would solve this. Internally the `writer` field of `StreamWriter` would need to be changed to some more complicated writer, e.g. ```rust enum MaybeBuffered<W> { Raw(W), Buffered(BufWriter<W>) } ``` I'd be willing to make a PR for either of those changes, but wanted to ask you whether these make sense and would be accepted. This is quite important and blocking for me at the moment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
