FWIW arrow-rs may split RecordBatch in order to comply with gRPC message size limits. The threshold is configurable [1] and clients desiring full control can still use the lower-level IPC interfaces [2], however, many gRPC implementations, and some proxies, will by default reject messages larger than a couple of MB.
Kind Regards, Raphael [1]: https://docs.rs/arrow-flight/latest/arrow_flight/encode/struct.FlightDataEncoderBuilder.html#method.with_max_flight_data_size [2]: https://docs.rs/arrow-ipc/latest/arrow_ipc/writer/struct.IpcDataGenerator.html On 15 November 2023 20:22:28 GMT, Nick Hughes <n...@intervaliq.com> wrote: >Good to know! Thanks David. > > > >On Wed, Nov 15, 2023 at 12:02 PM David Li <lidav...@apache.org> wrote: > >> The C++/Java/Go/Python Flight implementations do not chunk the data for >> you, so you will get the full batch. >> >> On Wed, Nov 15, 2023, at 14:40, Nick Hughes wrote: >> >> Hi, >> >> If I write a RecordBatch to a stream from a Flight server >> using MetadataRecordBatchWriter.write_batch and then attempt to read this >> from a client using FlightStreamReader.read_chunk, is it guaranteed that >> the returned FlightStreamChunk will contain the original RecordBatch in its >> entirety, or is there a possibility it may be partitioned over several >> chunks? >> >> Thanks, >> >> Nick >> >> >> >>
