lidavidm commented on pull request #7387: URL: https://github.com/apache/arrow/pull/7387#issuecomment-642173870
So, gRPC has a default limit on the max size of any message on the wire. Flight disables this limit to allow you to write large record batches. However, we've found that it's still useful to enforce a (less restrictive) limit on the server side to help control the memory usage of a Java Flight server (so this applies to clients sending data via DoPut). Once you do that though, a Flight client must then limit the sizes of record batches it writes. While there are things like `RecordBatch.nbytes`, these aren't perfect estimates (e.g. it doesn't account for slicing/it really measures the memory usage of the underlying buffers). Being able to write a payload would help by letting the application construct a payload, check its size, then write it. In the common case, the payload is under the size limit and can be written, avoiding duplicate serialization. (Of course, this would need to get exposed to Python to be useful...) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
