alamb commented on code in PR #10522:
URL: https://github.com/apache/arrow-rs/pull/10522#discussion_r3703283498
##########
arrow-ipc/src/reader.rs:
##########
@@ -1804,6 +1804,36 @@ pub(crate) enum IpcMessage {
},
}
+/// Upper bound on how much memory a single message length is allowed to
reserve before any
+/// of the bytes it promises have been read.
+///
+/// Message lengths come from the stream itself, so they cannot be trusted: a
truncated or
+/// corrupted stream can declare a body of arbitrary size. Reserving that up
front turns a
+/// malformed input into an allocation failure, which aborts the process
rather than
+/// returning an error the caller can handle. Beyond this size the buffer
grows as the data
+/// arrives instead, so an implausible length costs one bounded allocation and
then fails as
+/// a short read.
+///
+/// The value trades the size of that bounded allocation against how large a
body still gets
+/// read in a single allocation: bodies up to this size behave exactly as
before, larger ones
+/// grow geometrically (`MutableBuffer::reserve` doubles) and pay the
reallocations.
+const MAX_PREALLOC_BYTES: usize = 64 * 1024 * 1024;
+
+/// Reads exactly `len` bytes of message body, without reserving `len` before
reading it.
+fn read_body_bounded<R: Read>(reader: &mut R, len: usize) ->
Result<MutableBuffer, ArrowError> {
Review Comment:
this seems reasonable -- basically it reallocates in 64MB chunks when
reading
for a 1GB message, this will result in quite a few reallocations, but I
suppose the idea is that such message sizes are rare.
I was thinking about some usecase reading 8K rows record batch where each
row has a 1MB document -- that is 8GB and would likely reallocate 128K times,
which is probably not great for performance
Is there some way to use a built in rust function (or maybe employ a
doubling strategy) or something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
