Cintu07 commented on PR #11224: URL: https://github.com/apache/arrow-rs/pull/11224#issuecomment-5874268854
rolling the block id costs one compare per row. the loop already appends into values and measures the length after, so the check is start_offset + decoded_len against the cap, right next to the one this pr adds. when it trips, the value that just landed gets copied into a fresh buffer and the block id goes up. one memcpy per 2 GiB. make_view takes that block id instead of the hardcoded 0. values_capacity is the other half of it. it sums every long value in the batch, and that sum is what asks for the oversized allocation, so each buffer wants min(remaining, cap). the explicit error still earns its place, a single value longer than the cap has nowhere to roll to. @kita-renji no apology needed. happy to take the multi-buffer part as a follow-up, or leave it with you if you would rather carry it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
