mzabaluev opened a new pull request, #11751: URL: https://github.com/apache/arrow-rs/pull/11751
# Rationale for this change The `async_reader` implementation has a bug where, if the requested range ends in the middle of a sync marker, the reader does not keep reading the block after the sync marker. If the range starts in the middle of a sync marker, the following block also gets skipped. Combined, this means that if a file is split into ranges so that the second range starts where the first range ends, and that split point happens to be inside a sync marker, the following block gets skipped entirely. # What changes are included in this PR? * **Marker cut by the range end**: if no whole sync marker is found, the reader fetches up to 15 more bytes past the requested end. It searches them together with the last 15 bytes it already had, so any marker it finds there must have started inside the range. * **Where a range stops**: a new `requested_end` field keeps the original end while `range.end` grows with each extra fetch. After each block, the reader works out where the following sync marker starts and stops if that is at or past `requested_end`. * **Builder**: a range that ends exactly at the header's end no longer finishes immediately, because the header's own sync marker starts inside it. The doc comments are the shortened versions from the last commit. * **Bug fix for small blocks**: the `DecodeVLQHeader` path threw away any bytes after the block data, including the sync marker, whenever the 20-byte header fetch held the whole block. Blocks with less than about 20 bytes of data would then fail. The `*every_split_point*` test, which uses 1-row blocks, would hit this. Instead, header fetches go through normal decoding, and with it `finishing_partial_block`. Every extra fetch moves `range.end` forward, so the loop still terminates. # Are these changes tested? Added tests (created by Miha, modified by Claude) exercising cases when the read range splits a sync marker. # Are there any significant user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
