ump45nose opened a new pull request, #884: URL: https://github.com/apache/arrow-rs-object-store/pull/884
# Which issue does this PR close? Closes #880. # Rationale for this change Since #628, `LocalFileSystem::read_range` zero-fills the whole buffer (`buf.resize(requested, 0)`) before reading into it with `read_at`. For large ranges that memset is pure overhead, as the bytes are overwritten straight away. # What changes are included in this PR? - On unix, add a small `read_at_uninit` helper that calls `pread` (`pread64` on Linux/Android) directly into the spare capacity of the `Vec`, then bumps its length by the number of bytes actually read. Each read is capped at `c_int::MAX - 1` bytes like std does (macOS rejects larger reads). `libc` comes through the `nix` dependency that the `fs` feature already pulls in, so there are no new dependencies. - The read loop in `read_range` is unchanged otherwise: it keeps retrying on `Interrupted`, stops on EOF, and reports `OutOfRange`/`InvalidRange` the same way as before. - Windows has no equivalent positioned read into uninitialized memory, so it keeps zero-filling and using `seek_read` as today. - New test `read_range_reads_exact_bytes`, which reads several ranges (empty, 1 byte, unaligned, >1 MiB, whole file) of a 3 MiB file with position-dependent content, plus a range that runs past EOF. A quick local benchmark (Linux x86_64, file in page cache, median of repeated `read_range`-style reads): | range size | zero-fill + `read_at` | uninit `pread64` | |---|---|---| | 16 KiB | 6.21 GB/s | 6.48 GB/s | | 64 KiB | 6.27 GB/s | 6.86 GB/s | | 256 KiB | 6.04 GB/s | 6.72 GB/s | | 1 MiB | 6.15 GB/s | 6.85 GB/s | | 8 MiB | 5.58 GB/s | 6.74 GB/s | I went with the `pread` option from the issue rather than switching to `seek` + `read_to_end`, because it keeps positioned reads (no shared file cursor) and works for every range size. Tested with `cargo test --features fs --lib local::` and `cargo clippy` (default, `--no-default-features --features fs`, `--all-features --all-targets`), plus `cargo clippy --no-default-features --features fs` for the `x86_64-apple-darwin`, `x86_64-pc-windows-msvc` and `i686-unknown-linux-gnu` targets. # Are there any user-facing changes? No API changes; local range reads get a bit faster on unix. AI assistance disclosure: I used an AI coding assistant to help write and benchmark this change. I reviewed the code and test results myself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
