rustyconover opened a new pull request, #11374:
URL: https://github.com/apache/arrow-rs/pull/11374
# Which issue does this PR close?
- Closes #9837.
# Rationale for this change
When an entire IPC stream is already in memory (an HTTP body, a
memory-mapped file, a shared-memory segment, or `bytes::Bytes` from object
storage), `StreamReader<R: Read>` is the natural API to reach for, but it
copies every message body into a fresh allocation. `StreamDecoder` fed a single
owned `Buffer` is already zero-copy, since it slices the input for each column
buffer, but it has a push-based API that is awkward for this case.
Microbenchmark (`Float64Array`, 10M rows, in-memory stream, see #9837):
| reader | time | throughput |
|---|---|---|
| `StreamReader<Cursor<&[u8]>>` | 33 ms | 2.4 GB/s |
| `StreamDecoder` + `Buffer` | 0.004 ms | (aliases input) |
# What changes are included in this PR?
- New `arrow_ipc::reader::BufferStreamReader`: wraps `StreamDecoder` with an
owned `Buffer` and implements `Iterator<Item = Result<RecordBatch,
ArrowError>>`, like `StreamReader`
- `try_new(Buffer)` and `try_new_from_bytes(bytes::Bytes)`; the schema is
decoded eagerly so `schema()` is infallible
- Like `StreamReader`, a stream that ends on a message boundary without an
EOS marker is accepted; a truncated message returns an error
- New `StreamDecoder::is_finished()`
- `bytes` added as a direct dependency of `arrow-ipc` (already a dependency
of `arrow-buffer`, so no new crate in the dependency tree)
# Are these changes tested?
Yes:
- Single and multi-batch round-trips, a schema-only stream, and
dictionary-encoded batches
- A pointer-aliasing test confirming the returned column buffer points
inside the input allocation
- An empty buffer, a stream without an EOS marker, and a truncated message
- The `bytes::Bytes` constructor and a doctest
# Are there any user-facing changes?
New public APIs: `BufferStreamReader` and `StreamDecoder::is_finished`. No
breaking changes.
Written with AI assistance; all changes reviewed by the author.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
